Enhancing Zero-Shot Black-Box Optimization via Pretrained Models with Efficient Population Modeling, Interaction, and Stable Gradient Approximation
Authors: Muqi Han
Xiaobin Li
Kai Wu
Xiaoyu Zhang
Handing Wang
Venue: NeurIPS 2025 poster ()
Keywords: Meta Black-box Optimization
Black-box optimization
Pretrained Optimization Model
Neural Combinatorial Optimization
Primary area: optimization

Abstract: Zero-shot optimization aims to achieve both generalization and performance gains on solving previously unseen black-box optimization problems over SOTA methods without task-specific tuning. Pre-trained optimization models (POMs) address this challenge by learning a general mapping from task features to optimization strategies, enabling direct deployment on new tasks.

In this paper, we identify three essential components that determine the effectiveness of POMs: (1) task feature modeling, which captures structural properties of optimization problems; (2) optimization strategy representation, which defines how new candidate solutions are generated; and (3) the feature-to-strategy mapping mechanism learned during pre-training. However, existing POMs often suffer from weak feature representations, rigid strategy modeling, and unstable training.

To address these limitations, we propose EPOM, an enhanced framework for pre-trained optimization. EPOM enriches task representations using a cross-attention-based tokenizer, improves strategy diversity through deformable attention, and stabilizes training by replacing non-differentiable operations with a differentiable crossover mechanism. Together, these enhancements yield better generalization, faster convergence, and more reliable performance in zero-shot black-box optimization.

Enhancing Zero-Shot Black-Box Optimization via Efficient Population Modeling, Interaction, and Stable Gradient Approximation
Muqi Han Guangzhou Institute of Technology
Xidian University
Key Laboratory of Data and Intelligent System Security Ministry of Education mohan@stu. xidian. edu. cn
Xiaobin Li School of Artificial Intelligence
Kai Wu
School Artificial Intelligence Xidian Univeristy kwu@xidian edu. cn
Xidian Univeristy
221712147840stu. xidian, edu. cn
Xiaoyu Zhang
Handing Wang
School of Cyber Engineering
School of Artificial Intelligence
Xidian University
Xidian Univeristy hdwang@xidian edu. cn
xiaoyuzhangéxiddann an. edu.
Abstract
Zero-shot optimization aims to achieve both generalization and performance gains on solving previously unseen black-box optimization problems over SOTA methods without task-specific tuning. Pre-trained optimization models (POMs) address this challenge by learning general mapping from task features to optimization strategies, enabling direct deployment new tasks.
In this paper, identify three essential components that determine the effective- ness of POMs: (1) task feature modeling, which captures structural properties of optimization problems; (2) optimization strategy representation, which defines how new candidate solutions are generated; and (3) the feature-to-strates mechanism learned during pre-training. However, existing POMs often suffer from weak fcature representations, rigid strategy modeling, and unstable training.
mapping
To address these limitations, propose EPOM, an enhanced framework for pre- trained optimization. EPOM enriches task representations using a cross-attention- based tokenizer, improves strategy diversity through deformable attention, and stabilizes training by replacing non-differentiable operations with a differentiable crossover mechanism. Together, these enhancements yield better generalization, faster convergence, and more reliable performance zero-shot black-box opti- mization.
Introduction
Black-box optimization (BBO) problems ubiquitous in machine learning and engineering, The characteristic of BBO is that the algorithm can evaluate f(x) for any solution x; however, access to
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

additional information about f. such as Hessian matrices and gradients, is unavailable. BBO problems include hyperparameter optimization (HPO) [T] neuroevolution 2 neural architecture search (NAS) S and algorithm selection
[6] etc.
Traditionally, solving BBO tasks requires expert-designed heavily tuned optimization algorithms for each new problem, making the process inefficient and non-scalable. Thus, the paradigm of solving new optimization problems without any task-specific tuning extremely valuable.
Meta-learning, or learning to learn 7 particularly in the context of meta-black-box optimization [ learning optimize [10), covers a wide range of scenarios. It improves the performance of the optimizer on the target task by pre-training it on similar tasks. However, they typically exhibit poor generalization to new and unseen tasks. Although some methods such as LES [TT] and LGA 12 can generalize to novel settings, their performance still falls significantly short compared to state-of-the-ar optimizers specifically designed for these settings.
To better highlight our contribution, we introduce the concept of zero-shot optimization, which aims to simultaneously achieve strong generalization and high performance new optimization problems without requiring task-specific training. A particularly promising direction in this area is the development of pre-trained optimization models (POMs) [3]. These models are designed to learn canerrl-puppose optimization behaviors a wide variety of training tasks and effectively transfer that knowledge to previously unseen problems.
We view a POM as a function that maps task-specific features to optimization strategies. This process involves three key components: (1) Task feature modeling, which captures characteristics of the problem (e-g. normalized fitness values and centralized rankings); (2) Optimization strategy representation, which determines how candidate solutions are generated and refined; (3) Mapping mechanism, which learns the transformation from features to strategies during pre-training.
Although recent POMs have made progress, they remain limited in three areas:
The expressiveness of task features insufficient to generalize across diverse problem structures;
The strategy generation process lacks flexibility and diversity;
The training process suffers from gradient instability due non-differentiable operations.
To overcome these limitations, we propose EPOM (Enhanced Pretrained Optimization Model), which systematically enhances all three components:
Feature modeling is improved by cross- cooss-attention-aase based tokenizer that captures decision- variable-level information and encodes it into fixed-length task representations;
Strategy representation is enhanced through deformable attention, enabling dynamic and diversity-aware interactions among population;
Training stability is ensured by replacing sampling-based crossover operations with differentiable, weighted-sug sum mechanism, leading to smoother gradient flow.
By jointly improving feature modeling, strategy generation, and training robustness, EPOM achieves superior generalization and performance in zero-shot black-box optimization. This work contributes the growing body of meta-learning literature on pathway toward universal optimization agents.
methods, and offers a scalable
2 Related Work
Manually Designed Population- BBO Algorithms. Traditional population- based black-bon optimization (BBO) algorithms, such as genetic algorithms [14], evolution strategies [15], particle swarm optimization [16), and differential evolution [17], have been widely used to tackle optimization challenges. sfute-of-the-ee methods like CMA-ES [18] and L-SHADE [19] rely heavily expert knowledge and and-erorr development, resulting in limited flexibility, high development costs,
and suboptimal generalization 200
Meta-Learned Population-Base Based BBO Algorithms. Meta-learned BBO algorithms improve gen- cralizability through meta-learning, be categorized by their training strategies. One category

---

G
V
F
U
x
LMutM
Efficient
LCMM
Differentiable LCrM
SM
X
Tokenizer
LMuttM
Tokenizer
Figure I: The overall framework of EPOM. The LMutM Tokenizer and LCMM Tokenizer are designed generate input tokens for the Efficient LMutM and Differentiable LCrM, which are responsible for obtaining the mutant population V and the candidate population U*, respectively. Finally, the SM produces the offspring population based on X and Uº.
employs bi-level optimization 21 as seen in 22 which adapts optimizers specific task classes. LES [11 and LGA [12 use neuroevolution with self-attention mechanisms, but their reliance on external search algorithms such as OPENAI-ES [2 leads to slow training and limited scalabil- ity. Meta-MOGA (24] addresses multi-objective BBO problems by parameterizing mutation and crossover operators with multi-head self-attention and the selection operator with an MLP, whose parameters are optimized via evolutionary algorithm. Another adopts reinforcement learning (RL) 25). for example, Shala et al. [26] use RL to meta-learn policies for adjusting CMA-ES parameters, however, suffers from training instability. Moreover, Q-Mamba [4] offline learns meta-level policy for dynamic algorithm configuration over an optimization task distribution, while ConfigX (27 meta-learns universal configuration policy
modular evolutionary algorithm space. Alternatively. EvoTF (28) incorporates algorithm distillation, but requires large-scale datasets and struggles to surpass the performance of its source algorithms. Besides these, B20pt (29] and POM [13] leverage self-supervised learning, however, B20pt is restricted by iteration limits, and POM faces challenges in scaling with model size.
LLMs for Optimization. In the field optimization, series of optimization methods based on Large Language Models (LLMs) have emerged. They widely applied in areas such as NP. hard problems [30 31). algorithm evolution [32] 33] 34] 35| 36), reward design [37], and Neural Architecture Search (NAS) [38 39]. LLMs play crucial role in optimization, especially in sampling new solutions. However, as stated in 40 their optimization strategies rely on
natural selection mechanisms. limiting their effectiveness in numerical optimization scenarios. For example, LLaMoCo [41 and EoH [42) use LLMs to generate code for optimization problems. The former depends well-designel instructions and prompts, while the latter faces high evaluation costs, restricting practical applications. Moreover, TNPs (43, ExPT [44), and LICO (45) use transformer structures to address the BBO problem. Nevertheless, TNPs relies the context information of the target problem, and neither ExPT nor LICO can be directly applied to tasks with different dimensions from the training ones. summary, although LLM-based optimization methods have made progress. compared with pre-trained BBO methods, they have insufficient cross-task solution solution-generaa generation capabilities and poor generalizability, which remain key issues to be overcome in the optimization field.
3 EPOM
black-box optimization problem can be transformed as minimization problem, and constraints may exist for corresponding solutions: min f(x), s.l. E [4, w.). where x (x1, ..) represents the solution of optimization problem f. the lower and upper bounds 1 - (4,, l2. .la)
and 1 = (ui, W2, ... uua), and the dimension of A population consists of individuals, denoted as X X2,XX, xx
3

---

3.1
Review of POM
3.1.1 Main Parts
The POM primarily consists of three modules: Learned Mutation Module (LMutM), Learned Crossover Module (LCrM) and Selection Module (SM).
LMutM. LMutM generates the mutant population V through the multi-head self-attention
mechanism 46]
S + LMuIM(H'', V = X
where V", X' E Raxd H' E R and St E R=x" Here. n denotes the population size and denotes the problem dimension. X represents the parent population, H' [bij ...,, h, is obtained by tokenizing X and serves as the input to LMutM, where each token h encodes the mutation features of the corresponding individual x including: 1) ft: the normalized fitness| f fxx) of x/: 2) ff: the centralized ranking of x S is the weight matrix computed by the attention mechanism.
2) I.rr. LCrM generates the crossover rate er' = [ert, [ , erål ( cr² E RIx=, or is the crossover rate of the i-th individual) by an FFN 46 and performs the crossover operation obtain the candidate population U E Rxxd
cr' t LerM(Z"), U gumbel-softmazx(X (X', V'] cr')
(2)
Where Z' = [2i,
1 E R=x3 2 encodes the crossover features of the i-th candidate individual uf. including: 1) ft: the normalized fitness f(x{) of x/: 2) f:: the centralized ranking of x;; 3) simç: the cosine similarity between of and uf. The combell-sffmaa trick [47), which independently operates across dimensions by
xpx3;x, is the crossover feature matrix utilized by LCrM. Each token
taking [ vf,5]( 1 = 1, 2..... d) as two categories and [or cr.] as their respective probabilities, provides a differentiable approximation of the crossover operation.
3) SM. Finally, SM 48 I-to-/ selection strategy is executed between U and X to produce the next next-generation population X ++,, where X = SM(X', U*).
3.1.2 Issues
Issues with LMutM. 1) Insufficient and non-diverse strategies: The mutation strategies are charac- terized by the weight matrix Se derived from the self-attention mechanism, where each individual interacts with the entire population. However, such global interaction can hinder the efficiency and diversity of the mutation process. 2) Training instability: The mask operation randomly masks parts of S', which may obscure significant signals, thereby slowing down convergence and causing training instability. 3) Insufficient input features: The input H' of LMutM only includes fitness and ranking information, overlooking the distribution characteristics of the population. This limits the POM's convergence and generalization abilities.
Issues with LCrM. LCrM uses the gumbel softmax trick to address the non-differentiabbiity of crossover operation. However, the imprecise gradient approximation can lead to instability during model training.
3.2 Proposed EPOM
Our goal is to improve the strategic efficiency and trainability of POM. We have redesigned the LMutM and LCrM of POM and proposed the EPOM, as shown in Fig. [
3.2.1 Overall Architecture
First, the LMutM Tokenizer encodes X into fixed-dimennional tokens E' using cross-attention mechanism (46), similar in spirit to prior cooss-attention-baa based tokenization methods [49]. These tokens are then concatenated with the tokens H' from the original LMutM in POM form G
To avoid potential confusion between the abbreviations of Learned Mutation Module and Learned Crossover Module (formerly LMM and LCM) with Large Multimodal Models and Large Context Models, we have revised their abbreviations to LMutM and LCrM, respectively.

---

Subsequently, the Efficient LMutM takes G' as input and generates the mutant population V through two self-attention mechanisms. The first self-attention mechanism identifies, for each parent individual, the subset of population members it should attend to. The second mechanism, deformable self-attention module [50], computes the attention weight vector si for each individual i based on the keys corresponding to its selected individuals X Then, the mutant individual vद is obtained by multiplying s with X Next, the LCrM Tokenizer processes V in the same manner as the LMutM Tokenizer to generate F and then feeds [E*, F*] into the Differentiable LCrM. The Differentiable LCrM produces the candidate population U through two cross-attention mechanisms. The first mechanism determines, for each parent individual, the subset of mutant individuals from V it should attend to. The second mechanism, a deformable cross coos-attentionn module (50), computes the attention weights between each parent and the mutant individuals it attends to. The candidate individual is then obtained as a weighted combination of the parent and the attended mutants. Finally, X and U are passed the Selection Module (SM) to generate the offspring population X
3.2.2 Tokenizer
1) LMutM Tokenizer. In POM, the dimensions of the inputs (H' or Z') to LMutM and LCrM are and 3, respectively. However, such low-dimensional representations may fail adequately capture the characteristics of the optimization landscape, limiting the information available to LMutM and I.CM. To address issue 3) in Section 3.1.2 we design tokenizer that maps each individual in the population from the problem dimension to a fixed dimension de using a cross-attention mechanism:
x/ x
if even otherwise
append (x,,, 0),
K = reshape (-1,2))
(3)
Kr KT WiK bIk:
VT Kr x Wiv + bIV
é = Tanh! Qr x KT
x VT,
e reshape (ef, (-1,))
Vdiv
where x² E Rd is the i-th individual in the (-th generation, The append and reshope operations are functionally equivalent to their PyTorch counterparts. WIK E 2xxd1 Wiv E [2xdvv, bдズ and bi bias vectors with dimensions dięk and div, respectively. Q7 E BRdrxdiq represents the shared query across the population. All parameters, including WIK, WIv, bın, bıv, and Qr, are learnable.
By using Eq. 3), we obtain a dr x dimmeinnnl token for each individual x", and subsequently construct the population token matrix E' jei, e...... ... en Then we concatenate E² with the
token matrix H' from the LMutM of POM to form G H E*}, where G' E Roxd, and de 2 dr div.
2) LCrM Tokenizer. Similarly to the LMutM Tokenizer, the LCrM Tokenizer transforms the mutant population V into tokens Ft with a fixed dimension d, according to Eq. (3). where d, dr x div.
3.2.3 Efficient LMutM
For issues 1) and 2) in Section 3.1.2 we introduce Efficient LMutM, which leverages deformable attention mechanism to allow individuals to adaptively select a subset of individuals for information exchange. This approach avoids global interactions, improving the model's 's performance and diversity. Additionally, we incorporate a dropout layer that is active during both the training and testing phases within the EPOM, introducing more randomness to the model.
We adapt two self-attention mechanism the Efficient LMutM. The first self-attention identifies the individuals within the parent population that each individual should atttend to according to the input
5

---

tokens G The attended individuals can be computed using Eq. (4
QIM G¹ x W2Q + b2Q KIM G' W2k + b2K
QIM KT KiM
VMM = G¹ x Wav bav, Ñ, = Softmar
x IM,
(4)
vd2v
N, = top-p-sammpigg
where QQ W2K I2 Xd,x b2q, bok E Rdocx Wav E Rd xdous, and bav Rd denote learnable parameters, and dout is the fixed output dimension. If the population size S dout. the first dimensions are used to represent the selection probability of each individual. Otherwise, selection is restricted to the first dout individuals. The top-p-sampling function selects top-m; individuals via top-p sampling [3T], ensuring E pr) p. Applying top-p sampling to N yields the index matrix N. E R where k = max ) representing the selected individuals for each member of the population.
The second mechanism, deformable self-attention [50]. computes the attention weight vector st for each individual Then, we derive the mutant population V [v1, viz," vn where v st x X
Table 1: Training Functions for EPOM and POM.
We-
ID
Functions
Range
TF1
Σ a wil
E 10, 10). w E [ 10. 10
TF2
Σx (xx- ww) + (#i+1 wi+1)l
Ei [.,
w/
# E [- 10, 10). w E
10,
TF3
Σ, #
E [ 100, 100) w [-50, 50
TR4
max (2
Sis
100, 100) -50, 50
d-1
Σ (100 21112² (2: 1)2)
æ
100, 100)
[- [-50, 50, 50]
3.2.4 Differentiable LCrM
To resolve the issue in LCrM, we propose differentiable LCrM, replacing the gumbel-softmax trick with a weighted summation approach. Specifically, the differentiable LCrM takes [E', F') as inputs and generates the crossover weight ew' R The i-th candidate individual can be computed by Eq. (5).
u = cwfo x + cwo,, - meam (V!),
(5)
where V E RRixd denotes mutant individuals selected by parent individual x². The mean function
computes the average along each dimension of the input matrix. Through Eq. (5). we obtain the
candidate population U = [u u..... ]. The remaining task is handled by SM, which peforms
a one-to-one selection strategy to generate the offspring population (++
The training set. and training strategy of EPOM consistent with those of POM, as detailed in the Appendix A
After EPOM has been trained, we use Algorithm to solve new problems.
3.2.5 Loss Function
The loss function POM for the i-th training function consists of two components: 1) the difference in the mean fitness of adjacent generations, normalized I/; and 2) the mean standard deviation across the dimensions of the population I?. Ideally, 1 encourages POM to converge toward the optimal point, and ! encourages population diversity. However, in practical applications, we found that 1 has limited effect on population diversity, therefore, we replaced with crowding distance (22 The loss function of EPOM for the i-th training function can be discribed as Eq. 6

---

Algorithm Driving EPOM to Solve Problem
Input: Generations T, population size n,, BBO problem f.
Output: The optimal X found.
1: EPOM loads the trained parameter 0.
2: Randomly sample an initial population X of size n.
3: for = 0, 1..... .
do
4: Construct E based on X and f using LMutM Tokenizer.
5: G <- E*)
6: S' Efficient LM")M(G").
7: V f S' x X
8:
Build Ft based on V using LCrM Tokenizer.
9: U Differeentiab LCrM(E',
10: X f- SM(X', (*).
11: end for
1 17 + alea
Σ fi(x((()
Σ fi(x(w") XEXI-1
X
X--
fi(x (wo)))m)l++ Si(x -110
[2(x(-1onkk) konket)
xex
(6)
X
fi(x!rn (w0) - fi(x) (w
Σ f((x(u+) XEXI-1
|x--|
where X denotes the individual ranked i-th in ascending order (for a minimization problem) in the 1-th generation. a is a hyperparameter, and we found that setting it to 0.1 better balances the convergence of the population toward the optimal value and its diversity.
Experiments
4.1 Experimental Setup
-10
IPOM
BROM
IPON
POM
PON
-30
-40
LES
LES
LGA
LOA
LSHADE
LSHADE
LSHADE
40
100
(a) Fl d=30
(b) F2 d=30
(c) d=30
Figure 2: Part of the convergence curves of EPOM and other baselines. It shows the logarithmic
convergence curve these algorithms on functions in BBOB. See Appendix C for more details.
Baselines. Our core approach a population- based pretraining BBO algorithm;
therefore, we focus comparisons with other population-basel methods rather than non-population methods such as Bayesian optimization. Moreover, we do not compare with LLM-baser approaches 30 31
33
34
37
38
39),
as these methods lack zero-shot optimization capabilities.
Heuristic Population- BBO Algorithms. We compare against the following established population-based BBO algorithms: (a) DE (DE/rand///bbin) [53]: A classic numerical optimization algorithm. (b) ES (μ,A-ES): well-known variant of evolutionary strategies. (c) CMA-ES [18] Of- ten considered the of-e--of-the-rt method for continuous domain optimization in challenging settings (e.g., ill-conditioned, non-convex, multimodal problems). (d) L-SHADE 19 state-of-the-art variant of DE.

---

Table 2: Results of BBOB. EPOM is trained on TF1-TF5 with =10. The best results are indicated in bold, and the suboptimal results are underlined. The "+/=/-" at the bottom indicates that EPOM's performance is better/same/worse than the given baseline on the given dimension setting and function.
RPOM
POM
DE
CMA-RS
LSHADE
LGA
11 4.92--565.26::44
2. SITS-RO (Nllo0))
178-4113
7.908-466
97B-04)
1.288-0077
355-44)
A 91T (
125
F2 S.WE, 55. 55..278--55
6, 136 X5E-13 12)
2.1SE 24E 01)
25 0B)
8.55
G4E 62)
1.77E 1EFE-022212---0 (0)
1.45 021. 15E
1.75E 01 ENE 02)
F3
1.64 440-844-444
6.57E-0111
(ME-01)
1. 1.4THHIR 200+00)
47E-011
2.47843322
54E-4R)
7.12E+010.3111
31E+000
S WIE-ONI (TEE-A))
(5E+01)
R1
1.923-4003 M03
6 6MME-0TT
291--013
3. NSTSHIRA
di B-477
7.21840711
(771-40))
1,048500004 MMi-0000
6 11R1-N1 11 N0FAA0)
KSE-01)
F5
3.28E:
00.5 (01)
361E 13E (0)
5.1E:01:450
(0)
32220002
0J02E DODOLIDEE (00)
0:00: CONOA2E (60)
L9SE (0215.46) (1))
0.0GE (OGIDCIEE (X)
Rg
1,902
-7)
1,6EE 0511 (2E
(0)
5.29EH0B1L
74E (02)
T.4E 20T334E IXT
LILE 1IE
3I2KETO D2SETOUTCSSETT (Q])
F7
6.832-3707
3,758
1703
59E 135
IZE (12)
3.E HO0TT-00 61)
1.25E-016
1.30E-0244 B0E-001
6.97E-0112
F8
0,00E4 00.3 000
8,288 0044 968-
260
4.
4.215-05:6.8
85E+04)
3.25E
AAE--GE 404)
3.15E
401--22)
3.08E-012.51E44
301E-0N2.7HEE000
LIGE--01 (2E--2)
P9
1.36E40228
10E-01)
THE
7.05E
64E .04)
4.17E-0A1L.200
2.57--225-4MM (2)
1.SRE
THE (2)
F10
5.582
48.9 08-2-44--24444
(25 11E
3.50E: 0K) 14:1 (0)
2.33E:0706.20
30E (05)
3.39 LO5 (05)
1.
7.54E (0413.56))
2,07E
(0SI6.2HE
P11
3,992
SS0--000 155-011
5:20
6228-123
3.555-0311 21 215-07)
1.5AE-00 1364020
2.568-1255
(75
9958-122 1.7 (22
30
F12
1.4e2
NITEOON (TE-34)
188--0044
1.7E
2.0 112 10)-
9.59E-171A
F13
3.1E
21E--2:24446-25
L57E-00:6.256 200400)
9.60E-00X
K26-00)
2.44-00L
A1E-00)
2.40B-02/44 CMEE-02/4.05E0000
1.66-247..41---
T.000AA 070-00)
065+01))
141-400
P35
2.747-844795-45
(70-00)
201
4.270004
1.55 DSFw61)
250H-II 155-00)
Fi6
1.44=0015 25E-00
SITT (20-400)
A
241-000
2.1SE (68-000
L64E=ON:
-116-00)
2.4E-00L1 L LIT
1.46
24E
2.47-01C7.80 36E-00
2.4TE-016
USE
THE
117E-089.75--00 75E-02)
T4E-
2.216-1013 S.N.66E--0
27
SITH-OTC dAE+00)
1.1960000 065-011
2.00-0055
515-01
4ERE-OTT 2555-01
1. 225--00)
110
4011-4800.05 -01
HONE- 155-013
KESTIOUUE 100
1000WOR) 178-013
SHIT-O2EE (015-1))
A.AR 678--01
7.007-1022 355-01
F20
00:4
14E1
LOBE (0522
3.27 (000
3.13E
2.72E (0111.268
(OL))
0022 1000
30EE USE
8.50E1ON IcE.0))
2.ME HORIS. 34
1.44H-ONL 2OEEL22EEEE
H.ME (0022
7.35
470
57E--00
(3E-00)
82E+000
8.57E-0100 ONE-01)
UNLE-0055000 DOE
L145-602.17--00
L.GEE-OTEE, 4485 (0)
WODTEROOS TUE-
4.26
100E-002.222--0
3.51E4005
3,416-0013.80
3.75E
3.3HE IIE--EEE-
3.11-000-4008
4.38E--00 L.NE-0L)
2.2 (TE00))))))))) 3.20
2.0E
7.0GE
3.00E
11469-1188 170-15
4630wotts 518--07
2000000
1640 750-00
2001-17744 (965-01
F2
4.202 55.5 55.5533
TIT 1212.53E 12)
LIKE (116 19E1 (00) 2,248 1ONE DOE (00)
4.17E
1EEI010 ZHE (00)
7.54E
0014.39E 02)
4.5G (OO) 2.2 (40)
3.2RE: (MV5.755 5.5 BL)
F3
36E 0115 121 013
LIEE--.T. 02)
4.7E--0422 377
2.2 HO4DL 5HE HES)
6.1E--005
7ZE H031- ANE- (C)
SIZE HI3IS ESE- DI)
P4
5:005+300
1.396-0711 (71-070777
315+05)
2:90E-0000 25E-404
1.256-0041 A9B
15E--0311 2555-00
2-EE-1331 51E-10)
F5
1.52E
2254013
144 .024 (4E-0L)
2 2IEE-OU:142E400
1.64E-0022 13E 400)
LISE 402/2 K3E
LISE -001 SIE. (0)
4.832
0.32E-
1005 176- 10
2 2466 00:2.044 (2) 9.15E 109K2 22E (02)
2.5E 10211 ISSE A02)
4.DDE: (0155.00 HO0)
4.5TE, (02/4.5RE (I))
ETRIE 1OL7 J.T1EE/0 (T)
F7
9,292
121 53E
LIEE (6:2.21E:0))
6.L 1OAE ISE (03)
2.79E
53E (02)
1,56E:
(000).11100)
LABE (0313.998 (02)
PUARE (02) ESE: (C)
F8
0 600 (0000000-00)
165-477)
6.106-00122-
478-44)
1.3EEWDRC 40554033
4.88-0511 L44E-04)
9 WURIE-OHIS SEE-HN)
P9
6.41E-0009
201-000
LATE-DR 50EA
2.1BE
455-477
112E-050R
54E-04)
3.71--0511 1-38--041
LO7E--05 (SE-04)
F10
T.
2.54E-0004 04g .01)
S L.7
3.2HE-OBLL.
13E 407)
RZE -051 1.241.244--- (0)
J.KEE AOGIN S4E. (5)
7-41E-OR
455--471..045-- P12
LANESONI (955-09)
ISE- DESSHELLSSE 788 4.8E 105 191-10)
19 215 61 1-11)
112--901 I. 125-000
P13
TINNSANN 376-009
ISE-DE 565E 212
6.355-021
15B-02)
(7E401)
9.26--221 C. (22-00)
F14
1,WE
DOFE--000-57 576-05
4 STEEIIIA (2E+01) FARESHORS
14E-01
4115-
558-01)
10504000
P15
SIE 178-001)
O INTTWTTT COE
-01--
91B-04
P16
771 (4 728-013
& 41T+IICS (TT
-00 -01-
F17
145E
S.SME 0712 32E
I 26E ID TAE
17TE OZ))
5.7 HOOK 24E (0))
261B
12B OO)
IIE (01/2.6TE (K))
1.30E (OL ESE (N))
F18
6.0022661
KINE D544 25E-26
I. TTEEEIRLLLE (CE
Z60EHOME3
33E-133 H13) 265E-OTI4
37E-43)
1,00E-0EEL
25E+000
66EE-011 (6E:01)
IGEEHOUS JSE- (N))
F19
LA2254800 2.2144001.290000 (00
6 74E--4014 THE--13
5 STT-HDSS ACE+0T)
3.378
71B-99)
1998-4007
WEB-01)
1.00--0111 7587-19))
F20
J.5SE:
(0))
-..RE (ONT.1SE 01)
L.SEE B. (.5
S.ISE 25E (55) 3.08E 3.36 (04)
L.GG /04/2.00/ (4)
2.56E (04/7.5E2 (8)
P2
7.7ETOOR EEEETOLLAEETT 1000
AADEE 10117 STE (0)
2. TOE (22/4066100 OO)
1.22 1091 28E (02) 6,55E (0111 25E (01) 1.37:
012.-113/ (0)
TAGDE: (U(T-33E 0L)
T.2SEI DUP-2E 01)
P23
8
8,24E10101 01E-00)
SOSEEO (OOT) ZYE (00)
2.53E (12)) (00199
LASE 1084108 OZ) 7,028 (0112 (03)
7,0ZE (0U15.5E (MT)
SILLE HOUT THE (0)
P23
1.E--869.52E--11
8VE-4007 548-01)
4.84-00S
12B-01) 4228-00--NE-00
5.12E+00, 16B-01)
7.18E-00) (4SE-01)
P24
1.11E4003.224000
24E50001 2XE-12
128E--0::.44C4000
LAZE 75E-003 L61E46300
3.508-013.77E6-02
3.77--002266E-00
-14-
65W0
340/10
SWA110
These algorithms are implemented as follows: DE and ES using Geatpy (4). CMA-ES and IPOP- CMA-ES using emae and L-SHADE using pyade F
Pretrained BBO Algorithms. For comparison with EPOM, we select three recent meta-learned BBO algorithms: (a) POM [3) The Latest pretrained optimization model with the best performance. (b) LES III: A learnable evolutionary strategy that uses a data-driven approach to enhance generaliza- tion performance and search efficiency. (c) LGA [12] A data-driven learnable genetic algorithm that adapts to unseen optimization problems, search dimensions, and evaluation budgets.
For all algorithms, we train EPOM and POM the TS problem, with maximum of 100 evolution generations, n- = 100, and a problem dimension of 10.
Benchmarks BBOB [55 [6] is a well-established benchmark suite for evaluating optimization algorithms. It includes a diverse set of high-dimensional continuous functions, encompassing single- peak, multi-peak, rotated, and distorted functions, along with functions exhibiting characteristics such as Lipschitz continuity and second-order differentiability.
4.2 Results
We evaluate the generalization ability of EPOM on 24 BBOB benchmark functions with dimensions d 30 and d 100. Table 2 summarizes the performance comparison across all algorithms (see
8

---

5
2
No SM 7d0o0
1.550
EPOM
No Efficient LMutM
No Dropout
LMutM Tokenizer" 12kenizer122m
3:208
No LCrM Tokenizer
No Differentiable LCrM 1392
Figure 4: Results of ablation study on BBOB. The metric used to evaluate performance is the optimal value of the function found, with smaller values being better. Here, d 30.
Appendix Figures 6 and for detailed results). The fitness value of the best individual in the final generation for each function and dimension setting is reported.
Compared to POM, EPOM achieves superior results 32 out of 48 functions across dimen- sions d = 30 and 100 (66.7%), indicating consistent performance gains across both low- and high-dimensional settings. This demon- strates EPOM's improved scalability and bustness. Moreover, EPOM outperforms all baseline methods in most cases, highlighting its strong generalization capability across diverse optimization landscapes.
POM
CMAES
EPOM
LES
LGA
LSHADE
EPOM exhibits significantly better convergence properties, evidenced by its ability achieve extremely small objective values. In contrast, POM often struggles achieve comparable pre- cision, particularly in high-dimensional prob- lems. This highlights EPOM's enhanced ability to model population information and exploit fitness landscapes effectively.
Generation
Figure 3: The results of robot control in the Bipedal Walker task.
Bipedal Walker [57]. The Bipedal Walker task aims to optimize a fully connected neural network with d = 874 parameters over 800 time steps to enhance robot locomotion control. In Fig. 3 EPOM achieves rapid and high-quality convergence, whereas LSHADE exhibits lower effectiveness, and CMA-ES, DE, and LES suffer from premature convergence.
4.3
Ablation Study
The results of ablation study for the designed modules are shown as Fig. 4 Configurations include the following items: (a) No Efficient LMutM: where the Efficient LMutM is excluded, and simple DE/rand/I/bin mutation operator is employed; (b) No LMuIM Tokenizer: the LMutM Tokenizer is excluded; (c) No Diffrentiable LCrM: indicating the absence of the learnable crossover operation, using only binomial crossover; (d) No LCrM Tokenizer: the LCrM Tokenizer is excluded; (e) No Dropout: the dropout layer for St excluded; (f) SM: the 1-to-1 selection module excluded.
Upon observing the experimental results, we discovered that the performance of the model without SM deteriorated significantly. This is because the model loses its evolutionary direction. The impact of the modules can be roughly ranked as follows: Efficient LMutM >~ LMutM Tokenizer > Differentiable LCrM LCrM Tokenizer Dropout. This indicates that the Efficient LMutM introduces a more powerful representational ability to the model, addresses the problem of strategy inefficiency in POMs, and enhances the model's search capabilities. The LMutM Tokenizer can effectively extract the distribution characteristics of the population. The introduction of this characteristic significantly improves the model's decision-making ability, enabling better understand the fitness landscape. The Differentiable LCrM and LCrM Tokenizer can achieve information interaction between new and old individuals, strike an adaptive balance between exploration and exploitation, and enhance the generalization and convergence of the model.

---

025
032
628
W
231
5050
5275
240-
-235
(a) F1 step 1
(b) F1 step 50
F1 No.1 individual
(d) F1 No.50 individual
Figure 5: (a)(b) Visualization results of some information selection strategies of Efficient LMutM. The numbers 0-9 represent the population ranking, and the smaller the ranking, the higher the fitness. The population size here is 10 and parameter p for lop-p-sampling is set to 1. (c)(d) Visualization results of crossover weight some individuals of Differentiable LCrM. The label represents the generation, and the y label denotes the crossover weight of X'. The population size is 100. (a), (b), (c) and (d) are all tested on BBOB d = 30
4.4 Visualization Analysis
Analysis of Efficient LMut' To conduct an in-depth analysis of the Efficient LMutM strategies, visualize the evolution of S' through heat maps, as shown in Fig. 5 (further details can be found in Appendix D). These heat maps reveal two key insights: 1) Similar to the original version, Efficient LMutM effectively balances exploration and exploitation by dynamically adjusting the weights of individuals based on their performance; and 2) Efficient LMutM adaptively generates diverse mutant strategies in response to varying task landscapes and individual performance.
Analysis of Differentiable LCrM We visualize the crossover strategies of Differentiable LCrM. As shown in Fig. [5] (refer to Appendix Figures 15 19 for additional details), Differentiable LCrM demonstrates its dynamic crossover strategies, generating variable crossover weights based on the overall performance of individuals and the characteristics of task landscapes.
Conclusion
We propose the Enhanced Pretrained Optimization Models, novel framework designed to enhance the performance and robustness of POMs. The experimental results demonstrate that EPOM addresses the key limitations of POM, including insufficient task feature modeling, inefficient strategy genera- tion, and unstable training process. By introducing the LMutM/L Tokenizer, Efficient LMutM, and Differentiable LCrM, EPOM achieves of-te-of-thh-arr art performance in zero-shot optimization, particularly in high-dimensional landscapes. These advancements make EPOM a highly promising candidate for real-world black-box optimization problems, offering superior scalability, robustness, and efficiency compared to existing methods.
EPOM's ability outperform POM and other baselines across wide range of functions, especially in high-difficull scenarios, underscores its potential as a universal black-box optimizer. This work not only advances the state of the art in zero-shot optimization but also paves the way for broader adoption in machine learning and engineering applications.
However, the performance of EPOM in heterogeneous search spaces, involving optimization tasks with diverse data types (e.g., images, strings), remains underexplored. We identify this an important avenue for future work.
Acknowledgement
This work was supported in part by Natural Science Basic Research Program of Shaanxi under Grant 2025JC-QYCX-060, in part by the National Natural Science Foundation of China under Grant 62206205, 62472345, and 62471371. in part by the Young Talent Fund of Association for Science and Technology in Shaanxi, China under Grant 20230129, part by the Guangdong Hige-level Innovation Research Institution Project under Grant 2021 B0909950008, and in part by the Guangzhou Key Research and Development Program under Grant 202206030003.
10

---

References
[1] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning: methods, systems, challenges. Springer Nature, 2019.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:1712.0656 2017.
[3] Jiacheng Chen, Zeyuan Ma, Hongshu Guo, Yining Ma, Jie Zhang, and Yue-Jiao Gong. SYMBOL: Generating flexible black-box optimizers through symbolic equation learning. In The Twelfth International Conference on Learning Representations, 2024.
[4] Zeyuan Ma Zhiguang Cao, Zhou Jiang, Hongshu Guo, and Yuc-Jiao Gong. Meta-haakk-oo--opiiiiaaion through offline q- g-function learning. arXiv preprint ar Xii::2505,020100 2025.
[5] Qing Ye, Yanan Sun, Jixin Zhang, and Jiancheng Lv. A distributed framework for ea-based nas. JEEE Transactions on Parallel and Distributed Syxtems, 32(7):1753-1764, 2021.
[6] Hongshu Guo, Yining Ma, Zeyuan Ma, Jiacheng Chen, Xinglin Zhang, Zhiguang Cao, Jun Zhang, and Yuc-Jiao Gong. Deep reinforcement learning for dynamic algorithm selection: A proof-of prrincll- study differential evolution. IEEE Transactions on Systems, Man, and Cybernetics: Systems, pages 1-13, 2024.
[7] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEFE transactions on pattern analysis and machine intelligence, 44(9):51495516) 2021.
[8] Zeyuan Ma, Hongshu Guo, Yue-Jian Gong, Jun Zhang, and Kay Chen Tan. Toward automated algorithm design: A survey and practical guide to meta meta-black box opxtmmimiiztiinn IEEE Transactions on Evolutionary Computation, 2025.
[9] Xu Yang, Rui Wang, Kaiwen Li, and Hisao Ishibuchi. Meta-black-box optimization for evolutionary algorithms: Review and perspective. Swarm and Evolutionary Computation, 93:10183: 2025.
[10] Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang. and Wotao Yin. Learning optimize: A primer and benchmark. Journal of Machine Learning Research, 2389)))
59, 2022.
[11] Robert Lange, Tom Schaul, Yutian Chen, Tom Zahavy, Valentin Dalibard, Chris Lu, Satinder Singh, and Sebastian Flennerhag. Discovering evolution strategies via meta meta-black- box optimization. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation, pages 29 29-30, 2023.
[12] Robert Lange, Tom Schaul, Yutian Chen, Chris Lu, Tom Zahavy, Valentin Dalibard, and Sebastian Flenner- hag. Discovering attention-based genetic algorithms via meta-black-box optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 929-99999999 937, 2023.
[13] Xiaobin Li, Kai Wu, Yujian Betterest Li, Xiaoyu Zhang, Handing Wang, and Jing Liu. Pretrained optimization model for zero-shot black box optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[14] John H Holland. Genetic algorithms. Scientific american, 267( (6661667677,, 1992.
[15] Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9(2): 159-11599 2001.
[16] James Kennedy and Russell Eberhart. Particle swarm optimization. In Proceedings of ICNN'95- International Conference on Neural Networks, volume 4, pages 1942 1948. IEEE, 1995.
[17] Rainer Stor and Keaneth Price. Differential evolution-a simple and efficient heuristic for global optimiza- tion over continuous spaces. Journal global optimization, 11:341-359, 1997.
[18] Nikolaus Hansen. The cma evolution strategy: A tutorial. arXiv preprint arXiv: 1604.00772, 2016.
[19] Ryoji Tanabe and Alex S. Fukunaga. Improving the search performance of shade using lincar population size reduction. In 2014 IEEE Congress Evolutionary Computation (CEC), pages 1658-1665 1665, 2014.
[20] D.H. Wolpert and W.G. Macready. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1):67-82 1997.
11

---

[21] Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimization for learning and vision from unilied perspective: A survey and beyond. IEEE Transactions Pattern Analysis and Machine Intelligence, (4(12):10045-10066 2022.
[22] Hugo Siqueira Gomes, Benjamin Léger. and Christian Gagné, Meta learning black-box ippulation-based optimizers. arXiv preprint arXiv:2103.03526 2021.
[23] Xingwen Zhang, Jeff Clune, and Kenneth O Stanley. On the relationship between the openai evolution strategy and stochastic gradient descent. arXiv preprint arXiv:1712.06664, 2017.
[24] Tianyu Li. Kai Wu, Xiaobin Li, Xiangyi Teng, and Jing Liu. Meta-moga: Meta-learning multt-objectivve genetic algorithm. In 2025 IEEE Congress on Evolutionary Computation (CEC), pages 1-4, 2025.
[25] Zeyuan Ma, Hongshu Guo, Jiacheng Chen, Zhenrui Li, Guojun Peng, Yue-Jiao Gong. Yining Ma. and Zhiguang Cao. Metabox: benchmark platform for meta-black-bbo optimization with reinforcement learnin.. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 10775-10795. Curran Associates, Inc., 2023.
[26] Gresa Shala, André Biedenkapp. Noor Awad, Steven Adriaensen, Marius Lindauer, and Frank Hutter. Learning step-size adaptation in cma-es. In International Conference on Parallel Problem Solving from Nature, pages 691-706. Springer, 2020.
[27] Hongshu Guo, Zeyuan Ma. Jiacheng Chen, Yining Ma, Zhiguang Cao, Xinglin Zhang, and Yue-Jiao Gong. Conligx: Modular configuration for evolutionary algorithms via multitask reinforvement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 26982-26990, 2025.
[28] Robert Tjarko Lange. Yingtao Tian, and Yujin Tang. Evolution transformer: In-context evolutionary optimization. arXiv e-prints. pages arXiv-2403, 2024.
[29] Xiaobin Li, Kai Wu, Xiaoyu Zhang, Handing Wang, and Jing Liu. B2opt: Learning to optimize black-box optimization with little budget. arXiv preprint arXiv:2304. 11777 2023.
[30] Bernardino Romera-Paredes. Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, pages 1-3, 2023.
[31] Elliot Meyerson, Mark J Nelson, Herbie Bradley, Arash Moradi, Amy K Hoover, and Joel Lehman. Language model crossover: Variation through few-shot prompting. arXiv preprint a02 2023.
[32] Fei Liu, Xialiang Tong, Mingxuan Yuan, and Qingfu Zhang. Algorithm evolution using large language model. arXiv preprint arX1v:231.11529, 2023.
[33] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.0344 2023.
[34] Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Ych, and Kenneth O Stanley. Evolution through large models. Handbook of Evolutionary Machine Learning, pages 331 331-366. Springer, 2023.
[35] Fei Liu, Xialiang Tong, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang. Zhichao Lu, and Qingfu Zhang. Evolution of heuristies: Towards efficient automatic algorithm design using large language mode, 2024.
[36] Niki van Stein and Thomas Bäck. Llamea: large language model evolutionary algorithm for automatically generating metaheuristics. IEEE Transactions on Evolutionary Computation, 2024.
[37] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.129:: 2023.
[38] Angelica Chen, David Dohan, and David R So. Evoprompting: Language models for code-level neural architecture search. arXiv preprint 2302. 2023.
[39] Muhammad U Nasir, Sam Earle, Julian Togelius. Steven James, and Christopher Cleghorn. Llmatic: Neural architecture search via large language models and quality-diversity optimization. arXiv preprint arXiv:2306.00 01 102, 2023.
[40] Beichen Huang, Xingyu Wu, Yu Zhou, Jibin Wu, Liang Feng, Ran Cheng, and Kay Chen Tan. Exploring the true potential: Evaluating the black-box optimization capability of large language models. arXiv preprint wxXix22444006294 2024.
12

---

[41] Zeyuan Ma. Hongshu Guo, Jiacheng Chen, Guojun Peng, Zhiguang Cao, Yining Ma, and Yue-Jiao Gong. Llamoco: Instruction tuning of large language models for optimization code generation, 2024.
[42] Fei Liu, Xialiang Tong, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang. Zhichao Lu, and Qingfu Zhang. Evolution of heuristics: Towards efficient automatic algorithm design using large language model, 2024.
[43] Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. 2023.
[44] Tung Nguyen, Sudhanshu Agrawal, and Aditya Grover. Expt: synthetic pretraining for few-shot experi- mental design. Advances in Neural Information Processing Systems. 36, 2024.
[45] Tung Nguyen and Aditya Grover. Lico: Large language models for in-context molecular optimization. arXiv preprint arXiv:2406. 18851, 2024.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention all you need. Advances neural information processing systems, 30, 2017.
[47] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 2016.
[48] Kai Wu, Penghui Liu, and Jing Liu. Deen: Automated evolutionary algorithms via evolution inspired deep convolution network, 2023.
[49] Zeyuan Ma, Jiacheng Chen, Hongshu Guo, and Yue-Jiao Gong. Neural exploratory landscape analysis for meta-black-boxx bow.acccbbboooopmmiittiiioo In The Thirteenth International Conference on Learming Representations. 2025.
[50] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 4893-4803, 2022.
[51] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv: 190409910 2019.
[52] Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and clitist multiobjective genetic algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computation, 6(2):182-197, 2002.
[53] Swagatam Das and Ponnuthurai Nagaratnam Suganthan. Differential evolution: A survey of the state-of- the-art. IEEE transactions evolutionary computation, 15(1):4-31, 2010.
[54] Jazzbin. Geatpy: The genetic and evolutionary algorithm toolbox with high performance in python, 2020.
[55] Steffen Finck, Nikolaus Hansen, Raymond Ros, and Anne Auger. Real-parrmmttr black-box optimization benchmarking 2009: Presentation of the noiseless functions. Technical report, Citeseer, 2010.
[56] Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. Coco: platform for comparing continuous optimizers in a black- box setting. Optimization Methods and Software, 14-14:1 2021.
[57] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeicch Zaremba. Openai gym, 2016.
[58] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
13

---

A Tasks, Loss Function & MetaGBT
Tasks. We form training task set TS (f((( [], and j E N], comprising 4N tasks derived from Table I in appendix, where wi denotes the task parameter influencing the function's 's landscape offset. Our selection of these functions for the training task motivated by their diverse landscape features.
MetaGBT. The pseudocode for MetaGBT is presented in Algorithm 2 Initially, we sample the EPOM parameter from a standard normal distribution. The objective of MeiaGBT is to iteratively update to bring closer the global optimum 0* In line 2, sample population for each task in TS. Lines 3, 4 and 5 involve the resampling of task parameters for all tasks in TS, thereby altering the task landscape, augmenting training complexity, and enhancing the learning of robust optimization strategies by EPOM. The final loss function (line 10) is determined by computing the average of the loss functions for all tasks. Subsequently, in line 12, we update @ using gradient- based optimizer, such as Adam [58]. The trained EPOM is then ready for application in solving an unknown BBO problem, as depicted in Algorithm
Algorithm 2 MetaGBT
Input: T. n. training set TS.
Output: The optimal 0.
1: Randomly sample the parameter 0 of EPOM.
2: while not done do
3: Sample [TS] populations of size to obtain the population set pop [X] X2,
_x xx7s
4: for 1,2..... STS] do
5: Randomly sample w' for the f; in TS.
6: end for
7: for = 1, 1,2,...
T do for 1....... TTS do
8:
9:
X < FPOM 110).. x-- wi,
10:
loss + (X,, x end for
11:
12:
A - Update using Adam based on
1 788
Σ loss
end for 14: end while
14

---

B Parameters
The primary control parameters of CMA-ES and L-SHADE are automatically adjusted. For LGA and LES, we utilized the optimal parameters provided by the authors without modifications. Other hyperparameters were tuned using grid search identify the optimal combinations, and multiple experiments were conducted accordingly. Detailed parameter settings are presented in Table Each experiment reports mean and standard deviation the results from various sets of experiments, with consistent population size of 100 across all trials. All experiments are performed on device with GeForce RTX 3090 24G GPU, Intel Xeon Gold 6126 CPU and 64G RAM.
Table 3: Detailed parameter settings for all baselines.
Algorilhm
setting
POM
dmm
1000
Standard Sertings for POM (M).
CMA-FS
Intal
apper handssione hound
Iaital
2/5 leperprameter, determine hyperpraaeee betweon (0.1, 1] nking a grid soarch
steper
sampling dimensional vecser use a grid search dateminn this presmncter, souch interval [1, 10,
lower, lower_bounds + (Fandn((d) + (upper bounds
lower bounds)). where randa (d) distribution
stands
memory. Mise
step
FS
relfin ure
We use grid woarch to determins this
Idu,,
5d
paramdter.
the search
interval is
Nacl
sep we determine hoperpirmeter between [0. using gol worcs, with sep
iopeparameter
using
DE
XOV = 0.5
we dotarmin: this iyportramttr berween [0. 1, 0.00 msng gnd worrch with of
EGA
All
We trained uptimal paramesers provided the authors.
15

---

-10
-20
-2
-30
-40
-40
L25
-00
LSHADE
LSA
LSHADE
16A
LSHADE
20
40
00
80
100
20
40
00
80
100
20
40
60
80
100
F1
(h) F2
F3
0
-10
2.0
EPOR
-20
PON
1.5
-30
LGA
LSHADE
0.5
LSHADE
50
LSHADE
40
100
80
20
80
(d)
(e)
(f) F6
EPOM
EPOM
POM
POM
DE
LES
LGA
LGA
PGN
-20
CMALS
DE
-30
100
(g) F7
(h) F8
(i) F9
-30
EPOM
EPOM
IPON
-20
PON
-20
POM
POM CMAES
CMALS
CHARS
-30
-30
(i) F10
(k) F11
(l) F12
Figure The log convergence curves of EPOM and other baselines on F1-F12. It shows the convergence curve of these algorithms on functions in BBOB with d = 30.
C
Visualization Results of BBOB
The visualization results of all BBOB experiments shown in Figure
16

---

-10
-10
20
CMASS
-15
CHASS
CMAES
-15
-20
LES
-40
LEG
LGA
-25
LOA
LOA
LSHADE
LSMADE
LSHADE
40
60
80
100
20
40
80
100
20
90
100
(a) F13
(b) F14
(c) F15
POM
CHAES
2.5
DS
ES
LES
LGB
-10
LSMADE
3.5
-20
LES
LES
-25
LOA
LSHADE
LGA
LSHADE
20
40
80
100
40
80
100
20
40
90
100
(d)
(e) F17
F18
EPOM
POM
EPOM
CHAES
POM
DE
LES
EPOM
LGA
POM
LSHADE
CHAES
LENADE
LGA
-2
LSHADE
20
40
20
120
F19
(h) F20
(i) F21
CHAES
POM
EPOM
1.2
CHAIS
POM
EPOM
CHAES
EPON
POM
DE
DE
DE
ES
LES
LES
2.5
LES
LSHADE
LSHADE
0.6
0
20
40
60
80
100
20
60
60
80
100
0
20
40
60
80
100
(i) F22
(k) F23
F24
Figure 7: The log convergence curves of EPOM and other baselines on F13-F24. It shows the convergence curve of these algorithms on functions BBOB with d 30.
17

---

-10
EPOM
EPOM
00
BPORT
CHARS
-30
CHABS
-25
PON
DE
-40
E5
-80
LES
-50
LGA
LGA
-10.
100
20
60
100
(a) F1
(b) F2
(c) F3
2.5
3.0
-10
2.5
2.0
PON
-20
CMARS
CHAES
CMAES
15
-90
LES
1.0
LES
LES
LGA
LSA
LGA
LSHADE
0.
LSHADE
-00
LSHADE
20
40
60
80
100
20
80
100
20
40
60
90
100
(d) F4
F5
(1) F6
-10
LPOR
POM
EPON
-20
DE
CMAES
E5%
-30
LGA
LSHADE
BSHADE
100
20
40
50
80
100
20
40
100
(h)
F9
10
-10
EPOM
EPOM
-10
IPOM
POM
PON
CMASS
EMASS
-80
(j) F10
(k) F11
F12
Figure The log convergence curves of EPOM and other baselines FI-F12. It shows the convergence curve of these algorithms on the functions in BBOB with 100.
18

---

11
-10
EPOR
-10
EPOM
PON
EPON
-15
-90
-20
LES
-40
ISHADS
-25
LGA
LSHADE
-00
LOA
LSHADE
40
100
80
100
80
100
(a)
(b) F14
F15
LSHADE
-5
POM
CHAES
EPOM
-10
-15
PON
EPOM
1.5
-0
-2
LSHADE
LSHADE
1.0
100
(d) F16
(e) F17
(f) F18
EPON
POM
EPON
CHARS
ES
LES
LSHADE
CMAES
(g) F19
(h) F20
(i) F21
4.0
EPON
CHASS
POM
BPOM POM
CHARS
CHAES
POM
EPOM
DE
2.0
DE
2.0
LES
ES
LES
LSA
25
LSHADE
LSHADE
10
0.5
20
40
60
80
160
20
40
80
80
100
20
40
80
80
100
(i) F22
(k) F23
F24
Figure 9: The log convergence curves of EPOM and other baselines on F13-F24. It shows the convergence curve of these algorithms on the functions BBOB with 100.
19

---

D Visualization Results of Efficient LMutM
The visualization results of Efficient LMutM are shown in Figure 10
20

---

0.15
0.05
0.00
0.10
(a) step
(h) F1 step 50
(c) F1 step 100
-20
(d) F2 step
(e) F2 step 50
(1) F2 step 100
415
30
415
(g) F3 step
(h) F3 step 50
(i) F3 step 100
(j) F4 step
(k) F4 step 50
(I) F4 step 100
215
111
(m) F5 step
(n) F5 step 50
(0) F5 step 100
Figure 10: F1-F5 Visualization results of some information selection strategies of Efficient LMutM. 0->9 represents the population ranking, and the smaller the ranking, the higher the fitness.
21

---

45
20
(a) F6 step
(h) F6 step 50
(c) F6 step 100
60
11
(d) F7 step
(e) F7 step 50
(I) F7 step 100
(g) F8 step
(h) F8 step 50
(i) F8 step 100
(i) F9 step
(k) F9 step 50
(1) F9 step 100
111
(m) F10 step
(n) F10 step 50
(o) F10 step 100
Figure 11: F6-F10 Visualization results of some information selection strategies Efficient LMutM. 0->9 represents the population ranking, and the smaller the ranking, the higher the fitness.
22

---

50
500
42
(a) step
(b) F11 step 50
(c) F11 step 100
40
(d) F12 step
(e) F12 step
(I) F12 step
50
gg) F13 step
(h) F13 step 50
F13 step 100
415
330
(j) F14 step
(k) F14 step 50
(1) F14 step 100
(m) F15 step
(n) F15 step 50
(0) F15 100
Figure 12: F11-F15 Visualization results of some information selection strategies of Efficient LMutM. 0->9 represents the population ranking, and the smaller the ranking, the higher the fitness.
23

---

(a) F16 step
(h) F16 step 50
F16 step 100
40
(d) F17 step 1
(e) F17 step 50
(1) F17 step 100
(g) F18 step
(h) F18 step 50
(i) F18 step 100
(i) F19 step
(k) F19 step 50
(I) F19 step 100
50
(m) F20 step
(n) F20 step 50
(o) F20 step 100
Figure 13: F16-F20 Visualization results of some information selection strategies of Efficient LMutM. 0->9 represents the population ranking, and the smaller the ranking, the higher the fitness.
24

---

(a) F21 step
(b) F21 step 50
(c) F21 step 100
-0.00
(d) F22 step 1
(e) F22 step 50
(1) F22 step 100
512
2313
455
400
gg) F23 step
(h) F23 step 50
F23 step
(i) F24 step
(k) F24 step
F24 step
Figure 14: F21-F24 Visualization results of some information selection strategies of Efficient LMutM.
0->9 represents the population ranking, and smaller ranking, the higher the fitness.
25

---

-5775
MM
0.32
M0
.5125
0.51
0.51
.5300
0.50
.5075
.550
0.49
.5025
20
100
100
(a) F1 step 1
(b) Fl step 50
(c) F1 step 100
-5550
0.575
-5125
0.55
53.00
0.54
5975
0.500
.5000
MMNU Unnu
0.450
0.51
MUM NW
(d) F2 step
(e) F2 step 50
(f) F2 step 100
0.530
0.505
0.500
0.56
0.490
0.54
0.485
0.52
100
(g) step
(h) F3 step 50
(i) step
0.520
0.52
0.56
0.515
0.51
0.54
0.52
h
m y
MJl
0.48
0.500
0.47
0.44
.00
60
100
20
80
100
20
40
60
100
(i) step 1
(k) F4 step 50
()) F4 step 100
0.57
0.500
.5025
0.54
0.53
0.496
.575
.550
0.51
0.50
20
40
60
80
100
20
40
60
80
100
20
40
60
60
100
(m) F5 step
(n) F5 step 50
(0) F5 step 100
Figure 15: FI-F5 Visualization results of crossover weight of Differentiable LCrM.
E Visualization Results of Differentiable LCrM
The visualization results of Differentiable LCrM are shown in Figure 15 19
26

---

0.60
0.51
Mp Pay hap
0.58
0.52
0.56
0.51
0.54
0.30
0.59
0.49
20
80
100
20
49
100
20
40
80
100
(a) F6 step
(h) F6 step 50
(c) F6 step 100
0.530
0.520
0.515
0.506
0.504
Aur Vormmoo
0.500
0.500
0.500
20
40
60
80
100
20
40
60
80
100
20
40
60
60
100
(d) F7 step
(e) F7 step 50
(1) F7 step 100
0.54
0.525
0.52
0.51
0.520
0.50
0.50
0.49
0.510
0.505
0.48
20
40
60
100
100
20
400
100
1.00
20
40
60
60
100
(g) F8 step
(h) F8 step 50
(i) F8 step 100
11
0.56
0.540
0.54
0.52
0.52
0.535
0.50
0.48
0.48
0.66
0.48
0.520
0.44
20
60
80
1.00
20
80
1.00
20
40
60
(j) FP step 1
(k) F9 step 50
(1) F9 step 100
0.540
0.55
0.54
0.530
0.53
0.525
0.51
0.50
0.10
0.502
(m) F10 step
(n) F10 step 50
(o) F10 step 100
Figure 16: F6-F10 Visualization results of crossover weight of Differentiable LCrM.
27

---

MM
0.54
0.530
0.525
0.50
0.520
0.50
0.515
0.49
0.46
0.10
0.46
20
80
100
40
50
100
20
40
0.0
80
100
(a) F1 step
(b) F11 step 50
F11 step 100
0.510
.5075
0.525
0.508
5080
0.520
5025
0.508
5000
0.504
0.510
0.502
4050
0.505
0.500
20
40
60
80
100
20
4.0
60
80
100
20
40
60
60
1.00
(d) F12 step 1
(e) F12 step 50
(1) F12 step 100
0.54
0.535
VK_
0.53
MM W Madepp
0.530
0.51
0.520
0.50
0.69
0.48
0.510
0.47
0.505
20
40
60
800
100
40
80
100
20
40
60
120
(g) F1.3 step 1
(h) F13 step 50
F13 step 100
0.54
0.54
MMM p
3.54
0.53
0.50
0.45
0.51
0.52
0.09
0.48
0.50
20
80
1.00
20
60
80
100
20
40
60
80
100
(i) F14 step
(k) F14 step 50
(1) F14 step 100
0.55
0.502
0.502
0.54
0.496
0.198
(m) FF5 step
(n) F15 step 50
(o) F15 step 100
Figure 17: F11-F15 Visualization results of crossover weight of Differentiable LCrM.
28

---

Wr My W M
0,520
0.54
0.515
0.53
0.52
8.54
0.505
0.52
0.50
0.495
0.50
20
80
100
100
20
40
80
100
(a) F16 step
(b) F16 step 50
F16 step 100
0.533
0.525
0.505
1 W
4M Myy Ma
ww
0.515
0.486
0.51.
0.505
0.500
0.485
0.485
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
(dl F17 step 1
(e) F17 step 50
(1) F17 step 100
0.56
0.53
0.56
0.55
0.54
3.53
0.50
0.52
0.50
0.48
0.48
0.50
0.47
20
40
60
80
100
20
400
80
100
20
40
60
80
120
(g) F18 step 1
(h) F18 step 50
F18 step 100
0.52
0.52
3.65
0.51
Mall
0.60
0.50
3.55
0.49
0.50
0.48
20
40
60
80
150
20
80
1000
20
40
60
300
(i) F19 step
(k) F19 step 50
(1) F19 step 100
0.515
0.55
My
0.530
0.505
0.56
0.505
0.500
0.52
0.490
(m) IF2 step
(n) F20 step 50
(o) F20 step 100
Figure 18: F16-F20 Visualization results of crossover weight of Differentiable LCrM.
29

---

0.51
0.56
0.50
0.50
0.55
0.49
0.49
100
100
(a) F21 step
(b) F21 Step 50
(c) F21 step 100
0.64
0.62
0.505
0.58
0.50
3.56
0.56
0.49
W WWWWWNNO
0.50
0.450
(d) F22 step
(c) F22 step
(f) F22 step 1000
.5550
0.53
0.515
.5255
53775
0.51
0.505
53500
0.50
.5300
0.990
(g) F23 step
(h) F23 step
F23 step
0.54
0.52
0.515
0.52
M / Ww mi
0.50
0.510
0.49
0.505
0.48
0.500
60
100
20
1000
40
100
(i) F24 step
(k) F24 step 50
(2) step step 100
Figure 19: F21-F24 Visualization results of crossover weight of Differentiable LCrM.
30

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
31

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [es]
Justification: All experimental settings have been described. Once the article accepted, the code will be released.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification:
Once the article is accepted, the code will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
swiiisiscoyyycyy for more details. //nips. httpsscccpppbccccceeceeeeee http://wiieseeeeeeeeeeeeeeeeeeee Caateees/oooSemmieeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
All experimental settings have been described.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
33

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///warpp.ccccaaaa/a//aaaaaa
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Ye]]
Justification:
See Conclusions.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: the paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
36

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
37


Reviews:
[{"review_id": "AX0PTCCOMs", "reviewer": "Reviewer_xpkG", "summary": "This paper addresses the problem of zero-shot optimization by identifying three limitations in existing Pretrained Optimization Models (POMs): (1) limited population modeling, (2) static information exchange, and (3) noisy gradient estimates. To tackle these challenges, the authors propose a new framework called Efficient Pretrained Optimization Models, composed of three components: LMM/LCM Tokenizer, Efficient LMM, and Differentiable LCM. Experimental evaluations are conducted to assess the effectiveness of the proposed approach.", "questions": "Please refer to Weakness.", "limitations": "Please refer to Weakness.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is generally well-written and easy to read at the sentence level.\n2. The identified challenges (e.g., noise, representation, and information flow) are relevant and important for improving zero-shot optimization performance.\n\nWeakness:\n\n1. Unclear Problem Definition and Background. The Introduction section lacks a clear and precise formulation of the research question. Readers unfamiliar with pretrained optimization may struggle to understand what exactly is being addressed. A more structured introduction that defines the zero-shot optimization task, its applications, and the role of POMs would significantly improve clarity.\n2. The motivation is unclear and unconvincing. The authors outlined three limitations of existing works in Pre-trained Optimization Models. However, the motivation for addressing the three limitations is not well-grounded. The current descriptions use abstract and general terms (e.g., \"distribution characteristics,\" \"convergence,\" \"generalizability,\" \"approximate gradients\") without clearly explaining their practical meaning or impact. For instance, the authors mentioned that \"POMs often fail to adequately capture the distribution characteristics of the population, which undermines the convergence and generalizability of the derived optimization strategies.\" In their statements, they typically used broad terms, while did not illustrate them in detail. Specifically, what specific distribution characteristics are important in modeling the population, and how do they affect optimization performance? Why should we capture these characteristics? Besides, \"The reliance on approximate gradients during training can destabilize the model, hindering its overall effectiveness.\" Why does relying on approximate gradients lead to model instability? Are there theoretical or empirical studies supporting this claim? Providing concrete examples, intuitive explanations, or visual illustrations would help clarify the rationale behind the design choices.\n3. Limited Novelty and Coherence. The issues addressed in this work (population modeling, information exchange, gradient estimation) have been studied individually in prior work, as the authors themselves acknowledge. While integrating these three aspects into a single framework could be valuable, the paper does not present strong evidence of novelty in either methodology or insight. Furthermore, the relationship among the three components is not well-articulated. The method appears as a collection of loosely related techniques rather than a cohesive, unified solution. The authors should elaborate on how these components interact and collectively contribute to solving zero-shot optimization.\n4. The experiments are a little hard to follow. For example, it is difficult to understand the performance of various methods from Table 2. It also hard to identify the effectiveness of the proposed method compared with existing methods.", "was_revised": true, "final_justification": "The rebuttal solves my concerns. I raised the related scores.", "rebuttal": "> Weakness 1: Unclear Problem Definition...\n\nThis article mainly addresses the zero-shot optimization problem for black-box optimization, focusing on its continuous optimization scenario. Based on the reviewer's suggestions, we have revised the second to fourth paragraphs of the Introduction to:\n\nTraditionally, solving BBO problems requires carefully designed or heavily tuned optimization algorithms for each individual task. This manual design process is costly, requires expert knowledge, and lacks scalability across diverse domains. To overcome these challenges, the paradigm of zero-shot optimization has emerged. \n\nZero-shot optimization refers to the ability to solve a previously unseen optimization task directly—without any per-task tuning or adaptation. That is, given an unknown objective function $f$, the optimizer is expected to perform well without trial-and-error adjustments. This generalization ability is crucial for real-world applications where human-in-the-loop tuning is impractical.\n\nA promising approach to zero-shot optimization is the use of Pre-trained Optimization Models (POMs) \\cite{li2024pretrained}, which learn general optimization strategies across a diverse training set of functions. POMs aim to transfer this knowledge to solve new BBO tasks efficiently and automatically.\n\n> W. 2: The motivation is unclear...\n\nPre-trained optimization models (POMs) are a universal optimization framework designed to achieve zero-shot optimization. Their core idea is to learn the mapping between task features and optimization strategies, enabling the model to directly output high-quality optimization strategies without fine-tuning for new tasks.\n\nWe summary this process into three key elements: task feature modeling, optimization strategies expression, and a mapping mechanism from task features to optimization strategies. First, the task feature set, typically consisting of normalized fitness values and centralized rankings of individuals, characterizes the structural characteristics of the optimization task. Its expressive power directly determines the POM's ability to perceive and adapt to different tasks. Second, the optimization strategies, as the output of the POM, must possess sufficient expressive power to cover search behaviors across diverse tasks. The mapping from features to strategies, learned during the pre-training phase, is key to achieving zero-shot performance.\n\nAlthough existing POM methods have achieved some success, significant bottlenecks remain in feature modeling, strategies expression, and training stability. To this end, this paper proposes systematic improvements around the three core elements mentioned above, building a POM system with enhanced generalization and training robustness:\n\nEnhanced Task Feature Modeling: To improve the expressiveness of task features, we propose the LMM/LCM Tokenizer. This module utilizes cross-attention to construct representations of decision variables of arbitrary dimensions and concatenates them with the original features, resulting in a richer and more structure-aware problem representation.\n\nEnhanced Optimization strategies Representation: We introduce deformable attention into the original LMM module, proposing the Efficient LMM, to enhance flexibility and information exchange during strategies generation, achieving significant performance improvements on multiple benchmark tasks.\n\nEnhanced Training Stability: To overcome the gradient estimation inaccuracies caused by Gumbel-Softmax, we design a differentiable crossover mechanism, Differentiable LCM. This replaces the original non-differentiable crossover process with a weighted summation, effectively improving the stability and convergence quality of the training process.\n\nIn summary, by synergistically enhancing feature modeling, strategies representation, and mapping mechanisms, this paper promotes the development of pre-trained optimization models with stronger zero-shot optimization capabilities and wider adaptability.\n\n> W. 3: Limited Novelty...\n\nPlease see our rebuttal to Weakness 2.\n\n> W. 4: The experiments are ...\n\nWe hope that the following explanation of Table 2 can increase its readability. \n- Header $d$ means the problem dimension (30, 100), F means test problems (F1, F2, ..., F24), and EPOM, POM, ..., LGA are the evaluated methods.\n- Bottom $+/=/-: x/y/z$ shows that among the $24 \\times 2$ tasks (24 BBOB functions in 2 dimension settings), EPOM is better than the comparison algorithm on $x$ tasks, comparable to the comparison algorithm on $y$ problems, and weaker than the comparison algorithm on $z$ problems.\n- Data For the data $a(b)$ in the table, $a$ represents the mean fitness of the last generation of population in the three average tests, and $b$ represents its standard deviation. The bold data represents the best among all methods, and the underlined data represents the second best or equally best.\n\nIf there are any other issues regarding our Experiments section, we would be happy to discuss them."}, {"review_id": "CiyVm3Bq9p", "reviewer": "Reviewer_gTGN", "summary": "This paper proposes the Efficient Pre-trained Optimization Models which embeds landscape cues into population tokens for richer modeling and seamless feature fusion, and introduces the deformable attention module to enable adaptive, diversity-promoting interactions. Experimental results demonstrate superior performance over both traditional BBO methods and existing pre-trained algorithms.", "questions": "1. What is the meaning and value of $\\bar{d}$ in Section 3.2.2? How to shape and reshape the solutions?\n2. In Section 3.2.3, what is the meaning of prob(x)? How to use the top-p sampling to obtain the $n \\times k$ indices in $N_t$?\n3. It seems that $m_i$ is different across $x_i$, how to determine the value of $m_i$ and why is they different?\n4. How to calculate $S^t$ using $N_t$?\n5. Since solutions are directly tokenized, can EPOM generalize across problems with different search spaces?\n6. The population size n and $d_{out}$ determine the shape of some of the network weights, they should be known values before training and testing. Why do lines 181-183 suggest uncertainty between $d_{out}$ and n?\n7. What is EPOM’s time efficiency compared to traditional BBO methods and pre-trained methods?", "limitations": "1. Though promising in the generalization across problem types and dimensions, EPOM might not be generalizable across population sizes.\n2. EPOM significantly reduces the required expert knowledge in BBO optimization, however, it might introduce more computational complexity nd slow down the optimization.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The proposed method efficiently embeds the optimization population and generates the next population using neural networks without handcrafted optimization strategies. This significantly reduces reliance on human effort and expert knowledge in algorithm design.\n2. The proposed loss function could balance the population diversity and convergence which is essential for effective optimization.\n3. EPOM demonstrates strong generalization ability across both synthetic tasks and neuroevolution tasks, as well as problems of varying dimensions.\n\nWeaknesses:\n1. The related work could be richer. Surveys for meta-learned BBO algorithms [1][2] could enrich the paper. Recent meta-learned algorithm ConfigX [3] and LLM for optimization method LLaMEA [4] are also worth mentioning. Besides, including NeurELA [5] which shares the similar idea of using Attention to embed population would also benefit the paper. \n2. The shapes of some network weights depend on population size n, hindering EPOM's generalization across different population sizes.\n3. In the Tokenizer, the 2-dimensional $H^t$ is concatenated with the $d_T \\times d_{1V}$-dimensional $E^t$ which has significantly higher dimension than $H^t$. It might dilute the contribution of $H^t$.\n\n\nOther issues:\n1. The reference for Q-Mamba links to MetaBox instead of its intended source, it might be a mistake.\n2. Section 3.1.1 2) LCM appears to lack an equation that utilizes introduced variables. Additionally, the shape of $W_{2V}$ in line 180 is incompatible for multiplication with $G_t$. In Equation (3) $K_{1M}$ might need a transpose.\n3. In Algorithm 1, line 9, $V^t$ should be a parameter of LCM().\n\nSuggested references:\n\n[1] Ma, Zeyuan, et al. \"Toward automated algorithm design: A survey and practical guide to meta-black-box-optimization.\" IEEE Transactions on Evolutionary Computation (2025).\n\n[2] Yang, Xu, et al. \"Meta-Black-Box optimization for evolutionary algorithms: Review and perspective.\" Swarm and Evolutionary Computation 93 (2025): 101838.\n\n[3] Guo, Hongshu, et al. \"Configx: Modular configuration for evolutionary algorithms via multitask reinforcement learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 25. 2025.\n\n[4] van Stein, Niki, and Thomas Bäck. \"Llamea: A large language model evolutionary algorithm for automatically generating metaheuristics.\" IEEE Transactions on Evolutionary Computation (2024).\n\n[5] Ma, Zeyuan, et al. \"Neural exploratory landscape analysis for meta-black-box-optimization.\" The Thirteenth International Conference on Learning Representations. 2025.", "was_revised": true, "final_justification": "The authors provided a through rebuttal that addressed my concerns. I keep my original assessment toward acceptance.", "rebuttal": "> Weakness 1: The related work ...\n\nWe appreciate the author's comment and will enrich our Related Work section according to the advice.\n\n> W. 2 The shapes of ...\n\nWe believe that the \"some network weights\" mentioned by the reviewer refer to the attention network used to identify the individuals to be focused on, as described in Section 3.2.3. In fact, for the following reasons, we set the output dimension $d_{out}$ of such networks to a fixed value independent of the population size (as stated in the submission: \"... and $d_{out}$ is the fixed output dimension.\", and we set $d_{out}$ to 100 in this submission):\n(1) To avoid additional computational overhead. When the population size $n > d_{out}$, we identify the individuals to be focused on only from the first $d_{out}$ candidates in the population, thereby avoiding any increase in the computational burden of deformable attention. When $n < d_{out}$,  we clip the network output and retain only the first n dimensions. This reason should also help address the reviewer's concerns in Question 6 and Limitation 1.\n(2) To preserve generalization ability. As the reviewer pointed out, if  $d_{out}$ were determined by the population size $n$, EPOM's generalization ability across different population sizes would be compromised. By setting it to a fixed value, we mitigate this issue to some extent.\n\n> W. 3: In the Tokenizer, ...\n\nWe also took this issue into consideration when designing the LMM/LCM Tokenizer. Therefore, we suggest that both $d_T$ and $d_{1V}$ should not be set too large. In this submission, we set $d_T = d_{1V} = 4$ , resulting in a 16-dimensional decision variable feature. When concatenated with the original 2-dimensional feature, its contribution would not be overly diluted.\n\n> Other Issues 1: The reference for ...\n\nWe thank the reviewer for pointing out this citation issue. We will correct it in the next revision.\n\n> O. I. 2: Section 3.1.1 2) ...\n\nWe sincerely thank the reviewer for carefully identifying this issue. We acknowledge that some equations related to variables such as $\\mathbf{cv}^t _ i$ were regrettably omitted in the current version. We provide the missing equations below:\n$$\n\\mathbf{cr}^t \\leftarrow LCM(\\mathbf{Z}^t) \\\\\n\\mathbf{r}^t _ i = rand(d), \\; \\; \\mathbf{cv} _ i^t = gumbel\\_softmax(cat(\\mathbf{r}^t _ i, tile(cr _ i^t, d))) \\\\\n\\mathbf{u}^t _ i = \\mathbf{cv} _ {i, 0}^t \\cdot \\mathbf{x} _ i^t + \\mathbf{cv} _ {i, 1}^t \\cdot \\mathbf{v} _ i^t\n$$\nWe thank the reviewer for pointing out these two writing errors. First, the dimension of $\\\\mathbf{W}_{2V}$ should be $(\\hat{d}_t, d_{out})$ , not $(n, d_{out})$. Then, we indeed missed the transpose operation on $\\mathbf{K}_{1M}$ \nin the attention computation in Equation (3).\n\nWe sincerely thank the reviewer once again for pointing out issues (a) and (b). We will make sure to correct them in the next revision.\n\n> O. I. 3: In Algorithm 1 ...\n\nTo clarify,  $\\mathbf{V}^t$ is not an input parameter of Differentiable LCM. Instead, it is processed by the LCM Tokenizer, which converts  $\\mathbf{V}^t$ into  $\\mathbf{F}^t$. The Differentiable LCM then operates on  $\\mathbf{F}^t$ and  $\\mathbf{E}^t$ to generate the offspring population  $\\mathbf{U}^t$. Thus, while  $\\mathbf{U}^t$ is indeed an input to the overall LCM pipeline, it is not directly used by Differentiable LCM.\n\n> Questions 1: what is ...\nIn this submission, we define $\\bar{d} = \\lceil \\frac{d}{2} \\rceil$. The operations of shaping and reshaping a solution are straightforward and are detailed as follows: \n1. Padding (if necessary):\nIf the dimensionality of the solution vector $\\mathbf{x}$ is odd, we append a scalar 0 to its end to obtain a new vector $\\hat{\\mathbf{x}}$ of length $d+1$; otherwise, we set $\\hat{\\mathbf{x}} = \\mathbf{x}$\n2. Shaping into a matrix:\nWe then shape $\\hat{\\mathbf{x}}$, which is of length $2 \\cdot \\bar{d}$, into a $2 \\times \\bar{d}$ matrix $\\hat{\\mathbf{K}}_T$.\n\nfor example, consider a 5-dimensional solution vector:\n$\\mathbf{x} = [0.3, 1, 4, 9, 7]$  \nSince $d=5$ is odd, we append 0 to obtain:\n$\\hat{\\mathbf{x}} =[0.3, 1, 4, 9, 7, 0]$\nThen we apply the $shape$ operation:\n$$\n\\hat{\\mathbf{K}}_T = shape(\\hat{\\mathbf{x}}) = \n\\begin{eqnarray}\n\\begin{bmatrix}\n0.3 & 1 & 4 \\\\\n9 & 7 & 0 \\\\\n\\end{bmatrix}\n\\end{eqnarray}\n$$\nThe $reshape$ operation is simply the inverse, i.e., flattening $\\hat{\\mathbf{K}}_T$ row-wise to recover $\\hat{\\mathbf{x}}$.\nBoth $shape$ and $reshape$ operations can be efficiently implemented in Python using tensor manipulation methods, such as $view()$ in PyTorch.\n\n> Q. 2: In Section 3.2.3 ...\n\nWe clarify the meaning of $prob(\\mathbf{x}^t_i)$ and the process of top-$p$ sampling as follows:\nFor a given individual $\\mathbf{x}^t_j$, we define $prob(\\mathbf{x}^t_i)=\\hat{\\mathbf{N}}_{t, (j, i)}$ as the importance score (i.e., probability) of $\\mathbf{x}^t_i$ being selected as a focused-on individual from the perspective of $\\mathbf{x}^t_j$. The full matrix $\\hat{\\mathbf{N}}_t$ stores all such pairwise importance scores between individuals in the population.  To obtain the final index matrix $\\mathbf{N}_t$, we perform the following 3 steps for each individual:\n\n1. Sorting:\nFor each row of $ \\hat{\\mathbf{N}}_t$, which corresponds to one individual, we sort all other individuals in descending order of importance scores.\n2. Top-$p$ sampling:\nFor the $j$-th individual, we progressively select top-ranked individuals until the cumulative score exceeds a predefined threshold $p$. Since the importance distributions differ across individuals, the number of selected individuals $m_j$ will also vary. $m_j$ is calculated by the equation below. \n$$\n\\mathbf{\\alpha} _ j = sort(\\hat{\\mathbf{N}} _ {t, (j, :)}) \\quad \\text{(in descending order)} \\\\\nm_j = \\min \\{ m \\;|\\; \\sum_{i=1}^{m} \\mathbf{\\alpha} _ j[i] > p \\}\n$$\n\n3. obtaining $\\mathbf{N} _ t$: \nLet $k = max\\{m _ j\\}^n _ {j=1}$. We collect the top-$k$ indexes in $\\mathbf{\\alpha} _ j$ for each individual, and stack them to form the final index matrix, $\\mathbf{N} _ t \\in \\mathbb{R}^{n \\times k}$.\n\nWe hope this explanation also addresses the reviewer’s concern raised in Question 3.\n\n> Q. 3: It seems that ...\nPlease see our rebuttal to Question 2.\n\n> Q. 4:\n\nWe should clarify that $\\mathbf{N} _ t$ is used to calculate $\\mathbf{V}^t$, instead of $\\mathbf{S}^t_i$. We would like to explain this process in individual-wise.\nGiven the  index vector $\\mathbf{N} _ {t, (j, :)}$, corresponding to the top-$k$ focused individuals of $\\mathbf{x}^t _ j$, as well as the full population $\\mathbf{X}^t$,  and their features $\\mathbf{E}^t$. Firstly, we collect the focused-on features $\\mathbf{E}^t _ j \\in \\mathbb{R}^{k \\times \\hat{d}_t}$ according to indexes in $\\mathbf{N} _ {t, (j, :)}$. Then, in a similar way with LMM in POM, we obtain the query vector $\\mathbf{q} _ j$, key matrix $\\mathbf{K} _ j$ by $\\mathbf{e} _ j$ and $\\mathbf{E}^t _ j$, respectively, and calculate $\\mathbf{S}^t _ i \\in \\mathbb{R}^{1 \\times k}$. Finally, we collect $\\mathbf{X}^t _ j \\in \\mathbb{R}^{k \\times d}$ in the same way as $\\mathbf{E}^t _ j$, and calculate $\\mathbf{v}^t _ j =\\mathbf{S}^t _ i \\times  \\mathbf{X}^t _ j$.\n\n> Q. 5\n\nAs showed in lines 261–265 of our submission, EPOM does demonstrate a certain degree of generalization capability across numerical search spaces. However, its ability to generalize across problems involving different data types remains an open question. We believe that the LMM/LCM Tokenizer has the potential to support such generalization, and this is indeed a promising direction for future research. We thank the reviewer for highlighting this important point and would be glad to further discuss and explore it in future work.\n\n> Q. 6 \n\nPlease see our rebuttal to Weakness 2.\n\n> Q. 7\n\nWe recorded the time consumption (in seconds) of EPOM, POM and CMAES when optimizing 24 BBOB functions in 30- and 100-dimensional settings respectively, as shown in the following table.\n||30|100|\n|---|---|---|\n|EPOM|22.29(0.22)|23.10(0.36)|\n|POM|**4.84(0.24)**|**5.57(0.23)**|\n|CMAES|11.34(0.24)|33.99(1.99)|\nIn summary, POM, thanks to its streamlined model structure, achieved the lowest time consumption in both settings. CMAES, however, lacks GPU acceleration, and while it consumes less time than EPOM in the low-dimensional setting, it consumes approximately 10.89 seconds more than EPOM in the 100-dimensional setting. For EPOM, the increased time and computational effort required for performance improvement, driven by its increased model complexity, is almost an inevitable price to pay.\n\n> Limitation 1\n\nPlease see our rebuttal to Weakness 2.\n\n> Lim. 2\n\nPlease see our reply to **Question 7**."}, {"review_id": "ZhsGZlIxXC", "reviewer": "Reviewer_NPVS", "summary": "This paper proposes Efficient Pretrained Optimization Models (EPOM), a method for solving black-box optimization problems without task-specific training. EPOM builds off of prior Pretrained Optimization Models (POM) in a few ways: The addition of an LMM/LCM tokenizer which embeds the population into fixed-length tokens to allow the model to capture more landscape cues, a deformable attention module for the LMM, and replacing the simple gumbel softmax LCM with a learned module that has stable gradients. Evaluation on a suite of black box problems show better performance and convergence compared to prior methods.", "questions": "1. Could you clarify the results in table 2?\n\nThe paper makes multiple claims about how EPOM is especially good with higher dimensions, d=100, but table 2 seems to suggest that it's worse in this case. In fact, it does worse than POM more often than not for d=100. Furthermore, there seems to be a miss match between the plots in figures 6-9 of the appendix and table 2 mentioned previously. I lean reject with these issues outstanding, but I'm willing to increase my recommendation based on the authors response.\n\n2. How much deviation from a crossover weight of 0.5 is significant?\n\nFigure 5(c)(d) shows very little deviation in the crossover weight (0.49-0.52), but the text claims that this is significant. I would imagine replacing this with a static 0.5 would lead to very similar performance. Looking at the appendix, there do seem to be cases where the crossover weight has much more variance, but these examples weren't selected for figure 5. Is there a pattern we can draw for when the crossover weight is close to 0.5 vs when it is not?\n\n3. How does the runtime of EPOM compare to POM?\n\nThe appendix discusses the compute requirements, but doesn't say how this compares to POM. Does EPOM require much more or less compute than POM?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The overall quality is good, the plots are well made with nice error bars and figure 1 is clear. Figure 4 is a bit confusing with small text and overlapping lines, I think a simple table would work better, an also a better explanation of which function is being optimized.\n\nThe clarity seems fair. The methods section is very clear with the description of each component and loss/bbox functions. There are a few things that seem unclear. First, there seems to be some discrepancies with figures 6-9 in the appendix and table 2. For example, f4 with d=100 shows better performance for POM in table 2, but has EPOM doing better in figure 8 (d). Additionally, section 4.2 claims EPOM is especially good with d=100, but table 2 shows that it's more often worse than POM for d=100 (POM is better 13/24 tasks). Finally, the crossover weight doesn't deviate much from 0.5 but the text doesn't justify if this is significant.\n\nOverall the significance also seems good. EPOM seems to converge much better in many problems which is promising, however it adds much more complexity and possibly compute. I have some concerns with the results mentioned previously that make me unsure if conclusions drawn are accurate. \n\nThe originality is also good. There are many novel adjustments made to POM and they are justified with ablations. The performance improvements would certainly be of interest to the larger community.", "was_revised": true, "final_justification": "My main concern with this paper was with unclear results in both the main table and appendix. The authors provided clarifications to my main concerns, which better explain the results and alleviate my worry. They also provide additional analysis on runtime and crossover weight which further justifies the method. For these reasons I now recommend acceptance.", "rebuttal": "> Weakness 1: Figure 4 is a bit confusing ...\n\nWe hope the following explanation will help the reviewer better understand Figure 4. First, Figure 4 is a critical difference diagram, which visualizes the performance differences among the given algorithms on specified datasets based on their average rankings. The results of our ablation study are summarized in this figure.  \nSpecifically, we remove each proposed module one by one and evaluate the corresponding variant of EPOM on 24 BBOB functions in 30 dimensions. For each function, we compute the ranking of all variants and then calculate the average rank, which is shown as the number above each polyline in Figure 4.  \nWe suspect that what the reviewer referred to as “overlapping lines” mainly concerns the lines representing the “No Differentiable LCM” and “No LMM Tokenizer” settings. This overlap is due to their similar average rankings. To address this issue, we will increase the figure’s size and resolution in the next revision, which should alleviate the visual ambiguity.\n\nAdditionally, following the reviewer’s suggestion, we have made the following table based on the average rankings in Figure 4, which we hope will further clarify the results and assist in interpreting the figure. \n| Settings | Average Rank |\n|----------|--------------|\n| EPOM | 1.6250 |\n| No LCM Tokenizer | 1.8750 |\n| No Differentiable LCM | 3.2500 |\n| No LMM Tokenizer | 3.2917 |\n| No Efficient LMM | 4.9583 |\n| No SM | 6.0000 |\n\n> W. 2: The clarity seems fair ...\n\n**(a)** We sincerely apologize for our oversight and the confusion it may have caused the reviewer. The discrepancies between Table 2 and Figures 6–9 are mainly caused by their different calculation methods. For each entry $m(n)$ in Table 2, $m$ and $n$ represent the mean and standard deviation of the fitness of the full population, for a given function, method, and dimension setting. Fig. 6-9 represent the optimization trajectory curves consisting of the best fitness in each generation. Once again, we sincerely apologize for not describing these two experimental setups clearly in our submission, and for any confusion this may have caused.\n\n**(b)** Following **(a)**, our conclusion are drawn based on the best fitness of the last generation of each method, because the core goal of black-box optimization algorithms is to find the optimal solution as much as possible, rather than to make all solutions close to the optimal. The best fitness of the last generation can better reflect both this goal and the best performance of a given algorithm. \n\n**(c)** We tested EPOM with constant crossover weight(0.5) and varying crossover weight on the BBOB in a 30-dimensional setting. On some functions, their performance is not much different, but on some functions, their performance is quite different, as shown in the following table (the best fitness of the last generation is recorded).\n\n||f1|f11 |f16|\n|---|---|---|---|\n|cr=0.5|1.69e-55(1.90e-55) |5.55e-39(7.85e-39) |21.99(3.60)|\n|varying cr|8.85e-54(5.45e-54) |4.24e-50(2.81e-50)|32.55(2.30)|\n\n> W. 3: however it adds much more complexity ...\n\nWe recorded the time consumption (in seconds) of EPOM, POM and CMAES when optimizing 24 BBOB functions in 30- and 100-dimensional settings respectively, as shown in the following table.\n||30|100|\n|---|---|---|\n|EPOM|22.29(0.22)|23.10(0.36)|\n|POM|**4.84(0.24)**|**5.57(0.23)**|\n|CMAES|11.34(0.24)|33.99(1.99)|\n\nIn summary, POM, thanks to its streamlined model structure, achieved the lowest time consumption in both settings. CMAES, however, lacks GPU acceleration, and while it consumes less time than EPOM in the low-dimensional setting, it consumes approximately 10.89 seconds more than EPOM in the 100-dimensional setting. For EPOM, the increased time and computational effort required for performance improvement, driven by its increased model complexity, is almost an inevitable price to pay.\n\n> Question 1: Could you clarify ...\n\nPlease see our rebuttal to Weakness 2 **(a)** and **(b)**.\n\n> Q. 2: How much deviation ...\n\nPlease see our rebuttal to Weakness 2 **(c)**.\n\n> Q. 3: How does the runtime ...\n\nPlease see our rebuttal to Weakness 3.\n\n> Paper Formatting Concern 1: There might be ...\n\nWe sincerely thank the reviewer for carefully identifying this issue. We acknowledge that some equations related to variables such as $\\mathbf{cv}^t_i$ were regrettably omitted in the current version. We provide the missing equations below:\n$$\n\\mathbf{cr}^t \\leftarrow LCM(\\mathbf{Z}^t) \\\\\n\\mathbf{r}^t _ i = rand(d), \\; \\; \\mathbf{cv} _ i^t = gumbel\\_softmax(cat(\\mathbf{r}^t _ i, tile(cr _ i^t, d))) \\\\\n\\mathbf{u}^t _ i = \\mathbf{cv} _ {i, 0}^t \\cdot \\mathbf{x} _ i^t + \\mathbf{cv} _ {i, 1}^t \\cdot \\mathbf{v} _ i^t\n$$\nWe will make sure to correct them in the next revision.\n\n> P. F. C. 2: I think figure 4 ...\n\nPlease see our rebuttal to Weakness 1."}, {"review_id": "3gDfA1znX7", "reviewer": "Reviewer_X9vv", "summary": "The work presented in this paper, titled \"Enhancing Zero-Shot Black-Box Optimization...\", introduces a framework referred to as EPOM. This framework is designed to improve upon a prior method, Pre-trained Optimization Models (POMs), for the task of zero-shot black-box optimization. The authors identify three primary weaknesses in the existing POM framework—namely, insufficient population modeling, inefficient information exchange, and unstable gradient approximation—and propose a corresponding set of architectural solutions. These solutions involve: (1) a new tokenizer module to create richer embeddings of the solution population; (2) a mutation operator, which they call an \"Efficient LMM,\" based on deformable attention to facilitate adaptive interactions; and (3) a crossover operator, which they call a \"Differentiable LCM,\" that utilizes a weighted-sum mechanism to improve the stability of the meta-training process. The authors provide an empirical evaluation on the BBOB benchmark suite, where their proposed EPOM model demonstrates superior performance compared to the baseline POM and other established optimization algorithms.", "questions": "My current rating is 'Borderline Reject' due to the severe issues with clarity and conceptual framing. For the paper to reach an acceptance status, the authors must undertake a major revision to address the following points. A successful revision that thoroughly resolves these issues would be necessary to reconsider my evaluation.\n\nRewrite for Clarity and Accessibility: The paper must be substantially rewritten. All bespoke acronyms, especially the misleading 'LMM' and 'LCM', must be clearly defined upon their first use in the Abstract and Introduction, or preferably, replaced with less ambiguous terms. The convoluted sentence structures should be simplified to make the core ideas accessible to a general machine learning audience.\n\nExplicitly Position the Work within Meta-Learning: The paper must situate itself correctly within the meta-learning literature. The introduction and related work sections need to explicitly discuss how \"Pre-trained Optimization Models\" relate to and build upon decades of research in meta-learning. The authors should either justify their novel terminology or adopt the standard language of the field.\n\nClarify Architectural Contributions: Given the complexity of the system, a clearer ablation is needed. What is the single most impactful component? Is the performance gain primarily from the deformable attention mechanism, the tokenizer, or the differentiable crossover? A more fine-grained analysis would help distill the core engineering takeaway of the paper.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe primary strength of this work lies in its empirical results. The experimental evaluation is extensive and appears to be robust, showcasing that the proposed EPOM system consistently outperforms a range of strong baselines, including the original POM architecture and classical methods like CMA-ES, on a challenging benchmark suite. The engineering effort is clearly substantial, and the final model is demonstrably effective at its stated goal.\n\nWeaknesses:\nDespite the strong empirical results, the paper is beset by significant weaknesses in its presentation, clarity, and conceptual positioning, which collectively form a significant barrier to its understanding and potential impact.\n\nClarity and Terminology: The writing, particularly in the Abstract and Introduction, is convoluted and appears almost intentionally gatekeeping. It is laden with bespoke jargon and acronyms that are not immediately defined, making the paper exceptionally difficult to penetrate for any reader not already deeply familiar with the specific baseline paper [7]. This lack of clarity is a major flaw.\n\nMisleading Acronyms: The use of the acronyms 'LMM' and 'LCM' for \"Learned Mutation Module\" and \"Learned Crossover Module\" is exceptionally poor practice and must be addressed. In the broader machine learning community, LMM almost universally stands for Large Multimodal Model, while LCM has been recently associated with Large Context Models or Latent Consistency Models. Defining these overloaded acronyms deep within the paper (Section 3.1.1) is insufficient. This choice is not merely confusing; it is actively misleading and suggests a lack of awareness of the wider field.\n\nConceptual Positioning: The work is presented as \"zero-shot optimization\" via \"Pre-trained Optimization Models,\" creating an artificial distinction from the well-established field of meta-learning. A reader well-versed in meta-learning is left to wonder why this new terminology is necessary when the described process—training a model on a distribution of tasks to generalize to new tasks—is the very definition of meta-learning. This failure to connect to, or even acknowledge, the foundational concepts of the field it operates in is a critical conceptual weakness.\n\nIn summary, while the engineering is sound, the poor writing, confusing terminology, and flawed conceptual framing make the work hard to digest and will severely limit its potential impact and its ability to be integrated into the broader discourse on meta-learning.", "was_revised": true, "final_justification": "See my response to rebuttal for full details. \n\nTLDR; Authors addressed most of my concerns satisfactorily and I am now happy to accept the paper.", "rebuttal": "> Weakness 1 Clarity and Terminology ...\n\nWe rewrote the Abstract and Introduction to highlight the motivation, novelty, and coherence of the paper.\n\nAbstract\nZero-shot optimization aims to achieve both generalization and performance gains on solving previously unseen black-box optimization problems over SOTA methods without task-specific tuning. Pre-trained optimization models (POMs) address this challenge by learning a general mapping from task features to optimization strategies, enabling direct deployment on new tasks.\n\nIn this paper, we identify three essential components that determine the effectiveness of POMs: (1) task feature modeling, which captures structural properties of optimization problems; (2) optimization strategy representation, which defines how new candidate solutions are generated; and (3) the feature-to-strategy mapping mechanism learned during pre-training. However, existing POMs often suffer from weak feature representations, rigid strategy modeling, and unstable training.\n\nTo address these limitations, we propose EPOM, an enhanced framework for pre-trained optimization. EPOM enriches task representations using a cross-attentive tokenizer, improves strategy diversity through deformable attention, and stabilizes training by replacing non-differentiable operations with a differentiable crossover mechanism. Together, these enhancements yield better generalization, faster convergence, and more reliable performance in zero-shot black-box optimization.\n\n\nIntroduction\n\n... Traditionally, solving BBO tasks requires expert-designed or heavily tuned optimization algorithms for each new problem, making the process inefficient and non-scalable. Thus, the paradigm of solving new optimization problems without any task-specific tuning is extremely valuable.\n\nMeta-learning, or learning to learn [1], particularly in the context of meta-black-box optimization [2] or learning to optimize [3], covers a broad range of scenarios. These approaches often leverage information from the target task to enhance the optimizer’s performance on that specific task. However, they typically exhibit poor generalization to new, unseen tasks. Even though some methods—such as LES \\cite{lange2023discovering1} and LGA \\cite{lange2023discovering}—can generalize to novel settings, their performance still falls significantly short compared to state-of-the-art optimizers specifically designed for those settings.\n\nTo better highlight our contribution, we introduce the concept of zero-shot optimization, which aims to simultaneously achieve strong generalization and high performance on new optimization problems—without requiring task-specific training. A particularly promising direction in this area is the development of pre-trained optimization models (POMs). These models are designed to learn general-purpose optimization behaviors across a wide variety of training tasks and effectively transfer that knowledge to previously unseen problems.\n\n\nWe view a POM as a function that maps task-specific features to optimization strategies. This process involves three key components:  \n(1) **Task feature modeling**, which captures characteristics of the problem (e.g., normalized fitness values and centralized rankings);  \n(2) **Optimization strategy representation**, which determines how candidate solutions are generated and refined;  \n(3) **Mapping mechanism**, which learns the transformation from features to strategies during pre-training.\n\nAlthough recent POMs have made progress, they remain limited in three areas:  \n- The expressiveness of task features is insufficient to generalize across diverse problem structures;  \n- The strategy generation process lacks flexibility and diversity;  \n- The training process suffers from gradient instability due to non-differentiable operations.\n\nTo overcome these limitations, we propose **EPOM (Enhanced Pretrained Optimization Model)**, which systematically enhances all three components:\n- **Feature modeling** is improved by a cross-attentive tokenizer that captures decision-variable-level information and encodes it into fixed-length task representations;\n- **Strategy representation** is enhanced through deformable attention, enabling dynamic and diversity-aware interactions among population members;\n- **Training stability** is ensured by replacing sampling-based crossover operations with a differentiable, weighted-sum mechanism, leading to smoother gradient flow.\n\nBy jointly improving feature modeling, strategy generation, and training robustness, EPOM achieves superior generalization and performance in zero-shot black-box optimization. This work contributes to the growing body of meta-learning literature on learning-to-optimize methods, and offers a scalable pathway toward universal optimization agents.\n\n> W. 2: Misleading Acronyms ...\n\nThank you for your suggestion. We have revised the two terms to: Learned Mutation Module (LMuM) and Learned Crossover Module (LCrM) to distinguish them from the existing concepts.\n\n> W. 3: Conceptual Positioning ...\n\nWe thank the reviewer for raising this important point. While our work is indeed related to meta-learning, we use the term zero-shot optimization to emphasize a more specific setting: directly generating optimization strategies for unseen tasks without task-specific adaptation, based solely on task features.\n\nMeta-learning, particularly meta-black-box optimization [1], encompasses broader scenarios such as algorithm selection, configuration, and solution manipulation. These settings do not align exactly with our goal of optimization strategy generation with strong generalization and superior performance. Similarly, terms like learning to optimize often imply inner-loop adaptation, which our method avoids.\n\nThus, introducing the term zero-shot optimization helps clarify our contribution: achieving both generalization and performance gains over SOTA methods without per-task training. We explicitly define this term in the introduction (Lines 25–27) and compare with meta-learning baselines in both related work and experiments, where our method demonstrates superior effectiveness.\n\nTherefore, the use of this terminology is intentional and necessary, rather than a conceptual flaw.\n\n[1] Hospedales, T., Antoniou, A., Micaelli, P., & Storkey, A. (2021). Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9), 5149-5169.\n\n\n[2] Z. Ma, H. Guo, Y. -J. Gong, J. Zhang and K. C. Tan, \"Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimization,\" in IEEE Transactions on Evolutionary Computation, doi: 10.1109/TEVC.2025.3568053.\n\n[3] Chen, T., Chen, X., Chen, W., Heaton, H., Liu, J., Wang, Z., & Yin, W. (2022). Learning to optimize: A primer and a benchmark. Journal of Machine Learning Research, 23(189), 1-59.\n\n> Question 1: Rewrite for ...\n\nBased on your suggestions, we will simplify complex sentences.\n\n> Q. 2: Explicitly Position ...\n\n In the introduction, we discussed the differences and connections between pre-trained optimization base models and meta-learning. For details, see the second and third paragraphs of the Introduction section.\n\n> Q. 3: Clarify Architectural Contributions ...\n\nWe thank the reviewer for their suggestions. We will enrich the ablation study in the revision and conduct in-depth analysis of each module."}, {"review_id": "k5XfCc4rY9", "reviewer": "Reviewer_P2q1", "summary": "This paper proposes EPOM (Efficient Pretrained Optimization Models), a new architecture for zero-shot black-box optimization. It improves on previous POMs by introducing (1) richer population tokenization (LMM/LCM Tokenizers), (2) deformable attention for better adaptive interaction between individuals (Efficient LMM), and (3) Differentiable LCM for more stable training gradients. The method is extensively benchmarked on BBOB test functions and an RL locomotion task, achieving improved performance and convergence over state-of-the-art population-based optimization methods.", "questions": "- Can you provide more justification for not comparing EPOM to LLM-based optimization approaches? Are there settings where such methods might be competitive?\n\n- Have you tested EPOM on real-world scientific or engineering problems (e.g., materials discovery, control design), beyond synthetic BBOB tasks?\n\n- What is the computational cost of EPOM during training and inference, compared to POM or other methods? Is the method practical for problems with extremely large dimensions?\n\n- Does EPOM require large-scale pretraining datasets? How would it perform in low-data regimes?", "limitations": "Yes, included in the conclusion section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Addresses an important and challenging problem: efficient zero-shot black-box optimization.\n\n- Proposes a technically sound architecture with improvements over existing POM methods.\n\n- Uses attention mechanisms and improved tokenization to enhance performance.\n\n- Shows strong experimental results on BBOB benchmarks (d=30 and d=100).\n\n- Provides good ablation studies and analysis of module contributions.\n\nWeaknesses\n\n- The innovations are mostly architectural — no fundamentally new learning paradigm.\n\n- Experiments are limited to BBOB and a simple RL control task — lacks validation on more realistic scientific or engineering problems.\n\n- No comparison to recent LLM-based optimizers (though the authors argue they are weaker for this problem class).", "was_revised": true, "final_justification": "The authors addressed most of my concerns, so I will keep my positive score to support the acceptance of this paper.", "rebuttal": "> Weakness 1: The innovations ...\n\nOur submission first summarizes the principle of strong generalization of POM and then enhances it. The details are as follows:\n\nTheir core idea is to learn the mapping between task features and optimization strategies, enabling the model to directly output high-quality optimization strategies without fine-tuning for new tasks.\n\nWe summary this process into three key elements: task feature modeling, optimization strategies expression, and a mapping mechanism from task features to optimization strategies. First, the task feature set, typically consisting of normalized fitness values and centralized rankings of individuals, characterizes the structural characteristics of the optimization task. Its expressive power directly determines the POM's ability to perceive and adapt to different tasks. Second, the optimization strategies, as the output of the POM, must possess sufficient expressive power to cover search behaviors across diverse tasks. The mapping from features to strategies, learned during the pre-training phase, is key to achieving zero-shot performance.\n\nAlthough existing POM methods have achieved some success, significant bottlenecks remain in feature modeling, strategies expression, and training stability. To this end, this paper proposes systematic improvements around the three core elements mentioned above, building a POM system with enhanced generalization and training robustness:\n\nEnhanced Task Feature Modeling: To improve the expressiveness of task features, we propose the LMM/LCM Tokenizer. This module utilizes cross-attention to construct representations of decision variables of arbitrary dimensions and concatenates them with the original features, resulting in a richer and more structure-aware problem representation.\n\nEnhanced Optimization strategies Representation: We introduce deformable attention into the original LMM module, proposing the Efficient LMM, to enhance flexibility and information exchange during strategies generation, achieving significant performance improvements on multiple benchmark tasks.\n\nEnhanced Training Stability: To overcome the gradient estimation inaccuracies caused by Gumbel-Softmax, we design a differentiable crossover mechanism, Differentiable LCM. This replaces the original non-differentiable crossover process with a weighted summation, effectively improving the stability and convergence quality of the training process.\n\nIn summary, by synergistically enhancing feature modeling, strategies representation, and mapping mechanisms, this paper promotes the development of pre-trained optimization models with stronger zero-shot optimization capabilities and wider adaptability.\n\n> W. 2: Experiments are limited ...\n\nWe evaluate the performance of EPOM and POM on the planner mechanic arm problem, a engineering problem that has been widely used to evaluate the performance of BBO algorithms. The optimization goal of this problem is to search for a set of lengths $\\mathbf{L} = [l_1, l_2, \\cdots, l_n]$ and a set of angles $\\mathbf{\\alpha} = [\\alpha_1, \\alpha_2, \\cdots, \\alpha_n]$ to minimize the distance from  to the target position based on a given $r$, represents the distance from the target point to the origin of the mechanic arm . In our experiment, we let $n=30$, $r=100$, fix the length $l_i = 10$, and search for the optimal angles only.  \nWe used POM and EPOM to optimize the problem for 100 generations each, and recorded the best fitness (minimum distance) of the last generation. The optimal value achieved by EPOM was **2.63(1.07)**, and that of POM was 8.09(2.98).\n\n> W. 3: No comparison ...\n\n LLM-based optimizers lack zero-shot optimization capabilities, so comparison with them is meaningless. Related work describes this capability as being related to pre-trained models.\nThis may not be our weakness.\n\n> Question 1: Can you provide ...\n\nPlease see our rebuttal to **Weakness 3**.\n\n> Question 2: Have you tested ...\n\nPlease see our rebuttal to **Weakness 2**.\n\n> Q. 3: What is the ...\n\nWe recorded the time consumption (in seconds) of EPOM, POM and CMAES when optimizing 24 BBOB functions in 30- and 100-dimensional settings respectively, as shown in the following table.\n|Method|d=30|d=100|\n|---|---|---|\n|EPOM|22.29(0.22)|23.10(0.36)|\n|POM|**4.84(0.24)**|**5.57(0.23)**|\n|CMAES|11.34(0.24)|33.99(1.99)|\nIn summary, POM, thanks to its streamlined model structure, achieved the lowest time consumption in both settings. CMAES, however, lacks GPU acceleration, and while it consumes less time than EPOM in the low-dimensional setting, it consumes approximately 10.89 seconds more than EPOM in the 100-dimensional setting. For EPOM, the increased time and computational effort required for performance improvement, driven by its increased model complexity, is almost an inevitable price to pay.\n\nEPOM can generalize well across dimensions. We tested the performance of POM and EPOM on BBOB in a 300-dimensional setting and recorded their best fitness in the last generation on each function, part of the results are shown in the following table.\n\n|method |f1|f8|f15|f22|\n|---|---|---|---|---|\n|EPOM|2.88e-54(3.21e-54)|5.24e-18(7.41e-18)|3.05e-51(4.31e-51)|83.78(0.45)|\n|POM|1.40e-10(6.54e-11)|9.55e-6(5.89e-6)|1825.98(247.68)|79.65(0.54)|\n\n> Q. 4: Does EPOM ...\n\nIn our submission, EPOM was trained on only five functions under the 10-dimensional setting (as shown in Table 1). Compared to the 24 BBOB functions in the 30- and 100-dimensional settings used for testing, the training set can be considered a low-data regime. Despite this, EPOM demonstrates strong performance in the experiments, indicating that it does not require a large-scale pretraining datasets."}]

Author rebuttal:

Meta-review: This paper presents Efficient Pretrained Optimization Models (EPOM), a novel framework for zero-shot black-box optimization. The authors identify three key weaknesses in existing Pretrained Optimization Models (POMs)—poor population modeling, static solution interaction, and unstable gradient estimation—and introduce a series of well-motivated architectural improvements to address them. The core contributions include a new tokenizer that better captures landscape cues, a deformable attention module to promote solution diversity, and a differentiable crossover mechanism for more stable training. Together, these components create a more robust and efficient optimization model.

I recommend this paper for acceptance. It's a technically solid work that demonstrates strong empirical results on challenging benchmarks, clearly advancing the state of the art. The reviewers initially raised several important concerns, particularly regarding the clarity of the writing, motivation, and experimental results. However, the authors' thorough rebuttal successfully addressed these points, convincing all reviewers to converge on a positive recommendation. The paper tackles a significant problem, and its contributions are valuable to the optimization community, and I believe even more broadly to the NeurIPS community at large. I encouraged the authors to make sure they carefully integrate the excellent feedback from the reviewers and the discussion phase to further polish the paper for the camera-ready version. Great work!
Decision: Accept (poster)