SOO-Bench: Benchmarks for Evaluating the Stability of Offline Black-Box Optimization
Authors: Hong Qian
Yiyi Zhu
Xiang Shu
Shuo Liu
Yaolin Wen
Xin An
Huakang Lu
Aimin Zhou
Ke Tang
Yang Yu
Venue: ICLR 2025 Poster (2025)
Keywords: Offline Optimization
Black-Box Optimization
Stability
Benchmarks
Primary area: datasets and benchmarks

Abstract: Black-box optimization aims to find the optima through building a model close to the black-box objective function based on function value evaluation. However, in many real-world tasks, such as the design of molecular formulas and mechanical structures, it is perilous, costly, or even infeasible to evaluate the objective function value of an actively sampled solution. In this situation, optimization can only be conducted via utilizing offline historical data, which yields offline black-box optimization. Different from the traditional goal that is to pursue the optimal solution, this paper emphasizes that the goal of offline optimization is to stably surpass the offline dataset during optimization procedure. Although benchmarks called Design-Bench already exist in this emerging field, it can hardly evaluate the stability of offline optimization and mainly provides real-world offline tasks and the corresponding offline datasets. To this end, this paper proposes benchmarks named SOO-Bench (i.e., Stable Offline Optimization Benchmarks) for offline black-box optimization algorithms, so as to systematically evaluate the stability of surpassing the offline dataset under different data distributions. Along with SOO-Bench, we also propose a stability indicator to measure the degree of stability. Specifically, SOO-Bench includes various real-world offline optimization tasks and offline datasets under different data distributions, involving the fields of satellites, materials science, structural mechanics, and automobile manufacturing. Empirically, baseline and state-of-the-art algorithms are tested and analyzed on SOO-Bench. Hopefully, SOO-Bench is expected to serve as a catalyst for the rapid developments of more novel and stable offline optimization methods. The code is available at \url{https://github.com/zhuyiyi-123/SOO-Bench}.

Published as a conference paper at ICLR 2025
SOO-BENCH: BENCHMARKS FOR EVALUATING THE STABILITY OF OFFLINE BLACK-BOX OPTIMIZATION
Hong Qian, Yiyi Zhu, Xiang Shu, Shuo Liu, Yaolin Wen, Xin An, Huakang Lu, Aimin Zhou" East China Normal University, China (hgian, amzhou) @cs ecnu. edu. en, yiy zhu ecnu wcnu.eduu edd.
Ke Tang
Yang Yu Nanjing University, China Polixir Technologies, China yuy@nju. edu. en
Southern University of Science and Technology, China tangk3@susteeh.. edu. on
ABSTRACT
Black-box optimization aims to find the optima through building model close the black-box objective function based on function value evaluation. However, many real-wol world tasks, such the design of molecular formulas and mechanical structures, it is perilous, costly, or even infeasible to evaluate the objective function value of actively sampled solution. In this situation, optimization can only be conducted via utilizing offline historical data, which yields offline black-box optimization. Different from the traditional goal that is pursue the optimal solution, this paper emphasizes that the goal of offline optimization is to stably surpass the offline dataset during optimization procedure. Although benchmarks called Design- Bench already exist in this emerging field, it can hardly evaluate the stability of offline optimization and mainly provides real-world offline tasks and the corresponding offline datasets. To this end, this paper proposes benchmarks named S00-Bench (i.e., Stable Offline Optimization Benchmarks) for offline black-box optimization algorithms,
to systematically evaluate the stability of surpassing the offline dataset under different data distributions. Along with SOO-Bench, we also propose stability indicator to measure the degree of stability. Specifically, S0O-Bench includes various real-world offline optimization tasks and offline datasets under different data distributions, involving the fields of satellites, materials science, structural mechanics, and automobile manufacturing. Empirically, baseline and state-of-the-an algorithms are tested and analyzed on SOO-Bench. Hopefully, SOO-Ber -00-Benchh3 is expected to serve as a catalyst for the rapid developments of more novel and stable offline optimization methods. The code is available at hhtps! 7/qithub com zhuyiy 23/5000-///0
INTRODUCTION
Black-box optimization (BBO) is widely used in scientific and engineering fields, for example, hyper-parameter tuning (Soock et al. 2012). BBO only needs function evaluation and does not require explicit function expression. To search for the optimal solution, fundamental ingredient in model-based BBO is constructing surrogate model to approximate the black-box objective function based on actively sampled solutions and their evaluated function values. However, in many real- world tasks, such as exploring drug molecular structures (Gaulton al. (012) and designing hardware mechanical structures YYazdanbakhsh et al. 202T |Reagen et al., 2017). it is perilous, costly or even infeasible actively sample solutions and evaluate their objective function values. In these cases, optimization can only be conducted via utilizing a limited number of offline historical data. That is to say, optimization algorithms need to make full use of the offline dataset in hand to learn the surrogate model and determine the good solution Jin et al. (021)) This yields offline model-based black-box optimization, abbreviated as offline optimization
Trabucco et al. 2021 Tan
2025
"Corresponding author.

---

Published as a conference paper at ICLR 2025
A significant challenge in offline optimization is the narrow distribution of data within the offline dataset (Trabucco et al. 2021: Brookes et al. 2019; Fannjiang & Listgarten 2020 CCen et all 2022,
2023 Yu et a |021 Q1 et al. 2022 Fu & Levine 2021p. In many real- situations, we can only use narrow data distributions for offline optimization due to data collection strategies. For instance, when adjusting mechanical structure parameters, the offline dataset typically includes only the test data that the experimenter already possesses. However, these test data may be influenced by the subjective opinions of the experimenters and may not evenly cover the solution space.
Current research on offline optimization algorithms primarily focuses on finding the optima of a black-box function TTrbbucc et al., 2021, Brookes al. 2019; Fannjiang & Listgarten 2020 Chen et al..
2022: 2023; Yu et al. 202 Qi et al, [2222] Fu & Levine, 2022,, Yuan et al., 2023| Zbu et al.
2025 However, the surrogate model in offline optimization can be increasingly misled in the area
uncovered by the offline dataset due to the narrow distribution issue. It means that the surrogate model could greatly overestimate the objectíve value of optima far from the offline dataset covered regions Trabucco et al. 202T) and results in degradation during the optimization procedure (Lu et al. 2023p. Under this observation, we point out that stability is equally important as optimality for a comprehensive evaluation of offline algorithms. Herein, stability refers to the algorithm' 's ability to consistently surpass the offline dataset as much as possible during the optimization process without being misled by the narrow data distribution. To the best of our knowledge, this paper is the first work to highlight stability as one of the core objectives in offline black-box optimization and conduct a systematically study of it. Moreover, in real-world scenarios. the degree of narrow distribution varies and is difficult to be estimated beforehand. Therefore, assessing an algorithm's 's stability and optimality under different levels of narrow distributions also crucial. Unfortunately, existing benchmark for offline optimization, called Design-Benche (Trabucco et al. 2022 have limitations
and inabilities in these aspects. The narrow distribution in Design-Benncte is artificially constructed and fixed, and there is established indicator for evaluating the stability of offline optimization algorithms. The field of offline optimization urgently requires benchmark capable of comprehensive algorithm evaluation in stability and optimality, so to boost the developments of stable offline optimization (SOO) approaches.
In response to the aforementioned demands, this paper further proposes benchmarks named SOO- Bench to evaluate offline optimization algorithms' stability and optimality. Specifically, SOO-Bend provides (1) werldworld offline optimization tasks including satellites, materials science, structural mechanics and so on, (2) customizable narrow distribution levels to tailor the difficulty levels of offline optimization datasets, (3) a novel stability indicator called Stability
indicator
(SO) to measure the stability and optimality of algorithms. Besides, SOO-Bench also introduces the constrained offline optimization problems. Empirically. baseline and state-of-the-art (SOTA) algorithms are tested and analyzed on S00-Bench. By incorporating these features, SOO-Bench is able to provide a more comprehensive evaluation of offline optimization algorithms. The codes of c00-Bench can be found at https:t://........... //t::////ttttt com/
The subsequent sections respectively recap the related work, present the preliminaries, introduce the proposed SOO-Bench, depict the tasks and datasets in benchmarks, show the empirical analysis, and finally conclude the paper.
RELATED WORK
Although offline BBO has received widespread attention due to its application in real-world problems, there is currently only one comprehensive benchmark suite in this emerging field. Specifically, a benchmark platform called Design-Bench (Trabucco al.] 2022). which is unfortunately no longer maintained now, provides basic environments for offline optimization testing. It covers offline black- box optimization tasks in multiple fields, such as neural architecture search (oph & Le 2017 DNA sequence design (A et al. 2016), drug discovery (Gaulton et al., 2012), and robot design (Ahn et al. 2019; Brockman et al. 2016). Although Design-Bench was launched as the first comprehensive benchmark suite, it only provides the basic interfaces for offline optimization, fails to analyze the narrow distribution of offline datasets fully, and does not provide a corresponding test environment. Besides. SDDObench Zhong et al. 2024 evaluates streaming data-driven evolutionary algorithms, focusing on the need for standardized test suites in dynamic optimization, which distinguishes it from our approach. So, it is urgent to propose new benchmark to address the aforementioned issues.
2

---

Published as a conference paper at ICLR 2025
Construct Offine Datasets
Offline Optimization & Find Solutions for Online Application
Sampled distribution
Sample
Black Blx finction
Offline data
Online
Original distribution
Surrogate model
Best
Application
Probutlity
Optimum
by Offline BBO
Tasks
Offline Dataset
Offline Optimization Method
IN Y
x
Recommendstory Solutions
{(m, 96
Leaming Surrogare Model Opeimization
Figure I: The process offline black-box optimization. The left sub-figure shows the process of collecting offline data from real black-box objective function. The right sub-figure shows the process of an offline optimization algorithm finding satisfied solutions for online application.
In order to address the narrow distribution of offline datasets, various offline black-box optimization algorithms are proposed handle it. For example, COMs (Trabucco et al. 2021) proposes a conser- vative model to penalize the proxy function value far from the offline dataset solution. IOM (Qi et al.] 2022) adds an inertial regularization term to force the optimized model to maintain similar represen- tations under different data distributions, thereby alleviating the model's 's performance degradation on the data far from the offline dataset. CbAS (Brookes et al. 2019) models the generative model and uses variational autoencoder (Kingma & Welling 2014 build model to find the optimal solution in the trust region within an acceptable uncertainty region. CL-DDEA (Huang & Gong 2022) introduces contrastive learning model enhance data-driven evolutionary algorithms by focusing on pairwise comparisons rather than absolute fitness values, while CC-DDEA (Gong et al. ) employs a hierarchical surrogate model and cooperative coevolution address. Although relevant algorithms are designed to address the narrow distribution of offline datasets and attempt to find better solutions areas near the offline datasets, there currently no comprehensive benchmark suite available to test these algorithms comprehensively.
3 OFFLINE BLACK-BOX OPTIMIZATION
3.1 PROBLEM STATEMENT
Let f: X' - I be black-box function, where X C Rd is d-dimensional solution space. In black-box optimization, the target to find an optimal solution x" that maximizes f. which can be written as xx arg m fxx). is called objective function and X' is called solution space. However, in the offline optimization scenario, direct interaction with the objective function only be performed by accessing static offline dataset D (x,, y: 1 containing N solutions and their objective values. As shown in Figure ! most offline optimization methods use 'D to train model fo(m) to fit the objective function via supervised learning: 0t arg ming Ei (fo( y:)², where O represents the parameters of the model. Subsequently, the solution appp for online optimization found by the learned model fo (x)) For example, use gradient ascent iterated T times to find the optima of the surrogate model, i.e.,
is not allowed, and optimization
x (t+1) f- (1) 1 (x))==
1,2,
(1)
where Taap (T) is the solution output at the terminating condition set by certain time step T for the online application. Since the exact value of T at which the optimization should stop in practical applications is unknown, and in order to avoid repeatedly adjusting the number of optimization steps for different datasets, is important that all offline black-box optimization algorithms are designed to remain stable throughout the process. This stability ensures that the worst value of the online evaluation remains relatively favorable during the optimization process.
An offline BBO method gradually learns the characteristies of the offline dataset by training model and then inferring the online objective function. Thus, even if the offline dataset contains only local optimal solutions, the offline optimization algorithms take advantage of the information within
3

---

Published as a conference paper at ICLR 2025
OfTinne Dulisct with
Customiized Narrow Ditribution
Objective Function Value
Hemose
Frequeney
Toperia
Alguriihm Algoritim
M
Optimal Solution in Offline
Objective Function Value
Optimization Step
(a) illustration customized distribution.
(b) An illustration of SO on different algorithms.
Figure 2: The illustrations of narrow data distribution issue consider and address in SOO-Bench and the motivation of stability indicator.
the dataset, identify relations among variables, especially analyze the characteristics of these local optima, and subsequently discover new solutions that surpass the best solution in the dataset.
3.2 NARROW DISTRIBUTION OFFLINE DATASETS
In offline BBO, the data distribution of offline datasets usually does not cover the entire solution space. In real-world tasks, experimenters often obtain offline datasets through expensive real experiments. For some reasons, such experimenters' subjectivity when constructing offline datasets where personal biases, preferences for certain parameter settings, or prior knowledge might lead to uneven sampling- the data distribution of offline datasets cannot evenly cover all solution values in the entire solution space. For example, as shown in the "Sampll" of Figure [] the black curve represents the distribution of evaluation values of solutions uniformly sampled from the solution space, while the blue curve represents the distribution of evaluation values of the offline dataset. It can be observed that the solutions in the offline dataset mainly lie in a narrow region, which results the out-of- distribution (OOD) issue (Kumar & Levine 2020 Trabucco al.) 2021 Brookes et al. 2019 Chen et al. 2023) studied previous work. As a result, some areas of solution space lack prior data information, which causes the algorithm to be unable to model these areas accurately. This narrow distribution of offline datasets presents the greatest challenge for offline optimization problems.
Therefore, when the offline BBO methods select solution far from the offline dataset as the optimal solution in the solution space, it must exercise extreme caution. Specifically, when providing the optimal solution, the algorithm should consider the trade-off between attempting to find better solution to surpass the optima of the offline dataset and not moving too far away from the offline dataset ensure stability. Hence, the data distribution of the offline datasel directly determines the difficulty level of the offiiee oplimization stability. Controlling the distribution of the offline dataset is particularly significant for comprehensively testing the stability of offline BBO algorithms.
4 THE PROPOSED 2OO-BENCE
This section introduces SOO-Bench, more comprehensive benchmark suite evaluate the stability of offline black-box optimization algorithms. Compared with previous work, S00-Bench considers the customized distribution of offline datasets, allowing construct offline optimization tasks of different difficulties. SOO-Bench for the first time introduces the stability indicator, enabling quantitative evaluation of algorithmic stability.
4.1 CUSTOMIZED NARROW DISTRIBUTIONS OFFLINE DATASETS
The core issue of offline optimization is that the data distribution of offline datasets is usually narrow. The narrower the data distribution covers the entire objective function search space, the more difficult the offline optimization is. To offer benchmark suite for comprehensive algorithmic stability and optimality performance evaluation, SOO-Bende provides offline optimization tasks with adjustable difficulty, achieved by customizing the distribution of offline datasets.
4

---

Published as a conference paper at ICLR 2025
First, we provide black-box ground-trute truth oracle objective function for each task. An initial dataset is obtained by uniformly sampling and evaluating the objective function. Then, the initial dataset is sorted by the value of the objective function to obtain its distribution, shown in the curve in Figure 2 (a). Finally, datasets of different difficulty levels are constructed based on this sorted initial dataset. Specifically, the top-n% solutions by objective function value removed to increase the difficulty of finding high-quality solutions, and the bottom-m* solutions are removed to increase the sparsity of the dataset. Through the above steps, an offline dataset with narrow distribution in real tasks is constructed by removing solutions, as illustrated in Figure 2 (a).
4.2 THE PROPOSED INDICATOR FOR MEASURING STABILITY
Another problem in offline BBO is the lack of indicators to evaluate algorithm stability, which directly affects the performance of the optimal solution found by the algorithm. We point out that the stability of an offline algorithm is reflected by its ability to continuously and stably optimize solutions that exceed the best solution in the offtine dataset throughout the entire optimization process. For example, algorithms A and B can find solutions surpassing the offline dataset's optimal solution, as shown in Figure 2 (b). However, due to the poor stability of Algorithm B, although it found better solution at the end of the optimization process, it failed to maintain this solution throughout the entire optimization process. In reality, when an offline BBO algorithm provides an online solution, is often unclear which step of the optimization process should be considered. Therefore, the requirement for the algorithm to maintain a stable and high-quality solution throughout the entire optimization process becomes a critical factor affecting its effectiveness.
SOO-Bench first introduces the Stability-Optimallit Optimality indicator (SO) to measure algorithmic stability. Specifically, for a task with an offline optimal solution #OFF (i.e., the best solution in the offline dataset), after T optimization steps, the online evaluation value of the optimal solution provided by
the algorithm is denoted as fxx), where = 1,2,.... T. and SO is then defined follows:
SO 2- 2-01(()-5(0)
(2)
SI(t) + Ol(t)
S
8
and O1 = Si SI(() S22
(3)
where S = ET= f(x;) represents the cumulative sum of the evaluation values curve, S = T. ((ZOFF) represents the product of the evaluation value of the optimal offline solution and total optimization steps T, and S2 = T max, f (x:;) represents the product of the evaluation value of the optimal solution found by the algorithm and total optimization steps T..
Intuitively, as shown in Figure 2 (b), the Optimality Indicator (OI) reflects the ratio of the area under the algorithm's 's evaluation curve to the area under the offline optimal solution curve, indicating the algorithm's 's optimality. On the other hand, the Stability Indicator (SI) represents the ratio of the area under the algorithm's 's evaluation curve to the area under the best solution found by the algorithm, reflecting stability by measuring how closely the algorithm's performance aligns with the ideal solution. When evaluation values are stable and approach the optimal value, the SI value nears 1, signaling better stability. Conversely, significant fluctuations or deviations from the optimal value result in a lower SI. The OI value increases as the algorithm approaches the global optimum and decreases when the algorithm's performance is consistently below the optimal offline solution. Since stability refers to the algorithm''s 's ability to consistently surpass the offline dataset as much as possible during the optimization process, we must take both Ol and SI into consideration. This results in the proposed
indicator (SO) to measure algorithmic stability, as shown in Eq.
Sometimes, when we assign different weights to Ol and SI, indicating our degree of preference of OI and SI, it is expressed as
01(1) ST(() SO w() ST(() + (1-((t)) OI(t)*
where w E (0, 1). If = 0.5, SO,, degenerates to SO. w > 0.5, the significance of OI is greater. If < 0.5, the significance of SI is greater. When > 0.5. to improve SO... a desirable offline optimization algorithm should prioritize the improvement of OL. This means that the offline optimization algorithm should place more emphasis on the algorithm's ability of optimality. When
5

---

Published as a conference paper at ICLR 2025
Table An overview of the tasks in the proposed SOO-Bench. SOO-Bench includes several tasks, which are divided into four benchmarks. These tasks have discrete and continuous design spaces and constrained and unconstrained situations to suit offline black-box optimization. Nvar indicates the number of variables and Noon means the number of constraints. The symbol "/" means that the global optimal function value of the real-world black-box function is unknown.
Benchmarks
Tasks
Nvar
N con
Optimum
gtopx 1
Cassini
6
4
4.9307
gtopx 2
Cassini
22
0
8.383
gtopx 3
Messenger (reduced)
18
8,6299 0
gtopx 4
Messenger (full)
26
1.9579 0
GTOPX
gtopx 5
GTOC
8
-1581950 6
gtopx 6
Rosetta
22
0
-1.3433
glopx 7
Cassini 1-MINLP
10
3.5007 4
cec
Optimal operation
7
14
-4,52912
Process flow
3
3
1.07654
CEC
cec
Process synthesis
2
2
2
cec
Three-bar
2
3
2.63896
cec
Welded beam
4
5
1.67022
protein 1 protein 2
TF Bind 8 TF Bind 10
8
0
/
PROTEIN
10
0
< 0.5, desirable offline optimization algorithm should place more emphasis the algorithm's ability of stability. In this work, we employ a linearly decreasing weight function, i.e., w(() max T. This setting w() ensures that a desirable offline black-box optimization algorithm should focus improving optimality (i.e., OI) during the early steps of optimization process, while in the later steps it should shift its priority to stability (i.e., SI). This procedure can prevent the performance degradation caused by out-of-distribution issue. The setting (() can be adjusted according to the user's specific needs.
4.3 EXTENSIVE TASKS, DATASETS AND BASELINES FOR EVALUATING STABILITY
SOO-Bench provides an extensive real-world benchmark environment, and all task details are introduced in Section 5 Besides, SOO- -eoOBBenchee reproduces variety of baseline algorithms and tests their performance through experiments, which are introduced in Section 6 We provide users with more customizable APIs. In addition to controlling task difficulty by adjusting the distribution of offline datasets, users can also create noisy offline datasets to test algorithm performance specifically.
TASKS AND DATASETS DESCRIPTION
This section presents the details of all tasks and datasets. An overview is shown in Table !| Despite the tasks vary a lot, we provide a unified API on our datasets. Each task in SOO-Bench provides ground-truth oracle objective function f(x). In addition offering basic offline datasets, also includes an API for constructing offline datasets, allowing users to create datasets with different difficulties. We provide a detailed description of the tasks the four benchmarks Appendix A which is placed in the supplementary material, Our benchmarks and example implementations are available at http:p///ggttbbbb.o...////u///// 22111233//0OOc-/---000
httmm//huyyyyy1.13
GTOPX: Space Mission Optimization (Schlueter et al. 202T). This paper addresses a set of real-world space mission trajectory problems (European Space Agency and Advanced Concepts
Team 2020), designed numerical black-box optimization problems. The objective is to minimize
the total velocity variation during interplanetary space missions. Our benchmark consists of seven GTOPX tasks. Satellite missions particularly challenging, especially when the oracle evaluation encounters an invalid value, which returns as NaN, we indicate it as the worst value in the offline dataset. If the solution found the offline optimization algorithm results in such value, the true

---

Published as a conference paper at ICLR 2025
objective function value is poor, and the algorithm proves to be very unstable. The license for this dataset is GNU General Public License.
CEC Task: Industrial and Design Optimization (Kumar al. 2020). We select five real-world constrained problems, which are maximization tasks with constraints that are feasible when they greater than equal to zero. Constrained tasks crucial in real life. When normalizing or denormalizing these tasks, the solutions found may exceed the variable boundaries. If some methods are applied set bounds, the solutions may be suboptimal. Even if the algorithm demonstrates good stability, if the optimization results are poor, the overall outcome is still considered unsatisfactory. We aim for the offline optimization algorithm find high-quality solutions while maintaining stability throughout the optimization process. The license for this dataset is CC-BY 4.0 License.
DNA Sequence Optimization (Trabucco al. 2022b. The variable space comprises sequences of four categorical variables. The objective of the two tasks is to identify the optimal 8-nucleotide DNA sequence that exhibits the highest binding affinity a specific transcription factor. Discrete tasks play a crucial role in offline optimization. However, in these tasks, offline optimization may identify non-existent molecules. The license for this dataset MIT License.
PROTEIN:
EXPERIMENTS
To simulate a more realistic data distribution, we choose dataset size that is 1000 times the variable dimension. At the same time, to further simulate the narrow distribution, the missing m% near the worst objective function value and the missing n% near the optimal objective function value are used to ensure that the data volume small and missing near the optimal value, as shown in Figure 2 (a). In this paper, we select the middle 50% of the data (i.e., m% n% 50%) to construct simulated dataset as a reasonable bascline without leveraging any prior knowledge. Furthermore. when constructing the constraint dataset, set the ratio of satisfying constraints not satisfying constraints as : 3. Given the high flexibility and customizability of the proposed benchmark, users can modify the data volume, m%, n%, and constraint proportions as needed. Meanwhile, the proposed benchmark can adjust both the data volume size and the distribution of solutions in the offline dataset. For more details, please refer to our Github.
6.1 OFFLINE OPTIMIZATION ALGORITHMS
We test range of baselines and SOTA offline optimization algorithms on each of task. Specifically, we compare with two categories of algorithms: (1) those that address unconstrained problems, including classic baselines: BO-qEI (Wilson al., 2017). CMA-ES (Hansen) 2006). and Offline
BBO methods: autofocusing CbAS (Fannjiang & Listgarten 2020p. TTDDEA Huang et al. 2021).
ARCOO (Lu et al. 2023), Tri-mentoring (Chen et al. 2023). and (2) those that address constrained problems, including CARCOO, CCOMs, DDEA-PF, DDEA-SPF (Huang & Wangl 2021a). Since classical methods lack specific methodologies for offline optimization problems, they cannot be directly applied to such problems. Therefore, we introduce guided training agent model to provide optimization guidance through agent prediction, thereby indirectly solving the offline optimization problem. In this section, we briefly discuss these algorithms and evaluate them in the next section. Due to page limitation, the implementation codes links are listed in Appendix B
BO-qEI: Offline Bayesian optimization is performed to minimize learned surrogate function (x)) by training an ensemble of neural network models. Candidate solutions are generated with a Gaussian Process model and labeled with values from f(x)). We employ the quasi-expected Improvement (qEI) acquisition function within the BoTorch framework (Balandat et al. 2019) CMA-
ES: We compute the value of learned surrogate function f(x) the samples w,, which is obtained from the distribution N (14, Σx) at an iteration t. We then adapt the covariance matrix to refine the belief distribution, repeating this process multiple times. Autofocusing CbAS: Autofocusing CbAS learns a density model (x) of æ to approximate the data distribution, then gradually use importance sampling to re-training f(x) under current distribution Pr (æ)), and adapts it towards the optimized solution. ARCOO: ARCOO constructs the surrogate model f(x) combined with the energy model, which is used to characterize the risk of degradation. After construction, a risk suppression factor is applied control the risk. Tri-mentoring: This approach constructs three
7

---

Published as a conference paper at ICLR 2025
Table 2: Overall results for GTOPX unconstrained tasks with 128 solutions and 100th percentile evaluations are averaged over eight runs. f ((FFF) represents the optimal objective function value in the offline dataset. FS (i.e., final score) refers to the function value found by an offline optimization algorithm at ( 50 during the optimization process, which consists of total of = 150 optimization steps. this case, 25% of the values are missing near the worst value and another 25% near the optimal value.
199
1:33mm
133
114rs
11dum
115 Dh-4
6.50.0
Lokos
130
155mm
1-28
145mm
113kmm
13
T0004
34hm
800me
-83 (bacc
-Soce
100ma
18tmes
209 Serie
6.3me
6.0mm
Table 3: Overall results for GTOPX constrained tasks with 128 solutions and 100th percentile evaluations. In this case, 25% of the values are missing near the worst value and another 25% near the optimal value. Details are the same as Table 2
CHIFA
GROPK
CAR000
Litelanne
1400%
0.0kee
696 Pige
8'9,am
377
M
surrogate models and uses ranking supervision signals for mutual mentoring. After that, we can use adaptive soft-labeling to learn more accurate labels. TTDDEA: By dividing the offline dataset into three equal parts randomly, we build three surrogate models respectively. We select high- confidence pseudo-label to fill in each other's s datasets, then retrain the surrogate model, generate offspring, and repeat this multiple times. CCDDEA: CCDDEA designs hierarchical surrogate-joint learning model to be able to guide the evolving population to search at different granularities and then optimize the global and local subspace levels in a cooperatively coordinated evolutionary manner. CARCOO: This is simple constrained version of ARCOO. We incorporate the degree of constraint violation into the risk assessment and consider solutions that violate the constraints as high risk. CCOMs: This is reproduction version of the PRIME (Kumar et al., 2022) method. Since we do not find its codes, we simply implement PRIME according to the paper, that is, making the objective function value that violates the constraint as bad as possible. DDEA-PF & DDEA-SPF: It combines common constraint processing techniques with offline data-driven evolutionary algorithms to handle constrained optimization problems. This is achieved by constructing proxy models for constraints and objective functions, respectively, and using constraint processing techniques to process constraint proxy models and objective proxy models.
6.2 RESULTS AND ANALYSIS
This scenario represents normal settings real-world conditions where the maximum and minimum values of the objective values are proportionally missing. We conduct a total of T = 150 optimization steps. In the main experiment, we present results only at optimization steps / = 50, with additional results available in the Appendix In the unconstrained optimization case, we have four tasks in GTOPX. As shown in Table 2 due to space limitations, we only present the table for n = 25% and m 75%. from which two key observations can be obtained.
Baseline methods like BO-qEI and CMAES have shown the ability to compete with offline BBO methods in terms of optimality and stability. This result is consistent with the results in the Design Bench Trabucco et al. (202)).
Existing offline BBO methods (c.g., ARCOO and Tri-mentoring) perform well across different benchmarks and different distributions, such as gtopx 2 and gtopx 4 in table 2
TTDDEA and CCDDEA can perform well different tasks, except for gtopx 2 and gtopx 3, and even CCDDEA is the best algorithm on mujoco 2 task in Appendix [ gtopx 2 and 3 is more complicated.
which is because the space of
In the constrained optimization case, we have three tasks in GTOPX. As shown in Table we can obtain two key observations:
8

---

Published as a conference paper at ICLR 2025
Experimental results show that simply constructed offline BBO methods for handling constraint problems are successful on both unconstrained and constrained tasks. However, we find that DE-PF and DE-SPF perform poorly on satellite missions. Poor SO and OI values indicate that the algorithm is prone to fall into infeasible solutions during the optimization process. This result suggests that directly combining online constraint handling techniques with offline evolutionary methods may have difficulty capturing features in complex tasks with sufficient accuracy.
We find that simply modifying the previous unconstrained offline optimization method cannot achieve good optimization results and stability on complex satellite missions, which shows that a better optimization algorithm needs be proposed.
In real-world applications, we not only pursue the optimal solution (such as the 100th percentile). but may also choose the median (50th) or a worse solution (Oth percentile). especially when facing uncertainty or limited resources. At this time, the stability of the optimization algorithm is particularly important. Even if worse solution is chosen, we hope that the algorithm can stably improve the results and avoid the instability or risks caused by over-optimization. For example, in the drug design task, we hope that even the worst solution is relatively good to avoid the drug from having a bad effect on the human body.
In real-worid scenarios, we often do not know the distribution of the dataset. We simulate this by keeping the total missing proportion constant while varying n% and m%. Based Appendix C the optimization stability of BO-qEI gradually decreases as n% increases. In contrast, ARCOO initially even shows some improvement but generally remains stable. Tri-mentoring performs best at
n = 20%, m = 30%, but its performance is subpar at other times. In unconstrained benchmarks, we can obtain three observations:
Classical baselines like BO-qEI, which originate from online optimization scenarios, are highly sensitive to data missing in the optimal search region. In contrast, offline BBO methods are rela- tively stable, demonstrating their advantage in different narrow distributions and validating their effectiveness in real-world applications.
After 150 steps of optimization, ARCOO often achieves the best performance despite experiencing some degradation. This can be attributed to ARCOO's use of learned energy model assess the risk of generating solutions during the optimization process. By leveraging this risk to control the step size during gradient ascent, ARCOO maintains a certain degree of stable performance improvement.
we observed that while some algorithms exhibited strong stability, their performance tended to degrade as the number of optimization steps increased (i.e., from 100 steps to 150 steps). This decline can be attributed to these algorithms encountering OOD regions as the optimization progresses, leading to poorer results.
Although CCOMs are relatively stable in some cases, they often find poor solutions and stagnate, indicated by sub-optimal optimization results. The reason may be that the constraints are too strict, making the algorithm less inclined to explore outside the current solution space.
CONCLUSION AND DISCUSSION
This paper proposes SOO-Bench, a benchmark suite promote stable oflline black-box optimization. SOO-Bench provides a variety of real offline optimization tasks. Considering that the narrow distribution of offline datasets is a vital challenge of offline optimization, SOO-Bench provides APIs that can customize the distribution of offline datasets to construct tasks with different difficulties, thereby comprehensively testing the performance of algorithms. Notably, the significance of stability of offline BBO and introduces an indicator called Stability Improvement (SI) to quantify the stability. Finally, SO0-Bench reproduces offline optimization baselines and conducts experiments evaluate algorithmic stability and optimality.
emphasizes
There are still some limitations of SOO-Bench. First, the variety tasks we provide is still not rich enough, and there are many unexplored black-box optimization scenarios (e.g.. robotic control). Second, we emphasize that optimization stability is crucial in the offline optimization, and the proposed indicator can trade off between the optimality of the output solution and the stability of the optimization process. But we believe that other reasonable indicators still exist. The future work will explore real-world scenarios by providing new datasets and tasks and investigate range of data

---

Published as a conference paper at ICLR 2025
size percentages (i.e., different degrees about the missing m% near the worst value and the missing n% near the optimal value) to conduct a more comprehensive analysis. Meanwhile, we will further improve SOO-Bench develop more realistic narrow distribution methods to simulate real offline datasets, and propose comprehensive evaluation system containing richer evaluation indicators.
ACKNOWLEDGMENTS
The authors would like thank the anonymous reviewers for their helpful and valuable comments. This work is supported by the National Natural Science Foundation of China (No. 62476091).
ETHICS AND REPRODUCIBILITY STATEMENTS
Ethics. This paper does not involve human subjects, personal data, or sensitive information. All datasets used for testing are publicly available, and no proprietary or confidential information has been utilized. We take responsibility for any potential violation of rights and for ensuring compliance with data licensing requirements.
Reproducibility. Experimental settings are described in Section with further details of the methods included in Appendix A The datasets utilized in this paper are all publicly available and open-source. The link to our anonymized repository that includes codes, datasets, documents, demo and license can be found at http:: /stpps//ggitutte com/ zhuyiyi 12101100-1000000011 Bench
REFERENCES
Barrera Luis A, Vedenko Anastasia, Kurland Jesse V, Rogers Julia M, Gisselbrecht Stephen S, Rossin Elizabeth Woodard Jaie, Mariani Luca, Kock Kian Hong, and Inukai Sachi. Survey of variation in human transcription factors reveals prevalent dna binding changes. Science, 351(6280): 1450,1454, 2016.
Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. ROBEL: robotics benchmarks for learning with low-cost robots. Proceedings of the 3rd Annual Conference Robot Learning, pp. 1300-1313, Osaka, Japan, 2019.
Maximilian Balandat, Brian Karrer, Daniel R Jiang, Samuel Daulton, Benjamin Letham, Andrew Gor- don Wilson, and Eytan Bakshy. Botorch: Programmable Bayesian optimization pytorch. arXiv preprint arXiv: 10999900004... 117, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. CoRR,
2016.
David H. Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In Proceedings of the 36th International Conference on Machine Learning, pp. 773-782, Long Beach, CA, 2019.
Can Chen, Yingxue Zhang, Jie Fu. Xue (Steve) Liu, and Mark Coates. Bidirectional learning for offline wiitt model-bard based optimization. In Advances in Neural Information Processing Systems 35, pp. 29454 29467, New Orleans, LA, 2022.
Can Chen, Christopher Beckham, Zixuan Liu, Xue (Steve) Liu, and Chris Pal. martolel-mentoring for offline model-based optimization. In Advances in Neural Information Processing Systems 36, New Orleans, LA, 2023.
European Space Agency and Advanced Concepts Team. GTOP Database global trajectory problems and solutions. httppp/:/www..eaaaaaaaaa///////a 2020. Archived webpage.
global optimisation
Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-based design. In Advances in Neural Information Processing Systems 33, pp. 12945-12956. Virtual Event, 2020.
10

---

Published as a conference paper at ICLR 2025
Justin Fu and Sergey Levine. Offline model-based optimization via normalized maximum likelihood estimation. Proceedings of the 9th International Conference on Learning Representations. Virtual Event, 2021.
Anna Gaulton, Louisa J. Bellis, A. Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, and John P. Overington. ChEMBL: A large-scale bioactivity database for drug discovery. Nucleic Acids Research, 40: 1100-1107, 2012.
Yue-Jiao Gong, Yuan-Ting Zhong, and Hao-Gan Huang. Offline data-driven optimization at scale: A cooperative coevolutionary approach. IEEE Transactions Evolutionary Computation, 2023.
Nikolaus Hansen. The CMA evolution strategy: A comparing review. In Towards a New Evolutionary Computation Advances in the Estimation of Distribution Algorithms, volume 192, pp. 75-102. 2006.
Hao-Gan Huang and Yue-Jiao Gong. Contrastive learning: An alternative surrogate for offline data-driven evolutionary computation. IEEE Transactions on Evolutionary Computation, 27(2): 370-384, 2022.
Pengfei Huang and Handing Wang. Comparative empirical study on constraint handling in offline data-driven evolutionary optimization. Applied Soft Computing, 110:107603, 2021a.
Pengfei Huang and Handing Wang. Comparative empirical study on constraint handling in offline data-driven evolutionary optimization. Applied Soft Computing, 110:107603, 2021b.
Pengfei Huang, Handing Wang, and Yaochu Jin. Offline data-driven evolutionary optimization based on tri-training. Swarm and Evolutionary Computation, 60: 60:100800, 2021.
Yaochu Jin, Handing Wang, and Chaoli Sun. 2021.
Evolutionary Optimization. Springer,
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations, Banff, Canada, 2014.
Abhishek Kumar, Guohua Wu, Mostafa Z. Ali, Rammohan Mallipeddi, Ponnuthurai Nagaratnam Suganthan, and Swagatam Das. A test-suite of non-convex constrained optimization problems from the real-world and some baseline results. Swarm and Evolutionary Computation, 56:100693, 2020.
Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. In Advances in Neural Information Processing Systems 33, pp. 5126-5137, Virtual Event, 2020.
Aviral Kumar, Amir Yazdanbakhsh, Milad Hashemi, Kevin Swersky, and Sergey Levine. Data-driven offline optimization for architecting hardware accelerators. Proceedings of the 10th International Conference on Learning Representations, Virtual Event, 2022.
Huakang Lu, Hong Qian, Yupeng Wu, Ziqi Liu, Ya-Lin Zhang, Aimin Zhou, and Yang Yu. Degratation-resistann offline optimization via accumulative risk control. In Proceedings of the 26th European Conference on Artificial Intelligence, pp. 1609-1616, Kraków, Poland, 2023.
Han Qi. Yi Su, Aviral Kumar, and Sergey Levine. Data-driven offline decision-making via invariant representation learning. In Advances in Neural Information Processing Systems 35, pp. 13226- 13237, New Orleans, LA, 2022.
Brandon Reagen, José Miguel Hernández- Hobano, Robert Adolf, Michael A. Gelbart, Paul N. What- mough, Gu- Yeon Wei, and David M. Brooks. A case for efficient accelerator design space exploration via Bayesian optimization. In IEEE/ACM International Symposium on Low Power Electronics and Design, pp. 1-6, 2017.
Martin Schlueter, Mehdi Neshat, Mohamed Wahib, Masaharu Munetomo, and Markus Wagner. GTOPX space mission benchmarks. SoftwareX, 14: 100666, 2021.
11

---

Published as a conference paper at ICLR 2025
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical Bayesian optimization of machine learning algorithmss. In Advances in Neural Information Processing Systems 25, pp. 2960-2968, Lake Tahoe, NV, 2012.
Rong-Xi Tan, Ke Xue, Shen-Huan Lyu, Haopu Shang, Yao Wang, Yaoyuan Wang, Sheng Fu, and Chao Qian. Offlin model-based optimization by learning to rank, 2025.
Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In Proceedings of the 38th International Conference on Machine Learning, pp. 10358-10368, Virtual Event, 2021.
Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-Bench: Benchmarks for data-driven offline model- -ased optimization. In Proceedings of the 39th International Conference Machine Learning. pp. 21658 21676, Baltimore, MD, 2022.
James T. Wilson, Riccardo Moriconi, Frank Hutter, and Mare Peter Deisenroth. The reparameteriza- tion trick for acquisition functions. CoRR, abs//772,,04444 2017.
Amir Yazdanbakhsh, Christof Angermüller, Berkin Akin, Yanqi Zhou, Albin Jones, Milad Hashemi, Kevin Swersky, Satrajit Chatterjee, Ravi Narayanaswami, and James Laudon. Apollo: Transferable architecture exploration. CoRR, abs/2102.01123, 2021.
Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. RoMA: Robust model adaptation for offline model-based optimization. In Advances in Neural Information Processing Systems 34, pp. 4619- 4631, Virtual Event, 2021.
Ye Yuan, Can Chen, Zixuan Liu, Willie Neiswanger, and Xue (Steve) Liu. Importance aware co- teaching for offline masel-based optimization. In Advances in Neural Information Processing Systems 36, New Orleans, LA,, 2023.
Yuanting Zhong, Xincan Wang, Yuhong Sun, and Yue-Jiao Gong. Sddobench: A benchmark for streaming data-driven optimization with concept drift. In Proceedings of the 2024 Genetic and Evolutionary Computation Conference, pp. 59-67, 2024.
Yiyi Zhu, Huakang Lu, Yupeng Wu, Shuo Liu, Jin-Wen Yang, and Hong Qian. Constrained offline black-box optimization via risk evaluation and management. In Proceedings of the 39th AAAI Conference on Artificial Intelligence, Philadelphia, PA, 2025.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France, 2017.
12

---

Published as a conference paper at ICLR 2025
APPENDIX
This appendix structured into four sections improve understanding and facilitate the repro- duction of our benchmark research. Task Description Section elaborates on the objective function, constraints, and variables used in our study. The Experimental Details Section provides comprehen- sive information the methodologies, code implementations, and resource usage of all methods compared. Experimental Results Section thoroughly diseusses the research results and findings in detail. Finally, the Hyperparameter Analysis Section itemizes the specific settings and config- urations utilized in the experiments. Our benchmark and comparison algorithms are available at https: /ttpt/gtttt.u./ee///////.e...a com/ Zhuylyi 1etpyyyy-11///100 123/11000 Bench
A TASK DESCRIPTION
In this section, we detail task description in SOO-Bench. Specifically, we answer (1) what the goal and background of each task? (2) where are the tasks from? (3) what do the variables and constraints in each task represent?
A.1 GTOPX: SPACE MISSION BENCHMARKS
For those interested the task, more information can be found on the official website for the benchmark. To access detailed descriptions, methodologies, and data related the task, please visit the following URL: htt.:http:// www wtwwwwwwmm. wttp:/wwac... midaco so solver. ver wwwerr.cominneii com/index. phpp happ//but/ benchmarks/ /topx This page provides essential resources for understanding the scope and requirements of the benchmark challenges.
A.1.1 GTOPX 1: CASSINI
The Cassini benchmark simulates an interplanetary mission targeting Satum, with the mission's aim being to enter a specific orbit around Saturn characterized by an intracentric radius of 108,950 km and eccentricity of 0.98. The primary goal of this benchmark is minimize the total AV, change in velocity, required throughout the mission, which includes the launch and orbital insertion maneuvers. This scenario utilizes six decision variables and incorporates four constraints that set maximum limits on the proximity of the center during four flyby maneuvers. The variables' descriptions of this task include var, and var2.
A.1.2 GTOPX 2: CASSINI 2
The Cassini benchmark, which models complex interplanetary missions to Saturn, including critical maneuvers like the Deep Space Maneuver (DSM), presents more challenging scenario than the gtopx benchmark. Unlike Cassini 1, which focuses on orbital insertion, the primary objective of this benchmark to achieve rendezvous with Saturn, aiming minimize the total A V required throughout the mission. This benchmark involves handling 22 decision variables. The variables' descriptions of this task include varı, var2, varg, var4, vars, varg, and var7.
A.1.3 GTOPX 3: MESSENGER (REDUCED)
The third benchmark, known as Messenger (reduced), a simulation of interplanetary missions to Mercury, excluding any resonant flybys of the planet. The main objective of this benchmark is to minimize the total AV over the course of the mission. involves 18 decision variables and the explanation of the variables is given in Table 4
1.1.4 GTOPX 4: MESSENGER (FULL)
The fourth benchmark. titled Messenger (full), models interplanetary missions to Mercury, incorporat- ing resonant flybys the planet. This benchmark aims to minimize the total V incurred throughout the mission. It features 26 decision variables, which includes all variables in Table 2
13

---

Published as a conference paper at ICLR 2025
Table 4: Description of variables in GTOPX tasks Schlucter et 2021
Variablc
Descriptions
gtopx
gtopx 2
var1
Initial day mcasured from 1-Jan 2000
var2
Time interval between events (e.g. departure, fly-by, capture)
varg
Initial excess hyperbolic speed (km/S)
/
var.
Angles of excess velocity a hyperbolic trajectory
X
varg
Fraction of the time interval after which DSM occurs
X
4
varg
Radius of flyby (in planet radii)
X
var.
Orientation of the trajectory angle in the planet's 's B-plane approach vector
X
A.1.5 GTOPX 5: GTOCI
The fifth benchmark, known as GTOCI, models complex space mission involving multi-gravin assists to asteroid TW229. The objective of this mission is to maximize the change in the semi- major axis of the asteroid's orbit. This benchmark incorporates 8 decision variables and includes constraints that set limits on the proximity to the center during each of the four flyby maneuvers. The variables description includes varj and varg.
A.1.6 GTOPX 6: ROSETTA
The sixth benchmark, named Rosetta, emulates multi-gravity-assiteedd space missions to Comet Saturyymoo--Grrsimennll
including the execution of a DSM. The primary objective of this benchmark is to minimize the total A required throughout the mission. It encompasses 22 decision variables, and description of variables are shown in Table
A.1.7 GTOPX 7: CASSINII-MINL.P
The last benchmark, cassinil-minl. the original C instance, the order of planetary flybys was fixed as Venus- Venus-Farth-Jupiiter Cassinil-MINLP treated all four flybys discrete decision variables. Every planet in the solar system (plus the dwarf planet Pluto) is a viable option to fly by any of the four planets. The benchmark involves 12 decision variables and further considers four constraints. Descriptions of all variables are in Table !!
mixed integer extension of Cassinil instance. While in
A.2 CEC TASK: INDUSTRIAL AND DESIGN PROBLEMS
For detailed information and access to the resources associated with the task, please visit the provided URL: https://github. 2020-RN-ConsE-
Suu--anthhaaaaaa wwn--eddtttt. impt zation This link leads to the GitHub repository, where you can find all the necessary files, including source code and documentation, to understand and engage with the benchmark suite effectively.
A.2.1 CEC 1: OPTIMAL OPERATION ALKYLATION UNIT
The initial benchmark test is termed "Optimal Operation of Alkylation Unit". This test focuses optimizing the octane number of the olefin feed in an acidic environment, with the main goal to enhance the alkylating product. The benchmark involves decision variables and incorporates 14 constraints, which are designed to limit onboard fuel and launcher performance. The problem can be formulated follows.
14

---

Published as a conference paper at ICLR 2025
Minimize:
(()) = -00.035x) + 1.715x + 10.0x2 + 4.0565xzg 0.063x3325)
subject to:
+ 0.1. - 5,
(2( 1.1088z, + 043033322)4 0.00660332, 23 S 0,
(z) 6.66173269x - 56.9666922 + 172.398788; - 10000 - 201.20592mg S 0
(x - 000700220000000222444440 50.85075 - 25 S 0,
Tz 0,
(7) = 161.88900,,44 489510..0zz 23 S 0,
97(#) 0.33Ir7 44.333333 25 0,
98 (7) 0.0225562; - 1.0 0.00759527 S
(5)
99 (x) 0.00061.3 1.0 0.0005x, 0,
910 (7 0.819672x - 33 + 0.819672 < 0,
911 ( 11559.9922200:224. 23 0,
912 2 1020.4082r422 + +.1.:22--22244444 1.2244898832 10000002 S 0,
913 (7 6.25 + 6.25a 7.625m3
100000 S
914 1.22x3 .6 + 1.0
with bounds:
1000 < æ1 - 2000, 0 S S 100,
2000 23 < 4000, 0 < σ1 100,
0 5 26 < 100,0 0 5 dσ < 20,
x7 S 200.
A.2.2 CEC 2: PROCESS FLOW SHEETING PROBLEM
The second benchmark is characterized as a non-convex constrained optimization problem. This benchmark incorporates three decision variables and is governed by three constraints. is noted for having an optimal objective function value of f(x) 1.07654. The formulation of this problem can be shown follows.
Minimize:
f(z) = -.. 0.8 xz)²
subject to:
(z) exp (z) 0.2)
x2 S 0,
(6)
(2(#) = x2 + 1.12, < -1.0,
(z) = I1 I3 5 0.2.
with bounds:
2.22554 T2 < 11,0.2 51,23 (0.1).
A.2.3 CEC 3: PROCESS SYNTHESIS PROBLEM
The third benchmark. referred to the Process Synthesis Problem, is defined as non-convex constrained optimization problem. This benchmark features two decision variables and is subject to two constraints. It is recognized for achieving best objective function value of f(x) 2.0. The
15

---

Published as a conference paper at ICLR 2025
problem can be defined as follows.
Minimize:
f() 22 + 2x,
subject to:
(z) - -x2 + 1.25 < 0,
(2(z) = I1 + S 1.6.
with bounds:
S 31 S 1.6,
22 E {0,1}.
A.2.4 CEC 4: THREE-BAR BAR TRUSS DESIGN PROBLEM
The fourth benchmark, known as the Three-bar Truss Design Problem, originates from the field of civil engineering and involves complex constrainted setup. The primary objective of this problem is to reduce the weight the truss structure. The constraints are based on the stress limits for each bar, leading to problem characterized by a linear objective function, two decision variables, and three nonlinear constraints. This benchmark's optimal objective function value is f(x) = 2.63896. The problem can be defined below.
Minimize:
(()) = 1 ( 2V
subject to:
32
(1( 28201 + v 22
P - S 0,
92(T) = 82 + V211
P 0 $
21231 + /222
(8)
93(
P o S 0. I1 + /202
where,
(= 100, P 2, and = 2.
with bounds:
0 < 11, 22 < 1.
A.2.5 CEC 5: WELDED BEAM DESIGN
The fifth benchmark, known as the Welded Beam Design, primarily aims to minimize the cost of constructing welded beams. This engineering challenge is defined by four variables and five con- straints, which guide the development of the beam. The most acclaimed outcome of this benchmark recorded with an best objective function value of f(x) 1.67022. The description of this problem
16

---

Published as a conference paper at ICLR 2025
is shown below.
Minimize:
(() 0.0481 12z34 (12 + 14) + 1.10471xf.2
subject to:
(1 21 34 S 0,
(2(T) -((x) 6max S 0,
(x P S Po(),,
(T Tmax 2 (Z))
95 (T) (T) max S O.
where,
7
722 + 7!12 2T'T"" 22
7"
RM
P
2R
v V2r211
22 + L
M
R
12 +
11 + Taw3
2
J= =
22
I1 + 13
2)
v /2n122
2
6PI.
6PT,3
4.013Faraa
33
F
(0)
5(z)
E
Pc(T)
6L2
1
2L
V
4G
L 14in, P 6000lb, 30.10 psi, maax 30, 000psi, Tmax
13, 600psi,
G 12.10 psi, 8max 0.25in
with bounds:
0.1 < 23, 12 10
0.1 < F4 S 2
0.125 S 01 2.
A.3 PROTHIN: DNA SEQUENCE OPTIMIZATION
The tasks TF Bind 8 and TF Bind 10 explore the affinity of various DNA sequences for several human transcription factors. In our study, we specifically target the transcription factor SIX6 REF RI, with the aim to design an nucleotide sequence that exhibits high binding affinity to this factor. The datasets for TF Bind 8 and TF Bind 10 encompass all possible combinations of nucleotides for sequences of lengths and 10, respectively, offering a comprehensive basis for assessing sequence effectiveness. You can access the data and code by visiting the URL https //github.
despaccco/deiinnnnne bench bench
B EXPERIMENTAL DETAILS
In this section, we provide further details about the experiments, including the URL for the code of the comparison algorithms, and the computational resources used.
B.1 METHODS
This section introduces approaches for offline optimization. To establish baseline for future comparisons, we benchmark several recent offline MBO algorithms across our tasks. Existing methods fall into three main categories: forward, generative, and evolutionary methods. Forward methods focus on training robust surrogate models to combat adversarial optimization of inputs, followed by gradient- based maximization. Generative methods sample solutions from learned generative models with regularization. Evolutionary methods neural networks and other techniques to learn proxy models, which are then solved using evolutionary algorithms. In addition, we also introduce traditional black box optimization methods for comparison. For evaluation, we consider three key
17

---

Published as a conference paper at ICLR 2025
components of offline optimization: model architecture, learning algorithm, and search algorithm, as outlined below.
B.1.1 TRADITIONAL BLACK BOX OPTIMIZATION METHOODS
BOIqEI Balandat et al. 2019 We perform offline Bayesian optimization by fitting Gaussian Process using learned surrogate model (az). Next, we employ the quasi-Expected Improvement ac- quisition function to propose candidate solutions for efficiency. After the Bayesian optimization cycle is completed, we select the best candidates a from the dataset D. The procedure of BO- is shown in Algorithm We use the implementation from https: http://www..oooooonntttaaaaaaaae
com www.oo/rrannnottbbuuooooo www.sseeineeeeoo..m
design base h
CMA-ES Hansen (2006): We perform offline optimization using the Covariance Matrix Adap- tation Evolution Strategy (CMA-ES), which adapts the covariance matrix of a multivariate normal distribution to guide the search. At each iteration, candidate solutions are sampled from the distribution, and the distribution parameters are updated based on selected high- performing solutions. After completing the optimization cycle, we select the best candidates from the dataset D. The procedure of CMA-ES is shown in Algorithm 1, We use the implementation from cree/masterr/es. base wews/cbaaa
wttp://oooobbmoae www.annnnnnarrcoooooooaooooooooe desi wtw..e...bbbeeeeeeeee......eeeee
B.1.2 FORWARD METHODS
ARCOO learns both surrogate model and energy-based model, which characterizes the risk of degradation to address the out-of of-oostrrbbution issue. The optimizer each step is regulated by risk suppression factor derived from the energy-based model. The The implementation can be found at https: /Etps://githus /github
ARCOO Lu
2023
procedure is outlined in Algorithm
com/ Scm//uhuakangg/Ae (aaCuakang/ARCOO
Tri-mentoring Chen et al. 2023
Tri-mentoring trains three surrogate models on the offline
dataset and utilizes majority voting generate consensus labels. To mitigate potential errors in the consensus, an adaptive soft-labeling module is applied. The complete algorithm is presented in Algorithm The implementation can found at htttttss//gthhui com GGchen1997)
parallel mentoring
B.1.3 GENERATIVE METHOOSS
Autofocusing CbAS Brookes et (2119: The Autofocusing CbAS iteratively updates search model for optimal design by focusing on high-quality solutions. It employs importance sam- pling to weight samples based on their likelihood of meeting desired conditions, allowing for targeted optimization. Using weighted Maximum Likelihood Estimation (MLE), the search model is refined, while an oracle model is retrained to adaptively reduce bias as the design space is explored. The procedure of Autofocusing CbAS is shown in Algorithm 5 We use the http://ttttuubooooooootaattaaaaa from httpapanwwiioooooorrorg////////d http://aateeeeeaaa.c...eaaeaeeee
design baselines/ cbas
Algorithm 1 Offline Bayesian Optimization Trabucco et al. (2022)
1: Train a surrogate model based on offline dataset D.
2: Select the top 1 1% initial designs D = (X,, yt) from the offline dataset 'D.
3: for = 1...K do
4: Find æ, by optimizing the quasi- munsi-Exppeceed-lppoeeeee acquation function over the Gaussian arg maxe u(x 2 11.--)).
Process:
5: Sample the Surrogate function: y: = fxxz).
6: Augment the data D (D1a-1, (2,, y:) and update the Gaussian Process.
7: end for
18

---

Published as a conference paper at ICLR 2025
Algorithm 2 Covariance Matrix Adaptation Evolution Strategy (CMA-ES) Hansen (2006)
1: Initialize mean mo, step size do, population size A, initial covariance matrix Co
2: Set learning rates for mean (j)), covariance matrix (a), and step size adaptation (3)
0, 2, do 4: Sample candidate solutions z! o( for i 1. I....,A
3: for each generation
5: Evaluate fitness f(x;") for each candidate x
6: Select the top μ candidates with the best fitness values
7: Update mean m as the weighted average of the top μ candidates:
(t)
M++1
w.Sm
8: Update covariance matrix C(+1 to adapt the search distribution:
C++ = (1- +
Wi
D
(1)
mi
(a. m+
T
9: Update step size 0++1 based on the success rate of candidate solutions:
B (1woo -1)
|pPlll
0++1 o. exp
10: Update evolution path Pa for step size adaptation
11:
end for
12:
return Best solutions found during the optimization process
B.1.4 EVOLUTIONARY METHODS
TTDDEA Huang 202T TTDDEA is offline optimization method that combines tri-training with evolutionary algorithms. It uses three Radial Basis Function Networks (RBFNs) as surrogate models to predict fitness scores and generates high-confidence pseudo-labels augment limited training data. By applying an evolutionary optimization process, TT-DDEA iteratively updates these surrogate models and selects high- performinfo candidate solutions for the next generation. This approach leverages semi-superven supervspe learning and multi-model ensembles to address data scarcity in offline environments. The algorithm can seen Algorithm 5
We use the implementation from DDEA
https: thtps:
thub. com/ Hand http:/aannnnnnwana.nmmmm www.aggDgrouugu//gg//p
CCDDEA Gong et al. (2023): CCDDEA combines hierarchical surrogate models with cooperative coevolution to solve large-scale optimization problems. It uses global model (HM) and local models (LMs) to guide search different levels. Local searches are enhanced with gradient-based and evolutionary operators, while global communication merges uub-populations. The dynamic space division strategy improves convergence by shifting focus from local to global optimization. The procedure of CCDDEA is shown in Algorithm. The implementation can be found in https 7/githubs www..th.....caa....omommc. wwt.:occmmaaaoc.e.. Laaim//LaGongg//eo cc- ddea
B.1.5 CONSTRAINED METHODS
We select two primary categories of offline constraint algorithms: deep learning- constrained offline methods and evolution-base constrained offline methods. For our experiments, utilize classic methods from both categorics.
This is a simplified constrained version of ARCOO. We integrate the degree of constraint violation into risk assessment, treating solutions that violate constraints as high risk. CARCOO employs the three models (i.e., surrogate model, energy based model to characterize OOD risk, an mased model to characterize constrained risk) to train the models on a training dataset. The algorithm can be found in Algorithm
CARCOO:
19

---

Published as a conference paper at ICLR 2025
Algorithm 3 Accumulative Risk Controlled Offline Optimization (ARCOO) Lu et al. (2023)
Input:Ofiline dataset D, learning rate 77, maximum Langevin dynamics step K, Langevin dynamics stepsize A, and initial momentum m.
1: Initialize dual-head model that consists surrogate head fo(x) and energy head (z).
2: for each training epoch do
3: Update fa())) using MSE loss:
0 0
4: Sample high-risk distribution ((x)) by Langevin dynamics: q(x) LD>(D((():: K), i...
Ik - zk-1 + XV fo(xx=11) + wk: k = I,..,,,,
where wi ~ (O, )), and xo p(x)). Sampling starts from the low-risk empirical distribution p(x) over the offline dataset.
5: Update (x) using contrastive divergence loss:
V (I) (I)(
where h6 is derived from (x)
6: end for
7: Let P be empirical distribution batch of the high-quality solutions in D, and Q LDs(P; K).
8: Calculate the risk suppression factor:
Rs(x) = m(Ea - Ex(x)))Eg Ex)
9: for = to I' do
It f- _t-1 + Ro (2: R
11: end for
return Final solution Tupp xr for online application.
Algorithm 4 Tri-mentoring for Offline Model-based Optimization Chen al. (2023)
Input:
The static dataset D, the number of iterations T. the optimizer OPT(.).
1: Initialize zo as the design with the highest score in D.
2: Train proxies (-), f (-). and (-) D with different initializations.
3: for L + 0 to T - do
4: Sample K neighborhood points at xa as S(æ.).
5: Compute pairwise comparison labels ÿ 8 and d for the three proxies on S(x;).
6: Derive consensus labels: v majority_votingg (g gn (e))
7:
for proxy in (-A f (-), (8) (.) do
8:
Initialize soft-labels as consensus labels: 88 = gV.
9:
Inner level: fine-tum the proxy with Eq. (8).
10:
Outer level: learn more accurate soft-labels ÿ with Eq. (9).
11:
Mentor proxy using the optimized soft-labels is with Eq. (8). end for
12:
13: Form more robust ensemble as fo(x) = f()=))
Gradient ascent: #++1 = I; + :OPT(V..o((.)).
15:
end for
16: return The high-scoring designs x² ØT.
DDEA-PF & DDEA-SPF Huang & Wang (2021b): DDEA-PF and DDEA-SPF are two data-driven evolutionary algorithms for handling constraints in optimization tasks. DDEA-PF employs penalty functions to manage constraints, adjusting the penalty to push solutions towards feasibility. On the M
D (x)))
other hand, DDEA-SPF focuses on adaptive penalty, i... fitness(x)
f(x) + Dmax.
20

---

Published as a conference paper at ICLR 2025
Algorithm 5 Autofocused Model-based Optimization (Autofocusing CbAS) Brookes et al. (2019)
{ Vi }i)]i=i oracle model class, Pa(y x) with parameters, B, that can be estimated with MLE; search model class, pa( with parameters, 0, that can be estimated with weighted MLE or approximations thereof; desired constraint set, S (e.g.. S = {y 2 yr }); maximum number of iterations, T; number of samples generate, m; EDA-specific monotonic transformation, (.).. Initialization: Obtain (x( by fitting to {x; }i= with the search model class. For the search model, set Poço) (a:) pa))).. For the oracle, Pε(0) (y æ), use MLE with equally weighted training data.
Input: Offline dataset,
1: for t = 1,...,T do
2: Sample from the current search model, {at Ji=1 P)()- Vi fl....;m. m).
(P E | =! Vi fl....,mi. m).
3: 02
1==1 and 4: Fit the updated search model, Potcj (ax), using weighted MLE with the samples, {- (m their corresponding EDA weights, {u, )m
5: Compute importance weights for the training data, w; < Poce) (x,), i ,,..., n.
6: Retrain the oracle using the re- reiiheet training data,
B(t) + arg max
my log Pu x;).
BEB T2
7: end for
8: return The most promising candidates among (0
(t) Fm Jt=1
Algorithm 6 Tri- Fri-riinnng Data-Driven Evolutionary Algorithm (TTDDEA) Huang et al. (2021)
Input: Separate offline data L1, L2, and L3, trained RBENs Mi, M2, and M3. current population P. population size Q.
1: for i 1 - do
2: Use the models M1. M2. M3 to predict the fitness of population, written as f2. fq.
/2, B. and f
3: end for
4:
for 1- do
5: Find the high-confidence data æ, as in Equation (9).
6: Calculate its pseudo label ÿ; in Equation (10).
7: Update L, to L by (æ,, Di) as in Equation (11).
8: end for
9: Train RBFNs M1, M2, and M3 with L L2 and L/g-
Use the evolutionary algorithms to obtain promising candidates.
11: return The most promising candidates found by the evolutionary algorithms.
DDEA-PF is presented in Algorithm 9 The implementation is from https://gittub.co
faardinolaan2DDrrron//Gosstsrere Haniiintt-aannnln Handl ing Off
CCOMs: This simple experimental version of PRIME (Kumar al. 2022 We use 0* arg ming Exrgg (fo (z:) - y1)² - =, [fo (z, ]] This means that in addition
to fitting the surrogate model, we also make the values of points that violate the constraints as small as possible. Then use the conservative model COMs (Trabucco et al. 202 surrogate model to train. Pseudocode can be found at Algorithm 10
B.2 COMPUTATION RESOURCES
The computing resources required for the research described in this paper relatively modest, requiring only a single Nvidia GeForce RTX 3090 GPU. The experiments were efficiently completed using this powerful graphics card, almost all within a 24-hour timeframe.
21

---

Published as a conference paper at ICLR 2025
Algorithm 7 Offline Data-Driven Optimization at Scale: A Cooperative Coevolutionary Approach (CCDDEA)
Gong et al. (2023)
1: Input:
The offline data;
The size of complete population:
Th, Ty: The maximum generations of higher-level and lower-level optimization;
giitt: The initial number of groups:
T. The interval for updating the number of groups;
T,: The interval for re-dividing the sub-spaces;
0: The learning rate of gradient descent;
'top: The control parameter for top-ranked random merging in the cooperative search;
2: Output: The best solution
3: Initialization: i, f 0 (index of T,)), 9 - 9init
4: P < Latin hypercube sampling
5: for i <- Tи do
6: if 1 and i mod Tg and then
7:
g g
8:
1 0
9:
end if
10:
i, mod T and g then
11:
+ division rules
(HM, LM)) HSJL(D,
13:
end if
SP +- split according for SP, E SP do
15:
SP; Lo a, LM,) end for
17:
18:
if g > then
19:
P Higherr.eveSSeach((55.
HM, LMs)
20:
end if
21:
i, f- ip +1
22:
end for
23:
return P(O)
C EXPERIMENTAL RESULTS
Here, present detailed results across various tasks. For each task, evaluate 128 solutions the 100th percentiles. In this paper, we define our areas (i.e., S, S1, S2) as the region formed by doubling the distance between solutions (i.e., the optimal algorithm for each step in the optimization process is in 100n,S, the optimal offline solution in Si, and the optimal solution found by the algorithm in 12) and algorithm's 's optimal solution and relative the offline optimal solution, shown in Eq. Additionally, we assess the algorithms' robustness by providing results at the 50th and Oth percentiles. Among the various algorithms, ARCOO, CMAES, and Trimentoring emerge as the top performers. However, most algorithms experience decline in performance as the number of optimization steps increases from 100 to 150. This highlights the necessity for further advancements the stability of offline black-box optimization methods, particularly for addressing these challenging tasks.
Due to significant computational cost associated with the CMAES algorithm, which requires more than 24 hours to complete, this paper focuses on reporting the results after only 50 optimization steps.
The experimental results in the unconstrained scenario show that ARCOO and CMA-ES algorithms are the most balanced and stable on various benchmark functions, demonstrating high performance and stability. Although CCDDEA performs well in some cases, the stability of the results is poor. This may be because previous offline evolutionary algorithms did not consider the problem of dittof-distriuution generalization.
Since CbAS a generative model that samples the design space and optimizes based on the con- ditional distribution of the target attribute, the final result is primarily determined by the model's
22

---

Published as a conference paper at ICLR 2025
Algorithm 8 Constrained Accumulative Risk Controlled Offline Optimization (CARCOO)
Input:Ofiline dataset D. learning rate 77, maximum Langevin dynamics step K, Langevin dynamics stepsize A, and initial momentum m.
1: Initialize dual-head model that consists surrogate head fo(x) and energy head (2).
2: for each training epoch do
3: Update fa())) using MSE loss:
0+0- 0+0-
4: Sample high-risk distribution ((x)) by Langevin dynamics: q(x) LD>(D((():: K), i..,
Zk - zk-1 + XV fo(xx=11) + wk: k = 1,..,,,,
where wi ~ (O, )), and xo p(x)). Sampling starts from the low-risk empirical distribution p(r) over the offline dataset.
5: Update (x) using contrastive divergence loss:
V (KI(p(())]])<<<))) (l(((a)))l [/k/e//))|hce)))-kkkkddddllllll)
where he is derived from Ex(x)
6: Update Er (2) using contrastive divergence loss:
T V (L.(p()))) (LL(()))))))
where h is derived from (a)), j(xz) is solutions that satisfy the constraints in the dataset, ((x) is solutions in the dataset that do not satisfy the constraints
7: end for
8: Let P be an empirical distribution over a batch of the high-quality solutions in D, and Q =
LD, K).
9: Calculate the risk suppression factor:
RĄ(x) m(Es (x Ex) R-(x:) = m(FFs - Fr (r(
- Ftp)
10: for t 1 to T do
11: Kt f- xt-1 + (m 1)+ V V (x
2
12: end for
13: return Final solution Tapp = ør for online application.
conditional distribution and the adaptive sampling strategy, rather than the step size. When the seed fixed, the result derived from the conditional distribution regardless of the size of the optimization step, leading to consistent outcomes. Furthermore, due to the inherent complexity of satellite missions, the VAE model struggles effectively learn the prior distribution, preventing from surpassing the offline optimal.
The experimental results in the constraint scenario show that the DE-PF and DE-SFP algorithms do not perform well on various benchmark functions. In addition, the performance of DE-PF and DE-SFP on multiple benchmark functions is almost the same, showing consistent performance. The performance of the CCOMS algorithm fluctuates greatly in different dimensions and functions.
Overall, the CARCOO algorithm has good performance and stability when dealing with constrained optimization problems, providing important reference for subsequent algorithm optimization. However, CARCOO and CCOMs often fail to find a suitable feasible solution due to the small number of data points and insufficient neural network training.
23

---

Published as a conference paper at ICLR 2025
D HYPERPARAMETER ANALYSIS
We outline guidelines for hyperparameter selection for methods evaluated in the benchmark. These principles help in offline tuning for new tasks. Begin by identifying essential hyperparameters, like learning rate and batch size, and select tuning method such as Grid Search or Random Search. We run experiments to find the satisfied hyperparameter configurations with performance metrics like accuracy and F1 score.
D.1 HYPERPARAMETERS OF BO-QEI
The primary tunable components of Bayesian optimization include the objective function and the parameters within the optimization loop. The objective functtinn commonly optimized with maximum likelihood method, allowing validation log-likelihood regression error to be tracked directly. Training continues until the validation loss is minimized, ensuring good generalization beyond the training dataset. This tuning method is fully offline since it exclusively uses static task dataset. In this study, we employ a Gaussian Process model combined with the quasi-Monte Carlo Expected Improvement acquisition function. For in-depth understanding of various Bayesian optimization strategies and their associated hyperparameters, detailed information can be found in the Torch documentation at http::///otorrec..rr///obee
D.2 HYPERPARAMETERS OF CMA-ES
The primary configurable elements of Covariance Matrix Adaptation (CMA) techniques include the trained objective function and the evolution strategy parameters. Typically, the objective function trained using maximum likelihood approach to ensure optimal performance. For further details and comprehensive information, refer to the open-source CMA-ES implementation and its extensive documentation available at lnttp:::: /ttp:/gittt. com/ CMA ES pyccal In this research, we adopt the default settings for CMA-ES provided by this implementation, with the parameter set to 0.5.
D.3 HYPERPARAMETERS AUTOFOCUSING CBAS
Autofocusing CbAS has one main tunable components, involves verifying the performance index, which plays crucial role in generating the generalization capability of the learning model. By ensuring that the performance index stays above a certain positive threshold, we can ascertain that the algorithm is well-tuned and performing optimally. In this paper, have this threshold to 0.9. This careful selection helps maintain the reliability and effectiveness of the autofocusing CbAS approach.
Algorithm Data-Driven Evolutionary Optimization with Penalty Function (DDEA-PF) Huang
Wang (2021b)
1: Initialize population P with size Q
2: Set the degree of violation of the jth constraint (x)),, penalty coefficient r;.
3: for each generation T do
4: for each individual x in P do
5: Evaluate fitness using the objective function with penalty for constraint violation:
fitness(x) f(x) + +r max gj (x)1')
6: end for
7: Perform selection, crossover, and mutation in population P find best feasible solutions.
8: end for
9: return Best feasible solutions x TT.
24

---

Published as a conference paper at ICLR 2025
Algorithm 10 Constrained Conservative Objective Models for Offline Optimization (CCOMs)
Input: Offline dataset D, the learning rate of OOD gradient ascent 7), trade-off coefficient a. Initialization: Surrogate model fe.
1: for = 1 to training steps do
2: Sample (æo, y) D
3: Find (r(()) via gradient ascent from zo: #1+1 + fo 1=
u(xx)) =
EmEED (==)r(
4: Minimize a();a) with respect to 0.
L(0;a) = BEyDD (fo - 1) -alExa [fo(2oo) +alluao) [fo(tt] -E Nine [so (x
レ 0 - 100-VVoLie (V)((8,a)
5: end for
6: Initialize optimizer the optimum in D:
7: Find x* via trust-region gradient ascent from ;:
2++1 It V Lo
where :=) 8: Return the solution xx Tr.
:=
D.4 HYPERPARAMETERS OF ARCOO
The Accumulative Risk Controlled Offline Optimization (ARCOO) method two hyperparameters: accumulative risk control and Langevin dynamics steps. The former represents the level of total risk tolerance, with an initial momentum set at 0.2, reflecting a moderate risk tolerance. The latter involves sampling by 64 steps of Langevin dynamics, determining the distance between high-risk solutions and observed solutions.
D.5 HYPERPARAMETERS TRI-MENTORING
We tuned two critical hyperparameters: the number of neighborhood samples and the number of optimization steps. Due the algorithm's 's robustness to the number of neighborhood samples, we set this parameter to 10. To maintain consistency, standardized the number of optimization steps to 100 across all experiments.
D.6 HYPERPARAMETERS OF TTDDEA
In this evolutionary offline optimization task, we retained the original hyperparameters because they were carefully tuned for mutation rate, crossover probability, and selection pressure. These settings ensure a balanced exploration and exploitation process, preventing premature convergence and maintaining diverse solution population. Keeping them unchanged guarantees the algorithm's robustness and reliable performance.
D.7 HYPERPARAMETERS CARCOO
The hyperparameters in the constrained ARCOO (CARCOO) method need be consistent with the original ARCOO settings. Therefore, we have configured them to exactly match the hyperparameters used in the original ARCOO method.
25

---

Published as a conference paper at ICLR 2025
D.8 HYPERPARAMETERS OF CCOMS
Constrainted conservative objective models have four main tunable parameters. The first parameter is the degree which the objective model is allowed overestimate the objective value for off-manifold designs, set to 2 to ensure a conservative approach. The second parameter is the number of gradient ascent steps, chosen to be 100, balancing optimization efficiency and computational effort. The third parameter the learning rate, step sizes are appropriately scaled to the problem's 's complexity. The final parameter is the constraint trade-off coefficient, set to ensuring an equal emphasis on optimizing the objective and satisfying the constraints. These parameters are crucial for fine-tuning the model's performance within the desired constraints.
2vd, where d is the dimension of the design space, ensuring
D.9 HYPERPARAMETERS OF DE-PF& & DE-SPF
This hyperparameter identical the one used in TTDDEA, maintaining consistency with the origi- nal paper. However, due to the algorithm's 's suboptimal performance in handling out-of-distribution (OOD) problems, we have adjusted the number of iterations to 50 in this study. This modification aims to improve the algorithm's 's efficiency and effectiveness in addressing the specific challenges posed by OOD scenarios.
Table 5: Overall results for GTOPX unconstrained tasks with 128 solutions and 100th percentile evaluations are averaged over eight runs. ((orF))) represents the optimal objective function value in the offline dataset. FS (i.e., final score) refers to the function value found by an offline optimization algorithm at the final step of specific optimization steps (i.e., 50. 100, 150). The symbol "." means that the algorithm cannot complete the corresponding task within 24 hours. In this case, 0% of the values are missing near the worst value and another 50% near the optimal value.
196 Namm
110
114
TRED0A
175
100mm
201 Séouc
-460-
0700 mp
1.4
0
20100
005
NHeSe))
20
1250mm
185ca
m35
113
-0 40
CHAAS
03.80 m0
100rms
1280m
125m
1ihmm
Lokree
11.m
LASERUNE Now
7018
ABTER poe
15.90 cmm
12
-2375uu
GTOFT
CBAS
10:0::1
cm
1:70am
1380
13:0
100000
CMAAT
-408um
199.
1:00m
137km
1.8kc
1.7kmm
127
26

---

Published as a conference paper at ICLR 2025
Table 6: Overall results for GTOPX unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
126
-275 2h
Werk
CBAS
13000
133
CHAVES
48 1aa
1350
44.30
142
$4.00
135
81.00
200
7
SAL Marre
1M
58
125
108
DTOPCC
15lam
155
CHAALS
Labie
40000
+1.3mmm
15km
140m
1350
34.5 mm
21618
210
8000
158
318
000gm
117/01
14
069
$77
-11
1.80u
203
Table 7: Overall results for GTOPX unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Sig..
1.7 Lac
1
1
CMALS
4030m
1,7
100mmm
A8000
<1i8mm
Libres
125
Ci
12hr
43.4
240
15 Mare
02 8pm
313
14.00
15
-1812
The maaki
137
34
17
180
1Mamn
CMALS
100m
042
121.0rmm
AB000
8.00m
125res
12bun
1iboa 4230
Libea 60ree
Milans Witler
1000
6.30mm
12
020us
39 Mure
195gm
17gmm
34.m
177
19
178kmm
27

---

Published as a conference paper at ICLR 2025
Table 8: Overall results for GTOPX unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
(10gr1
44
91.0
40:00
115
1.0kmn
47 Mu
CALES
48:00
418 Moae
OIVES
16.3 mm
150me
118me
12hrm
122
134m
Table 9: Overall results for GTOPX unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
69/00
1100
aud
10k
8.3mm
105mmm
718
AB00O
6012
Labrrw
123mmm
LiBre
AB000
1150
12:30
350
Tal
STOPSS
09/6
100
10kme
1000
Lodow
-150Wine
013mm
630me
355mm
305
28

---

Published as a conference paper at ICLR 2025
Table 10: Overall results for GTOPX unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 10% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
180
1.bbus
130-im
160mm
-25.07
m00mm
100
TED0EA
220
136c
632
185
173
-31
100
4R000
$1 51.9
1800
120
-000
1:5mm
120
D1 -
110 um
100
OCEODA
400
352
TECEA
6.3
-2 10
22 3ini
2108
4B000
136m
18
130
OCTO0A
453
TEDEA
4100mom
3300ma
43
18%
12
5014
Lm
135
6.5mms
-22 Miris
4.50m
+0.0m
438
4.70
-155
-530mm
1.50mm
11148A
116.3
1.30m
18hm
640-
0.1
Libue
6.30u
-5130
1:20g
4211ece
1,2mme
Table 11: Overall results for GTOPX unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 20% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Tap
GEOPLE
-VC 20 10
037
Muss
155
1.
(MAASS
42 33
007ecm
mm
-2.0
-00
-M6 10
1770ma
1.0mus
1.0mum
1.5mm
GEOKL
JPI
171 Mans
1100mm
19
11T Mas
100
10
111 Mum
TIE00A
15 Sl
25 20n
-344
1:2laan
8
18796 Mcc
(CTERA
196 Stame
15 Scow
190 Mlase
354 Murnn
J0MQu
13
Liliee
DEOORLS
111
171
1.
055
1Mum
14bass
1.5me
136mm
13eeis
18lacs
180ms
29

---

Published as a conference paper at ICLR 2025
Table 12: Overall results for GTOPX constrained tasks with 128 solutions and 100th percentile evaluations. In this case, 25% of the values are missing near the worst value and another 25% near the optimal value. Details the same Table 5.
A8000
-1 MutMart
1.5 100meo
3508
CBAS
-00 Mlace
-185446
100mmm
TEDDEA
39hoa
037
Nesca
025
A3000
+2.8mmm
CRAS
LAbrea
1ibme
100mm
Libree
(MALS
151
1,20m
Tal
23574
40
100mm
DHALS
40 Miren
1.3mmm
Libtes
Lilna
100
100
-12 Elan
Bluce
DMALS
8.5
-2
1:8mm
27 Ngui
Hluu
Table 13: Overall results for GTOPX unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 30% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
Sege Tal
100
190
AB000
8.12ms
#033
400
14 Aluae
1M
19:am
Tofuss
1770
187um
108ue
Taa
EX
CRAS
134 13
1144
100
1000
100
TIEDGA
1.6
-2200
A5000
62lme
127ai
Liblwe
121
12huu
100
E-0 Biu
CMNES
28
452
1100e
Extur
EST
130ma
130mm
100mm
10
100me
1100
100mm
1000m
TIE00A
-3640em
64ima
138ms
-0656000
100we
00leo
30

---

Published as a conference paper at ICLR 2025
Table 14: Overall results for GTOPX unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 40% of the values are missing near the worst value and another 10% near the optimal value. Details the same Table 5.
1iGerl
+De Shace
Made
15
30
S1
DI
S1
01
18
100
(3AAES
48033
GB
OCOOEA
110
U1B
13
TIDDRA
134 Str
3.52
04D
15
4B030
Liboww
-118
139
1.
Offace
-0.87 20.0
4.3
3M
150
002
6
085mae
1.00
0.7
1:00
1ibe
14ims
160ma
1-6ml
Table 15: Overall results for GTOPX unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 50% near the optimal value. Details are the same as Table 5.
TA
120
Talt
GTOCk:
-82/0rme
042mu
023mmm
2.54
600mme
038
6.3mm
-552 25522Hon
6.3mso
335mm
-.24
-3.0
158
140 100
187mm
124mm
125mm
mm
12km
124mm
190
(C002A
136 Sus
45
100ue
100
Mare
cm
34
100
1
-3
1/00m
TTEDGA
3 Sodera
<3 m0
firger)
ABC00
15kmn
2.50
055
Sud Strens
364m
100
130m
5:40
Mape
100
Liboo
110
110
19099 100
10 cmm
19
1.30mm
1000
TTE00A
-120m
456 70
<1alms
5.00
31

---

Published as a conference paper at ICLR 2025
Table 16: Overall results for GTOPX unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
StL
301
1000
1:00
1992
CMILES
5/0
A5000
E:0
0C2054
1300m
4 50mm
433mm
11 Moain
4.30
Ti
18132mm
412040044
11224
MAR
4000 Ca
132
120
CMAVES
308mm
180
203
$10
10.00 M0
$10
0072
113461
1.3mm
156 186
-7ANcs
-344us
-37
-3.4600
-18786c
-7/4mm
Table 17: Overall results for GTOPX unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Malk
50
S0
50
140
100
213 10
1000
100
110/001
177
47h
15000
1.00mm
100
200
118
(C30EA
488
<2
22500
<4. 24
41.2%
13000mus
24
100
185
+14 M00
125
100m
-1
180
400
188
100
341
180
Liboos
CMALS
301.2
4 450m
48
400
-100m
-13
ABC00
300
4223300
300mm
+12 Simus
8.0mm
300ee
15008mm
-112ce
<111
201
Ritwws
000
$30
000/es
111194
777
7X8re
217ux
230
6.9
32

---

Published as a conference paper at ICLR 2025
Table 18: Overall results for GTOPX unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
Tal
CRAS
129
Moca
4.00m
4..0
455mm
4.5imo
CHAASS
600
105res
TTace
128cv
10 Hlne
100
100
158
13544
-11A8
DTOPS
CBAS
134
177
1.20 mm
-0400
15000
003004
120:3-mm
3.1mm
0.3
02)mm
<3.3
4.780 94
48.5
11:00
11:00
11:0mm
1138
1000
2400
-200
-00
1488
1408
Table 19: Overall results for GTOPX unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
501
158
CMAZS
+24.90 m8
1.0mm
8.0mm
E00we
3.30
3
-02.0
AB000
-34 37
100m
8.0me
E90wes
(123044
50
10
10 Mome
11
5130pmm
TIEAGA
+.5d
4.70d
154
CBAS
199
Liflww
TTEDG)
-23
10000
1.0amm
00202A
A00
308
100 Shar
170 177.5ece
4.Naam
6.8eam
-2800nue
4.30pmm
4.%
N
-24 -20.8
4,00um
33

---

Published as a conference paper at ICLR 2025
Table 20: Overall results for GTOPX unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 10% of the values are missing near the worst value and another 40% near the optimal value. Details the same Table 5.
Mark
50,
T5
50
F5
20
CHAS
1.00mm
L80
LA0
LABre
140
180mm
1.0ma
108mm
L
090
8.
[
100
1180 All
187 11
Travauees
4DWooi
41 41.5mm
JATHin
Tel
1
100
110005
-06 STune
075
025-
004654
613mn
-1.0huu
-180mu
-130ma
-130
400 S0vi
-1 -1
-35
131 Nove
19
046
09hm
154 Mive
09/
Table 21: Overall results for GTOPX unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 20% of the values missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
DROPS
48000
218-000m
Muvs
11008A
1.0
-20 Blace
0.00
8.0
GOOPLE
434 Bires
440
140
401 10130-mm yea
6.40
444
J54 ml
64
148
Trevering
Tad.
G5087L4
600)0mu
4.30
Se
1
1100EA
4350
4,5
[l
18019)
152
100m
(C006)
812
E13m0
38657
-130u
4000
110054
-38 M-ni
11fme
19imo
ERomn
03Dea
J13 Stane
00
34

---

Published as a conference paper at ICLR 2025
Table 22: Overall results for GTOPX unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 25% of the values are missing near the worst value and another 25% near the optimal value. Details are the same Table 5.
-18 Aveci
Maure
F5
50
FS
50
50
4.22mo
140 05
100
(34AR5
134
151.5mm
151.5 11
19150
100
8:3
4AC00
May
11145
100
12
100
750
50305
4BC10
1Wocc
155
407
11577
$13
110mm
41tur
3M
13 Mune
18 Mun
10
25
TT30EA
13
430mm
13
117ma
Table 23: Overall results for GTOPX unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 30% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
GROFE:
ABCO0
-17 Minu
of
office
ad0low 400
adfice
1
4.30m
0.
130
4.8mm
72 2niii
37 um
62 uea
15
1980
198
185
190
180
185,
180
150 Pi.
198
17
Libur
TTEDDA
4050mm
350
10112a0
-00
Marve
035300
<508
<500
<50
<5 Silf
Tel
CBAS
132 Amme
10:0m
140mm
123
12hua
100mm
100mm
103me
123me
100 Mine
100na
10hos
110160
170
014
12 Doae
35

---

Published as a conference paper at ICLR 2025
Table 24: Overall results for GTOPX unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 40% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
100
1- 130
CHEASS
128 mi
800mm
40
AlA
134 thmm
LAB
13
120
20
2917
37 17
355
-1:13
-7423mm
-18555
DTOFPC
CBAS
18338
Tr
122 1000
1255
TI516A
1.45
24.0
208
3538
3838pm
3030pm
3830pm
3530mm
Table 25: Overall results for GTOPX unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 50% near the optimal value. Details are the same as Table 5.
11.3
33 13
36
CMASS
4.0mm
848ms
340mns
343
-U5 15
AB000
4100m
4477
3.500 4427
-0.30
40 23
4333
Nel 48
43
64
m. mm
50
31530
Th darii
SITETume
319
Mam
49
436 Muse
480 400
180
13
10000
0ragon
0tagon
013gm
310mm
110mm
000004
31238
AMS
000
200
315,0 mo
1Ebun
4.504mie
4840
435/0maa
40500mom
435.9rumm
36

---

Published as a conference paper at ICLR 2025
Table 26: Overall results for GTOPX unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
109 GIOENS
190
CAL
622
225
48
18
11
73 4i
110/
108
110
15
ARD
A
CBAS
N
-4008ms
002004
43hma
<25mm
4.
330m
TIB/6A
3,8mm
-120
1140
-12
-4330
Table 27: Overall results for GTOPX unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table
AB000
04lmm
34Beia
STolune
Mi
442
GIOF33
Mun
54
Bilur
Bilure
313
Litbon
10
27 Slnr
Litice
25
2611
140
X
Tiee
16
1
M4p
.74
(13464
198 18800/m
473
-508um
-14kmm
1aam
Musi
Jitas
-7 -1
-2.0p
37

---

Published as a conference paper at ICLR 2025
Table 28: Overall results for GTOPX unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as 5. 5.
Tal
Mark
CRAS
+29
-035
-305
40,0
20 m7
20,00mm
40/0mm
20.4 20.4
48 Maus
4154mm
CHIATS
400
400us
300
48.00 ml
012004
1763
200
1
+344
-3 Toleeeee
0TOP%
CBAS
100
15000
30.5un
11.0pm
002064
33275
<67
<00
20
4227ma
TI816A
4,3
3408
3400
Table 29: Overall results for GTOPX unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
Sepe
CRAS
158
1.00s
Munc
179 Taav
180
17879
180
-10 DOL
CMALS
4.55ms
18.40
AB000
+1 M.Bo
+1.
100 Buuue
00
431
TEEAGA
Tiee marii
8 116%
4 118%
1 13
4 116
110
4.80
-288
18
180
180
TTED0)
10000
11kgne
124
-22
00302A
455
-.hun
4.204
48 Nean
400
Woae
38

---

Published as a conference paper at ICLR 2025
Table 30: Overall results for GTOPX unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 10% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
1-130
DToFk:
-8.0 B8.
43456
4006G00
-03 PistP
45
[1rGe))
LAGrom
LAG
L00mm
Libic
10117
+1465
1.8
<BAS
122
3,55 voe
-.500
4.500
CHAAS
21.50 1p
AM000
4.40
-333
100
500.0
7.7
TI816A
41k3ure
42:5
42:5wee
42.5
Table 31: Overall results for GTOPX unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 20% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
.89
159
Tel
GROEN:
-0.5 man
<8 mis
42000m
4100mm
CMASS
-11.240
AB000
5450
0.0mm
400 1000
4N
3.5
41 ml-m
130m
25
1300
110
180m
3M
TTEDGA
1.20
+.2
-22d
Th
10
199
3 Milegge
1199%
1000
<RAS
115/um
319/um
18
149/ 119/ume
Lif
100
200%
TTE00
+22
mm
200
10000
123
11.0
00300A
1350m
115mm
13300
345m
1160
-1000
39

---

Published as a conference paper at ICLR 2025
Table 32: Overall results for GTOPX unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 25% of the values are missing near the worst value and another 25% near the optimal value, Details are the same as Table 5.
Sap Ariarl
4iimn
17h4
LIDAUS
400%
Eat
255
30
480
70094
$77,44
Table 33: Overall results for GTOPX unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 30% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
10
GROFK:
A8000
-.0 Moarl
Moart
T1 B0
14
+34 Drnn
153
12bus
(17709
MM
GROFF4
CEAS
1154
180
119
CMATS
18 1150
11
(
AB000
01260A
1130ns
1155
-35mm
40

---

Published as a conference paper at ICLR 2025
Table 34: Overall results for GTOPX unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 40% of the values missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
JM GROPKS
19
140 Sluce
340 3M
CHAASS
1200
12:00
13050
1005
7135
52
Tra
100
10:00am
100
12%
180
18
10000
Plase
30 Puan
190
-00
100 44.3mmm
140 (4.3
24 Boa
-40 2003h
445
444 240
Table 35: Overall results for mujoco unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 50% near the optimal value. Details are the same as Table 5.
Supe
c- 54
c- 100
1- 280
Talk
Mtase1
fellss)
777--
Masa
F5
G1
50
50.
BS
51
OE
50
S1
DE
50
S0,
BD
0.8.amm
120
111
1.2
655
1.35
E.44
CCEGRA
075,mm
106
1.1
6.7
653
1.77
1.31
1000
0.85
1.00s
1.17pm
EA5
I1DUEA
0.20ma
0.4
0,20eox
3,34mm 0.05cc
Bubber
0.50ec
Talk
Mhawe
ABC00
D42
035
Lifox
0.3on
100
112
672
612
1.30ie
413.04
1.80
47500
417.9
11125
OCEERA
675
1.5
676
125
335
012
060
0.0
82fmm
117..
135
640gpm
685
Table 36: Overall results for mujoco unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
Supe
50
100
1: 150
Talk
fiKlay)
Mala
15
61
50
50.
B5
51
50
90,
P%
S1
5D
50,
BD
0.80amm
111-m
135
B55
11
ES1.
CCTORA
077mm
077mm
100,0
655
1.0
098
877mm
661
IIDDEA
0.1aam
655.0*
5k3um
0.45am
3.0m
Tonewrrrg
Talk
3base
fikter)
80
0.9hee
LAlens
100
122
DAZess
B/3
0.5eac
180 23
CCEERA
015
0770
120mm
13
170053
1150n
60
6:7um
Treewoveg
100mmm
135mm
153mm
Mlae
Fillas
41

---

Published as a conference paper at ICLR 2025
Table 37: Overall results for mujoco unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Supe
50
c- 100
I: 350
Taxt
Mellay
Mall
F5
GI
50
50.
B5
30
91,
R
SI
C4
50
50_
RD
1dtger
108
133
B53
1.55
0000
077amm
1.00um
0X1EBA
0.3
677
1.5
0.5
0.7
1/
0.75-ox
TIOBEA
0.30ma
0.26mx
033
0.3
0.2
0.3ca
0.15
0.2
020
0.15-
0.30mm
123
1.3
D00
600
134ra
0.3
1.8rx
Talk
Mhasn
!iKlay
42000
083
1:00um
100
1:0
11
E66
FRAS
125
185
185
18548
186
13
J.3aam
325
0.90mm
0.Paa
100
123
18
1:0cmm
0.0amm
Table 38: Overall results for mujoco unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
Stps
C- 100
150
Talk
Show
Malie
1%
D1
50
50_
50
9%
800
0.41am
1Alam
109
(LDRRA
099
9.5kmm
Uillane
I1UDBA
0.20mm
Jubuae
9,2emm
Ionnnurng
1.
Talk
Majps
flelas)
100
BD
0.7
0
1.5
0.00
CBAS
-1.0raa
-0022
-1442
-00
Masa
(KaRS
144mm
0.30aa
0.5bem
1/00me
1.3oom
1.2
100
1ibbax
1.bras
Lillww
0.33ccm
Table 39: Overall results for mujoco unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
Skpe
54
c-
100
L- 250
Talk
fleles)
Masa
F5
G1
50
50.
B5
51
OE
50
20p
15
S1
DE
5D
SO, 6.T
BD
0.70mmm
0.Nom
1.00mm
160
2.33
6770umm
676
(90
4.5
1.
-77
1271amm
1.23c0
0.55cc
I1UUBA
-L4teon
-2.2
2.12-m
-150
-1.0
0.30me
123-o6
1.3
D04
0.0
Talk
ABC00
1.30cm
1000sm
1000um
Dadaua
0.3iaa
030
043mmn
3.5
125
0.0
1.80aa
30536
285
1.80
300M
3609
307.50m
307
OXEERA
066
111
017,0m
1Pame
100
1.7
Table 40: Overall results for mujoco unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 10% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
Sups
c 50
c 100
L: 150
Talk
Mala
1%
G1
50
50.
B5
51
30
90u
P
S1
50
BD
166mmm
111
135
B53
100
CCTORA
100
670
679
157mm
140rm
0
1 19
10k
666
I1DD6A
0:20am
Maam
0720mm
855mm
Edilans
0.8amm
0.3mn
Maa
6:
0.89vam
175
133
Eillne
0.2
Talk
3base
fikter)
80
(.75em
LABeer
1:3
1.4cr
0.0em
140wn
672
CRAS
130me
12030
175.4
1807 2
130 181
1889
Stu
125
19
118053
Treewrveg
110mmm
130mm
1.0pm
17h
EMum
42

---

Published as a conference paper at ICLR 2025
Table 41: Overall results for mujoco unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 20% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Supe
50
c 100
I: 350
Taxt
Melaay
Fuuee
Mall
GI
50
50.
65
OE
30
91,
R
S1
D4
50
5p.
RD
108
133
6.7kmmm
033
1.0g
075um
0.
19kg
6.03
0X1EBA
085mmn
0.Neexx
1.4
0.9ha
0.55mm
TIOBEA
0.30mn
0.35mm
3.52
0.30mmm
0.bba
0.36v
0.0
0.3
0,5
1.7
60
1.50
122rea
E10bm
0.5
0.90
123
Talk
!iKlay
42000
1:00um
096
13
0.5
pr 15097
6.73
FBAS
185
185
12007
186.7
6.70
108gm
13
17
122
100
Liflae
0.9uc
1/kuss
1.11am
Table 42: Overall results for mujoco unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 25% of the values are missing near the worst value and another 25% near the optimal value. Details are the same as Table 5.
Saps
100
150
Talk
Maioca froe
15
50
50_
50
9%
800
0.83amm
110
133mm
1/8um
(LDRKA
3.3acc
9.5kmm
1.
10
I1UDBA
1.30aa
44b
0.2bc
122
Ionnnurng
DM0
Talk
flelas)
1:0
DASeiS
0.
BD
1330
0.50xaa
CBAS
-1.0na
11000
119
13000
-09 201
-00
OKaRS
1,00mm
125
08bae
0.55pm
LAlaso
1.Bboe
1.2
133bor
105mmmm
0.55m
1275ac
1.3oas
Table 43: Overall results for mujoco unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 30% of the values are missing near the worst value and another 20% near the optimal value, Details are the same as Table 5.
Skpe
54
c- 100
L- 250
Tal:
Majos
fleles)
0
Masa
15
G1
50
50.
B5
51
OE
50
90,
F5
S1
DE
5D
SO, E.Kliom
BD
0.M0umm
0.81pmm
1:50mm
1.0
3.3
650
1.87
077
0.00mm
Liffiw
48.2
0.3
I1UUBA
0.30ma
0.12em
0.30mc
1m3
030est
0.4ecc
Talk
ABC00
6i1mes
100r
0.87aa
6:0
1.4
0.67
19631
.1643
-18621
192
-09 Mas
OXEERA
066
1.77mm
03
112
015
530p
021
0.90p
0.6u
075cm
106
133
650
077
1
Table 44: Overall results for mujoco unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 40% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
Sups
c 50
c 100
L: 150
Talk
Mala
1%
61
50
50,
B5
S1
CE
30
90,
P
SI
C4
50
BD
0.70mm
0.72mm
1.00p
140
1
1.01
435
1.5
100gm
CCTORA
0 0'Naam
1'00mm
667
mm
6.0mmmm
071
100m
477r
1
I1DD6A
0:20am
3 3.am
.18..
227m
2200
.28km
0.50
12Pra
832maa
Talk
Mbase
fikter)
Dabans
tolless
80
0.30aa
122
028
BA
133oe
840mn
CRAS
130mmm
3055
365
3009
302
307 M
0.47
100mpm
100
130mm
119083
3300 152mm
Treewrveg
07humm
177amm
19tax
100cmm
1hhmm
1/00cm
43

---

Published as a conference paper at ICLR 2025
Table 45: Overall results for mujoco unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 50% near the optimal value. Details are the same as Table 5.
Supe
c: 100
t: 350
Taxt
Mellay
Plaee
Mali
F5
GI
50
50.
30
90,
R
SI
54
50
50.
RD
694
8370mm
p04gm
6.68
0X1EBA
0:5mm
106
1i3xe
0.5
1i2
0.88
TTOBEA
0.20wa
0.5
0:8
3.5
0.5
0,50ma
0.2
0.53
0,5
0.
0.30au
1:50mm
100
1.77
1.3ra
0.5
0.5
Talk
30ase
!iKlay
42000
1000
1190
342
100
FRAS
188ces
(P5M
171
171
126M
121
138.5
050
11
18
605acm 1.34am
022ues
1300mm
100
1.4k
133pm
Lituee
L.Puun
Table 46: Overall results for mujoco unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
Stps
C- 100
150
Talk
Majaca Whow
Malie
1S
50
50_
50
DI
50
50%
80
0.30am
M
boflue
(LDRRA
0270am
100.0
1.00x
9.7amm
Milane
02b
I1UDBA
0.20
334
Ionnnurng
DAtees
0.30em
1300
Talk
flelas)
BD
0.3
050
3.5
0,85
5.55c
0.53
4.3
CBAS
-1.0ms
119.35
-39
-1.0
18824.00
OKaRS
1:21am0
110
1.3
0.5ramm
1iBbaz
845
1.4
bibans
12f
1.30
1050mm
0.4km
1.2aa
E.50cmm
Table 47: Overall results for mujoco unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Skpe
54
c- 100
L- 250
Talk
Majos
fleles)
Masa
15
G1
50
50.
51
OE
30
50p
15
S1
DE
50
S0, E.lliw
BD
0.37
0.82
063
536mmm
651
3.5
0.81mmm
-3.12a
-4 43.3000
4144co
1275mmm
I1U0BA
0,30
D22
2:13aca
0.3
0.7
0.3
133
108
1.58mm
800
Li2
100mm
0.00caa
034
Libro
Talk
ABC00
0000m
3.5--m
1090
0.3
3.9
6:00
0.1
1.30ue
115508
185.88
1.80
18607
186.7
CXEERA
0.00
676
0%
1.0mm
100
1-
1:55
040
083um
1.00
3.5
005
07h
117mmm
100
6M
130gm
070mm
045
100
Table 48: Overall results for mujoco unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
Sups
c: 100
L: 150
Talk
f(Kigy)
Mala
1%
61
50
50.
B5
CC
30
90
P
SI
50
50.
BD
077pm
670.
$7
B/A
000
672umm
CCTORA
077mmm
Mage
104
077
87
6/4
I1D06A
0.20am
4:30.m
Edfaan
4:58
Mur
413umm
41 M:00
0.3mea
130mem
0.5
1.3exa
Talk
Nbase
fikter)
Eilene
80
1000mm:
333
CRAS
(9)
291
1813 L*
195
196 Mian
19536.u
105%
Stu
477mm
677
671
07
110pmm
110
118053
.100
Treewrveg
07hues
118mmm
110cem
100
BW
110pmm
02tana
44

---

Published as a conference paper at ICLR 2025
Table 49: Overall results for mujoco unconstrained tasks with 128 solutions and 50th percentile evaluations, In this case, 0% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
Says
180
Talk
Maaac Phacss
P5
90
61:
M
90,
30
30)
B0
0.30a
B42
344mi
3:44ca
0.5643
Bulluoo
C0TOR4
07)
658
6/70pmm
6700
13
0.0
08
4.0
ITBNRA
0 M
27
43
480
4.80
Mun
4.5
Talt
Mapeo
80
<131
1.36am
-28
-204
-30
0.3bam
122
025-00
354
D3bes2
0.55ax
9.5
1.26e
TIDUBA
<00mm0
-122kcu
-328-14
4.38-m
+133-20
<1.3kan
-1.30um
+135kc
-1,85c
-3.5
0.2ees
0.30ea
335
0.90vua
0.3eux
10feeio
Table 50: Overall results for mujoco unconstrained tasks with 128 solutions and 50th percentile evaluations. this case, 10% the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
Talk
Mapes
ARC0O
1.30aam
(199)
(dllumm
0.8lux
6.99
UBAS
2918
0.3lax
-P1
Dhase
CHARS
124mmm
0.2
0.3
122
1.30cm
1.0kcx
1ibre
8.2ech
135
136ma
Libeet
ABC00
0.20xx
E.0
3.50ccm
0.99kx
0.99
E56
4.37
CBAS
.1.0ism
11202
117225
12922
13
-18.33
-18 188.5mm
18054
18854..
183
033am
1.5
677
03
0.7
100..
CMARS
8.77mm
.1absen
04800
34
34bwca
Table 51: Overall results for mujoco unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 20% of the values missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Skge
380
Tal:
Masi
15
50
50.
DI
Plecc
BD
0.37
3.4--
0.00a
4532 3333
45.12mm
43.3
-0
OCEERA
112
6.7
8.5em
1illi
6.8
CRAMES
1:3
11eve
1:50m
Libee
13hos
Mileoo
Tell
Mipwon
f(Keay)
48700
08Pae
MMgan
000gp
BMW
Mige
037um
CBAS
-1.0mam
112 18248-0
.155 68
110 60
+18 85mas
-19007
18607
195 Shea
CHAES
0.2
0.77mm
116mm
110
120
0.9ims
0.2mm
1.14mm
140
125
6.50um
6:00
1.77mm
1.8imm
1.0km
10kmm
45

---

Table 52: Overall results for mujoco unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 25% of the values are missing near the worst value and another 25% near the optimal value. Details are the same as Table 5.
Sage
54
6- 105
L- 380
Task
Maase 0.4
Mkite
15
90
31:
90.
15
30
30:
BD
0.30ma
0.77mm
0:00u
679
379
D.50umm
67T)
543
078
07%
837
037
0.0
6
0.00am
1.30pm
890
183
6.77
677
1:0
090
87amm
4.8
18
10k
67%
ITBDRA
0 770ee
606
--
Mage
010
Talt
Mapea
80
40Phaa
020
333
8200
333xc
100
020
-12boam
.0000
201
.0122
-10
0318BA
0.500
0.75mx
134
055
153mmm
133
0.55mm
9.3
TIDUBA
0.3kmm
Bib--a
Bdiios BM
0.31mm
07
0.
140pe
100
16
185ux
10llw
03
045
108
6.9
Table 53: Overall results for mujoco unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 30% of the values missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
Skge
0 3Miui
Mass
50
50
DI
50.
0.30rax
3.0hxx
0.00
BD
0.72mm
3.50
5.13
0.09vex
48.97
0.0
OCTERA
0.Niea
135
1.5
6.7
1.5ma
1.0
TTDBEA
0.30mm
0.56
433--
0:32 pou
6023
m.3
0.25
033
0.15.
110
100
1.0
640
100r
Tall
Mapxo
HiKlay)
40100
011
MMgae
1000
(BAS
1'3uus
194 118
199 uma
11
195
100
195 Saac
Mun
CHABS
0.
0.77wm
1100m
100
125
0.3
0.50mm
100mm
0.6
125
100
1.00mm
0.00
03
1.00
100g
0.44
Table 54: Overall results for mujoco unconstrained tasks with 128 solutions and 50th percentile evaluations. In this case, 40% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
0 3.3
ABLDO
Utfiaee
Lakase
35
CBAS
<3133cc
33.29am
bibwu
-8% 10000
-0. MouDo
0.0exx
-5 44.33cc
-5.33mm
36.3
CHAES
07
0.30ma
130
1iboo
077kmm
118mm
100
105mm
658
44am
1.5mmm
64)
Tal:
f(4
ARC00
033
0.77mm
007
697
1:30pm
347
033
0.80
Silbuum
CBAS
.1.0ee
283 Bbic
-33 Shn
<20 Siice
-3004am
-3fr22
<80 25
Shoce
1i80es
D2heua
1.8
0.30ccx
0.35ec
802
CHAES
0.200
0.8em
055
355-u
TI0DEA
4:20u
-84
-831-am
-031
0.3mm
355
BS0
3.9eco
000
006
46

---

Table 55: Overall results for mujoco unconstrained tasks with 128 solutions and Oth percentile
evaluations. In this case, 0% of the values are missing near the worst value and another 50% near the optimal value. Details are the same as Table 5.
Mego
t- 190
180
Taol
Megao
Mancc
P5
50
60.
PS
54
CE
30
90,
F5
51
DI
50 Iddess
50,
BD
0.deun
Ed4
075mm
0.Neec
800
1.0
0.0
0.3
0.80ms
Ld
0.87
ITDDEA
0.9mmm
0.4bum
643-.
0:4
0.3
044
[43
3.3
R.Macw
Tall
Mapeo
RD
Ramm
643
1000
844
0.
0 3.90
0 0.00
E
.130mea
-172,44022
-128com
112.99
-27508
-12.9
-13423
CXIEBA
0.2
1.50
13
TIOBEA
4.5emn
0.4bou
B48
3.5
0.55mm
3,5kxa
40Pren
0.00nsm
677
3.5
0.87
4.3rc
Table 56: Overall results for mujoco unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
Nape
51
109
1- 180
Tank
Meer
80
40-
BD
3.55-m
0.50vm
0.50
0,50ms
077cmm
0.0
128 -m
6.77
653
1.08
0.00
0.77mm
E.5
ITDGEA
0.0mmm
0.50mm
0:53
0.31
0.30
033 433
030
0.33
Tal:
80
9:30um
0.30aa
0.3ton
IdSun Lilan
0.70mm
65
109mm
Echana
9.7amm
1100BA
0.5rmm
022
32
9,2
A17
0.55mm
0.30a
8:6
35
0.3
8.5kmm
4.3gm
3,77mm
0.0
Table 57: Overall results for mujoco unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Says
Talt
Mipao
Meine
15
90
10:
OB
30
90,
15
al
DI
MO
B0
844c08
3440m
3446c
E44
OXEERA
1330em
800cmm
100
1.5ma
Libiro
ITBREA
013gmm
600
onl.x
085
E
Tal:
3
80
1.Sham
3.5
1.Mham
-10
110
Slnce
0X10EA
135
104
1.40cc
0.90
0.34on
Lobano
T100BA
.130ne
-008-0
-328-m
-12B-co
-2.2
<200kcc
4100mam
<1
400
4uthoo 072
-2,00 6.7E-um
0.5emm
024.u
3.4
3:8--om
0.87
0.30cca
Table 58: Overall results for mujoco unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
Mego
1990
Tall
Migwo
fleiay)
50
50
90
AR700
BD
0.27ee
3
0.20
(13600
0/deus
026ms
CCEERA
0.7cmi
0.7
1330e
800
120
1.5
0.5mmm
104
ITDDEA
A77ixa
4008
-050ma
8.37mm
4.5
Taxl
Mapwo
flelas)
RD
0.77mm
810
505
3.00m
38
00
CETORA
017mmm
656
016
077rr
6T1
TIDDEA
-1.6mea
-L15mm
-.12-
-130-a
-188-cm
-1.440a
-1.4
-Liivwn
-1.18am
+115ma
-115
-2.15mm
415rem
0.6mn
345-m
(.30am
4.15ma
0.35caa
0.5
47

---

Table 59: Overall results for mujoco unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
Mego
t- 190
1- 180
Tall
Mepao
Mancc
P5
50
60-
PS
51
CE
30
50,
15
51
DI
40
50,
B0
0.5mma
4.12-a
4:12-s
0.dees
077
0.20ax
333-u
B
0.87
0.30ra
0.3baa
0.0
1.2
600 a
ITDDEA
0.3mm
4.0cum
4.64..m
4.54.m
EdMaa -500
.585
1.63.
2.34cm
0.3uam
Tall
Mapeo
RD
10pm
110g
0 10
0.10
-13mea
-28.85
-128cam
<20
-20
-20
CXIEBA
1ilmee
D00
TIOBEA
3.12-hh
-100
-1.03
-190
-1.80
4.0
0.00vm
0.10ve
B/6
0.3
4.30s
021
Table 60: Overall results for mujoco unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 10% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
Mape
51
109
1- 180
Tank
Meer
80
40-
BD
0.33mm
0.56mm
3.5--m
0.50mm
(.50mm
0,50ma
071
1.50mm
0.00
124
6.7
650
1.00
0.5
0.5
1.
E.3
ITDGRA
0 0.3am
10
0.30mm
6:50
017
0,3
0.38
Tal:
M
80
022han
0:5
9.55mm
0.
0.27pm
0.70em
IdPhamm
0.3kan
658
1.4
9.2hmn
14ban
tohu
1100BA
413
0.25
-006 Bic..
-23
JUP
Julfacs
035
0.5kpm
0.50pm
056
4.7mm
47
071
877
Table 61: Overall results for mujoco unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 20% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Mays
Tal:
Mipeo
1%
90
M
OB
30
90.
15
DI
MO E44 6.77vl1
BD
Libase
3444co
OXEERA
104-cm
1.3
0.55
12bba
035wmn
4.77mm
14hes
ITBREA
0 0
11
mm
0
0
4 M.
Tal:
3
80
J0MMan
(.11am
188
0X10EA
0.0bo
(175ee
100mmm
0.2hoon
13han
0.3emm
T100BA
-647-a
-020
-3.2
0.30mx
335
3.25
0.94
Table 62: Overall results for mujoco unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 25% of the values are missing near the worst value and another 25% near the optimal value. Details are the same as Table 5.
Meyo
199
Tall
Megao
50
50
50,
AR700
3.Mux
BD
0.77ed
3.3
0.77
(337000
0.30ax
(36m3
CCEERA
0.5ema
123
083-04
315
674
144
4.5
478
ITDDRA
023
033
433
433--4
D:3
400
408
-008
-2.0
Tall
Magwo
flelas)
8D
0.7mm
0:7gpm
6:3
$33
$35
0.30am
440mm
1.
CETORA
088mm
0.3mmm
175pmm
858
077
11
675
TIDDEA
433hnn
-047-u
4.0-4
-033
-028-aa
-3.2
43jmn
4.22ma
4:22
43
0.50ma
0.52mm
646
345
0.85aa
4.77ma
67h
48

---

Table 63: Overall results for mujoco unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 30% of the values are missing near the worst value and another 20% near the optimal value. Details are the same Table 5.
Says
150
Task
Napae Cheam
Milree
80
10
90.
40
10.
M0
3.
0.3aea
(100uc
07
0.00m
1:5
655
355
6.7
676
44
0.90
0900
4.7
1/8
140
6.7
ITBDRA
07
0.5
855
4.5
660
0.0mm
0d1
05
2.53
Tal:
Mapes
80
0.35am
018.0
11:4
0thras
0.1MAA
0.35am
0.35am
1ibaso
625
352
9.5kmm
120
T100BA
4375em
4:33-04
433ca
-338cca
-033
-23
4.Phax
-SAbbo
413menn
B43oaa
34heix
026
328
0.3
4.00wc
Table 64: Overall results for mujoco unconstrained tasks with 128 solutions and Oth percentile evaluations. In this case, 40% of the values are missing near the worst value and another 10% near the optimal value. Details the same Table 5.
Says
15m
Talt
Mapec
Melee
P5
90
10.
90.
DI
30
B0
4113am
4.10
bibas
41
48 41
4114
0XIERA
0.36mm
0.30m
0.5aca
1.3ma
10hers
IED0EA
01ewen
433-ma
A.23mmm
-2/0
Tal:
Mapea
80
4.8iaa
002
0000
440
6.0
4,5mmm
4.90m
4.0mmm
2.00
2huce
-30
-12barm
-Mic2
0.10EA
0.17ea
0.37mm
0.55am
102
T100BA
<USbeo
<1555mmm
-3857-40
1234cca
-252
<032-1 0.2ms
0.7ewn
0.2
0.32kc
Table 65: Overall results for CEC constrained tasks with 128 solutions and 100th percentile eval- uations. In this case, 0% of the values are missing near the worst value and another 50% near the optimal value. Details are the same as Table 5.
Mene
10
ML
12066
2 3.80kvon
1.00
Liftes
1.00mm
600
60.08
0200
78.00
710
DET
0.8
-300ma
4.00 AM
4.2k
3M4 2M
048
180ms
300
1600
47kam
74lom
12
101
110
750
49

---

Table 66: Overall results for CEC constrained tasks with 128 solutions and 100th percentile eval- uations. In this case, 0% of the values missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
100
CFFT
CARATOO
1/0k
AM
2630 m1:
19:0
-852
-00.5
7700
43 12.30
No3
3.704
CARCOO
300
45hmn
DEST
<8/0
-3.0
Table 67: Overall results for CEC constrained tasks with 128 solutions and 100th percentile eval- uations. In this case, 0% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
DAFE
1,0
Lob
1.0
1/0
00S77
-00
-25
25 Sh
3533mm
-2.00
431 S0:1
C001
10011000
415
194 14
152/0
000-
-300
118
15 1EB
$00
+1.00
12:2
1.20pm
-2.8g
417
Table 68: Overall results for CEC constrained tasks with 128 solutions and 100th percentile eval- uations. In this case, 0% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
50
50.
54
1,0bu
mnoe
1000
1000
100
FRNNS
-0 20
40.33
-3
25.00
(1 Bl
31.00
Hoop
5007
04
FANIL
30 S0a
4.50
Talk
04F
448
50

---

Table 69: Overall results for CEC constrained tasks with 128 solutions and 100th percentile eval- uations. In this case, 0% of the values missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
100
1800
100
100
1.00
CRFTS
45-3
FRIMI
-66 Sieae
CARCOO
-000
4377
FAINE
+60 40300
Table 70: Overall results for CEC constrained tasks with 128 solutions and 100th percentile evalu- ations. In this case, 10% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
Meris
50
50,
F5 -13elune
01550
1,00
1.
1.8
1.30ckns
100
10km
1,0m
1.3m
100mm
11231mm
11.5
15
351
Bat
19.00
CARCOO
260
6.30
4,30
4.300
14,003
4.30
4.360
4.302
-27
A25
362
315
4.00%
Bank
Stute
-558mm
11
4.80mm
400
6.0
4:40m
7730gm
3.3pm
1.0m
Table 71: Overall results for CEC constrained tasks with 128 solutions and 100th percentile evalu- ations. In this case, 20% of the values missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
191 C3PTT
CARCBO
1.0r
100ra
1.0mm
1.30m
1,0ka
DEF9
7558
015M0
441l-cn
Boss
2400
500
110 Torm
110
+3 Piliie
334 Soree
300
5DC.5
1999
3035ma
FROVE
Murn
51

---

Table 72: Overall results for CHC constrained tasks with 128 solutions and 100th percentile evalu- ations. In this case, 30% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
100
CRFT
CAROOO
1.0mmm
FRINE
15.0
16
HeKarl
CAROOO
4.00
02.
Talk
CBS4
CaR000
330ma
3.Nc
333
FAINL
422
4 3alm
48 20
4 3ioce
-24ma
16:00mm
-2
4.4
-2 -220mmm
Table 73: Overall results for CEC constrained tasks with 128 solutions and 100th percentile evalu- ations. In this case, 40% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
90
MC
BEST
4.206-
1.00
mm
1.0mm
1.0mm
10
CXR000
Mane
41.
11 11
-11 11.3
171
10.0 11.0gm
11.9%
18
11:3
11
+13
17%
m
19
15
27
CAR000
Talk
415
48205mam
Table 74: Overall results for CEC constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 50% near the optimal value. Details are the same as Table 5.
Bale Iletay!
CDCT
50L
2
1.brus
100
10
1.0mm
FREME
0435
66
-0.00
44 41
4100
40.00
40.35
2/0
(CARCOO
43.3
-300
-2.4aa 4dinc
4 Siiie
-60
Sivck
1999
FREVE
18:2m
CARC0O
-35lmw
<0.00rm
4.00mm
4.50mm
-000.
52

---

Table 75: Overall results for CEC constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
100
CFFT
CARATOO
6024,mm mm
90 A
<9
mm
14
me
2.10m
13
13
68:0
28.2
-14.5 18.5
18
10
DARS
CARCOO
3:0
DEST
4033mm
41.80
1500
43.07mm
-550
-1.33 -3331
Table 76: Overall results for CEC constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Sope
100
TM
GECT
DEFF
LeBre
L00
1,0mes
-200
Takk
-1.0
DEST
-4
46-5 54.50
24.30
44.7
FADIE
CARCO
41/3
7.00
[WKurl
S440
18-3pmm
15
15.0
16 Mot
63 m0
-24Rme
+1 41.lmm
#400
1248
12
124
1248
Table 77: Overall results for CEC constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
50
CARCOO
1:00pmm
10
18
!
CARCOO
PAUB mus
137
43
44.3
6500
Tell
DE2T
<18 38
218.5
<38
311 M0
1,0 10000
FREUE
400kres
310
-0.5
53

---

Table 78: Overall results for CEC constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
Dape
116
108m
1.0
100
160
04 mi
24
-2.04
1018
P100LA0
311
313
FRIME
214 Maas
216
211
311 Mace
217 417
217.4
-31 m0
Bal
CAR000
4500
4500
16.02ve
2
4000
Table 79: Overall results for CEC constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 10% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
190
CANCTIO
110
100
180mm
1.00
100
Bel
030-3
323
24
CARCIO
4330mm
-37.0
425
200
A200
127.66
4305mm
CTC
2127mm
500mm
5,00
30 Baee
4A MA15
2120gu
2222
242
Table 80: Overall results for CEC constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 20% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table
Race Bol
150
CDC
Bolk
DET9
St Auc
-28 Niami
-308
CCACCTO
4815
40A2240
-30 20
400
J08 18:1
130
-08 18
100
1100 Turm
OET
304 Menii
1999
40.28
54

---

Table 81: Overall results for CEC constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 25% of the values are missing near the worst value and another 25% near the optimal value. Details are the same as Table 5.
C
Hefurr
CARC00
m
am
18:0
mm
CARO00
$4.50
0505
900
BEST
CAR000
330m
S
335
3
-08.2
1796g
20
7
7
700
2
Table 82: Overall results for CEC constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 30% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table
Mastc
50
DAFS
100
1/8
4.312
100
1,0mm
-10
15 -8.m
TRDE
+1845ma
DEFF
<1100
FADEE
MC1
TRNE
-230001
OEFF
33136
191 19136
-66 3
J36 Stenco
46 10
-M6 Me
334
DEATI
-347
TIDE
Table 83: Overall results for CEC constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 40% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
<18m)
51
50
F5
OK
50
50u
75
94
Of
4.30
100
100nus
LiBrro
4 3000
100
100
1,0mes
100
CAR000
+22.3 FAM
15
Tall
DC2T
25.2
40
41
1999
N611
190 Plams
47% Waee
<298
40.3
55

---

Table 84: Overall results for CEC constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 50% near the optimal value. Details are the same as Table 5.
100
CFFT
CARATOO
2.8ookvou
1,0k
CAATO
0193
91 (5
+255mm
15.10
4.77
431
18.7
CARCOO
47
DEST
408
cM
408
200
Table 85: Overall results for CEC constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table
1,0r
Lob
1.0
1/0
10h-
-1.0
44.5
466 mm
-2.0
11..0
15
11
16 M.
1800
000+
4.3003
400
4.00
11
SEA
4.00mm
4.50
Table 86: Overall results for CEC constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Mates
50
50.
54
1,0bus
mnoe
1000
1000
14.1 JA
FRINS
State
46.3 40-5
-3
44.7
Hoop
1.50
315
215
315
21048
211
23
31100
FAINL
+1.31 113014
+3% Stris
Talk
04F
120.0
56

---

Table 87: Overall results for CEC constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
Sapg Balk
UM
188
Talt
Mure
10 7g
6200
218.1 18
712.00 00.
333
FRIM
<16 30
210
211
CAR00O
16
18
FAIN
46 48.5
Table 88: Overall results for CEC constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
30.
10
D0STT
-2.00 40
1.boxx
1/8
2 3.80k-oon
100mm
300m
1/0k
1.
1.0mm
1.00m
13
4100
6100
Bd
000.5
313
315
313.00
1
1.1el
60
Bal
030.5
2.21
4,30
Table 89: Overall results for CEC constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 10% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table
Rape Bol
CDC
Back
030-2
0419
Miaa
25
24
24.55
24.11
25
1999
140
FROM
Notu
Bot
530.5
30
57

---

Table 90: Overall results for CEC constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 20% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Mere
P4
1100mm
1.
L
1.8
1
100
21
BE5R0
5482
04 42
>6 20
-4.50
110.0
10
10007
-117
+18
100.8
400.3
190
+3 32
18052
<00 mm
4005
Bat
0B04
1364
PARCTR
DE5R
4738
J8 Mta
Table 91: Overall results for CEC constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 25% of the values are missing near the worst value and another 25% near the optimal value. Details are the same as Table
100.0
10
1305
010
417
12:00
33100
Table 92: Overall results for CEC constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 30% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table
1377
100m
108mm
100m
10.00
58

---

Table 93: Overall results for CEC constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 40% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
1301
140
38
110
40.90
111 411
41.8
100
43.
Table 94: Overall results for GTOPX constrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 50% near the optimal value. Details are the same as Table 5.
CTONCE
Table 95: Overall results for GTOPX constrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
Table 96: Overall results for GTOPX unconstrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values missing the worst value and another 30% near the optimal value. Details are the same as Table 5.
700
59

---

Table 97: Overall results for GTOPX constrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
Table 98: Overall results for GTOPX constrained tasks with 128 solutions and 100th percentile evaluations. In this case, 0% of the values missing the worst value and another 10% near the optimal value. Details are the same as Table 5.
AM
0F813
Table 99: Overall results for GTOPX constrained tasks with 128 solutions and 100th percentile evaluations. In this case, 10% of the values missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
FEM
Table 100: Overall results for GTOPX constrained tasks with 128 solutions and 100th percentile evaluations. In this case, 20% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
60

---

Table 101: Overall results for GTOPX constrained tasks with 128 solutions and 100th percentile evaluations. In this case, 30% of the values are missing near the worst value and another 20% near the optimal value. Details are the same Table 5.
2024
12
2.0
Table 102: Overall results for GTOPX constrained tasks with 128 solutions and 100th percentile evaluations. In this case, 40% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
2.28
Table 103: Overall results for GTOPX constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values missing the worst value and another 50% near the optimal value. Details are the same as Table 5.
18
COAR
61

---

Table 104: Overall results for GTOPX constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
LONAM
Table 105: Overall results for GTOPX constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
1491
Table 106: Overall results for GTOPX constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
Table 107: Overall results for GTOPX constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 0% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
CAB22
13.0
Table 108: Overall results for GTOPX constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 10% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
GEPLT
FCM
62

---

Table 109: Overall results for GTOPX constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 20% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
20 HEMTRN JEMT
Table 110: Overall results for GTOPX constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 25% of the values are missing near the worst value and another 25% near the optimal value. Details are the same as Table 5.
Table 111: Overall results for GTOPX constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 30% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
Table 112: Overall results for GTOPX constrained tasks with 128 solutions and 50th percentile evaluations. In this case, 40% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
111mm
01920
100
40
4000
40T
Table 113: Overall results for GTOPX constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 50% near the optimal value. Details are the same as Table 5.
OVE11
#110
T4k
63

---

Table 11:: Overall results for GTOPX constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
DEVET
2016
Table 115: Overall results for GTOPX constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Table 116: Overall results for GTOPX constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
766 Ruck
20 2016
1945
Table 117: Overall results for GTOPX constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 0% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
14
DEEET
Table 118: Overall results for GTOPX constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 10% of the values are missing near the worst value and another 40% near the optimal value. Details are the same as Table 5.
PS16
3th
Other
DEEST
64

---

Table 119: Overall results for GTOPX constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 20% of the values are missing near the worst value and another 30% near the optimal value. Details are the same as Table 5.
Table 120: Overall results for GTOPX constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 25% of the values are missing near the worst value and another 25% near the optimal value. Details are the same Table 5.
9M
Table 121: Overall results for GTOPX constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 30% of the values are missing near the worst value and another 20% near the optimal value. Details are the same as Table 5.
43
1944
Table 122: Overall results for GTOPX constrained tasks with 128 solutions and Oth percentile evaluations. In this case, 40% of the values are missing near the worst value and another 10% near the optimal value. Details are the same as Table 5.
100
65


Reviews:
[{"review_id": "3ZPp4gvYfh", "reviewer": "Reviewer_x8NF", "summary": "This paper introduces SOO-Bench, a new benchmark for offline black-box (BB) model-based optimization (MBO) algorithms, which allows for evaluating the algorithm stability, i.e., the ability of an algorithm to consistently and continuously improve upon existing solutions from the offline dataset. The authors frame stability as one of the core objectives of offline BB-MBO. Offline here stands for the fact that the optimization has to rely only on historical data, without having access to evaluating new solution candidates; an important property of the offline data is usually a narrow distribution that is exhibited, possibly because of the bias induced in the data collection, such that the search space is not equally covered. This is at the same time a big challenge for the offline BB-MBO algorithms. SOO-Bench allows for customizing the narrowness of the offline data distribution and proposes a stability indicator (SI) to quantify how stable an algorithm is. The benchmark consists of various real-world tasks and datasets, including a possibility to vary the difficulty of the dataset by aforementioned customization. A range of algorithms (for unconstrained and constrained optimization) are empirically evaluated on the proposed benchmark, giving insights into their stability measures.\n\nEDIT: I raise my score in light of the rebuttal.", "questions": "Could the authors clarify what is meant by ‘a guided training agent model’ which assists classic baselines (I assume BO and CMA-ES)?\nHow are the algorithms that are benchmarked selected?\nFinally, correcting for the OOD data is widely used in machine learning. It would be interesting to see how the value of the stability indicator of an algorithm would change after incorporating some of these techniques (e.g., for covariate shift) into the algorithm.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The presentation of the paper is clear, with clearly stated contributions. The proposed stability indicator is a novel and useful measure for offline BB-MBO. A wide range of structurally and behaviorally different BB-MBO algorithms is evaluated, showing that they greatly vary in terms of stability.", "weaknesses": "The authors claim multiple times that black-box optimization requires building a surrogate model from the available data (samples and their evaluations). This is not always true, as black-box optimization is very often addressed by model-free methods, which are typically iterative randomized optimization heuristics (i.e., population-based algorithms), usually stemming from the evolutionary computation field. CMA-ES is one such example: an evolution strategy that does not learn a surrogate per se, but adapts the probability distribution by means of updating the covariance matrix. It may be helpful to clarify early in the paper that you are focusing specifically (or mostly) on model-based approaches, while explaining the inherent differences of the algorithms you are considering. Furthemore, the details of the evaluated algorithms could be made clearer; related to this, it is not clear how the algorithms are selected in the first place. It would be good to include a brief description of each evaluated algorithm, perhaps in an appendix, and explain their criteria for algorithm selection. This would help readers better understand the scope and rationale of the benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0AApMa6nVJ", "reviewer": "Reviewer_Lkg5", "summary": "This paper proposes a new benchmark suite for offline optimization. The two key contributions are (1) the ability to adjust the left-out portion (top m% and bottom n%) in the construction of the offline dataset; and (2) a new evaluation metric called SI that measures the stability of the recommendations. The paper shows that existing methods have varying performances when adjusting the dataset difficulty via controlling the parameters m and n. Several experiments also reveal the unstable behaviors of previous methods. Finally, compared to the Design-Bench (which is the default benchmark for offline opt until recently), the proposed benchmark also offers several new constrained optimization tasks.", "questions": "Please see above", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper points out some clear weaknesses of the Design-Bench and proposes practical solutions to fix them. Empirical results point out some interesting insights and fill in missing gaps in current offline opt research. The paper is also clearly written and well-motivated.", "weaknesses": "- I get the point of the SI metric, but don't quite agree with its formulation. Other than the thresholding with respect to the best offline value, how is it fundamentally different from computing variance? The way I see it, even if we incorporate the best offline value into the formulation, the SI metric alone is not enough to reflect performance. For example, let's say the best offline value is 0, a recommendation of 0.1, 0.1, 0.1 .... 0.1 will obtain better SI score than -0.1, 10, 10, 10, 10 .... A better way might be to treat the list of recommendations as a distribution, then perform hypothesis testing to check if the mean of this distribution is greater than the best offline value. The p-value would then indicate the performance level. It doesn't seem like there is a need to reinvent the wheel here.\n\n-  One would expect higher difficulty to be achieved by leaving out high values in the offline dataset. This seems to be the case with the FS metric (table 5 in appendix). In the Design-Bench paper I think the bottom 30% of the entire dataset was selected as the offline data, which is likely harder than most of the settings presented. Even at this difficulty, the baseline methods are already quite competitive -- each new paper will report a different set of result (typically indicating that it is best). Personally, I think the focus should be on making the benchmark more challenging without severing limit the size of the offline dataset.\n\n- Additionally, the fluctuating performance of baseline methods suggest that randomness plays a huge factor in offline opt. I would be more interested in designing benchmarks to eliminate this factor.\n \n- Finally, can the authors elaborate on the technical challenge behind creating this baseline? To me, both the new metric and the customizable dataset construction seem like straight-forward wrappers that can be added to the Design-Bench, without remaking it from scratch.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3u4KU6O19q", "reviewer": "Reviewer_9Mjy", "summary": "The authors present an innovative benchmark for evaluating the stability of offline data-driven optimization problems. They employ a customized data distribution approach that removes points near optimal and suboptimal values, and introduce a Stability Improvement (SI) indicator to measure stability. Their evaluation compares various offline optimization algorithms — six for unconstrained problems, plus four for constrained ones. The results reveal diverse performance levels across the benchmark's instances, showcasing its effectiveness in assessing stability.", "questions": "- There appear to be some typos, specifically \"TTDEEA\" in Table 2 and Section 6.2, and \"TTDEA\" in Appendix C. \n\n- For the new SI indicator, how should we best assess an algorithm's performance? In which cases would a lower FS be advantageous, and when should we prioritize a higher SI?\n\n- I also have a question regarding the paper’s submission history. I was one of the reviewers for its NeurIPS submission this year, where I saw that all reviewers ultimately gave positive scores. However, I was surprised to see that the authors decided to withdraw the paper. Could you share the reason for this decision? Additionally, I noticed that some specific revisions the authors intended to make during the NeurIPS rebuttal are not reflected in the current ICLR submission.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper provides key insights into the stability of offline optimization methods, advancing this emerging field.\n- It is clearly written and well-structured.\n- The related work section is thorough, and the introduction effectively contextualizes the research.\n- The benchmark is detailed, with clear explanations of its tasks and datasets.\n- Open-source availability enhances interpretability and reproducibility, backed by comprehensive experiments.", "weaknesses": "Please refer to the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m5jc4uhraj", "reviewer": "Reviewer_8L7a", "summary": "This paper proposes SOO-Bench, a benchmark suite to promote stable offline black-box optimization. Authors integrate several real offline optimization tasks and provide offline optimization tasks with different customizable difficulties by narrowing the dataset distribution. Besides, authors introduce an indicator called Stability Improvement (SI) to quantify the stability of the algorithms and conduct experiments to evaluate algorithmic stability and optimality.", "questions": "See weaknesses part.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper is claimed to be the first work to disclose stability as one of the core objectives in offline black-box optimization.\n\n2. The paper integrates multiple real-world tasks with a unified API.\n\n3. In the experiments, authors conduct comprehensive evaluation on state-of-the-art baselines and provide thoughful insights.", "weaknesses": "1. Narrowing data distribution by removing top and bottom solutions cannot address the uneven cover in real-world task dataset. As mentioned in the paper, personal biases, preferences for certain parameter settings, or prior knowledge might lead to uneven sampling in solution space, not the fitness space. Removing top and bottom solutions cannot reflect such situations since the solutions in the middle may still evenly cover the solution space, and the solutions sampled unevenly may also cover both top and bottom solutions. Therefore, narrowing data distribution by removing top and bottom solutions cannot address the uneven cover in real-world task dataset, perhaps narrowing the distribution in solution space could be better.\n\n2. The Stability Improvement indicator proposed in this paper is imperfect in measuring the algorithm stability. According to Equation (2), the way to obtain highest SI is to achieve a solution better than the best solution in the offline dataset and at the first step and retain it till the end. In this case the SI can be 1, but such precocious behavior can lead to poor performance (i.e., COMMs in the experiments has high SI and poor performance). If we consider SI as an indicator only about stability and has nothing to do with performance, another case where the performance keeps increasing and achieves the best performance at the last step (i.e., Algorithm A's curve before achieving objective value a in Figure 2(b)), its SI can be lower than some unstable algorithms. However, we know that in this case, it is a stable algorithm with stable performance improvement along the optimization. In a word, the proposed SI cannot reflect the algorithm stability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)