SOO-Bench: Benchmarks for Evaluating the Stability of Offline Black-Box Optimization

Authors: Hong Qian, Yiyi Zhu, Xiang Shu, Shuo Liu, Yaolin Wen, Xin An, Huakang Lu, Aimin Zhou, Ke Tang, Yang Yu
Primary area: datasets and benchmarks
Keywords: Offline Optimization, Black-Box Optimization, Stability, Benchmarks

Abstract:
Black-box optimization aims to find the optima through building a model close to the black-box objective function based on function value evaluation. However, in many real-world tasks, such as the design of molecular formulas and mechanical structures, it is perilous, costly, or even infeasible to evaluate the objective function value of an actively sampled solution. In this situation, optimization can only be conducted via utilizing offline historical data, which yields offline black-box optimization. Different from the traditional goal that is to pursue the optimal solution, this paper emphasizes that the goal of offline optimization is to stably surpass the offline dataset during optimization procedure. Although benchmarks called Design-Bench already exist in this emerging field, it can hardly evaluate the stability of offline optimization and mainly provides real-world offline tasks and the corresponding offline datasets. To this end, this paper proposes benchmarks named SOO-Bench (i.e., Stable Offline Optimization Benchmarks) for offline black-box optimization algorithms, so as to systematically evaluate the stability of surpassing the offline dataset under different data distributions. Along with SOO-Bench, we also propose a stability indicator to measure the degree of stability. Specifically, SOO-Bench includes various real-world offline optimization tasks and offline datasets under different data distributions, involving the fields of satellites, materials science, structural mechanics, and automobile manufacturing. Empirically, baseline and state-of-the-art algorithms are tested and analyzed on SOO-Bench. Hopefully, SOO-Bench is expected to serve as a catalyst for the rapid developments of more novel and stable offline optimization methods. The code is available at \url{https://github.com/zhuyiyi-123/SOO-Bench}.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 5

Reviews and discussion:
<thread>
creation date:
1730208165804

thread_type:
official_review

<thread_object>
creator: reviewer_8l7a

<content>
summary:
This paper proposes SOO-Bench, a benchmark suite to promote stable offline black-box optimization. Authors integrate several real offline optimization tasks and provide offline optimization tasks with different customizable difficulties by narrowing the dataset distribution. Besides, authors introduce an indicator called Stability Improvement (SI) to quantify the stability of the algorithms and conduct experiments to evaluate algorithmic stability and optimality.

strengths:
1. This paper is claimed to be the first work to disclose stability as one of the core objectives in offline black-box optimization.

2. The paper integrates multiple real-world tasks with a unified API.

3. In the experiments, authors conduct comprehensive evaluation on state-of-the-art baselines and provide thoughful insights.

weaknesses:
1. Narrowing data distribution by removing top and bottom solutions cannot address the uneven cover in real-world task dataset. As mentioned in the paper, personal biases, preferences for certain parameter settings, or prior knowledge might lead to uneven sampling in solution space, not the fitness space. Removing top and bottom solutions cannot reflect such situations since the solutions in the middle may still evenly cover the solution space, and the solutions sampled unevenly may also cover both top and bottom solutions. Therefore, narrowing data distribution by removing top and bottom solutions cannot address the uneven cover in real-world task dataset, perhaps narrowing the distribution in solution space could be better.

2. The Stability Improvement indicator proposed in this paper is imperfect in measuring the algorithm stability. According to Equation (2), the way to obtain highest SI is to achieve a solution better than the best solution in the offline dataset and at the first step and retain it till the end. In this case the SI can be 1, but such precocious behavior can lead to poor performance (i.e., COMMs in the experiments has high SI and poor performance). If we consider SI as an indicator only about stability and has nothing to do with performance, another case where the performance keeps increasing and achieves the best performance at the last step (i.e., Algorithm A's curve before achieving objective value a in Figure 2(b)), its SI can be lower than some unstable algorithms. However, we know that in this case, it is a stable algorithm with stable performance improvement along the optimization. In a word, the proposed SI cannot reflect the algorithm stability.

questions:
See weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730345307526

thread_type:
official_review

<thread_object>
creator: reviewer_9mjy

<content>
summary:
The authors present an innovative benchmark for evaluating the stability of offline data-driven optimization problems. They employ a customized data distribution approach that removes points near optimal and suboptimal values, and introduce a Stability Improvement (SI) indicator to measure stability. Their evaluation compares various offline optimization algorithms — six for unconstrained problems, plus four for constrained ones. The results reveal diverse performance levels across the benchmark's instances, showcasing its effectiveness in assessing stability.

strengths:
- The paper provides key insights into the stability of offline optimization methods, advancing this emerging field.
- It is clearly written and well-structured.
- The related work section is thorough, and the introduction effectively contextualizes the research.
- The benchmark is detailed, with clear explanations of its tasks and datasets.
- Open-source availability enhances interpretability and reproducibility, backed by comprehensive experiments.

weaknesses:
Please refer to the Questions section.

questions:
- There appear to be some typos, specifically "TTDEEA" in Table 2 and Section 6.2, and "TTDEA" in Appendix C. 

- For the new SI indicator, how should we best assess an algorithm's performance? In which cases would a lower FS be advantageous, and when should we prioritize a higher SI?

- I also have a question regarding the paper’s submission history. I was one of the reviewers for its NeurIPS submission this year, where I saw that all reviewers ultimately gave positive scores. However, I was surprised to see that the authors decided to withdraw the paper. Could you share the reason for this decision? Additionally, I noticed that some specific revisions the authors intended to make during the NeurIPS rebuttal are not reflected in the current ICLR submission.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730672881260

thread_type:
official_review

<thread_object>
creator: reviewer_lkg5

<content>
summary:
This paper proposes a new benchmark suite for offline optimization. The two key contributions are (1) the ability to adjust the left-out portion (top m% and bottom n%) in the construction of the offline dataset; and (2) a new evaluation metric called SI that measures the stability of the recommendations. The paper shows that existing methods have varying performances when adjusting the dataset difficulty via controlling the parameters m and n. Several experiments also reveal the unstable behaviors of previous methods. Finally, compared to the Design-Bench (which is the default benchmark for offline opt until recently), the proposed benchmark also offers several new constrained optimization tasks.

strengths:
The paper points out some clear weaknesses of the Design-Bench and proposes practical solutions to fix them. Empirical results point out some interesting insights and fill in missing gaps in current offline opt research. The paper is also clearly written and well-motivated.

weaknesses:
- I get the point of the SI metric, but don't quite agree with its formulation. Other than the thresholding with respect to the best offline value, how is it fundamentally different from computing variance? The way I see it, even if we incorporate the best offline value into the formulation, the SI metric alone is not enough to reflect performance. For example, let's say the best offline value is 0, a recommendation of 0.1, 0.1, 0.1 .... 0.1 will obtain better SI score than -0.1, 10, 10, 10, 10 .... A better way might be to treat the list of recommendations as a distribution, then perform hypothesis testing to check if the mean of this distribution is greater than the best offline value. The p-value would then indicate the performance level. It doesn't seem like there is a need to reinvent the wheel here.

-  One would expect higher difficulty to be achieved by leaving out high values in the offline dataset. This seems to be the case with the FS metric (table 5 in appendix). In the Design-Bench paper I think the bottom 30% of the entire dataset was selected as the offline data, which is likely harder than most of the settings presented. Even at this difficulty, the baseline methods are already quite competitive -- each new paper will report a different set of result (typically indicating that it is best). Personally, I think the focus should be on making the benchmark more challenging without severing limit the size of the offline dataset.

- Additionally, the fluctuating performance of baseline methods suggest that randomness plays a huge factor in offline opt. I would be more interested in designing benchmarks to eliminate this factor.
 
- Finally, can the authors elaborate on the technical challenge behind creating this baseline? To me, both the new metric and the customizable dataset construction seem like straight-forward wrappers that can be added to the Design-Bench, without remaking it from scratch.

questions:
Please see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1731158928603

thread_type:
official_review

<thread_object>
creator: reviewer_x8nf

<content>
summary:
This paper introduces SOO-Bench, a new benchmark for offline black-box (BB) model-based optimization (MBO) algorithms, which allows for evaluating the algorithm stability, i.e., the ability of an algorithm to consistently and continuously improve upon existing solutions from the offline dataset. The authors frame stability as one of the core objectives of offline BB-MBO. Offline here stands for the fact that the optimization has to rely only on historical data, without having access to evaluating new solution candidates; an important property of the offline data is usually a narrow distribution that is exhibited, possibly because of the bias induced in the data collection, such that the search space is not equally covered. This is at the same time a big challenge for the offline BB-MBO algorithms. SOO-Bench allows for customizing the narrowness of the offline data distribution and proposes a stability indicator (SI) to quantify how stable an algorithm is. The benchmark consists of various real-world tasks and datasets, including a possibility to vary the difficulty of the dataset by aforementioned customization. A range of algorithms (for unconstrained and constrained optimization) are empirically evaluated on the proposed benchmark, giving insights into their stability measures.

EDIT: I raise my score in light of the rebuttal.

strengths:
The presentation of the paper is clear, with clearly stated contributions. The proposed stability indicator is a novel and useful measure for offline BB-MBO. A wide range of structurally and behaviorally different BB-MBO algorithms is evaluated, showing that they greatly vary in terms of stability.

weaknesses:
The authors claim multiple times that black-box optimization requires building a surrogate model from the available data (samples and their evaluations). This is not always true, as black-box optimization is very often addressed by model-free methods, which are typically iterative randomized optimization heuristics (i.e., population-based algorithms), usually stemming from the evolutionary computation field. CMA-ES is one such example: an evolution strategy that does not learn a surrogate per se, but adapts the probability distribution by means of updating the covariance matrix. It may be helpful to clarify early in the paper that you are focusing specifically (or mostly) on model-based approaches, while explaining the inherent differences of the algorithms you are considering. Furthemore, the details of the evaluated algorithms could be made clearer; related to this, it is not clear how the algorithms are selected in the first place. It would be good to include a brief description of each evaluated algorithm, perhaps in an appendix, and explain their criteria for algorithm selection. This would help readers better understand the scope and rationale of the benchmark.

questions:
Could the authors clarify what is meant by ‘a guided training agent model’ which assists classic baselines (I assume BO and CMA-ES)?
How are the algorithms that are benchmarked selected?
Finally, correcting for the OOD data is widely used in machine learning. It would be interesting to see how the value of the stability indicator of an algorithm would change after incorporating some of these techniques (e.g., for covariate shift) into the algorithm.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731911515426

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers and Revision Submitted.

comment:
We would like to thank the reviewers for their helpful comments. We are encouraged they find our idea to be novel (**Reviewer x8NF**, **Reviewer 9Mjy**, **Reviewer 8L7a**), the work well-motivated (**Reviewer Lkg5**), and the contribution clear and meaningful for ML (**Reviewer x8NF**, **Reviewer Lkg5**, **Reviewer 9Mjy**). We are glad they point out it is a easy-to-use framework (**Reviewer 9Mjy**, **Reviewer 8L7a**), the new indicator called Stability Improvement (SI) is novel and interest (**Reviewer x8NF**). Furthermore, there is consensus among the reviewers about the well-structured nature of our paper. In light of the reviewers' comments, we have revised our manuscript, supplementary material as detailed below.

We have carefully considered all of the feedback provided, and in response, we have made the following revisions to our manuscript to address the reviewers' concerns and suggestions:

- Enhance the description of the evaluated algorithms (Appendix B.1).
- Add different setting of our benchmark, including different task sizes and diverse distributions within the solution space (Appendix D).
- Evaluate new offline optimization algorithm CCDDEA on our benchmark (Section 6.1, Table 2, 5, 9, Appendix B.1).
- Fix minor issues on grammar and typos.

All changes have been highlighted in blue in the manuscripts.

**The definition of offline optimization:** Offline optimization aims to make full use of the offline data and finally recommend potential high-quality solutions to be applied online without actively querying the solutions online. Due to the exploration nature of global optimization, optimizers may tend to try unseen out-of distribution solutions for potentially better performance (i.e., find much better solutions than the offline dataset). As the optimization procedure proceeds, the quality of solutions may tend to decline. It is difficult to determine when to stop the optimization procedure and return the final solution without active evaluation feedback. Previous work has suggested that by specifying the number of stopping steps for optimization, good results can be achieved by adjusting the step size or modifying the model for experiments. However, this is often not feasible in real life. Therefore, we hope to maintain a steady improvement throughout the optimization process, or at least maintain stability so that the solutions at any step will not be too bad.

We want to restate our contribution:

1. **Customization:** SOO-bench allows users to extensively customize data distributions, constraints, and other parameters, providing unparalleled flexibility and control for thorough experimental analysis.
2. **Novel Metric - Stability Improvement (SI):** We introduce SI to assess algorithm stability during optimization. This metric is vital as the timing for applying optimization results online is uncertain, requiring algorithms to be stable throughout the process.
3. **Diverse Tasks:** Our benchmark includes a wide range of tasks, from simple to complex and from unconstrained to constrained, catering to various research needs.

If any of the reviewers have any further questions, we would be pleased to answer them.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper considers the problem of offline model based optimization and provides new benchmarks and stability metric to evaluate the algorithms. Offline optimization is an active and important area of research, so this work could be potential useful to the community. However, I have following concerns shared by two other reviewers.  

- I found the paper's contribution to be limited. Reviewer Lkg5 pointed this out as well and authors replied one contribution as **in-depth thinking on the essential goals of offline optimization (i.e., the goal of offline optimization is to stably surpass the offline dataset during optimization procedure)**. I believe this is a basic requirement for this problem setting well known to all of us working on this problem and cannot be highlighted as a new contribution. 

- One huge challenge with previous benchmarks in offline optimization (primarily design-bench and another multiobjective benchmark released recently)  has been the difficulty of installing and setting them up because of mismatched dependencies and code errors. There are multiple open issues on design-bench page describing this (https://github.com/brandontrabucco/design-bench/issues). Unfortunately,  looking at the requirements.txt and README description of the code provided with the paper doesn't seem to give much confidence about this aspect. A benchmark is most beneficial to the community if they can use this easily. This is the most important aspect authors can improve on.