AutoOED: Automated Optimal Experimental Design Platform with Data- and Time-Efficient Multi-Objective Optimization
Authors: Yunsheng Tian
Mina Konakovic Lukovic
Michael Foshey
Timothy Erps
Beichen Li
Wojciech Matusik
Venue: ICLR 2022 Submitted (2022)
Keywords: optimal experiment design
bayesian optimization
multi-objective optimization
software platform

Abstract: We present AutoOED, an Automated Optimal Experimental Design platform powered by machine learning to accelerate discovering solutions with optimal objective trade-offs. To solve expensive multi-objective problems in a data-efficient manner, we implement popular multi-objective Bayesian optimization (MOBO) algorithms with state-of-the-art performance in a modular framework. To further accelerate the optimization in a time-efficient manner, we propose a novel strategy called Believer-Penalizer (BP), which allows batch experiments to be accelerated asynchronously without affecting performance. AutoOED serves as a testbed for machine learning researchers to quickly develop and evaluate their own MOBO algorithms. We also provide a graphical user interface (GUI) for users with little or no experience with coding, machine learning, or optimization to visualize and guide the experiment design intuitively. Finally, we demonstrate that AutoOED can control and guide real-world hardware experiments in a fully automated way without human intervention.

Under review as a conference paper at ICLR 2022
AUTOOED: AUTOMATED OPTIMAL EXPERIMEN- TAL DESIGN PLATFORM WITH DATA- AND TIME- EFFICIENT MULTI-OBJECTIVE OPTIMIZATION
Anonymous authors Paper under doublc-blind review
ABSTRACT
We present AutoOED, an Automated Optimal Experimental Design platform powered by machine learning to accelerate discovering solutions with optimal ob- jective trade-offs. To solve expensive multi-objective problems in data-efficient manner, we implement popular multi-objective Bayesian optimization (MOBO) algorithms with state-of-the-ar performance in modular framework. To further accelerate the optimization a time-efficient manner, we propose novel strategy called Believer-Penallee Penalizer (BP), which allows batch experiments to be accelerated asynchronously without affecting performance. AutoOED serves as testbed for machine learning researchers quickly develop and evaluate their own MOBO algorithms. We also provide graphical user interface (GUI) for users with little no experience with coding, machine learning, or optimization visualize and guide the experiment design intuitively. Finally, we demonstrate that AutoOED can control and guide real-world hardware experiments in fully automated way without human intervention.
INTRODUCTION
Optimal Experimental Design (OED) problems in science and engineering often require satisfying several conflicting objectives simultaneously. These problems aim solve a multi-objective opti- mization system and discover a set of optimal solutions, called Pareto optimal. Furthermore, the objectives are typically black-box functions whose evaluations are time-consuming and costly (e.g.. measuring real experiments or running expensive numerical simulations). Thus, the budget that de- termines the number of experiments can be heavily constrained. Hence, an efficient strategy for guiding the experimental design towards Pareto optimal solutions is necessary. Recent advances in machine learning have facilitated optimization of various design problems, including chemical design (Griffiths & Hernández-LLoatte 2017), material design (Zhang al.) 2020). resource al-
location (Wu et al.) 2013), environmental monitoring (Marchant & Ramos 2012). recommender systems (Chapelle & Li (011) and robotics (Martinez-Cantin al. 2009). machine learning
concept that enables automatic guidance of the design process is Bayesian optimization (Shahriari al., 2016). This concept is extensively studied in the machine learning community from a theo- retical aspect and in the single-objective case. However, its practical applications in multi multi-objective problems are still not widely explored due to the lack of easy-to-use and open- software.
In this paper, we present AutoOEn an open-source platform for efficiently optimizing multi- objective problems with a restricted budget of experiments. The key features of AutoOED include:
Data-efficient experimentation: AutoOHD employs state-of-the-ard MOBO strategies that rapidly advances the Pareto front with small set of evaluated experiments.
Time-efficient experimentation: AutoOED supports both synchronous and asynchronous batch optimization to accelerate the optimization. We propose novel and robust asyn- chronous optimization strategy named multiple workers experiments in parallel, but their evaluations drastically vary in time.
(BP), which is instrumental when
!Code, screenshots, detailed documentation and tutorials can be found at httpa://at teaittes.goog..
com/ viiew/autccedd

---

Under review as a conference paper at ICLR 2022
An easy-to-use graphical user interface (GUT) is provided to directly visu- alize and guide the optimization progress and facilitate the operation for users with little or experience with coding, optimization, or machine learning.
Intuitive GUI:
Modular structure: A highly modular Python codebase enables easy extensions and re- placements of MOBO algorithm components, AutoOED can serve as a testbed for machine learning researchers to easily develop and evaluate their own MOBO algorithms.
Automation of experimental design: The platform is designed for straightforward in- tegration into fully automated experimental design pipeline long the experiment evaluations (either in simulation or physical) can be controlled via computer programs.
2 RELATED WORK
Bayesian optimal experimental design Optimal experimental design (OED) is the process of de- signing the sequence of experiments to maximize specific objectives in a data- or time-efficient man- ner. Therefore, Bayesian optimization (BO) (Shahriari et al., 2016)) is usually applied to find optimal solutions with a minimal number of evaluations. Essentially, BO relies on surrogate modely like the Gaussian process (GP) to accurately model the experimental process and proposes new experimental designs based on defined acquisition functions that trade-off between exploration and exploitation. Popular choices of the acquisition functions include Expected Improvement (EI) (Močkus [975),
Upper Confidence Bound (UCB) (Srinivas et al., 2010), Thompson Sampling (TS) (Thompson 1933). Bayesian OED has found success wide range of applications (Greenhill et 20200) and
the main methodology of AutoOED. To further speed up when evaluations can be carried out in parallel, asynchronous BO approaches have been developed (Ginsbourger et al. 2010 Kandasamy et al. 2018 Alvi et al., 2019). However, all of the previous literature focuses single-objective BO rather than the multi-objective scenario. In this paper, we extend several single-objective asyn- chronous BO methods to multi-objective versions and propose a novel asynchronous method named Believer-Penalizer (BP) with the stablest performance on multi-objective benchmark problems.
Multi-objective Bayesian optimization MOBO is developed to optimize for a set of Pareto optimal solutions while minimizing the number of experimental evaluations. Early approaches solve multi-objective problems by scalarizing them into single-objective ones using random weights (Knowles (006)). Instead of scalarization, some acquisition functions are proposed to com- pute a single objective, e.g., entropy-based or hypervolume-basce lakaria et al.] 2019 Emmerich & Klinkenberg 2008) Daulton et al. 2020a). Alternatively. MOBO can be solved by defining a separate acquisition function per objective, optimizing using cheap multi-objective solvers (usually evolutionary algorithms like NSGA-II (Deb et al. (002)) and finally selecting one or a batch of designs to evaluate next (Bradford et al. 2018 Belakaria & Deshwal
(Russo & Van Roy] 2014 Be-
2020 Konakovic Lukovic et al. 2020). AutoOED implements many of them in a modular way and allows easily changing modules an unified MOBO framework (see Section 3.2)
Open-source Bayesian optimization platform There many existing Python libraries for
Bayesian optimization including Spearmint (Snoek et al. (012), HyperOpt (Bergstra et al. 2013
GPyOpt (authors) 2016). GPflowOpt (Knudde et al. 2017). Dragonfly (Kandasamy et al.
2020
AX (Bakshy et al.] 2018). Optuna (Akiba et al." 2019), HyperMapper (Nardi et al,
2019
BoTorch (Balandat et al. 2020a). SMAC3 (Lindauer et al. 202T and OpenBox (Li et al. 2021
These Python libraries are designed for general applications and have different algorithmic features supported. The feature comparison between AutoOED and these libraries shown in Table and is further discussed in Section 5.2 However, they are all targeted for experts in coding without an intuitive GUI. In contrast, there are also software platforms that provide intuitive user interface and visualization to specific domain experts but the platforms cannot be used for other general applica- tions, for example, Auto- OCto-QChemm (Shields et al.] (021)) for chemical synthesis and GeoBO (Haan 2 for geoscience. Combining powerful Bayesian optimization algorithms and an intuitive GUT, AutoOED is designed to be a general optimization platform that can be easily used by anyone for applications in any field.
The comparison is based on AutoOED's core features. "~" means the package only supports single multi- objective algorithm rather than modular multi-objective framework with several state-of-the-art algorithms.
2

---

Under review as a conference paper at ICLR 2022
Table I: Feature comparison between different Bayesian optimization platforms
Multiple objectives
Multiple domains
Asynchronous optimization
External evaluation
Modular framework
Built-in visualization
Name
GUI
Spearmint
GPyOpt
GPtlowOpt
Dragonfly
BoTorch
AutoOED
DATA-EFFICIENT MULTI--ORJECTIVE OPTIMIZATION
3.1 PROBLEM FORMULATION
Optimal experiment design problems involving mul- tiple conflicting objectives can be formulated as a multi-objective optimization design parameters as data- and time-efficient as possible. More for- mally, we consider optimization problem over a set of design variables X' C. IRd, called design space. The goal is to simultaneously minimize m 2 ob- jective functions fi....fmm
Pareto Set
Pareto Front
R
F(X)
X
Design Space
Performance Space
X - R. Representing the vector of all objectives as f(x) (fi (x) fm. (x)). the performance spuce then an m-dimeeniooal image f(X) C Rm. Conflict- ing objectives result in set of optimal solutions rather than single best solution. These optimal solutions are referred to as Pareto set P, C X' in the design space, and the corresponding images in performance space are Pareto front P, f(P.) R.
To measure the quality of an approximated Pareto front, hypervolume (Zitzler & Thiele 1999) is the most commonly used metric in multi-objective optimization (Riquelme al. 2015). Let Py be a Pareto front approximation an m-dimensional performance space and given a reference point
RT the hypervolume H(P,) ) is defined H(P,) J 1 H where (P,)) {2 P)))).
the relation operator of objective dominance and H(P)) Dirac delta function that equals 1 if H(Pr) and 0 otherwise. The higher the hypervolume, the better Py approximates the true Pareto front.
Z 31 < i 1Pyl:
3.2 MODULAR ALGORITHM FRAMEWORK
Evaluate proposed designs
Surrogate model
Acquisition function
Multi-objective solver
Selection
Expected Improvement Upper Confidence Bound
NSGA-II MOEA/D
Hypervolume Improvement Uncertainty
Noural Network
fr
Obarrorians
Uncortirry
Angrastton valoe
Observaioos Approximate Pareto set and front
Prento fromt
Selected
Fit surrogate models for each objective evalusted data
Define f that trudes off mean and uncertuinty of surrogates
Propose hatch of designs to evaluate next
Figure 1: Algorithmic pipeline and modules of multi-objective Bayesian optimization.
3

---

Under review as a conference paper at ICLR 2022
Multi-objective Bayesian optimization (MOBO) is a data-driven approach that attempts to learn the black-box -ox objective functions f(x) from available data and find Pareto optimal solutions in an iter- ative and data-efficient manner. MOBO typically consists of four core modules: (f) an inexpensive surrogate model for the black-box objective functions; (ii) an acquisition function that defines sam- pling from the surrogate model and trade-off between exploration and exploitation of the design space; (iii) a cheap multi-objective solver to approximate the Pareto set and front; (iv) a selection strategy that proposes single or batch of experiments evaluate next. These four modules (see Figure 1 are implemented as core and independent building blocks of the AutoOEDs, making it highly modularized and easy to develop new algorithms and modules. The whole pipeline starts from given small dataset or a set of random evaluated samples, then it works iteratively by propos- ing new design samples and evaluating them until the stopping criterion is met.
For each module in this framework, AutoOED supports following choices:
Surrogate model:
Gaussian process, neural network (multi-layer perceptron), Bayesian neural network (DNGO (Snoek al., 2015)
Acquisition function: Expected Improvement, Probability of Improvement, Upper Confi- dence Bound, Thompson Sampling, identity function
Multi-ohjective solver: NSGA-II, MOEA/D, ParetoFrootDiscovery (Schulz et al. 2018
Selection: Hypervolume improvement, uncertainty, random, etc.
Stopping criterion: Time, number of evaluations, hypervolume convergence
class 18ERO (N080)
class US2MD_K1 81 (M0801
class DGEMO (MD201
[Bradford et a1, 2018]
[Bllakaiia and Deahwal 2020]
[Lukovic et al, 2020] spec "urzogale''s "gp", "aqqusitionn' 'identicy', 'salves's 'diszovary', "selection"s "direct',,
spec
apec
"suzrogale": "gp", aaquisition' "solver" "naga2', "salectiont" "hvi',
surcogate's 'gp', acquisition's 'ei', solver': : "nsga2', sellcction *uncertainty',
Code Example 1: Creating algorithms in AutoOED by simply specifying module combinations.
Based on this framework, we implement several popular and state-of the-art art MOBO methods, in- cluding ParEGO (Knowles] (006)) MOEA/D EGO (Zhang et al.] 2209), TSEMO (Bradford et al.) 2018). USeMO (Belakaria & Deshwal 2020). DGEMO (Konakovic Lukovic et al. 2020). DGEMO exhibits state-of-the-ar performance for data-efficient, multi-objective problems with batch evalua- tions to the best of our knowledge. With necessary modules of the MOBO framework implemented, the algorithms can be easily composed by specifying the choice of each module and inheriting the base class MOBO, see Code Example [] Supported choices of each module can be found in our documentation. Users can select an algorithm from this library that best fits the characteristics of their physical system or optimization goals, or they can easily create new algorithms by specifying novel combinations of existing modules in just a few lines of code.
TIME-EFFICIENT MULTI-OBJECTIVE OPTIMIZATION
4.1 BATCH OPTIMIZATION
While standard MOBO optimizes for the Pareto front in a data-efficient manner, often, when mul- tiple experiment setups are available, evaluations can be executed in batch by parallel workers to further speed up the whole optimization process. To leverage this speed-up. all the algorithms in AutoOED are implemented to support batch evaluation.
However, if parallel workers evaluate in different speeds, some workers are left idle when they finish evaluations earlier than others. Therefore, asynchronous optimization is desired to maximize the utilization of workers and is able evaluate many more designs than synchronous optimization in a fixed amount of time, as also illustrated by Kandasamyy et al.] (2018) and Alvi et al.] (2019). Nevertheless, while some designs are being evaluated (i.e., busy designs), how propose the next
4

---

Under review as a conference paper at ICLR 2022
design that (i) avoids being similar to the busy designs and (ii) incorporates knowledge from busy designs to reach better regions in the performance space is the key question that we want to explore.
To develop efficient asynchronous strategy for multii-bjeective optimization, borrow ideas from previous literature in the single-objective setting.
f2
A
Penalizing
Belleving
>
<
M
Lewer hound
Postesioe std
Postenor
design Evallated desigs
Busy design
Current Pureto front
Proposed design
Hypervolume improvement
(a) The failure case of KB when believing overestimated busy designs.
(b) The sub-optimality of LP in multi-objective scenario when be- lieving busy designs affects the selection result.
Figure Analysis of KB and LP strategies for asynchronous optimization.
4.2 FAILURES OF EXISTING STRATEGIES
Kriging Believer (KB) Ginsbourggrr et al. 2010) is simple yet effective approach that believes the performance of busy designs is their posterior mean of the surrogate model when optimizing for new designs. In other words, treats the mean prediction of the busy designs as their real performance and eliminates their posterior variance to prevent acquisition functions from preferring those regions. However, failure case happens when it believes an overestimated design, it might become difficult to find designs better than this overestimated one and make further improvement, see Figure 2aj Especially, when the posterior mean of the busy design is extremely small and even exceeds the lower bound of the objective, subsequent optimization can hardly find better solution. In other words, subsequent optimization will only propose more overestimated designs with even lower predicted performance to "make improvement" even though they are farther from the ground truth and drive the optimization away from the real meaningful regions. This overestimation issue has not been studied in the past literature to the best of our knowledge, though KB is still the strategy used in popular BO packages (Kandasamy et al.) 2020 Balandat et al.) 20206).
Local Penalization (LP) (González et al. 2016) is another widely used approach that directly penalizes the nearby region of the busy designs prevent similar designs from being evaluated next. However, extending this approach to the multi-objective scenario sometimes leads to sub-optimal selection of new designs, as explained in Figure 2b Intuitively, this sub-optimality comes from the failure of leveraging the accurate predictions from the surrogate model. Consider when selecting the best design evaluate from a of candidate designs (A and proposed by the multi-objective solver using hypervolume improvement criterion, while a busy design is in evaluation. LP penalizes the nearby regions of the busy design the design space but has no control over the performance space, which means that designs with similar performance as the busy design could still be selected (design A). Ideally, if the surrogate prediction of the busy design is certain, we can leverage this to avoid proposing designs with little performance gain. For example, simply believing the prediction of the busy design leads to selecting design B that has a higher hypervolume improvement.
4.3 BELIEVER-PENANL
STRATEGY
In conclusion, we observe that the failure case of KB is due to the trust of uncertain predictions while the sub-optimality of LP comes from not believing the certain prediction. Therefore, we propose a novel strategy BP that naturally combines KB and LP by applying KB to designs with certain predictions and LP to designs with uncertain predictions. Here, certainty of prediction is simply defined as the posterior standard deviation from the surrogate model which can be Gaussian processes, Bayesian neural networks other type of model that computes standard deviation of
5

---

Under review as a conference paper at ICLR 2022
any analytical expression determine the certainty threshold for applying KB or LP, in practice, we find a simple probabilistic form which works well: (x(
predictions. Though the idea of BP is general and one
max(1 20; (x), 0) where P, is the probability of believing for the i-th objective and o, is the posterior std x from the surrogate model the i-th objective. Because the objective data is normalized zero with mean unit variance before fitting the surrogate models, o, (x) is generally between 0 and I. As a result, BP generalizes more robustly than both KB and LP most of the benchmark problems as empirically demonstrated in Section 6.1
THE AUTOOED PLATFORM
5.1
OVERALL WORKFLOW
Manual Mode
Update visunlization
Propose designs
User
GUT
Optimization
Manual Evaluation
Database
Perform experiments
Auto Mode
Store resulss
Update visualization
Datahase
Automatic Evaluation
Internet
Update suass
Propose designs
Assign telks
Scheduler
Optinsization
Figure 3: The manual and automatic workflows of the platform design.
As full-stack software, the overall workflows of AutoOED are presented in Figure 3 mode and auto mode are distinguished by whether the experimental evaluations need to be done manually or can be done automatically through programs. Key components in this workflow include:
The manual
GUI: An intuitive graphical interface between users and optimization, evaluation and database, with many powerful functionalities supported described in Section 5.2
Optimization: Once the user sends the optimization request, the MOBO algorithm will be rumning in back-end and proposing next promising designs.
Evaluation: AutoOED supports different ways of performing evaluations: manually by hand and automatically by evaluation programs. The evaluation module receives designs proposed by the optimization algorithms and outputs the corresponding objective values.
Database:
The SQL database stores information of each design including design parame- ters, predicted and real objective values a transactional manner which can be distributed.
Next, we illustrate how the workflows of AutoOED combine these individual components.
Manual mode When the experimental evaluation must be performed by hand, the manual mode has to be selected. In this case, users need to interface with both the GUI and evaluation. When GUI receives the optimization request from the user, launches the optimization worker to propose designs for the user to evaluate. Users will see the proposed designs from the database GUI, then they can enter the evaluation results in the same interface. Once AutoOED receives the evaluation results, visualizations and statistics will be updated to inform users the latest status.

---

Under review as a conference paper at ICLR 2022
Auto mode If the evaluation program is available (in Python/C/C++/Malab), AutoOED can tomatically guide the experiments by alternating between optimization and evaluation through a scheduler. In this case, upon receiving the optimization request and stopping criteria from the user, the scheduler will automatically repeat the optimization-evaluatie criteria is met. The scheduler automatically launches evaluations after designs are proposed by optimization workers, and restarts optimization after evaluations are done, thus the whole loop of experimental design can be executed in an automated way without human intervention.
cycle until one of the stopping
5.2 FEATURE COMPARISON WITH OTHER PLATFORMS
we compare AutoOED with existing BO platforms according to the following important criteria. For other features of AutoOED, please refer Appendix A
As shown in Table D
Graphical user interface and visualization The GUI guides the user through a set of simple steps configure the problem, such as the number of design and performance parameters, the parameter bounds and constraints, parallelization settings, and selection of the optimization algorithm without the need of coding. The GUI also includes a real-time display of the design and performance space which allows users to easily understand the current status of optimization. We also support display- ing and exporting the whole optimization history (including database and statistics). All previous platforms do not offer such convenient GUI and even the visualizations need to be written by the user, except GPyOpt has built-in tool for plotting the acquisition function and convergence.
Multiple objectives and multiple domains For multi-objectivity, GPflowOpt implements HVPOI (Couckuyt et al.] 2014) and Dragonfly implements MOORS (Paria et al.) 2020) without the flexibility of incorporating other algorithms modules. BoTorch supports MESMO (Belakaria (et al. 2019), gEHVI (Daulton et al.] (020b) and qParEGO. Although algorithm implementations difter across platforms, AutoOED covers wider range of algorithms and incorporates them into more unified modular framework. Except continuous designs, AutoOED supports discrete, binary, categorical designs and mix of them by applying discrete or one-hot transformation in fitting and evaluating the surrogate model, as suggested by Garrido-Merchán & Brrnanndez-Lobal 2020).
Asynchronous optimization As demonstrated in Section 4 AutoOED supports different asyn- chronous techniques including KB, LP and BP, while Dragonfly and Bo'Torch only implements KB and all other platforms do not support asynchronous optimization.
External evaluation There are many real-world experimental design problems where the experi- mental evaluation must be performed by hand external lab equipment thus it is hard write an analytical objective function code. Though simple to implement, surprisingly, among all of the existing platforms surveyed in this paper, only GPyOpt supports evaluating externally and suggest- ing designs to evaluate purely based on a given dataset. AutoOED allows users to see the suggested designs and enter the evaluation results easily in the database interface, as described in Section 5.1
EXPERIMENTS
We conduct experiments across 12 standard multi-objective benchmark problems: ZDT1-4 Zitzler al. 2000). DTIZI-4 (Deb et al. 2005). OKA1-2 (Okabe et al., 2004) and VLMOP2-3 (Van Veld- huizen & Lamont 1999). For each algorithm, we run experiments with 20 different random seeds and a budget of 100 samples. The initial 20 samples of each run are generated by Latin hypercube sampling (McKay et al. 1979). We measure as the performance criterion the log of the difference between the hypervolume of the ground-truth Pareto front and hypervolume of the best Pareto front approximation found by the algorithms (thus lower is better). The curves averaged over 20 random seeds and the variance shown as shaded regions. Detailed problem information and hy- perparameters described in Appendix B Additional ablation studies included in Appendix D
6.1 ASYNCHRONOUS MOBO ALGRIIHTM
is effective, we compare four asynchronous MOBO algorithms on all benchmark problems. Async simply ignores the busy designs while optimizing asynchronously
To test whether
7

---

Under review as a conference paper at ICLR 2022
2.1
0.4
0.20
10
0.2
0.50
0.0
0.5
0.45
-22
02
20
80
90
220
40
90
100
50
80
102
300
dttr1
dtt2
dited
Log Sohhoordaaaee afterre
4.2
4.0-
4.5
-1-
4.4
-30
-3.
olal
0ko2
vimop2
Vimop3
hohngewe ure arlonnc
-1.00
3.0
2.22
0.9
1.50
50
-15
100
of corpple
10 Mumberr of curgles
Nambon of ca
of cirplee
Asymc
Asyme KB
Asymi
Anync BP
Figure 4: Performance comparison between variants of asynchoronous MOBO algorithms.
and the remaining algorithms are described in Section 4 Figure 4 shows that Async BP consistently outperforms other methods and follows the best of Asyne KB and Async LP.
6.2 PERFORMANCE COMPARISON ACROSS PLATFORMS
:d11
2002
2013
2d54
Log wog.oteeeeeeeeooo urooee
0.4
0.5
0.2
0.5
18-
0.
6-2
22
130
20
60
100
80
300
dil21
d.12
0013
d:124
-05-
-0.-
4.5
4.0
3.8
2.5-
+1.1
20
130
20
100
300
stmop2
1.2
1.5
2.0
1.0
1.1
1.0
2.0
0.
22
40
50
90
100
20
43
22
20
100
20
40
60
92
100
23
42
50
00
300
ToTorch
GPflowOpt
Bragontly
Hypermapper
OpenBox
AuboOED
Figure 5: Performance of AutoOED on the benchmark problems compared to other BO platforms. The result of BoTorch on VLMOP3 is omitted as the algorithm fails to stop within 24 hours.
We compare AutoOED against other open-source BO platforms with multi-objective optimization capabilities, including BoTorrc, GPflowOpt, Dragonfly, HyperMapper and OpenBox on the afore- mentioned benchmark problems. Our baseline implementation follows the default recommended
8

---

Under review as a conference paper at ICLR 2022
settings in the original documentation and tutorials. For BoTorch, function. In Figure [] we demonstrate the competitive performance of AutoOED against other BO platforms using our benchmark. AutoOED takes major lead in several challenging problems such as ZDTI, ZDT3, and VLMOP3, which shows that our platform handles high-dimensional MOBO problems very well with the proposed asynchronous BP strategy. Besides, the performance of Au- toOED is generally stable and ends either the best or comparable on most benchmark problems. We conduct additional ranked and paired comparisons between AutoOED and all the baseline plat- forms in Appendix to further demonstrate AutoOED's ' robustness and competitive performance.
the gEHVI acquisition
6.3 REAL-WORLD OPTIMAL EXPERIMENTAL DESIGN
Fanden
316
13M
3746
(a) Experiment setup.
(b) Discovered solutions.
(c) Hypervolume.
Figure 6: Physical setup, optimization process and solutions of the PID heater control experiment.
We further demonstrate the real-world applicability of AutoOED through applying to optimize a PID heater controller to have optimal response time and minimal overshoot in a fully automated way. Details of the experimental setup can be found in Appendix B33 To automate the experiment, simply link the Python evaluation program of this experiment setup AutoOED through GUI then start the iterative optimization. Finally, the results are exported as shown in Figure (6) where a set of solutions are discovered with optimal trade-off's between minimal response time and minimal overshoot. Using MOBO algorithms provided by AutoOED is able to discover better designs com- pared random sampling at the presence of evaluation noise (temperature measurement error, lack of precise initial temperature control, fabrication differences between heater blocks). This example, with all the components that people can easily buy off-the-shelf, serves simple proof of concept that AutoOED is applicable to optimize real physical systems. More examples, including optimiz- ing material structure based on FEM simulation and optimizing physical motor's rotation, can be found in our documentation with detailed instructions on how to interact with GUI and compose the evaluation program for fully automated OED.
CONCLUSION AND FUTURE WORK
We introduced an open-source platform for automated optimal experimental design of multi- objective problems. The platform of modular structure, facilitating the implementation of dif- ferent multi-objective Bayesian optimization (MOBO) algorithms and enabling both data- and time- efficient optimization. In addition, the platform includes a novel strategy for asynchronous batch optimization for improved time efficiency. We performed extensive experiments on standard bench- mark to demonstrate the robust performance against other relevant methods. Furthermore, we con- ducted real-world physical experiments to showcase the automated usage of our platform.
From the research and engineering perspective, future work includes implementing additional fea- tures, such as expensive constraint handling, advanced noise handling and extending AutoOED's framework to incorporate even wider range of MOBO algorithms. From a practical standpoint, are also interested in applying AutoOED to automate more real-world experimental design prob- lems science and engineering. And we believe AutoOED will gradually lower the barrier between MOBO research and practical applications.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework, 2019.
Ahsan S Alvi, Binxin Ru, Jan Calliess, Stephen Roberts, and Michael A Osborne. Asynchronous batch bayesian optimisation with improved local penalisation. arXiv preprinl arXiv:1901. 10452, 2019.
The GPyOpt authors. GPyOpt: A bayesian optimization framework in python. http://github. com/ CheffieedML/GPPM
2016.
Eytan Bakshy, Lili Dworkin, Brian Karrer, Konstantin Kashin, Benjamin Letham, Ashwin Murthy, and Shaun Singh. Ae: A domain-agnostic platform for adaptive experimentation. 2018.
Maximilian Balandat. Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorrh: A Framework for Efficient Monte-Carke Bayesian Optimization. In Advances Neural Information Processing Sys- tems 33, 2020a. URL
neur meurips.... cc ccc/papper/222/hast/ Abstract. html
E5b1b89d98b (at.49982226677722255555333
Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. Bo'Torc: A Framework for Efficient Monte-Carlo Bayesian Optimization. Advances in Neural Information Processing Systems 33, 2020b. URL
http://arxiv org. wwwpgg/abs/191/ 0ttpb///99000053
Syrine Belakaria and Aryan Deshwal. Uncertainty aware search framework for multi-objective bayesian optimization. In AAAI Conference on Artificial Intelligence (AAAI), 2020.
Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. Max-value entropy search for multi- objective bayesian optimization. In Advances in Neural Information Processing Systems, pp- 1823-7833. 2019.
James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference machine learning, pp. 115-123. PMLR, 2013.
Eric Bradford, Artur M Schweidtmann, and Alexei Lapkin. Efficient multiobjective optimization employing gaussian processes, spectral sampling and genetic algorithm. Journal of global optimization, 71 71(2):407-438, 2018.
Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Wein- berger (eds.), Advances in Neural Information Processing Systems 24, 2249- 2257. Curran Associates, Inc., 2011. URL http://papers nips cc/paper/ pdf
Ivo Couckuyt, Dirk Deschrijver, and Tom Dhaene. Fast calculation of multiobjective probability of improvement and expected improvement criteria for pareto optimization. Journal of Global Optimization, 60(3):575-594 2014.
Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hyper- volume improvement for parallel multi-objective bayesian optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.). Advances in Neu- ral Information Processing Systems, volume 33. pp. 9851-9864. Curran Associates, Inc., 2020a. URL https://pproeedingss neurips. cc/ pttpp//pppp/////11ee. Sag.aaddeeeeeeeSSSoacoaaaa377---
Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume im- provement for parallel multi-objective bayesian optimization. arXiv preprint urXiv:2006.05777 2020b.
Kalyanmoy Deb, Ram Bhushan Agrawal, et al. Simulated binary crossover for continuous search space. Complex systems, 9(2):115-144 1995.
10

---

Under review as a conference paper at ICLR 2022
Kalyanmoy Deb, Mayank Goyal, et al. A combined genetic adaptive search (geneas) for engineering design. Computer Science and informatics, 26:30-45, 1996.
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist mul- tiobjective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation, 6(2): 182-197, 2002.
Kalyanmoy Deb, Lothar Thiele, Marco Laumanns, and Eckart Zitzler. Scalable test problems for cvolutionary multiobjective optimization. In Evolutionary multiobjective optimization, pp. 105- 145. Springer, 2005.
Michael Emmerich and Jan-willent Klinkenberg. The computation of the expected improvement in dominated hypervolume of pareto front approximations. Rapport technique, Leiden University, 34:7-3, 2008
Eduardo C Garrido-Merchán and Daniel Hernández-Lobato. Dealing with categorical and integer- valued variables in bayesian optimization with gaussian processes. Neurocomputiing 380:20-35, 2020.
Michael A Gelbart, Jasper Snoek, and Ryan P Adams. Bayesian optimization with unknown con- straints. arXiv preprint arXiv:1403.5607, 2014.
David Ginsbourger. Rodolphe Le Riche, and Laurent Carraro. Kriging is well-suited to parallelize optimization. In Computational intelligence in expensive optimization problems, pp. 131-162.
Springer, 2010.
Javier González, Zhenwen Dai, Philipp Hennig, and Neil Lawrence. Batch bayesian optimization via local penalization. In Artificial intelligence and statistics, pp. 648-657. PMLR, 2016.
Stewart Greenhill, Santu Rana, Sunil Gupta, Pratibha Vellanki, and Svetha Venkatesh. Bayesian optimization for adaptive experimental design: A review. IEEE access, 8:13937-13948, 2020.
Ryan-Rhys Griffiths and José Miguel Hernández-Lobato. Constrained bayesian optimization for automatic chemical design. arXiv preprint arXiv:
2017.
Sebastian Haan. Geobo: Python package for multi-objective bayesian optimisation and joint inver- sion in geosciences. Journal of Open Source Software, 6(57):2690, 2021. doi: 10.21105/joss. 02690. URL
211.1...1111sssss 02690
Kirthevasan Kandasamy, Akshay Krishnamurthy, Jeff Schneider, and Barnabás Póczos. Parallelised bayesian optimisation via thompson sampling. In International Conference on Artificial Intelli- gence and Statistics, pp. 133-142. PMLR, 2018.
Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabas Poczos, and Eric P. Xing. Tuning hyperparameters without grad students: Scalable and robust bayesian optimisation with dragonfly Journal of Machine Learning Research, 21(81):1-21 2020. URL BaLLhtp:///// //jmrr. org wwtpp/paasss/.///122 223 html
Joshua Knowles. Parego: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation, 10(1): 50-66, 2006.
Nicolas Knudde. Joachim van der Herten, Tom Dhaene, and Ivo Couckuyt. GPtlowOpt: A Bayesian Optimization Library using TensorFlow. arXiv preprint arXiv::711103384 2017.
URL http:/htt///aaatt org/ http:/abs1111.....0 03845
Mina Konakovic Lukovic, Yunsheng Tian, and Wojeiech Matusik. diversity-guide multi-objective bayesian optimization with batch evaluations. Advances in Neural Information Processing Sys- fems, 33, 2020.
Yang Li, Yu Shen, Wentao Zhang, Yuanwei Chen, Huaijun Jiang, Mingchao Liu, Jiawei Jiang, Jinyang Gao, Wentao Wu, Zhi Yang, and et al. Openbox: A generalized black-box optimization service. Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery Data Mining, Aug 2021. doi: 10. 10.114//447548.4466666 URL httLht/t://eee www.ddww......o.oooooooooooooooe wtw...orgg11.1.1... 75447548 3467061
11

---

Under review as a conference paper at ICLR 2022
Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp. Difan Deng, Car- olin Benjamins, René Sass, and Frank Hutter. Smac3: A versatile bayesian optimization package for hyperparameter optimization. 2021.
Roman Marchant and Fabio Ramos. Bayesian optimisation for cnn monttmmr monitoring. pp. 2242-2249, 10 2012. ISBN 378-1-4673-1737 doi: 10.1 RAL.10//RRO.2202268555555
environmental
Ruben Martinez-Cantie Cantin, Nando Freitas, Eric Brochu, Jose Castellanos, and Arnaud Doucet. A bayesian exploration-exploitation approach for optimal online sensing and planning with visu- ally guided mobile robot. Auton. Robots, 27:93-103, 08 2009. doi: 10. 1007/s//1 1001-000-----1000-0-000000000000
Michael D McKay, Richard J Beckman, and William J Conover. Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21(2):39-245, 1979.
Jonas Močkus. On bayesian methods for seeking the extremum, Optimization techniques IFIP technical conference, pp. 400-404. Springer, 1975.
Luigi Nardi, Artur Souza, David Koeplinger, and Kunle Olukotun. Hypermapper: a practical design space exploration framework. In 2019 IEEE 27th International Symposium on Modeling. Anal- ysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pp. 425-426, 2019. doi: 10.1109/ASSOTS 2019000052210
Tatsuya Okabe, Yaochu Jin, Markus Olhofer, and Bernhard Sendhoff. On test functions for evolu- tionary multi-objective optimization. In International Conference on Parallel Problem Solving from Nature, pp. 792 802. Springer, 2004.
Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. A flexible framework for multi- objective bayesian optimization using random scalarizations. In Ryan P. Adams and Vibhav Gogate (eds.), Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, vol- ume 115 of Proceedings of Machine Learning Research, pp. 766-776. PMLR, 22-25 Jul 2020. .tml
URL https: httph:tt////ottteee /tae:://praa http:///occeeenng.ee/ee ml wtw.pnnsss............... wwtppsssss//pppppp
Nery Riquelme, Christian Von Lücken, and Benjamin Baran. Performance metrics in multi-objective optimization. In 2015 Latin American Computing Conference (CLEI), pp. 1-11. IEEE, 2015.
Daniel Russo and Benjamin Van Roy. Learning to optimize via information-directe sampling. In Advances in Neural Information Prucessing Systems, pp. 1583-1591, 2014.
Adriana Schulz, Harrison Wang, Eitan Grinspun, Justin Solomon, and Wojciech Matusik. Interactive exploration design trade-offs. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018
B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. de Freitas. Taking the human out of the loop: review of bayesian optimization. Proceedings of the IEEE, 104(1):148-175, 2016.
Benjamin J Shields, Jason Stevens, Jun Li, Marvin Parasram, Farhan Damani, Jesus I Martinez Alvarado, Jacob M Janey, Ryan Adams, and Abigail G Doyle. Bayesian reaction optimization as tool for chemical synthesis. Nature, 590(7844):89 2021.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, pp. 2171-2180. PMLR, 2015.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimiza- tion in the bandit setting: no regret and experimental design. In Proceedings of the 27th Interna- fional Conference on International Conference Machine Learning, pp. 1015-1022, 2010.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-29) 1933.
12

---

Under review as a conference paper at ICLR 2022
David A Van Veldhuizen and Gary B Lamont. Multiobjeetive evolutionary algorithm test suites. In Proceedings of the 1999 ACM symposium on Applied computing, pp. pp. 351-357, 1999.
J. Wu, W.Y. Zhang, S. Zhang, Y.N. Liu, and X.H. Meng. A matrix-based bayesian approach for manufacturing resource allocation planning in supply chain management. International Journal of Production Research, 51(5):1451-146, 2013. doi: 10. 1000000002053333222226666
URL
https://doi.org: org.lo...rr/11 10. 000.0000000043..222222 201..43...1....2......... 693966
Qingfu Zhang, Wudong Liu, Edward Tsang, and Botond Virginas. Expensive multiobjective opti-
mization by moea/d with gaussian process model. IEEE Transactions Evolutionary Computa- tion, 14(3):456 2009.
Yichi Zhang, Daniel W Apley, and Wei Chen. Bayesian optimization for materials design with mixed quantitative and qualitative variables. Scientific Reports, 10(1):1-13, 2020.
Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE transactions on Evolutionary Computation, 3(4):257- 271, 1999.
Eckart Zitzler, Kalyanmoy Deb, and Lothar Thiele. Comparison of multiobjective evolutionary algorithms: Empirical results. Evolutionary computation, 8(2):1331155, 2000.
13

---

Under review as a conference paper at ICLR 2022
A
PLATFORM DETAILS
A.1 ADDITIONAL FEATURES
Cross-platform AutoOED is cross-platform software that be installed executable
computers with either Windows/MacOO//Limus system. Installing from source code is also supported for extra flexibility.
Surrogate prediction Users may find AutoOED useful not only in optimization but also in predic- tion. addition the set of optimal solutions, our platform's 's final product the learned prediction models of the unknown objectives, which can be used easily from GUI to predict the objectives for given design from the user. The prediction provides the users with more insights into the poten- tial outcomes of experiments. It helps them better understand the optimization problem to make informed decisions and guide the optimization process towards their preference.
Constraints Besides, AutoOED supports inexpensive black-box constraints on the design space addition to the bounds. handle expensive black-box design constraints or objective constraints, it is possible to learn constraint model similar to learning the objective function, implemented by Spearmint (Gelbart et al. 2014).
B EXPERIMENT SETUP
B. BENCHMARK PROBLEMS
In this section, we briefly introduce the properties of each benchmark problem, including the di- mensions of the design space X' Rdd and performance space f(X) C IR"", and the reference points we use for calculating the hypervolume indicator, which are shown in Table 2 We perform 20 independent test runs with different random seeds for each problem on each algorithm. For each test run of one problem, generated by Latin hypercube sampling (McKay et al.] 1979) using a same random seed. To have vector containing the maximum
the same initial set of samples for every algorithm, which is
fair comparison, simply set the reference point E R value each objective over the initial set of samples {x1, }
max 1sisk
max Jm (x:)). isisk
Table 2: Basic descriptions of all the benchmark problems.
Name
d
m
ZDTI
30
(0.9699, 6.0445)
ZDT2
30
(0.9699, 6.9957)
ZDT3
30
(0.9699, 6.0236)
ZDT4
10
2
(0.9699, 199.6923)
DTLZI
6
(360.7570, 343.4563)
DTLZ2
(1.7435, 1,6819)
DTLZ3
6
(706.5260, 746.2411)
DTLZ4
6
(1.8111, 0.7776)
OKA1
2
(7.4051, 4.3608)
OKA2
(3. 1315, 4.6327)
VLMOP2
6
(1.0, 1.0)
VLMOP3
2
3
(8. 18.69 53.2348, 0.1963)
B.2 HYPERPARAMETERS
Here present all the common hyperparameters that AutoOED uses in the experiments.
14

---

Under review as a conference paper at ICLR 2022
Surrogate model We use the same Gaussian process model as the surrogate for all experiments. We use zero mean function and anisotropic Matern 1/2 kernel, which empirically is numerically stable than popular Matern 5/2 kemel in our experiments. The corresponding hyperparameters are specified in 3 which are suggested by TSEMO.
Table 3: GP hyperparameters.
parameter name
value
initial I
(1....
1) Rd
range
(v10-3
v109)
initial of
(v10-3, v10s)
of range
initial on
10-2
on range
(e-6,
Multi-objective evolutionary algorithm The cheap NSGA-II solver employed in AutoOED's MOBO algorithms by default uses simulated binary crossover (Deb et al. (9955) and polynomial mutation (Deb et al. 1996) for finding the Pareto front of acquisition functions. The initial popula-
tion is obtained from the best current samples determined by non-dominated sort (Deb et al. 2002
The other hyperparameters are specified in Table
Table 4: NSGA-II hyperparameters.
parameter name
value
population size
100
number of generations
200
crossover 1ec
15
mutation 7m
20
B.3 REAL- WORLD EXPERIMENT SETUP
In our real-world experiment setup, overall, PID controller is employed to regulate the temperature of the heater block with proportional, integral, and differential constants. To find the optimal set of constants a number of heating cycles are done asynchronously with the controller using AutoOED. The experiment comprised of setting the PID constants, then heating the block up to a set temper- ature, and keeping them at the set duration for 2 minutes. During this time the response time and overshoot are measured. After the heating cycle, the heater is then cooled back down to a starting temperature to prepare for another test with new PID constants.
Specifically, the experimental setup is comprised of 3 heaters with identical dimensions and charac- teristics. Each heater is comprised of heater block, heating element, temperature sensor, solid-state relay, power supply, and a controller, shown in Figure 7 To run an experiment, AutoOED sends the PID constants to a controller that is free. Next, the PID controller becomes active and starts regu- lating the temperature of the heater. The temperature sensor measures the temperature of the heater block. Depending on the constants of the PID controller and the temperautre of the heater block, the controller turns the heating element on or off via the solid-state relay. After period of 2 minutes where the PID controller is active, the controller stops actively regulating the temperature of the heater allowing the heater to cool. Next, the controller starts to monitor the cooling of the heater via the temperature sensor. It monitors it until the heater block cools to temperature below a threshold. The amount of time it takes cool the heater block depends the temperature that the block was heated to during the active period. Once it sufficiently cools, the controller sends the calculated overshoot and response time to AutoOED and notifies that is ready another experiment.
15

---

Under review as a conference paper at ICLR 2022
Heating Element
Henter 3
Heator
Hester
Temp.
Soliah Stare 3tdaay
PTD Controller
PID Controllr
PID Contreller
Figure 7: A schematic of the setup used for the real-world experiment.
C
ADDITIONAL COMPARISONS
BoTorch
GPflowOpt
Dragonfly Hypermapper
OpenBox
AutoOED
Figure 8: Performance rank of platforms the 12 benchmark problems (lower is better). The box extends from the lower to the upper quartile values, with solid line at the median and dashed line the mean. The whiskers that extend the box show the range of the data.
We conduct ranked and paired comparisons between AutoOED and the baseline platforms based on the 12 benchmark problems, shown in Figure and and Figure 9 The performance rank compar- ison in Figure suggests that BoTorch, OpenBox and AutoOED outperform other platforms by great margin overall. While BoTorch and OpenBox share better median rank, AutoOED appears to be the stablest platform that consistently ranks between and on all problems and has higher bound performance than BoTorch and OpenBox. Figure y also suggest that AutoOED has a competitive performance to BoTorch and OpenBox, but outpertorms other baselines on a wider range of benchmarks.
16

---

Under review as a conference paper at ICLR 2022
BoTorch 0
10
10
0
10
Number benchmarks outperform
GPflowOpt
-10
0
2
6
-8
-10
Platform A
Dragonfly
Hypermapper
-10
-12
OpenBox
0
8
12
0
AutoOED
10
12 Hypermapper
-00
BoTorch
GPfooOpt
Dragonfly
OpenBox
AutoOED
Platform
Figure 9: Performance comparison between each pair of platforms the 12 benchmark problems. Each value the matrix shows the number benchmarks that platform (associated with the row) outperforms platform B (associated with the column), the higher the better.
2.00
0.49
0.3
0.59
2.00
0.2
1.50
0.50
1.0
0.
07-
1.00
10
15
10
15
20
25
-.2
19000000 wwe.eee dftecros
-0.0
4.5
-0.0
-0.5
4.4
-0.8
-1.
10
20
10
20
10
15
70
70
vimop2
3.2
1.50
-1.00
1.2
-3.25
0.5
0.8
0.50
-2.25
19
15
Time
Tima
Time
Figure 10: Performance comparison between synchronous and naive asynchronous MOBO algo- rithms.
17

---

Under review as a conference paper at ICLR 2022
D ABLATION STUDIES
D.1 SYNCHRONOUS AND ASYNCHRONOUS MOBO
Following the experiment settings in Section we addtionally compare the performance of syn- chornous and asynchronous MOBO. As shown in Figure [0] they achieve similar hypervolumes whereas asynchronous MOBO spends less than half of the time of its synchronous counterpart.
D.2 BATCH SIZE
X022
2011
009 www.ewooeeeeeeeooo Merene
211
0.50
055
2.0
02
0.4
0.3
18-
0.1
20
33
1.50
20
40
100
20
42
60
80
350
022
4.0
0.4
4.0
-0.9
2.-
-080
4.4
-1.
30
60
60
80
1000
20
80
05
60
1.50
30
40
60
100
33
42
60
50
300
stmop?
2.1
1.5
2.0
1.0
1.0
2.4
0.5
-1.
0.5
0.-
27
100
20
20
40
100
20
45
60
00
300
plas
sampliss
of samplles
Abymc
Joync
Asymc
Figure 11: Performance comparison between variants of asynchoronous MOBO algorithms with a batch size of 4.
Ablation studies are also conducted on the batch size in asynchronous MOBO. For this category of experiments, we repeat our practice in Section 6.1 while changing the batch size 4 and 16, respectively. The results are demonstrated in Figure I1 and Our proposed BP strategy maintains
its relative lead in VLMOP3 and performs comparably with other variants the rest of the test problems.
D.3 ACQUISITION FUNCTION
Lastly, we evaluate the asynchronous MOBO variants using the El acquisition function. Although the change in acquisition function has clear influence hypervolume growth, the proposed BP variant still performs favorably in problems such as ZDT2, DTLZ2, and VLMOP2. The performance of BP on the other problems remain comparable the alternative strategies.
18

---

Under review as a conference paper at ICLR 2022
<d.1
A012
Upg www.w//onooooooo
0.4
0.20
0.5
0.35
02-
0.59
2.0
0.-
0.40
dttr
dttr
cho3
dited
Log bogoooateee deeggece
-0.3
-0.2
5.0
-1.0-
4.0
0.4
0.5
4.5
4.2
1.5
Dike
oka2
stmoop2
vimup3
1.2
-0.15
3.5
2.00
1.3
3.0
1.0
2.9
0.9
0.8
0.7
60
100
100
100
Asymic
Asymic KB
Asymic
Adynz BP
Figure 12: Performance comparison between variants of asynchoronous MOBO algorithms with batch size of 16.
2dk1
2d22
2dh3
2014
2.18
2.55
0.1-
2.05-
0.3
-0.9
35
dids
dile4
-2223
50
-3.99
4.0
0.275
3.4
+2.399
4.2
-3.375
4.2
35
oka1
6ka2
vimus2
vimopp
3.2
-0.7
3.5
-0.3
0.5
-0.3
0.5
-1.3
-0.5
60
100
100
Asymi
Asynic KB
Asymi LP
Adynz BP
Figure 13: Performance comparison between variants of asynchoronous MOBO algorithms with EI as acquisition function.
19


Reviews:
[{"review_id": "kv3slUskf_o", "reviewer": "Reviewer_iLNv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors have designed a generalized platform for further users to quickly design and evaluate their own MOBO algorithms, i.e., the proposed MOBO strategies are applied for data-efficient experimentation; The proposed Believer-Penalizer (BP) optimization strategy is introduced for time-efficient experimentation. The authors organize their work in a well-reading manner. However, the main concern of this paper is that not sure whether it has enough scientific contribution, for instance, the authors claim that they proposed the state-of-the-art strategy named BP while in the paper it is only presented in a very naive way: a novel strategy Believer-Penalizer (BP) that naturally combines KB and LP by applying KB to designs with certain predictions and LP to designs with uncertain predictions. To be honest, I do not have very clue that how authors combine these two together, and it lacks clear proof that in theory how this kind of combination could work, simply experiment results can not be adopted as evidence.  ", "summary_of_the_paper": "The authors present an Automated Optimal Experimental Design platform aiming at accelerating discovering solutions with multi-objective problems in a data-efficient manner. The main contribution of the work is introducing a novel strategy for batch experiments acceleration.\n", "summary_of_the_review": "this paper is more engineering-oriented, lacking convincing theoretical works.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "g-ATKhCc_6h", "reviewer": "Reviewer_f9tL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are several things to like about this paper:\n\n1. The problem studied in this paper is well-motivated. Multi-objective (MO) problems in the context of Optimal Experimental Design (OED) are pervasive. OED in science and engineering often require satisfying several conflicting objectives simultaneously. However, this area of MO-OED has received relatively little attention from the research community.\n2.This paper is well-organized and clear written. The authors start from stating the problem formulation. Then, the authors present the algorithm framework for AutoOED.\n3. The claims are well supported by four multi-objective tasks on 12 public datasets. \n\nHowever, I found that there are several shortcomings:\n\nMinor issues: \n- in general: better to use Pareto optimal solutions rather than Pareto-optimal solutions\n- page 5: I think the authors can directly use BP for Believer-Penalizer as it has been specified in previous section\n- in general: I think it is better for the authors to double check about the redundant specifications about the abbreviations throughout the paper\n- page 9: importantance should be importance", "summary_of_the_paper": "In this paper, the authors present AutoOED, an open-source platform for efficiently optimizing multiobjective problems (MO) with a restricted budget of experiments. The platform automatically guides the design of experiments to be evaluated. AutoOED is\nbuilt upon multi-objective Bayesian optimization (MOBO).  To accelerate the optimization in a time-efficient manner, the authors propose a strategy called Believer-Penalizer (BP) that allows batch experiments to be accelerated asynchronously without affecting performance. The authors also provide a graphical user interface (GUI) for users to visualize and guide the experiment design intuitively. Finally, we demonstrate that AutoOED can control and guide real-world hardware experiments in a fully automated way without human intervention. Finally, the authors demonstrate that AutoOED\ncan control and guide real-world hardware experiments in a fully automated way without human intervention.\n", "summary_of_the_review": "The problem studied in this paper is well-motivated. Multi-objective (MO) problems in the context of Optimal Experimental Design (OED) are pervasive. I like it very much that the authors would open source such a powerful and convenient framework to complement the current state-of-the-art packages. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "coVEOKMNMEV", "reviewer": "Reviewer_QfiR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Correctness and Clarity\n\nOverall, the paper is very well written and structured, and easy to follow. Since it is a system paper, the authors focus more on the system parts and describe the advantages and use cases of the system. The only novel contribution from a methodological point of view is Section 4.2 on the Believer-Penalizer strategy for asynchronous parallel optimization. The straightforward and simple idea makes sense; but the threshold for deciding between KB and KP is not well motivated and alternatives are not evaluated.\n\n### Technical Novelty And Significance\n\nThe system itself is well designed and implemented. So, from an engineering point of view the system is well justified. However, the system nevertheless gives me the feeling of “yet another Bayesian Optimization package”; over the last years there were so many of them. AutoOED's selling point is the asynchronous, multi-objective optimization which not many systems actually support. However, here AutoOED mostly implements existing techniques and adds with BP only a minor contribution overall. \n\nIn the appendix, the authors honestly point out that AutoOED only scales well to 2 or 3 objectives. Although I agree that this is a fairly common use case, there are many applications with more objectives. I encourage the authors to look into the field of many-objective optimization.\n\n### Empirical Novelty And Significance\n\nThe empirical results are the weak point of the paper. In Section 6.1, the authors compare BP with KB and LP. They claim that ```BP consistently outperforms other methods```. However, Figure 4 shows that BP is not always the best, sometimes it is the worst performing one (dtlz3). A clear gap between BP and the other baselines cannot be found in any of the benchmarks. I think that a fair statement would be that BP is a robust approach and performs well on average.\n\nThe same issue applies to the comparison to other packages. The authors claim ```AutoOED is generally stable and ends up either best or comparable on most benchmarks```. However, Figure 5 shows that AutoOED underperforms on 6 out of 12 benchmarks and is only clearly the best system on 4 benchmarks. \n\nAlthough I like the benchmark on a real-world benchmark, I have the impression that the benchmark is a bit cooked up and lacks the connection to a real real-world benchmark. Furthermore, AutoOED is only compared against random search on this benchmark. \n\nI appreciate the ablation study in the appendix. However, this gives me the impression that the results for the comparison for KB vs LP vs BP is a bit cherry-picked and it is even more unclear whether BP performs well on other settings of AutoOED.\n\n### Questions to the Authors\n\n1. How does AutoOED perform on single objective benchmarks?\n2. You mention “convergence” of AutoOED twice. How is convergence measured or determined?\n3. How can a neural network (MLP) be used for BO since it is not a probabilistic model? (see list of surrogate models in Section 3.2)\n\n### Minor Remarks\n\nI strongly disagree with the following statement:\n\n``` However, they are all targeted for experts in coding without an intuitive GUI, which means that the users need to write hundreds lines of code defining the design space, performance space, surrogate models, acquisition functions, optimization solvers, etc. to finally start the optimization.```\n\nIn fact, more and more tools get easier to use and most of the listed tools don’t require hundreds of lines; nevertheless some of them do.\n\nIn “Open-source Bayesian optimization platform” I’m missing many more open source BO packages, incl. AX, HyperOpt, SMAC, Scikit-optimize, HyperMapper, OpenBox or Optuna. Some of them even support multi-objective optimization. I encourage the authors to extend this list accordingly.\n", "summary_of_the_paper": "The paper presents a package for black-box optimization and is specifically designed for the optimization of experimental designs. To this end, the authors build upon multi-objective Bayesian Optimization which allows to obtain good points within a few function evaluations and also enables to obtain a Pareto-Front of non-dominated points. To have an efficient, asynchronous parallelization, the authors propose Believer-Penalizer for choosing the next point while others are still being evaluated. The main idea is to choose either Kriging Believer (KB) and Local Penalization (LP) based on a threshold on the uncertainty in the posterior distribution of the probabilistic surrogate model. In the experiments, the authors show that their package achieves often a fairly good performance.", "summary_of_the_review": "Overall, AutoOED is a well engineered BO package and it has some strengths and rare features such as multi-objective optimization. However, the novelty is rather minor, there are only little new ideas in the package and the claims in the experiments are clearly overstated. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "FbPneI9lMi", "reviewer": "Reviewer_Bmro", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern about this work is that it is more like an assembly of software design docs and a manual for an open-source platform. There is very limited methodology contribution from this work. The main contribution of this work is the highly modularized design, compared with existing Bayesian optimization platforms, e.g., Spearmint, Dragonfly, etc.\n\nAnother issue is that the authors claim at the beginning that ‘The goal is to simultaneously minimize m>=2 objective functions’ but later explain that ‘AutoOED is designed for cases when the number of objectives equals 2 or 3’.  It is misleading to present the platform in this way and might confuse readers with the impression that this is a generic platform to solve multi-objective optimization. Also, it is strange that AutoOED does not apply for only one objective, which should be naturally covered by an MOO algorithm.\n\nI am also surprised that the authors did not cite the following work though it is only on arXiv but strongly correlated with this manuscript.\n\nTian, Yunsheng, et al. \"AutoOED: Automated Optimal Experiment Design Platform.\" arXiv preprint arXiv:2104.05959 (2021).\n", "summary_of_the_paper": "This paper proposed and explained an open-sourced experimental design platform powered using Bayesian multi-objective optimization to accelerate discovering solutions with optimal objective trade-offs. The authors present how to use this platform in detail. ", "summary_of_the_review": "Overall, this is a decent document and manual for MOO software package/platform introduction but not a scientifically methodological paper for ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews are of good quality. The responses by the authors are commendable, but reviewers remain of the opinion that the scientific contribution of the paper is limited, no matter how strong the software engineering contribution may be.
Decision: Reject