Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed and explained an open-sourced experimental design platform powered using Bayesian multi-objective optimization to accelerate discovering solutions with optimal objective trade-offs. The authors present how to use this platform in detail. 
main_review: My main concern about this work is that it is more like an assembly of software design docs and a manual for an open-source platform. There is very limited methodology contribution from this work. The main contribution of this work is the highly modularized design, compared with existing Bayesian optimization platforms, e.g., Spearmint, Dragonfly, etc.

Another issue is that the authors claim at the beginning that ‘The goal is to simultaneously minimize m>=2 objective functions’ but later explain that ‘AutoOED is designed for cases when the number of objectives equals 2 or 3’.  It is misleading to present the platform in this way and might confuse readers with the impression that this is a generic platform to solve multi-objective optimization. Also, it is strange that AutoOED does not apply for only one objective, which should be naturally covered by an MOO algorithm.

I am also surprised that the authors did not cite the following work though it is only on arXiv but strongly correlated with this manuscript.

Tian, Yunsheng, et al. "AutoOED: Automated Optimal Experiment Design Platform." arXiv preprint arXiv:2104.05959 (2021).

summary_of_the_review: Overall, this is a decent document and manual for MOO software package/platform introduction but not a scientifically methodological paper for ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present an Automated Optimal Experimental Design platform aiming at accelerating discovering solutions with multi-objective problems in a data-efficient manner. The main contribution of the work is introducing a novel strategy for batch experiments acceleration.

main_review: The authors have designed a generalized platform for further users to quickly design and evaluate their own MOBO algorithms, i.e., the proposed MOBO strategies are applied for data-efficient experimentation; The proposed Believer-Penalizer (BP) optimization strategy is introduced for time-efficient experimentation. The authors organize their work in a well-reading manner. However, the main concern of this paper is that not sure whether it has enough scientific contribution, for instance, the authors claim that they proposed the state-of-the-art strategy named BP while in the paper it is only presented in a very naive way: a novel strategy Believer-Penalizer (BP) that naturally combines KB and LP by applying KB to designs with certain predictions and LP to designs with uncertain predictions. To be honest, I do not have very clue that how authors combine these two together, and it lacks clear proof that in theory how this kind of combination could work, simply experiment results can not be adopted as evidence.  
summary_of_the_review: this paper is more engineering-oriented, lacking convincing theoretical works.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a package for black-box optimization and is specifically designed for the optimization of experimental designs. To this end, the authors build upon multi-objective Bayesian Optimization which allows to obtain good points within a few function evaluations and also enables to obtain a Pareto-Front of non-dominated points. To have an efficient, asynchronous parallelization, the authors propose Believer-Penalizer for choosing the next point while others are still being evaluated. The main idea is to choose either Kriging Believer (KB) and Local Penalization (LP) based on a threshold on the uncertainty in the posterior distribution of the probabilistic surrogate model. In the experiments, the authors show that their package achieves often a fairly good performance.
main_review: ### Correctness and Clarity

Overall, the paper is very well written and structured, and easy to follow. Since it is a system paper, the authors focus more on the system parts and describe the advantages and use cases of the system. The only novel contribution from a methodological point of view is Section 4.2 on the Believer-Penalizer strategy for asynchronous parallel optimization. The straightforward and simple idea makes sense; but the threshold for deciding between KB and KP is not well motivated and alternatives are not evaluated.

### Technical Novelty And Significance

The system itself is well designed and implemented. So, from an engineering point of view the system is well justified. However, the system nevertheless gives me the feeling of “yet another Bayesian Optimization package”; over the last years there were so many of them. AutoOED's selling point is the asynchronous, multi-objective optimization which not many systems actually support. However, here AutoOED mostly implements existing techniques and adds with BP only a minor contribution overall. 

In the appendix, the authors honestly point out that AutoOED only scales well to 2 or 3 objectives. Although I agree that this is a fairly common use case, there are many applications with more objectives. I encourage the authors to look into the field of many-objective optimization.

### Empirical Novelty And Significance

The empirical results are the weak point of the paper. In Section 6.1, the authors compare BP with KB and LP. They claim that ```BP consistently outperforms other methods```. However, Figure 4 shows that BP is not always the best, sometimes it is the worst performing one (dtlz3). A clear gap between BP and the other baselines cannot be found in any of the benchmarks. I think that a fair statement would be that BP is a robust approach and performs well on average.

The same issue applies to the comparison to other packages. The authors claim ```AutoOED is generally stable and ends up either best or comparable on most benchmarks```. However, Figure 5 shows that AutoOED underperforms on 6 out of 12 benchmarks and is only clearly the best system on 4 benchmarks. 

Although I like the benchmark on a real-world benchmark, I have the impression that the benchmark is a bit cooked up and lacks the connection to a real real-world benchmark. Furthermore, AutoOED is only compared against random search on this benchmark. 

I appreciate the ablation study in the appendix. However, this gives me the impression that the results for the comparison for KB vs LP vs BP is a bit cherry-picked and it is even more unclear whether BP performs well on other settings of AutoOED.

### Questions to the Authors

1. How does AutoOED perform on single objective benchmarks?
2. You mention “convergence” of AutoOED twice. How is convergence measured or determined?
3. How can a neural network (MLP) be used for BO since it is not a probabilistic model? (see list of surrogate models in Section 3.2)

### Minor Remarks

I strongly disagree with the following statement:

``` However, they are all targeted for experts in coding without an intuitive GUI, which means that the users need to write hundreds lines of code defining the design space, performance space, surrogate models, acquisition functions, optimization solvers, etc. to finally start the optimization.```

In fact, more and more tools get easier to use and most of the listed tools don’t require hundreds of lines; nevertheless some of them do.

In “Open-source Bayesian optimization platform” I’m missing many more open source BO packages, incl. AX, HyperOpt, SMAC, Scikit-optimize, HyperMapper, OpenBox or Optuna. Some of them even support multi-objective optimization. I encourage the authors to extend this list accordingly.

summary_of_the_review: Overall, AutoOED is a well engineered BO package and it has some strengths and rare features such as multi-objective optimization. However, the novelty is rather minor, there are only little new ideas in the package and the claims in the experiments are clearly overstated.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors present AutoOED, an open-source platform for efficiently optimizing multiobjective problems (MO) with a restricted budget of experiments. The platform automatically guides the design of experiments to be evaluated. AutoOED is
built upon multi-objective Bayesian optimization (MOBO).  To accelerate the optimization in a time-efficient manner, the authors propose a strategy called Believer-Penalizer (BP) that allows batch experiments to be accelerated asynchronously without affecting performance. The authors also provide a graphical user interface (GUI) for users to visualize and guide the experiment design intuitively. Finally, we demonstrate that AutoOED can control and guide real-world hardware experiments in a fully automated way without human intervention. Finally, the authors demonstrate that AutoOED
can control and guide real-world hardware experiments in a fully automated way without human intervention.

main_review: There are several things to like about this paper:

1. The problem studied in this paper is well-motivated. Multi-objective (MO) problems in the context of Optimal Experimental Design (OED) are pervasive. OED in science and engineering often require satisfying several conflicting objectives simultaneously. However, this area of MO-OED has received relatively little attention from the research community.
2.This paper is well-organized and clear written. The authors start from stating the problem formulation. Then, the authors present the algorithm framework for AutoOED.
3. The claims are well supported by four multi-objective tasks on 12 public datasets. 

However, I found that there are several shortcomings:

Minor issues: 
- in general: better to use Pareto optimal solutions rather than Pareto-optimal solutions
- page 5: I think the authors can directly use BP for Believer-Penalizer as it has been specified in previous section
- in general: I think it is better for the authors to double check about the redundant specifications about the abbreviations throughout the paper
- page 9: importantance should be importance
summary_of_the_review: The problem studied in this paper is well-motivated. Multi-objective (MO) problems in the context of Optimal Experimental Design (OED) are pervasive. I like it very much that the authors would open source such a powerful and convenient framework to complement the current state-of-the-art packages.
review_title: null

Meta-review: The reviews are of good quality. The responses by the authors are commendable, but reviewers remain of the opinion that the scientific contribution of the paper is limited, no matter how strong the software engineering contribution may be.
Decision: Reject