Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: &nbsp;

The authors propose a novel Poisson process-based scheme for rank-based Bayesian optimization and investigate its performance on HPO and NAS tasks.

&nbsp;

overall_review: &nbsp;

**__MAJOR POINTS__**

&nbsp;

1. In terms of the MLP used to approximate the scaled rate parameter of the inhomogeneous Poisson process, a crucial question is whether the performance of PoPBO is robust to the training scheme/architecture of the MLP. Do the authors believe that a single fixed MLP architecture and training regime will remain performant across a broad range of problems or will the architecture have to be tuned? If so how will this tuning occur on each iteration of Bayesian optimization?

2. It does not seem as though HEBO is run on 2-D Branin in Figure 2a?

3. Is there a reason why HEBO and PPBO are not run on the HPO-Bench and NAS-Bench benchmarks?

&nbsp;

**__MINOR POINTS__**

&nbsp;

1. In section 1, when discussing the literature on preference BO, it may be worth referencing the recently introduced BOPE algorithm [2].

2. It would be great if ADAM was cited [3] given that it is used although one might think the paper has enough citations already :)

3. For the empirical evaluation, the standard error could be plotted in place of the standard deviation, this may make the trend in the empirical results visually clearer due to non-overlapping error bars.

4. In Figure 3, what are the units of time cost? Are the reported values wall-clock times or CPU times?

5. In Table 2, it might be misleading to imply that random search was invented in 2012!

6. In Figure 10, the plots do not seem to verify the claims made by the authors that PoPBO outperforms GP-BO?

7. In the algorithmic description the authors say that they optimize the MLP with SGD, however the experimental settings say that they use ADAM?

8. Line 256, it would be great if the benchmarks on which BOHB and BOHAMIANN are state-of-the-art is provided if they are described as such. (This is just a matter of taste relating to the literal definition of state-of-the-art!)

9. Where are the error bands for Figure 5?

&nbsp;

**__REFERENCES__**

&nbsp;

[1] Cowen-Rivers et al., [HEBO: Pushing the Limits of Sample-Efficient Hyperparameter Optimization](https://www.jair.org/index.php/jair/article/view/13643), Journal of Artificial Intelligence Research, 2022.

[2] Lin et al., [Bayesian Optimization with Preference Exploration](https://proceedings.mlr.press/v151/jerry-lin22a.html), AISTATS, 2022.

[3] Kingma and Ba, [ADAM: An Adaptive Optimizer](https://arxiv.org/pdf/1412.6980.pdf), ICLR 2014.

&nbsp;
review_summary: &nbsp;

The authors introduce a novel, rank-based Bayesian optimization scheme. The research arc is an important one for AutoML and the empirical evaluation (modulo the points/questions raised above) is solid. Furthermore, the paper is well-written. As such, I recommend acceptance with the scope for further increasing my score to strong accept if the outstanding points are addressed.

&nbsp;
actions_required_to_increase_overall_recommendation: &nbsp;

If the authors can address the points raised with regard to the empirical evaluation i.e. offer their thoughts on schemes for stabilizing MLP training as components of Bayesian optimization loops, supply a working link to the codebase, and explain why the PPBO and HEBO baselines are absent in some of the experiments I will be happy to increase my score from weak accept to strong accept.

&nbsp;
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The authors present a novel approach towards preferential Bayesian Optimization dubbed Poisson Process Bayesian Optimization (PoPBO). It is applicable for "relative" response surfaces, i.e. situations in which preferetial comparisons between candidates are available but not necissarily the concrete numerical values of the objective function under consideration. It relies on the assumption of a non-homogenous Poisson process. The authors introduce two acquisition functions, namely the Rectified Lower Confidence Bound (R-LCB) and Expected Ranking Improvement (ERI).
overall_review: Positive:
- The paper addresses an interesting problem and proposes a novel approach towards Bayesian Optimization with preference feedback. This potentially has a broad impact on AutoML as well as BO in general.
- The approach is empirically evaluated both on simulated and real-world benchmarks, including HPO and NAS. It shows competitive performance in terms of minimum regret with the considered baselines. The experiments seem to be conducted correctly and the results back the authors claims.
- In addition to the competitive performance, the Poisson process based BO surrogate model procedure only has a computational complexity of O(N²) for fitting, unlike the GP which has cubic runtime. 

Negative:
- The paper is very convoluted and packed with formal details. It is very difficult to comprehend it.
review_summary: The paper addresses an interesting problem, i.e. Bayesian Optimization when only preference feedback can be acquired. While it is difficult to comprehend the paper due to it's convoluted formal explanations, the derived approach seems sound to me. The experimental evaluation shows competitive performance both on simulated as well as real-world benchmarks.
actions_required_to_increase_overall_recommendation: - The first sentence of the abstract should not list SMAC as a surrogate model but rather the random forest used by SMAC
- There are some linguistical problems:
      - Capitalization, e.g. "Random Forest" vs. "random forests"
      - Punctuation
      - Missing use of indefinite and definite articles
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: This paper falls into the scope of preferential Bayesian optimization and it attempts to capture the relative performances instead of absolute performances. It describes a new Bayesian optimization framework called Poisson Process Bayesian Optimization (PoPBO). Authors proposed to directly fit the ranking based response surface based on Poisson process. This is unlike existing work, which tries to fit the absolute response surface first and then obtain the relative response surface. The proposed method is robust to noise. Besides, this paper also proposed two acquisition functions named R-LCB and ERI. These acquisition functions are derived from the vanilla LCB and EI for better exploitation-exploration trade-off. The computational complexity of PoPBO is much lower than that of GP-BO. The method outperforms many prior BO methods on both simulated functions and real-world benchmarks, including HPO and NAS.


overall_review: Strengths:

- The paper addresses an important problem: black-box optimization with only relative rankings of candidates. Absolute values can be hard to obtain in some cases. Besides, building absolute surface is not robust of noise. 
- This paper discuss limitations of existing work in detail and proposed a framework to mitigate these limitations. 

Weaknesses:
- The effect of parameter q in eq(7) is not discussed in detail. From fig.5, the q parameter has a huge influence to the final performance. In practice, we cannot try a list of different q parameters as the underline functions is expensive. That means, we will never be able to inspect the influence of parameter q. How could one set it correctly? 
- Continuing the above point. The influence of q is only discussed for a single synthetic function. Given this parameter is so important, without discussing it extensively, the reviewer will not be able to increase the score. 
review_summary: This paper addresses an important problem: black-box optimization with only relative rankings of candidates. The proposed method outperforms many prior BO methods on both simulated functions and real-world benchmarks, including HPO and NAS. This means that the PoPBO framework can be used to optimize hyperparameters and neural architecture search in AutoML more efficiently than previous methods. 

However the authors did not discuss the parameter q in R-LCB, this makes it difficult to justify the method's influence as it seems the performance is very sensitive to this parameter. 
actions_required_to_increase_overall_recommendation: 1. perform statistical test for the experimental results. It is hard to tell whether the proposed method is significantly better than other methods through mean plot. I would also suggest using median plot. 
2. discuss influence of parameter q for different functions.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This work suggests a ranking-based surrogate model using Poisson processes.  Instead of Gaussian processes or other surrogate models, it determines relative ordering of historical queries and make use of such ordering in next query selection.  The authors provide a new form of acquisition functions for the ranking-based surrogate model.  Finally the authors demonstrate experimental results with baseline methods as well as the proposed method.
overall_review: I think that this paper is quite interesting and novel.  However, I want to leave some comments.

* Could you elaborate on the contents in Section 3.2?  How does it estimate ranking and how does it work?

* Following the above questions, the authors should revise Section 3 of the paper.

* I am curious why multi-layer perceptron is used to estimate $\lambda_{\xi}$.  Did you try other models?

* Is there any guarantee that machine learning models are capable of estimating $\lambda_{\xi}$?

* Moreover, I would like to hear the implication of Equation (3), (4), and (5).

* Could you add a description of why $\sigma(x) = \sqrt{\mu(x)}$?

* I think that you missed the recent studies of the line of TPEs (i.e., likelihood-free Bayesian optimization).  You need to refer to [1, 2].

* I think you need to be consistent for a scale of y-axis. Some figures have a logarithmic scale, but the others are not.

[1] Tiao, Louis C., et al. "BORE: Bayesian optimization by density-ratio estimation." International Conference on Machine Learning, 2021.

[2] Song, Jiaming, et al. "A general recipe for likelihood-free Bayesian optimization." International Conference on Machine Learning, 2022.
review_summary: Please see the text boxes above.
actions_required_to_increase_overall_recommendation: I am now leaning towards a positive score.  But, if my questions will be resolved, I will increase my score more.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: - A ranking-based surrogate model  in the form of the Poisson processes, to enable BO using only the relative ranking of observations, as opposed to their (possibly unattainable) values. The Poisson processes model is approximated through an MLP 
- The adaptation of UCB and EI, and their gradients, to the proposed ranking-based model
- A resulting BO framework that is computationally cheaper $(\mathcal{O}(n^2)$ than the traditional $(\mathcal{O}(n^3)$
- Empirical results
overall_review: _Since the concerns are outlined in greater detail above, this section will be shorter._


### Pros
- __Model is novel:__ Poisson processes are a new type of model that inherently supports ranking-based data
- __The approach is straight-forward__: The approach is relatively simple and generally well-communicated.

### Cons
- __Acquisition functions have multiple hyperparameters__ though these should debatably be out-of-the-box (and thus not a significant contribution). Adding hyperparameters is seldom a good thing, and EI/UCB should not need additional ones given their principled foundations.
- __Introducing an MLP as a novel model type is not very novel.__ The nuances of the model training is not further explored in the paper.
- __The quality of the proposed model is not thoroughly assessed__ in terms of prediction ability, e.g. validation set log-likelihood. The first plot indicates that it predicts well, but that is all there is to assess the quality of the model's predictive capabilities. 
- __The noise evaluation is lackluster__ and does not benchmark against the proper methods. GP with sufficient noise modelling (what are the noise assumptions/prior in the GP?) and Noisy EI.
- __Preferential BO literature is not benchmarked against__, at all. This should be the most relevant comparison of all.

As such, this paper should not be accepted in its current format. Some premises (i.e. that GPs are not robust to noise) are unconventional, and something I personally disagree with. Discounting the rectified acquisition functions, the novelty is not very high. Moreover, there are too many outstanding questions with regard to the performance of the approach relative to its competition across several relevant dimensions, as outlined above.

review_summary: The paper presents Poisson processes, a novel tool for ranking-based Bayesian optimization. 

### Pros
- Poisson processes are a new type of model and a good idea
- The approach is straight-forward and generally well-communicated

### Cons
- Acquisition functions should be out-of-the-box
- Model is not thoroughly assessed
- The noise robustness theory and evaluation is lackluster 
- It does not benchmark against the relevant Preferential BO literature
- Introducing an MLP as a novel model type is, arguably, not very novel.

This paper should not be accepted in its current format. The novelty is not very high and there are severe outstanding questions regarding performance.
actions_required_to_increase_overall_recommendation: I will consider raising my scores if the concerns regarding the experimental setup and lack of relevant benchmarks are addressed. These have all been outlined above.
review_title: null

Decision: Accept(Main Track)