Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work proposes a likelihood-free Bayesian optimization strategy with a meta-learning scheme. Given multiple tasks, it trains task-agnostic and task-specific components in order to predict a probability that measures how likely a solution is, inspired by BORE and LFBO. By utilizing Bayesian logistic regression, it determines a query point. In addition, the authors use a gradient boosting model to predict a residual of the model. Eventually, the experiments demonstrate the effectiveness of the proposed method, compared to other existing methods.
strength_and_weaknesses: Here I describe the strengths and weaknesses of this paper.

### Strengths

It solves an interesting topic where multiple historical tasks are given, by applying a likelihood-free framework.

The proposed method is quite novel. In particular, a combination of some components such as mean prediction layer, residual prediction layer, and gradient boosting is interesting.

The purpose of the respective components and the respective loss functions is well-described.

### Weaknesses

Presentation and writing can be improved. The current version is okay, but I think it can be polished more.

Iteration budgets for the experiments are too low. I think that you should give a larger iteration budget in order to show the convergence of the algorithm tested.
clarity,_quality,_novelty_and_reproducibility: ### Questions

* For the experimental results, the variance (or standard deviation) of the experiments (i.e., shaded regions) is somewhat odd. Since $y$-axis is a log scale, the variance at the last of iterations should be larger than the variance at the beginning of iterations.

* A feature extractor can be called as ResNet? According to Appendix C, your feature extractor is different with ResNet. I think that the name should be changed.

* I would like to ask about gradient boosting. Which gradient boosting is used in this paper?

* Following the above question, if you use gradient descent, how did you optimize a function $C$ (Line 10 or Line 12 of Algorithm 1)?

* In the experiments, three proposed method, i.e., MALIBO wo GB, GB, and GB-TS, are tested. Does MALIBO wo GB include TS? Why did not you test MALIBO wo GB w TS or MALIBO wo GB wo TS?

### Minor issues

* In Page 4, $\max(y - \tau)$ seems like a typo; please fix it.

* In the caption of Figure 2, $x_t$ should be $\boldsymbol x_t$.
summary_of_the_review: Please see the above text boxes.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a meta-learning method for likelihood-free Bayesian optimization. In particular, the proposed approach is a combination of two existing solutions, a meta-learning approach from BaNNER and a likelihood-free LFBO, together with the gradient boosting method. The proposed solution is able to work with high-dimensional inputs and handle heterogeneous scales and noises across different tasks.

strength_and_weaknesses: The main strength of the paper is in the empirical performance which is shown to outperform two of the existing works, ABLR, and GC3P. However, it has several weaknesses as follows.

1. The technical solution is not novel since it is based mostly on the two existing works BaNNER and LFBO.

2. Although the paper claims that the meta-learned classifier can balance between exploration and exploitation, the proposed approach requires a gradient boost method to correct the errors. It means that the meta-learned classifier does not allow exploration properly. This is unlike multi-task GP, where the correlation between tasks can be learned from the data without any additional ad-hoc method (such as gradient boosting) for correction.

3. While Thompson sampling often relies on a good approximation of the posterior distribution, the proposed approach only uses the Laplace method to approximate the posterior distribution which is a very simple approximation method. Better approximation methods such as variants of MCMC and/or variational inference techniques should be applied.

4. The numerical representation of the task (z) is simply optimized to be close to a standard (multivariate) normal distribution. I am wondering if the proposed method can work with a multi-modal task distribution, i.e., prior tasks form 2 clusters where tasks in a cluster are similar and tasks between clusters are different. In a principled Bayesian approach such as multi-task GP, it can correctly correlate a new task with an existing task.

5. It is unclear to me about the choice of \tau (for prior tasks and the current task) in the proposed algorithm.

6. While the paper reviews a lot of related works on meta-learning for BO, the experiments only consist of 2 existing works while ignoring the others such as those using GP surrogates (e.g., running experiments on low-dimensional problems).

7. Section 3.1 lacks many well-known BO methods such as GP-UCB, predictive entropy search, max-value entropy search, and knowledge gradient-based methods.

8. Since BO is about sample efficiency, can gradient boosting work reasonably well with a small training dataset?
clarity,_quality,_novelty_and_reproducibility: As discussed above, the novelty of this paper is limited since it is based mostly on the two existing works BaNNER and LFBO. I also have several concerns about the proposed solution as elaborated in the above weaknesses. Regarding clarity, as the Laplace approximation is a well-known technique, the paper can reduce the explanation of the Laplace approximation to include more explanation on the regularization (to learn the task representation), the gradient boosting, and the choice of \tau.
summary_of_the_review: Due to the limited novelty, concerns about the proposed solution, and the lack of baselines in experiments, the paper may require further improvements to fit the conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of meta Bayesian optimization (BO), which aims to warm-start the BO process by exploiting knowledge from related tasks. In this paper, the authors propose warm-starting the acquisition function, which takes the form of a classifier in the likelihood-free BO setting. Gradient boosting is further incorporated to combat distributional shifts. 
strength_and_weaknesses: Strengths:

(+) The problem of (meta) Bayesian optimization is an important problem with several applications.

Weaknesses:

(-) The comparison with existing works is lacking, both in the techniques and in the empirical evaluations.

(-) The lack of a theoretical guarantee (which is common in the BO literature).

(-) The writing can be improved (as detailed below).  

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is generally hard to follow, many concepts require prior knowledge and are not sufficiently explained in the paper. More detail below.

Novelty: The novelty aspect is low as a similar idea has been explored by Volpp et al. (2020). The proposed meta-learning framework is almost a direct application of Berkenkamp et al. (2021).

Reproducibility: There is no code included with the submission.

summary_of_the_review: The paper presents an interesting idea to tackle an important problem. However, the difficulty or challenges of the problem are not highlighted enough. The comparison with related works is lacking. And the writing can be improved. 

Detailed comments:

- The idea of meta-learning an acquisition function has been explored by Volpp et al. (2020). Could the authors elaborate on the differences and advantages of the proposed algorithm to that of Volpp et al. (2020)? In addition, this should also be included as a baseline in the experiment section.

- The paper is generally hard to follow since the readers assume knowledge from several other key papers. I have to read several other works on the BORE framework and BaNNER to understand several parts of the paper. I encourage the authors to make the paper more self-contained by reintroducing concepts from essential related works like Berkenkamp et al. (2021), Tiao et al. (2021), and Song et al. (2022).

- When the input space is a simple Euclidean space without any structure (think of minimizing the function $f(x) = x^2 – 4x$), how does the feature embedding $h(\cdot)$ works? I suppose feature embedding is only useful when there is some structure in the inputs (e.g., images).

- Since BO is a black-box optimization algorithm, can the authors indicate more clearly what are the objective functions that we are optimizing in the experiment section? It is difficult for readers to comprehend and assess results without knowing what are we optimizing.

- From my understanding, UCB is one of the most commonly used acquisition functions, with a nice theory in BO. However, the paper did not mention or compare with UCB. Is there a good reason for that? Can the authors compare the proposed algorithm with GP-based BO methods with EI/PI and UCB?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new approach to perform transfer-learning for Hyperparameter optimization where offline evaluations are used to fasten the tuning on a new task. The method proposes to leverage recent work that uses classification to train surrogates by learning to classify good/bad configuration. In particular in this work, the classifier takes a combination of global (task agnostic) and local (task specific) features and is learned with a Bayesian Logistic Regression layer which is made possible with the use of several approximations. 
In addition, a gradient boosting approach is used to improve the final fit of the model at each iterations. Experiments are conducted on several tabular benchmarking suites  (hpobench, mlbench) and also artificial examples to study the effect of noise levels. The method proposed is show to be competitive or better than the baselines proposed.
strength_and_weaknesses: The strengths of the paper are:
* tackle a very relevant and impactful problem
* provides a novel technical contributions with several ideas that could be leveraged in future work (refinement with gradient-boosting, use of EI equivalent likelihood-free criterion)
* good coverage of experiments: a reasonable range of benchmarks/different blackboxes are considered 

Weaknesses:
* lack of runtime analysis: no runtime is given for how long the method takes to return suggestion (the appendix mentions 2048 epochs on top of that, a gradient boosting tree is fitted). However, this can significantly worsen the results if they were reported against wallclock time (as proposed in other works) since then almost all time will be spent in fitting models and the method would likely underperform random-search. Without additional details, one cannot assess if the method will have any practical relevance (a method taking 5 minute to suggest the next candidate will likely not be very applicable).
* the set of baselines is relatively considered is relatively small (only 2 transfer learning and 2 non transfer baselines). They are several baselines that could be easily added: BORE for non transfer, but also BORE/LFBO with search space pruned to bounding box of best previous evaluations [Peronne 2019]. Ideally, another additional transfer-learning baseline on top of this simple bounding-box approach would also be added in order to better assess the quality of the method regarding state-of-the-art (2 methods for transfer is a small number).
* details are lacking, some part of the methods were not clear to me:
  * 4.1: was difficult to get through while it is a key part of the paper, I would suggest adding the dimensions for the different variables and give the final expression of the classifier that sums its two input (which is given in 4.2). In any case, the text alone was not sufficient for me to get exactly how $\phi \dot z_t$ is obtained as I could see several ways to achieve this depending on input dimensions.
  * 4.2: the expression of L^LFBO given in Eq. (3) does not take arguments as input
  * 4.3: how exactly is the Gradient boosting combined with the classifier was not entirely clear to me. I would recommend to write it down formally rather than with words (given the current text description, there could be many ways on how a GB would be fitted to reduce the residuals)
* the set of ablations is small and several complexity are not justified (for instance the approximation done instead of the direct optimization of (3), see additional details for a bigger description on this.
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper is generally OK except for some details missing and some notations missing (should be easy to fix).
The quality of the paper is good although it could be improve by adding more baselines and ablations to show better that the method actually improves state-of-the-art. Finally, the paper provides an original contribution and references well previous work.

summary_of_the_review: The paper provides an interesting and novel method to a relevant problem with several ingredients that may be leveraged by future work (for instance using GB to improve surrogate quality). However, as it stands the experimental section has only too little baselines to really its performance against state-of-the-art (only 2 baselines for transfer) and some part of the models are unclear. I believe those could be potentially addressed before camera ready.

Additional details:
* many typos, it would be valuable to run a spell-check:
  * meta-leared
  * observaitons
  * This benchmarks
  * Coppola
  * ensmeble
* Eq (3): interestingly, the scale is back in the loss (but the surrogate dont have to predict values proportional to it)
* A potential ablation would be to use BORE to illustrate the benefit of using LFBO in your case, other possible ablations I was wondering about were: using only local/global part of the model, fitting (3) directly as mentioned in the begining of 4.2. The latter would be important to have given that otherwise all the complexity of the approximations (laplace and logits) would not be justified.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper is proposing a Bayesian optimization method, where the acquisition function is given via a classifier separating good and bad evaluations (observations split along a percentile), an idea introduced in TPE, generalized in BORE and other work. It makes use of transfer from previous tasks, again close to earlier related work.

As reviewers point out, there are some novel ideas here, for example the separation into task agnostic and specific parts to the surrogate, and the use of xgboost as a non-parametric representation of residuals. However, given previous work, these novelties are not enough to justify acceptance. The empirical evaluation is also somewhat limited, missing a number of baselines.

Short of a novelty that tackles a major limitation of prior work (e.g., in the high-dim BO context, or conditional spaces, like the original TPE tried to do), this work could become stronger by tackling harder real-world problems.

justification_for_why_not_higher_score: See reviews. The main issue is lack of novelty, given previous work. While there is one new idea (residual correction by way of xgboost), this is too little to justify acceptance.
Decision: Reject