Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In the paper, a probabilistic method is proposed for Bayesian optimization transfer learning (or meta-learning). A latent representation is modeled separately for each task. The surrogate used in BO will be approximated by a neural network with its input being the test point and the latent representation of the task. The meta-training procedure optimizes the prediction loss over the neural network weight and the latent representations of the source tasks. Additional regularizer is added to induce smooth latent representations. For a new task, predictions are made on the new data point using the surrogate model g_\theta(x,z) with a fully Bayesian treatment. The p(z|D,\theta) is approximated by Hamiltonian Monte Carlo. In the experiments, the proposed method is compared to random search, GP-BO, and adaptive Bayesian linear regression. 

The latent representation here could be regarded as some model hyperparameter for the neural surrogate, and the p(z|D,\theta) adaptively changes with the including of new data points into the dataset D. The meta-training procedure resembles that of VAE with z being something that can be directly optimized. The experiment shows that the proposed method performs better than the baselines. 

My biggest concern is the novelty of the proposed probabilistic approach. It appears that the framework is very similar to that of Neural processes where the latent representation is conditioned on the observations and the prediction is conditioned on the latent representation and the new input data. I think a very clear distinction between this method and Neural processes needs to be made. To me, the only difference is that the method in this paper proposes to directly optimize over z instead of over a variational posterior q(z|D), which is adopted from Ghosh et al. The Neural processes method is directly applicable to the BO problem, contrary to what it is claimed by the authors as “most of these meta-learning algorithms do not consider active learning as an application”. Therefore, NP should serve as the most relevant baseline to compare with in terms of experiments and methodology. 

Regarding the experiments, I am curious why the proposed method has a very large variance on the most simple task Forrester. Also, I am not sure why the initial performance of ABLR is even worse than GP-BO, which means meta-learning hurts its performance initially. Some explanation or ablations studies would be helpful here. One motivation of the work is faster transfer learning for BO by using the neural model. However, there is no runtime comparison with the baselines. 
Overall, the paper is easy to follow but lacks a direct comparison with NP which most resembles the proposed method. The experiments would be more convincing if more baselines (such as the NPs) could be included. 

Typo: Sec 5.1, strongly "form" any meta-learning model --> from

-------------After author's response--------------

As the other reviewers also pointed out, some baselines are missing and ablation studies on meta-data are missing as well. Meanwhile, my concern is that the presented idea in the paper looks very similar to that of the neural process, except the paper is optimizing over a point estimate of z while NP is optimizing over q(z|D). This remains unexplained in the author's response. Therefore I am keeping my original evaluation at the moment.
review_title: A probabilistic approach for BO meta-learning

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors introduce a technique for multitask Bayesian optimization based on meta learning. In general, there are two broad approaches to multi task learning in Bayesian optimization: sharing information between tasks by modelling the correlation between tasks (e.g., Swersky et al 2013), and sharing information between tasks via global weight sharing across over some embedding (e.g., Perrone et al., 2018). This paper falls in to the latter approach, learning a meta learning style task embedding (eqn. 1) to be able to make predictions over a new task given auxiliary task data (eqn. 2).

Overall, my biggest concern is whether the empirical comparison is as well set up as it could be. In particular, this paper compares primarily to ABLR. However, ABLR avoids comparing to standard multitask Bayesian optimization primarily for scalability reasons, using pooled datasets with tens or hundreds of thousands of examples for transfer learning. In this paper, as far as I can tell, each individual task in section 5.2 is limited to 128 and 512 examples respectively. The total dataset size here is significantly smaller, which (1) does not match the setting ABLR was originally used for, and more importantly (2) is well within reach of multitask BO.

First, before jumping in to specific details on this, let me admit outright that some details of the experimental setup in Section 5.2 are simply not clear to me. You say "For the meta training data, we sampled 128, and 512 points for GLMNET and Ranger respectively using the surrogate model." Does this mean you had 32 (number of meta datasets) * (128 data points) =4096 total hyperparameter observations before starting observation, or 128 total? I'm assuming the former? Or did you use the full set of hyperparameter evaluations? I also wasn't clear on what was meant by "Instead of validating on unseen tasks with unseen
hyperparameter settings, we reuse the training tasks, but with an unseen set of x." Does this refer just to the mechanism for setting BaNNER hyperparameters and not to how test tasks / data sets were selected, or were training tasks actually reused as test tasks?

If my assumptions about the empirical set up of the paper are correct (e.g., 4096 and 16384 meta examples), then my biggest issue is whether the playing field that ABLR was compared to was level. Part of the reason ABLR avoids comparing to the "directly learn correlations across tasks" style of BayesOpt that uses multitask kernels is an emphasis on scalability. In their OpenML tasks for warm start on SVM and XGBoost hyperparameter optimization, their meta dataset contains 6.5x10^4 and 5.9x10^5 data points respectively. In this paper, it seems as though the meta datasets are order(s) of magnitude smaller. This may not be the most efficient regime for ABLR, which can rely on larger feature extractors with more total pooled data.

Now, there's an argument to be made for a data efficiency improvement over ABLR (requiring less overall pooled data), but in these settings with on the order of thousands of pooled data points, it becomes trickier in my opinion to justify not comparing to standard multitask Bayesian optimization using multitask Gaussian processes, which works quite well in the data sparse regime and will have no trouble scaling to 32 tasks with 128-512 examples each.

Some of the standard error plotting (Forrester ensemble, GLMNET1471 in the main text, much of figure 8 in the supplementary materials) is clearly broken for some methods. Is this a bug in plotting or actually large standard errors spanning multiple orders of magnitude despite being over 256 trials? Additionally, the y axes on some plots (e.g., Figure 8 bottom row, Figure 9 bottom row) do not have appropriate scales.
review_title: Comparisons to standard Multitask BO in a relatively smaller data setting?

final_score: rating: 4: Ok but not good enough - rejection
review_content: SUMMARY OF REVIEW

This paper proposes a probabilistic meta-learning algorithm to establish an informative prior to be used for Bayesian optimization (BO). 

The technical contribution lies in the development of the probabilistic meta-learning algorithm instead of innovations made to BO's acquisition function: nearly the entire technical formulation in Section 4 pertains to meta-learning instead of BO. One may argue that this is a probabilistic meta-learning paper in "disguise".

If the aim of this paper is to set up an informative prior, it would be necessary to produce empirical comparisons with the state-of-the-art probabilistic meta-learning algorithms in terms of probabilistic predictions, which is lacking in this paper. In this regard, it would be necessary to perform experiments on higher-dimensional tasks/problems.

One drawback of their proposed work in the context of BO appears to be that unlike some existing meta-BO algorithms, it cannot exploit the GP posterior means and variances when made available from previous BO tasks due to equation 1.

There are a number of major concerns about the technical formulation and the experimental setup and results, as detailed below.


DETAILED COMMENTS

Page 3: The authors say that "Most of these meta-learning algorithms do not consider active learning as an application and focus on deterministic models g_theta." Exactly which probabilistic meta-learning algorithms in the same paragraph as this claim cannot be used for providing a probabilistic prior? Do existing probabilistic meta-learning algorithms not satisfy the requirements of data-efficiency and well-calibrated uncertainty estimates?

Pages 3 & 4: The authors have placed a prior on latent task embedding z but did not seem to consistently apply the Bayesian treatment on z: The task embedding z_t for each task tau_t is optimized using equation 1 without a Bayesian treatment, while the inference or adaptation (equation 2) applies the Bayesian treatment on z. Can the authors discuss and justify this inconsistency? How does it affect the choice of the optimized theta?

Equation 2: If g_theta approximates f, then should the posterior belief of g_theta(x) be an expectation of g_theta(x,z)? Furthermore, g_theta(x) is not defined.

Page 5: The authors say that "we use BOHB's multi-fidelity capabilities and scale the number of meta-training iterations and validation tasks used to compute the validation loss with the fidelity." How is the fidelity exactly determined? Are all the hyperparameters optimized prior to BO? Some hyperparameters need to updated as more BO samples are gathered in every BO iteration. How are these dealt with?

Page 6: The authors say that "We do not consider approaches like stacked GPs (Golovin et al., 2017) and weighted GP ensembles (Feurer et al., 2018) that do not scale to the large meta-data sets we consider." This lack of comparison, especially with the weighted GP ensembles approach, is not well-justified in the context of BO where it is costly to evaluate the unknown objective functions and hence we do not expect large number of samples. The authors are encouraged to perform this empirical comparison.

Page 7: What is the reason for introducing BaNNER-BLR and when would BaNNER-BLR be expected to perform better? The authors have discussed this only briefly in the second last paragraph of Section 5.2.

Figure 3: The fundamental difference between Algorithm 1 vs. GPBO lies in the former exploiting an informative prior based on probabilistic meta-learning (albeit not containing near-optimal points for meta-training) while the latter utilizes an uninformative prior. It is surprising to see that their proposed Algorithm 1 is not being able to undo the effects of the informative prior and hence performing worse than GPBO with an uninformative prior. The authors' justification in the second last paragraph of Section 5.1 was not able to allay my above concern. The use of an informative prior hurts in this case.

Appendix B: The authors seem to have used a large number of meta tasks T=256 and for some synthetic ensembles, a large number of datapoints N_t=1024 per task. In meta-learning, this seems fine. But, in BO tasks where functions are expected to be expensive to evaluate (Section 2), these numbers do not seem practical or reasonable. Can the authors present results with varying, smaller T and N_t?

The authors should consider performing empirical evaluation on hyperparameter optimization of the larger-scale CNNs, which is commonly seen in BO works.


Minor quibbles

Page 2: The *most* data-efficient class of algorithms for this setting are Bayesian optimization (BO) algorithms... Isn't this too strong a claim to make?

Page 3: adaption

Page 6: form

Figure 3: Hardmann, Ouadratic

Page 7: deliberately deliberately 

Page 8: verfy

Page 8: datatsets
review_title: Probabilistic meta-learning paper in "disguise"?

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes BaNNER, a new transfer learning approach to accelerate hyperparameter optimization based on related tasks. This is based on a generative model learning the data and task distributions. A number of experiments against standard BO and random search, as well as against ABLR, the most closely related transfer learning baseline, indicate that the proposed approach tends to find a good hyperparameter configuration more quickly.

Positives 

1. **Significance.** The paper tackles the cold-start problem in BO by leveraging related tasks. This is a highly relevant problem that fits an active line of research. The paper makes a step forward in this direction. 

2. **Reproducibility.** The experimental evaluation is very well detailed, with clearly defined error bars and a large number of repetitions (i.e., 256). Implementations of the models and code to reproduce the experiments will also be made evaluable, which will further foster reproducibility. 

3. **Clarity**. The paper is generally well-motivated, clear and easy to follow. 

Negatives

1. **Missing baselines.** The key concern with the current version of the paper is that more transfer learning baselines would be needed to fully justify the proposed method. Beyond ABLR, a number of additional follow-up and prior works have addressed the transfer learning problem. Many of these are indeed referenced in the related work, such as the initialization strategies to warm-start BO. It is true that these are different as they change the BO algorithm and not the underlying probabilistic model. Nonetheless, this is not a real reason not to compare against at least some of them. Without these comparisons, it is not clear if BaNNER should be the transfer learning method of choice, considering the many alternatives. One could argue that the proposed approach and warm-starting could be seen as complementary, but the authors would need to show in experiments whether there are mutual benefits from combining them or not. The same considerations hold for the following transfer learning HPO methods which prune or construct a search space based on previous related tasks:   
[Wistuba, et al.: Hyperparameter search space pruning–a new component for sequential model-based hyperparameter optimization. In Machine Learning and Knowledge Discovery in Databases, 2015]
[Perrone, et al.: Learning Search spaces for Bayesian optimization: another view of hyperparameter transfer learning. In NeurIPS, 2019.  ]

There are also some more recent closely related baselines that are not compared against nor discussed in the related work. For instance, the following work achieves transfer learning while scaling well in the number of hyperparameter evaluations:  
[Salinas, et al.: A Quantile-based Approach for Hyperparameter Transfer Learning. In ICML, 2020.]

  Lastly, while it is true that approaches like stacked GPs (Golovin et al., 2017) and weighted GP ensembles (Feurer et al., 2018) do not scale well to large datasets, they scale cubically only in the number of evaluations per task (as opposed to the total number of evaluations across tasks). In the considered experiments this is not necessarily unfeasible. To be comprehensive, one could also subsample the number of data points per task to make these baselines scale and be able to compare against them. It might be the case the using more datapoints will help BaNNER, but this needs to be shown; otherwise, the advantage of scaling to a large number of past evaluations remains theoretical but not demonstrated in practice. Varying the number of evaluations per task could also be seen as a useful ablation study (point 3 below).

2. **Non-challenging problems**. Apart from synthetic examples, the authors consider tuning generalized linear models and random forests with a low-dimensional hyperparameter space. While these are insightful experiments, most approaches for transfer learning in BO are also evaluated on more high-dimensional and ambitious tuning problems, including neural networks tuning/NAS problems. 

3. **Not enough ablation studies**. The paper focuses on useful performance comparisons but does not have enough ablation studies. These would give further insights into the proposed method. For instance, it is not clear how performance is affected by changing the number of hyperparameter evaluations available from related tasks. Also, it is not clear how robust BaNNER is to noisy or unrelated tasks. Without these empirical studies, one only gets a shallow understanding of the proposed approach.


Other comments

a. Figure 4 shows some specific datasets per ML method. I appreciate that all results are reported in the appendix. However, ABLR outperforms BaNNER on some datasets and several other datasets lead to very noisy results. To give a sense of overall performance, I would advise the authors to also report global metrics. One possibility is ranking different methods on each dataset and averaging the ranks (to account for heterogeneous loss values).

b. The authors report that ABLR performs best on a few example problems, such as GLMNET1489 and Ranger1485, and conjecture that this is because 1485 is a large dataset with many features and 1489 is more imbalanced. It is not clear to me why dataset dimension and class imbalance should favor ABLR over BaNNER. 

c. The method comes with a number of hyper-hyperparameters, which the authors learn on meta-data by running an additional BO loop powered by BOHB. From the perspective of deploying BaNNER, is this not too cumbersome in practice? Do the authors recommend a set of (robust) defaults for their method to be used off-the-shelf or is an additional BO loop always required?

Overall, I vote for rejection for the current version of the paper. The proposed idea is sound, promising and well-motivated, but the method is evaluated against a very limited selection of baselines and it lacks ablation studies. It is not clear with the current experiments how the BaNNER's performance is placed within the context of general transfer learning methods in the literature. I am ready to change my score if the authors compare to a wider range of baselines.

Minor:
1. Some in-line references are not in the correct format (e.g., "We refer to (Vanschoren, 2018)").
2. Typos in this sentence at page 3: "where we deliberately deliberately does not contain near-optimal points for meta-training".
review_title: The paper introduces a new approach to accelerate HPO via transfer learning, but a wider range of baselines are needed to demonstrate its benefits.

Meta-review: This paper propose an approach to probabilistic meta-learning for Bayesian optimization. The goal is to accelerate Bayesian optimization under the assumption that multiple related tasks require optimization. In terms of strengths, the paper addresses an important problem as it has applications to efficiently optimizing hyper-parameters over multiple related data sets or models. In terms of weaknesses, the proposed approach is closely related to neural processes, but this connection was not made in the original paper. The authors were unfortunately not able to provide additional insights or results regarding this point during the limited discussion period and as a result, the novelty of the method at the core of this approach is in question. The reviewers also note that the experimental designs and comparisons performed are limited. For some smaller problems, more standard baseline methods like multitask GPs should be applied. The authors have also not compared to a number of other recent methods aimed at scalable transfer learning for hyper-parameter optimization, as detailed in the comments of Reviewer 1. The reviewers agree that the paper is not yet ready for publication.
Decision: Reject