Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper:  HyperBO assumes the tasks are independent given the hyperparameters, unlike typical metalearning approaches which assume tasks are related. This allows for an efficient Kronecker decomposition of the kernel and thus linear, rather than cubic scaling, across tasks. 

Using this model, HyperBO performs BO as usual; maximize the acquisition function to obtain the next point to evaluate. HyperBO also makes the critical assumption of an offline pre-training of hyperparameters on a representative set of completed tasks; during optimization itself the hyperparameters are fixed. 

main_review: I have a few key concerns about this paper. 

- Why fixed hyperparameters? This is clearly the bottleneck of Metalearned BO, and if these hyperparameters are learned offline, this seems to (A) somewhat eliminate the strength of HyperBO which is the linear scaling per task ---obviously this still helps significantly during the offline training, but still a point of concern of mine, and (B) seems not robust, especially if the set of representative completed tasks is heavily biased. 

- HyperBO, in the experiments, uses the PI acquisition function. Is there a particular reason why this is? PI is quite greedy (even more than EI), so is there any intuition as to why PI is appropriate in this situation. 

- In Figure 2b, I am somewhat concerned about the empirical performance of HyperBO. Though it beats the baselines, it does so in a 4D search space, using thousands of tasks; this seems like overkill. The error bars are also all over the place. This is somewhat unfair of me to ask for I admit, but I am curious if a much simpler approach involving restricting the search space (given that it is fixed) will help (see the paper “Learning search spaces for Bayesian optimization, Perrone et al., 2019). I feel like there is definitely enough data for this to make a difference. 

- Also, the experiments only really concern one optimization problem involving optimizer hyperparameters. Though this one experiment is quite impressive in terms of the data involved, iit would be nice to see another experiment (say for tasks that might be easier like tuning a random forest). 

summary_of_the_review: I have some concerns about the assumptions used in the methodology, as well as the experiments, which leave a number of open questions. In particular, the fixing of GP hypers seems to largely remove the need for scaling, which is the primary strength of HyperBO. Furthermore, though the experimental set up uses a large amount of data to achieve somewhat unconvincing results in my mind, and only one optimization problem is presented (though worth noting, is thoroughly analyzed). Thus, I can't recommend acceptance at the time.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a Bayesian optimization method based on meta-BO. The motivation is tasks can share the same parameter structure and this shared information, e.g. correlation between tasks, can be transferred to new and similar tasks. An example is to optimize the the hyper-parameters of a same optimizer across different architectures and different datasets. This problem is a very important one in the community of Bayesian optimization and a reasonable method can lead to a potentially dramatic decrease in the required computation, especially when the objective function is very expensive. This work tries to overcome limitations of existing methods. For example, the method proposed in this work does not need to evaluate all objective functions associated with all tasks on the same parameters.
main_review: The reviewer appreciates the authors putting effort into the empirical evaluation of the proposed method. However, the proposed approach is not interesting to the Bayesian optimization community and is trivial to some degree. The reviewer believes that the targeting problem presented in this work is a very important one and an effective method could be of great practical value. 

In the abstract, authors claim that "data from similar functions" could lead to a better prior for GP. Obviously a better prior for GP is desirable and that is why the marginal likelihood is used to optimize  parameters of a GP. From such a claim, it is expected that an efficient method for BO will be presented by exploring novel similarities between tasks. However, throughout this paper, there is no definition of a similarity between tasks and tasks are treated as independent. This raises my concern on this work's novelty, which is my biggest concern on this paper.  

Authors claim that the critical difference between this work and standard BO algorithms is the initial learning process in line 2 of Algorithm 1. The corresponding likelihood of this approach is given in eq(2). I do not get the point how this approach is different from existing GP modeling and eq(2) is simply the unnormalized marginal likelihood for all data points since all tasks are assumed to be independent. Such a formulation is not only trivial to the GP community , but also to the empirical Bayes community. 


Additional (minor) issues:
1. the graphical model for GP in Figure 1 is wrong
2. there exist a lot of inconsistencies in this paper. In assumptions section, it is assumed the variance is known however the variance is a hyper-parameter in the marginal likelihood. 
3. lots of claims and statements are superfluous. For example, authors claim one limitation of existing approaches is the total number of BO iterations must be set in a manual way. However, throughout this paper, the number of iterations is still pre-defined. What is the point of saying this is a limitation while not touching it at all? Another example, authors claim  "interpretability of intermediate steps" is lost in existing methods, however, this problem is not touched either. 
4. Another contribution of this paper is a tuning dataset. I can see the value of such a dataset, however, failing to explicitly describe the required computation resources makes claiming this being a contribution less convincing.
summary_of_the_review: The proposed method is trivial. The theoretical part presented in this paper is very minimal and incremental.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper suggests a meta Bayesian optimization strategy that optimizes free parameters of GP including a prior function and noise variance, where multiple sets of historical observations are given. In particular, the proposed method chooses a free parameters using one of three approaches: (i) optimizing a marginal likelihood, (ii) measuring KL divergence, (iii) considering both marginal likelihood and KL divergence. The authors finally show the theoretical analyses on regret bounds and the numerical results on hyperparameter optimization.
main_review: ### Reasons to Accept

+ It is well-written and well-organized.
+ It solves a very interesting problem, which transfers a history to the current task in Bayesian optimization setup.
+ Compared the work by Wang et al. (2018b), it solves more realistic setups.
+ It provides promising numerical results and sound theoretical results.

### Reasons to Reject

- I do not think that it degrades the contributions much, but four-dimensional search space is relatively small, compared to other Bayesian optimization or hyperparameter optimization papers.
- Following the above point, is there any specific reason why the authors use four-dimensional search space? I do not think this algorithm is not scalable. Moreover, for example, batch size can be one of the meta-parameters to be optimized.

### Questions to Authors

1. Can you elaborate why the proposed method does not train a GP model every iteration, e.g., every $t = 1, \ldots, T$? I think that it can be possible without (relatively) expensive computational costs.
1. H* NLL does not use a matching dataset, right? If you did not use multi-task GP regression, which has an additional input to indicate task information, does H* NLL (i.e., optimizing Equation (2) with $D_N$) work appropriately? 
summary_of_the_review: I think that this paper addresses an interesting problem and suggests a novel method as described above. Thus, I would like to recommend acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper is concerned with speeding up Bayesian optimization by using evaluation data from previous, related tasks defined over the same configuration space. The authors propose to model the data from each experiment (or "task") by independent Gaussian processes, which all share the same mean and covariance function. This surrogate model can be learned from past data.

The paper also presents experiments on a fairly simple search space of 4 optimizer parameters. This is done for a bunch of datasets and NN models. And there is a pretty simple extension of theoretical results from (Wang, 2018b).
main_review: The problem of "warmstarting" HPO by making use of data from previous experiments is an obvious idea, and it has seen a large amount of past work, much of which the authors of this submission do not seem to be aware of, neither apparently was (Wang, 2018b) which seems more of a theoretical paper. In particular, there is quite a lot of work which uses GP models and scales linearly in terms of the number of past experiments, contrary to what is stated in the introduction. Two of the most interesting ones are maybe [1], [2]. The authors here cite (Perrone, 2018), which has these citations and more, so it is pretty odd the authors do not mention (or compare against) any of them.

Given the straightforward nature of what is proposed here (a setup closely related to what is done in [3]), I'd be quite surprised if for example [1] would not outperform it. After all, the assumption that data from experiments on quite different models can be modeled by the same mean and covariance function, is pretty strong. There are all sorts of issues with this idea, for example what if data from some tasks is much larger than data from others? Moreover, in what is proposed here, the surrogate model parameters do not even seem to be adapted to the current task, even as data from it becomes available. Here, methods like [1], [2] seem much more compelling to me, as they try to for example rank previous experiments by closeness to the current one. [1] is doing this without having to define any meta-features of the dataset, and also of course without relying on observations at the same configurations (given you model your data with a GP, you should certainly not need that anyway).

The experiments are not meaningful, because essentially all relevant prior work is missing for comparison. The authors more or less compare their proposal (in two variants) against a bunch of baselines, as if there was no revelant prior work. In fact, they even seem to invent on their own methods to compare against, such as "MIMO", in a way which has never been used for transfer HPO. Why? Please read about and compare against relevant prior work. Given they cite work (e.g., Perrone 2018), they should have been aware.

Apart from that, I also do not get much out of the experimental setup. Why was it chosen that way? Does it have any practical relevance? Does anybody else use this learning rate schedule, or was it just made up for this paper? I also did not find a discussion of a pretty critical point: how are the datapoints chosen for tasks you offline train on? In order to be realistic, these would have to be active choices themselves, because that is data we could have been obtained by running BO on them. Instead, my suspicion is that past data was sampled randomly, which would correspond to pure exploration (random search). Such data is obviously more valuable to obtain a good surrogate model fit, but also more expensive to obtain in the real world (one would have to run random search).

[1] Feurer etal: Practical Transfer Learning for BO, https://arxiv.org/abs/1802.02219
[2] Wistuba etal: Two-stage transfer..., ECML 2016
[3] Golovin etal: Google Vizier, KDD 2017
summary_of_the_review: This paper proposes a simple idea for wam-starting BO by fitting the parameters of a GP surrogate model on past data. Unfortunately, a lot of relevant prior work is ignored here and not compared against. Instead, the proposed approach is compared against simple baselines, as well as methods that mostly seem to have been made up (such as "MIMO").
review_title: null

Meta-review: This paper claims a practical improvement over one of earlier meta BO methods. Warm-starting BO or HPO by making use of data from past experiments or tasks seems to be interesting and useful for some applications. In fact, there are a large amount of work on this topic, but a lot of relevant prior work is ignored in this paper unfortunately. I appreciate the authors for making efforts in responding to reviewers’ comments. However, after the discussion period, most of reviewers had serious concerns in this work, pointing out that the proposed method is rather trivial and the comparison is made only against a simple baseline. It was also suggested to improve the experiments. While the idea is interesting, the paper is not ready for publication at the current stage.
Decision: Reject