Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Quality and Clarity**

 The authors clearly describe the problem and the proposed solution. The explanation in some parts can be improved (see Queries and Suggestions below) but overall the paper is well written.

**Originality and Significance**

While the problem of hyperparameter optimization is extremely important and well studied, the main contribution of this work appear to be some modifications to the transfer learning based approaches that leverage the performance of hyperparameter settings on related tasks to tune the hyperparameters for a new task. The techniques like Warm Start, Data Augmentation, and sharing of NN and GP parameters across all tasks are modifications that can potentially improve the practical performance of many approaches in this space and thus positioning this work as a set of techniques that can be applied to multiple models in this space might make it more impactful than the current approach of positioning it as a single model that can outperform other models.

**Strengths**

1. The idea of using task independent parameters for both the neural network and the GP appears to be novel and enables the model to learn from the data across all the old tasks and the initial data from the new task.

2. The data augmentation and warm start approaches are intuitive and appear to give clear gains in practice.

**Weaknesses**

1. I do not think it is correct to apply mini-batch SGD to the task-wise losses in the manner of Algorithm 1, since the individual losses (GP log-likelihood) will not be additive over the samples of a given task and so the gradient estimates from the mini-batch would be biased. It might work in practice but I would appreciate some discussion/clarification on this in the paper.

2. While using mini-batches is based on the rationale that the number of samples for each task might be large, the data augmentation approach proposed seems to be intended for the setting when "only a few examples are available". Thus it is not clear if the targeted setting involves few or many examples per task.

3. The data augmentation and warm start approaches are largely heuristic and it is not clear whether they will always be beneficial.

**Queries/Suggestions**

1. Please explain the mutation and crossover operations in the Warm Start approach more clearly (preferably with an example). I am not entirely clear on what they are doing.

2. Please provide a mathematical formulation of the normalized regret loss used to evaluate the methods in the Experiments.

3. Can the Warm-Start and fine-tuning steps be performed with the ABLR baseline as they have been performed with the Multi-Head GP baseline? If so, please include those results as well.

4. Please include some discussion/clarification on the points 1 and 2 under Weaknesses above.

**Comments after Author Response**

I thank the authors for their response. The explanation of the main approach has certainly improved and the details of task augmentation and warm start are much clearer now. I also appreciate the added discussion on bias in Stochastic Gradients for GP training. The reference cited does seem to indicate that the training will converge despite bias in gradients. The warm start and task augmentation approaches still seem a bit heuristic to me. The task augmentation approach seems to assume an inherent linearity in errors/noise for the metric of interest which may not hold in practice. The choice of loss function for warm start also seems rather arbitrary. It is not entirely clear why choosing $\mathbf{X}^{\text{(init)}}$ that minimizes (12) is a good idea. However as my main concern about the validity of SGD in training the model has been addressed and both the task augmentation and warm start approaches seem somewhat intuitive, at least at the idea level, I am increasing my score by one point.
review_title: A new approach for Hyperparameter Optimization using Few-Shot learning - Good empirical results but novelty over prior work does not appear significant

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update: I appreciate the response to address my concerns carefully. My major concerns were the lack of novelty and some unclear descriptions on details in methods and experiments, but most of them have been well addressed. At first, I didn't see the technical challenges when applying DKL or any multi-task GPs into hyperparamter optimization. After reading the response and the manuscript again, I'm convinced the task augmentation plays a critical role in this setting. And, additional information from the revised manuscript helps to understand the details in method and experiments. So, I increased my score by two points.

**Strengths**
Solving few-shot regression tasks is interesting for warm-starting Bayesian optimization.

**Weaknesses**
1. Technical novelty seems to be weak. I have some concerns on the proposed approach.
2. Compared to baselines, the experiments are not strong.
3. The presentation needs to be improved.

**Major comments**

“A deep kernel ϕ is used to learn parameters across tasks such that all its parameters θ and w are task-independent. All task-dependent parameters are kept separate which allows to marginalize its corresponding variable out when solely optimizing for the task-independent parameters.” -> It didn’t clearly state which parameters are task-dependent and which are not. \theta and w are the only learnable parameters by maximizing a marginal log-likelihood. So, which parameters are task-dependent?

Compared to few-shot regression in a meta-learning framework, I didn’t find the benefits of the proposed approach. In addition, the proposed method doesn’t have a regularzing effects for few-shot tasks, since there is no meta-learning phase and no parameters shared across tasks.

“In hyperparameter optimization, we are only interested in the hyperparameter setting that works best according to a predefined metric of interest. Therefore, only the ranking of the hyperparameter settings is important, not the actual value of the metric.” -> If we discretize all continuous hyperparameters, this argument makes sense. But, I couldn’t find the hyperparameter setting for AdaBoos, GLMNet, and SVM in the paper. How many discrete/continuous hyperparameters? And, how to determine the range for each hyperparameter? Did you follow the same setting in this paper? Initializing Bayesian Hyperparameter Optimization via Meta-Learning, M. Feurer et al. AAAI’15.
review_title: A nice application of deep kernel transfer in hyperparameter optimization.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes to simply learn/optimize the deep kernel parameters and hyperparameters of the GP using the data from all tasks (equation 9) and use such a deep GP kernel for BO. Instead of maximizing the log marginal likelihood of a single dataset (as is typically the case for a learning task), they propose to maximize the sum of log marginal likelihoods over the datasets of all tasks, which I view to be an incremental technical contribution. This paper is not about innovations in the acquisition function in BO. Their proposed approach outperforms the tested methods on 3 benchmark datasets.

A drawback of their proposed method (equation 9) is that in contrast to some existing meta-BO algorithms, it cannot exploit the GP posterior means and variances when such information is available from previous BO tasks. 

Considering the diversity of the types of datasets in the AdaBoost experiment, can the authors give an interpretation of the max likelihood estimates of theta and w in equations 8 and 9 in the context of this experiment?

For the GLMNet and SVM experiments, would it be possible to instead combine all the datasets over tasks and construct a *joint* likelihood over them in equation 9 instead of a sum of likelihoods over tasks? How would the results differ in this case?

For the experiments conducted, large amounts of data are drawn from many available previous BO tasks, especially for GLMNet and SVM. This seems to be in disagreement with the setting of BO where the unknown objective functions are expected to be costly to evaluate (e.g., in hyperparameter optimization). In the context of BO, it would be meaningful to consider the practical setting where only small amounts of data are available from a few expensive BO tasks. In this case, how would the proposed approach perform compared to the tested methods?

An empirical comparison with the state-of-the-art meta-BO algorithm: weighted GP ensembles (Feurer et al. 2018) should be included.

From the results presented in Table 1 and Fig. 2, it does not seem like only a few shots/BO iterations are needed for the experiments performed in this paper. How do the results compare when only a few shots are used?

Equations 9 and 10: Shouldn't the log marginal likelihood be over y's instead of f's?

Equation 13: Exactly how is the loss function L(f^(t),X) defined? The author mentions that it is a normalized regret. The exact expression is needed here. In particular, I have noticed that f^(t) is not in bold: does this mean that only 1 "test" point is selected per task t?

Page 6: Can the authors provide the exact details on "the settings being sampled in proportion to their performance according to the predictions"?

The authors can consider doing experiments on hyperparameter optimization of larger-scale CNNs, which is commonly seen in BO works.


Minor issues

Page 3: adaption or adaptation

The following reference would be relevant to the context of meta/transfer BO:

Z. Dai, B. K. H. Low, and P. Jaillet (2020). Federated Bayesian optimization via Thompson sampling. In Proc. NeurIPS.
review_title: Technical contribution is incremental

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper aims to solve the expensive Bayesian optimization from the view of few-shot learning by resorting to the usage of deep kernel learning for hyperparameter optimization. The major concerns regarding this paper are listed as below.

The description of GP in sec. 2.2 is confusing. The paper considers the noise variance \sigma_n^2 for K_n while ignoring it for K_* in eq. 5, which is inconsistent. The authors are recommended to refer to the GP model described in the book Gaussian Processes for Machine Learning by Rasmussen.

The second para of sec. 3 defines T source tasks, leaving no symbols for indicating the interested target task. Similarly, the reviewer cannot find the definition of target task in the following equations.

The paper states that the parameters \theta and w of the deep kernel are task-independent, and the task-dependent parameters are kept separate. Where is the definition of the task-dependent parameters? The authors should make it clear.

The reviewer did not clearly found the novelty of this paper. How to transfer knowledge from source tasks (simply sharing the parameters of deep kernel?), and what is the difference to existing transfer learning works in the studied few-shot Bayesian optimization framework?

review_title: Novelty is unclear and description is inconsistent

Meta-review: This paper uses deep kernel learning to develop a compelling framework for hyperparameter optimization in a few-shot setting, with empirically strong results. Please carefully account for all reviewer comments in the final version.

Decision: Accept (Poster)