Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper proposes a new Bayesian optimization (BO) based hyperparameter searching method that can transfer across different datasets and different metrics. The method is to build a regression model based on Gaussian Copula distribution, which maps from hyperparameter to metric quantiles. The paper shows that by leveraging this estimation using some specific (sampling) strategies, it is able to improve over other BO methods.

The high-level idea of this paper seems sound to me -- that improves standard BO to generalize across datasets and metrics by learning a mapping between the space of hyperparameters and metrics. While I am not an expert in this area, the derivation looks sound to me, and the evaluation results of this paper are comprehensive to show that CGP seems to outperform a number of methods.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper tackles the problem of black-box hyperparameter optimization when multiple related optimization tasks are available simultaneously, performing transfer learning between tasks. Different tasks correspond to different datasets and/or metrics. Gaussian copulas are used to synchronize the different scales of the tasks.

I have several reservations with this paper. First and foremost, it seems to be lacking a fair and trivial baseline (I will describe it below) that justifies the apparently unnecessary complicated path followed in this paper. Second, there are a few small incorrect or improperly justified technical details throughout the paper.


1) Mistaken/unjustified technical details:

- In equation 1, the last term seems to be constant. For each task, the function psi is not parametric, so its gradient is also not parametric and the input is the inverse of z, i.e., y, which is also fixed. So why is it included in the cost function? This sort of probabilistic renormalization is important in e.g. warped GPs because the transformation is parametric. In this case, I don't see the point. It can be treated as a normalization of the input data, prior to its probabilistic modeling.

- Before equation 1, the text says "by minimizing the Gaussian negative log-likelihood on the available evaluations (x, z)" But then, equation 1 is not the NLL on z but on y.

- In section 4.2 the authors model the residuals of the previous model using a powerful Matern-5/2 GP. Why modeling the residuals this way and not the observations themselves? The split of modeling between a parametric and non-parametric part is not justified.

- One of the main points of the variable changes is to normalize the scales of the different tasks. However, equations 1 adds together the samples of the different tasks (which, as pointed out by the authors might have different sizes). Even if the scales of the outputs are uniform, the different dataset sizes will bias the solutions towards larger datasets. Why would that be a good thing? This is not mentioned and doesn't seem correct: there should not be a connection between a dataset size and the prior influence of the corresponding task. In fact, this will have the same effect as if the cost had different scales for different tasks, which is precisely the problem that the authors are trying to avoid.


2) Trivial baseline

Given that the authors are trying to aggregate information about the optimal hyperparameters from several tasks, they should not compare with single-task approaches, but with the simplest way to combine all the tasks. For instance:
    a) Normalize the outputs of every task. This can be accomplished in the usual way by dividing by the standard deviation, or even better, by computing the fixed transform z = psi(y), separately for each task.
    b) Collect the z of all tasks and feed them into an existing GP black-box Bayesian optimizer.

This is a very simple way to get "transfer learning" and it's unclear that the extra complexities of this paper (copulas, changes of variable with proper renormalization when the transformation is parameter free, etc) are buying much else.


Minor improvements:

- Page 2: "is the output of a multi-layer perceptron (MLP) with d hidden nodes" Is d really the number of hidden nodes of the MLP? Or the number of outputs? Given that d is also the size of w, it seems it's actually the latter.

- Explain why the EI approach is used for the second model (with the GP), but not for the first model.

Edit after rebuttal:
“The term is not constant over z” -> Sure, it’s not constant over z. But z is constant. So the term is constant.

“The NLL is minimized in z and there is indeed no y in equation 1.” -> Sure, there’s no y in the equation, that’s correct. But it is still the NLL of y, and not the NLL of z.

About the new baseline: Instead of simply renormalizing using mean and standard deviation, I suggested above using the same z=psi(y) that is used in the paper for the normalization. Is that where the advantage of the proposed method is coming from?

"Note that this is orthogonal to the scale issues we focus on: larger tasks will have larger gradient contributions but the scaling we propose still allows us to learn tied parameters across tasks as their scales are made similar. " Both issues affect the scaling of the task, so I don't see how they can be orthogonal. Their scales are not made similar precisely because of the different sample sizes.

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The authors propose a new way of normalizing the labels of the meta-data. They propose a Thompson sampling strategy as a new hyperparameter optimization warmstarting strategy and an optimization method that leverages transfer learning.


The transfer learning baselines are surprisingly weak and show no improvement over the random search baseline for two of your tasks. You should consider stronger baselines. One interesting work is "Collaborative hyperparameter tuning" by Bardenet et al. which overcomes the problem of different scales by considering the problem as a ranking problem. You discussed further works in your related work. Another interesting work based on GPs is "Scalable Gaussian Process based Transfer Surrogates for Hyperparameter Optimization".
The warm-start GP currently seems to be your strongest baseline. However, I have doubts that it is implemented correctly. You say that you estimate the most similar dataset and then evaluate its best 100 hyperparameter configuration. First of all, I don't understand why you decided to choose 100 (DeepAR only has 220 configurations in total!). Second, this is not how this method works. Instead, you estimate the k most similar tasks and evaluate its best hyperparameter configuration. In your case k is upper bounded by 10 (number of tasks). This is notably smaller than 100 and gives more time to the Bayesian optimization which will likely improve your results.


You observe worse results on XGBoost and good ones on FCNet. You refer to Table 6 and connect it to the RMSE. This might be true but I have a much simpler explanation: the search space of FCNet is orders of magnitudes larger and it contains many configurations that lead to very high MSE and only few with low. Therefore, a random search will on average provide poor results where a warmstarted search will obtain decent results for the first iterations. XGBoost is less sensitive to hyperparameters such that the overall variance in the losses is smaller. Maybe you can provide some insights into the complexity of the optimization tasks (plot the distributions )and add it to the appendix?


Table 6 contains more datasets than Table 2. Why did you drop some tasks?


The aggregated results in Table 2 are nice but actually we are interested in the outcome after the search after a given budget. Can you add such a table?


I have few suggestions to improve the readability of the paper:
That there is a choice of copula estimators is mentioned at the very end of the paper. Can you add it to the section where you describe them first? In section 5.1 you already argue by referring to Table 6. However, Table 6 is explained first in section 5.2 which makes it hard to follow your argumentation.
You use the term "metric" to refer to objectives. This is confusing, you might consider changing this. You propose to use scalarization to address the multi-objective optimization problem. Why would an average of both objectives be the optimal solution? Does the unit you use to measure the time matter? What if you use machine learning algorithms that scale super-linear in the number of data points? How is this novel and why can't your baselines employ the same idea? A discussion of related work on autoML for multiple objectives is missing.
The second paragraph of section 5 is confusing. You cite some work, discuss it and then conclude that your setup is entirely different. Would any information be lost if you say that you precomputed the values?
review_title: Official Blind Review #1

Meta-review: This paper tackles the problem of transferring learning between tasks when performing Bayesian hyperparameter optimization. In this setting, tasks can correspond to different datasets or different metrics. The proposed approach uses Gaussian copulas to synchronize the different scales of the considered tasks and uses Thompson Sampling from the resulting Gaussian Copula Process for selecting next hyperparameters.

The main weakness of the paper resides in the concerns raised about the experiments. First, the results are hard to interpret, leading to a misunderstanding of performances. Moreover, the considered baselines may not be adapted (they may be trivial). This might be due to a misunderstanding of the paper, which would align with the third major concern, that is the lack of clarity. These points could be addressed in a future version of the work, but it would need to be reviewed again and therefore would be too late for the current camera-ready.

Hence, I recommend rejecting this paper.
Decision: Reject