Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
The authors of the paper propose a hyperparameter tuning algorithm that uses a simple non-probabilistic model to predict the performance of any new hyperparameter configuration on a target task. They use transfer learning from related tasks as well as learned meta-features for each task, and then simply pick the hyperparamater configuration predicted to perform best by this NN based model.

The authors provide results on 3 meta-datasets, each a slightly different hyperparamter optimization problem applied to the same 120 UCI datasets, thus providing relataed task for each of these three settings. The algorithm can be used in three modes Q1, as a zero-shot algorithm to immediately select the best performing hyperparameters Q2, as an initialization strategy used in the same way as Q1 but followed by a traditional sequential HPO algorithm, and Q3, as a standalone sequential tuning algorithm, by updating the model with any new evaluations, and then using it to suggest the configurations to evaluate next.


#### Comments / Questions
A crucial detail, that is only cursorly mentioned in the main test, is that the authors tune the architecture of their model to be adapted to the specific meta-dataset used (as described in table 5 in appendix D). I think this should be given more attention, having to tune the surrogate hyperparameters itself seems a fairly major limitation of the method and should be discussed. It is not clear exactly how many different architectures were tested, what the search space was, and how well any specific architecture would generalize to new meta-datasets, or even new datasets on the same meta-dataset.

Without more details on how the tuning of the surrogate architecture exactly happened, it does not seem possible to apply this method to other meta-datasets, making the paper incomplete a non-reproducable.

It is also not clear can the model can avoid getting stuck in local minima? I would be curious why this was not a problem with the Q3 experiments without the exploration usually given by a probabilistic model.

Algorithm 3 in the appendix mentions the use of an acquisition function, but the proposed model does not appear to be probabilistic, so it is not really clear what kind of acquisition function would be used here.

Normalized regret used for all results is never defined.

#### Pros
* Important use case/setting (expeciall Q1 and Q2)
* Simple idea, although not described so straighforwardly, I would avoid all the talk of a grey-box method and focus more on simplicity and the idea of providing a good and simple zero-shot NN model that is able to use meta-features in an effective way
* Meta-datasets useful for research are being published with the paper (or at any rate promised to be published upon acceptance)
* The experimental results look fairly competitive (although with some of the shortcomings and open questions I mention in this review)


#### Cons
* The authors tune their model architecture to be adapted to each specific meta-dataset, but this tuning process is not well described, making the method hard to reproduce.
* The algorithm is not very principled from a probabilistic standpoint when used in setting Q3, the used exploitation and exploration trade off is just an heuristic, and not guaranteed to provide good exploration. I would expect it to plateau at a much worse value than bayesian optimization methods. Honestly I would think the paper would be stronger without this use case described, and more more focus given to the results and the zero-shot case.
* The theory sections in the main text are fairly standard and could be shortened a lot, as well as not really needed to explain the method used, what is more relevant is the actual model used, in terms of architecture and tuning. In the current version the architecture is relegated to the appendix and the tuning absent.
* In general the paper is not really self contained, instead a lot of the important details are contained in the appendix. I would also suggest to be clearer about what the main contribution is.
* Any uncertainty quantification (confidence intervals or error bars) in the results is missing, thus it is difficult to assess how reliable the results are.


I think in the current state the paper is not ready to be published. However the idea is simple and promising. I would suggest the authors improve the description of the experimental results, comparing against some more zero-shot baselines. Also either the tuning process of the model itself should be described in more detail in the main text, or not be required (in case a more general architecture can be provided).



#### Typos/Other minor comments
It would be using always use the same term instead of simetimes probabilistic surrogate and other times response model.

The way l_hat and the selection of the surrogate is chosen by max likelihood is a bit convoluted, but as I said above, I don't think the theory sections in the main text add much insight into the method, just direcly describe the loss.

Section 4:
inputting -> taking as input
page 14 first paragraph:
to optimize outer loop -> to optimize *the* outer loop
Appendix B1:  learningJomaa -> missing space
Appendix Q1: typos in "but the time training required time of underlying model."
review_title: Promising idea, but too many weaknesses in current paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
The authors propose a new zero-shot hyper-parameter optimization method based-on the meta-learning framework. The proposed method incorporates two ideas from the meta-learning framework namely the task similarity based on the meta-features and the dataset identification. The former idea is used to achieve the requirement that responses to similar data sets should be similar.The latter idea has the role of preventing data of dissimilar tasks from being embedded in close proximity in the meta-feature space.

In this paper, the authors construct a new meta-dataset to evaluate the proposed method by compare it with various HPO methods (both zero-shot approach and sequential model based approach).

While the performances of various HPO tasks certainly seems to have improved, there are some concerns.

- How can you justify the proposed approach  which measures domain similarity in terms of distances between meta-features  (some kind of point estimation), rather than inter-distributional distances? My concern is that the information may be considerably dropped than measuring similarity by inter-distributional distance.

- The algorithm in the appendix doesn't have a section on learning meta features, how (and when) are these updated in the algorithm?

- Appendix B.1 defines a regularization term for dataset identification. 

  -- Although the range of \hat{s} is defined as {0, 1}, in (8) \hat{s} appears to be a continuous value. Are you using some kind of threshold to make it binary? 

  -- The authors argue that  "Without loss of generality, we use the Euclidean distance to measure the similarity between the extracted meta features, ... and \gamma = 1". What do you mean by "without loss of generality" here? Does this mean that other metrics and \gamma values will give same results?
review_title: Official Blind Review#4

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this papar, the authors formulated a new objective function for HBO, which included an additional regularization term based on the dataset similarity. The authors used the distance between the meta-features of selected datasets to measure this dataset similarity and assumpted that similar datasets should have similar hyper-paprameters.  The experiments are complete and demonstrate the advantages of using this new optimization formulation.

I have three major concerns as following.

1. In Appendix B.1, it mentioned that using the similarity regularization defined in Equation 4. alone is not sufficient to measure the dataset similarity, so an additional similarity metric Equation 9. is used. What is the impact of directly replacing the distance metric in Equation 4. with Equation 9?

2. Since the similarity regularization is computed using K selected datasets, and "for each meta-dataset the architecture of the models are selected based on their performance on their validation datasets", how to ensure that the proposed universal model trained on these K datasets can be generalized to other datasets.

3. It is suggested to re-organize the description of the proposed method. In the current version of the manuscript, many essential details are put in the appendix, such as the optimization of Eq. 5. 
review_title: This paper proposed a regularized regression using meta-feature embeddings of certain datasets as regularization for standard HBO.

final_score: rating: 7: Good paper, accept
review_content: The authors propose a novel approach to zero-shot transfer learning for hyper parameter optimisation (HPO). In contrast to standard HPO approaches, where HPs are found for each task by random/grid search or Bayesian Global Optimisation or sequential model based optimisation, the proposed approach learns a response model defined over the combined domain of datasets and hyper-parameters such that for a new task, for which training data and labels and some ranges of HPs are given, HPO can be done by querying the response model without having to evaluate the black-box function of the validation loss as a standard HPO would do. 

I think the manuscript is very well written, the motivation is sound, the related work section includes many relevant and well summarized examples of similar approaches, the notation and the presentation of the proposed approach are intuitiv and concise. 

One thing that could probably be improved a bit is the coherence of the text, I mean, I had to jump a lot between the Appendix and the paper to understand some necessary details. For instance the core of the approach in section 4, and eqs 3-5 are very difficult to understand without the information referenced in the last paragraph of that section. I wonder if it would be possible to mention some of the relevant information, e.g. how eq. 5 is optimized or how the dataset2vec features are learned, right in section 4. 

The experimental evaluation appears solid and comprehensive. I like how the authors investigated their approach in three different settings. Compared with a large number of alternative approaches the proposed approach achieves improvements in all HPO settings, ‘cold-start’, ‘warm-start’ and continuous learning of the response model. One question I had when thinking about the experiments is that the proposed approach relies on a number of parameters to be chosen well, in particular the HPs that determine the computation of the meta features and the response model. It appears that the response model used some basic settings that were not optimised for the task, but I still wonder how sensitive the learning of the response model is to the choice of hyper parameters. That includes the 'generalisation performance' of the dataset2vec model and all other ingredients of the response model, apart from the validation loss of the black/grey-box ML model.

When comparing the proposed approach with existing approaches, it seems that at least some of the aspects that make this work so well should also be captured in other approaches that use some dataset features in a transfer learning HPO setting. Some of them are mentioned in the related work section (Perrone et al, Volpp et al, Feurer et al, …) but especially the experimental setting of SEQUENTIAL GRAY-BOX FUNCTION OPTIMIZATION looks very similar to reinforcement learning approaches to HPO which include dataset features in their policy (e.g. https://arxiv.org/abs/1906.11527). Given these similarities and the strong empirical results it would be great to get a better understanding of which elements that are different to existing work contribute most. The ablation study in Appendix F.2 is great, and could be discussed in the main text especially in relation to other work. 

To summarise I think the experimental validation suggests that this approach has potential to improve the state of the art in HPO. Some relevant details of the method are not entirely clear, or at least could be better described in the main text. Also in the experimental evaluation it could made clearer that the results are not sensitive to the tuning of the, pretty complex, HPs of the response model itself.  

review_title: Interesting and efficient transfer learning HPO approach with some potential for improvement w.r.t clarity of model details, robustness of approach and relation to RL based HPO

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros:

This work proposes a solution for the zero-shot hyper-parameter optimization problem without access to observations of losses of the target response.

The proposed method is parallelizable and thus can reduce the required wall-clock time vastly.

Three new meta-datasets with different search spaces are proposed and served as a benchmark.

A battery of experiments and the completions comparison against state-of-the-art HPO solutions are conducted to demonstrate the performance of the proposed approach.


Cons:

This work may be treated as applying meta-learning to hyper-parameter optimization. That is, a model is introduced to learn how to learn a group of optimal parameters based on the given meta-features. The acquisition function, meta-features extraction, meta-learning optimization method, the dataset similarity loss, etc., are all borrowed from prior works.

The meta-features of datasets are directly computed by Dataset2Vec [1]. This paper points out that prior methods rely on engineered meta-features, but actually [1] can be applied to improve them as well. I suggest the authors discuss the situation where prior methods and [1] are combined to reveal the advantage of the proposed method.


[1] Jomaa, Hadi S., Lars Schmidt-Thieme, and Josif Grabocka. "Dataset2vec: Learning dataset meta-features." arXiv preprint arXiv:1905.11063 (2019).

review_title: This work applies meta-learning to zero-shot hyper-parameter optimization. No target response is required, and meta-features are extracted for the hyper-parameter estimation of a given dataset.

Meta-review: The paper provides a transfer learning approach to HPO. It builds and improves upon existing methods of zero-shot HPO where the high level idea is to use the outcomes of hyper-parameters on an offline collection of datasets in order to speed up HPO on a new dataset. On the plus side, the methods provided seem to be novel, and the results seem to be promising. The main issue is the writing and clarity of the paper, making it hard to be certain of the good qualities of the paper. Aggregating the reviews, the details are too spread out between the appendix and main body, the techniques require more motivation behind them, and important details of the experiment are somewhat vague. The authors provided a modified version which is definitely a step in the right direction, however, it does not seem to be enough. I think this is a solid paper based on a promising idea. However, given the almost unanimous agreement about that crucial gap in clarity even after the modified version was uploaded, I recommend rejecting the paper. 
Decision: Reject