Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors propose using Transformer to imitate the hyper-parameter optimization. The authors claim this is the first work in the area, and claim the proposed method exceeds several classic HPO methods.
strengths_and_weaknesses: Strength:
1. very comprehensive evaluations.
2. seemly reasonable idea.

Weakness:
1. limited novelty: perhaps applying transformer to HPO could be counted as a novel point, but I feel this is not enough.
2. The paper is difficult to read, please try to improve the readability. Many places in the paper tries to impress the readers with sophisticated terms even on very simple concepts.

questions: 1. how do you justify max_i\in{1:t}(y_i-y_rand)/(y_max-y_rand) a good evaluation metric? Why don't you use the y_i and plot as a range? 

3. Fig.4 made a few strong claims, especially that OPT-Former performs better than GP-UCB. I'm trying to understand the underly causes. Here is my guess: opt-former is trained on datasets that potentially contain tasks with similar distributions tested in Fig.4. The advantage of GP-UCB is to start without any priors and gradually approximate the underlying function contours by sampling. Without any prior data, I'm surprised that OPT-Former can perform better than GP-UCB. I'd be happy to see if I'm wrong, and it will be compelling if the authors can provide an anonymous link to the code for a quick comparisons. (key factor for me to improve the score)

4. The methods used in this paper does not fully capture the landscape of black box optimization solvers today. The authors may find the following repo to be useful. (feel free use at your discretion, it is just a suggestion)

a. https://botorch.org/

b. https://github.com/facebookresearch/nevergrad

c. https://github.com/facebookresearch/LaMCTS

d. https://github.com/uber-research/TuRBO
These repos have encapsulate several exciting BBO algorithms today.

limitations: no limitation found
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper describes a new meta-learning approach for hyperparameter optimization (HPO) based on a transformer model. The model is trained on offline generated data that includes the metadata that characterizes the optimization problem, for example the search space and  and the history of observed trials, i.e function values and input configuration. During inference time, the model can be combined with HPO policies, such as Thompson sampling or upper confidence bounds to suggest new hyperparameter configurations.
strengths_and_weaknesses: 
### Reason for overall rating

Current transfer learning approaches for HPO are limited to a fixed search space and the same underlying machine learning model and only transfer knowledge across different datasets. This paper presents, to the best of my knowledge, the first approach that enables meta-learning across these different dimensions. While I don't think the method is ready for a practice yet, the paper marks a first important step towards more universal HPO methods.


### Strengths

The paper aims to learn a more general meta-learning approach for HPO, that generalizes not only across datasets, but also machine learning methods and search spaces. This, in theory,  allows to access a much large amount of offline data and allows to generalize across different domains.

Overall, I found the different parts of the paper, e.g tokenization, inference and decoding of the model, well motivated and clearly explained.

`The empirical evaluation of the paper contains sensible set of baselines. Also the ablation study provides convincing insights in the proposed approach.



### Weaknesses

It remains a bit unclear how well this model generalizes beyond the training data. For example, what would happen if the method is applied to other problem domains, such as neural architecture search or general gradient-free optimization problems. Similarly, how does the method scale with the dimensionality of the search space?

The dataset generation seems a bit ad-hoc. Is it really necessary to include trajectories of such a large variety of optimizers or would it be sufficient to limit to few state-of-the-art optimizers? This could potentially reduce the dataset size and would allow to us a smaller architecture.

The paper could elaborate on the pre-training of the model. For example, how did different design decision of the network architecture effect downstream performance? How difficult was the pre-training, e.g did you have to restart from previous checkpoints, etc ? 



questions: - Section 6.2: How do you compute the predictive distribution p(y|...)? My understanding is that the transformer only predicts discrete outputs with [0, Q)

- Section 6.4 prior policy: How is Random Search combined with Thompson sampling (Random Search-TS)?

- What was the computational budget to train the transformer model and how long did it train?

- Do you plan to open-source the dataset and the code to reproduce the results?
limitations: While the method improves across a set of baselines, it does not improve yet over more sophisticated algorithms such as Vizier on real world datasets. I assume that it would also not outperform current state-of-the-art methods that early stop poorly performing configurations, such as Hyperband or BOHB. However, I think this is fine for a research paper, but it would not be sufficient for production.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies hyperparameter transfer with metadata in an "open-set" setting -- allowing different configuration spaces across tasks. A transformer-based hyperparameter tuner, namely OptFromer, is proposed to predict policy and response function values (e.g., validation performance) in a sequence-to-sequence training style, where the learned policy maps text-based metadata to pre-discretized hyperparameter configurations. To my best knowledge, the proposed method is the first HPO method that learns prior knowledge from the collected text-based configurations. Experimental results on one collected real-world dataset and two public benchmarks were provided in terms of the policy behavior imitation, response function prediction, and HPO. 
strengths_and_weaknesses: Pros:
- The proposed hyperparameter prior learning method is orthogonal to existing GP-based hyperparameter transfer methods in two folds: 1) it relaxes the limitation of sharing the same configuration space across different tasks, and 2) the data-driven approach given by training a transformer model significantly improves the efficiency. 
- It is interesting and novel to learn prior knowledge from text-based metadata. The seq-2-seq supervised learning framework is technically sounded with proper practical treatments. Moreover, the transformer structure is well-motivated to capture both symbolic and numerical manipulation.
- While some necessary implementation details are missing in the manuscript, the augmented HPO policy with Thompson Sampling provides a good implementation similar to offline RL. 
- Extensive experimental results demonstrate the effectiveness of the learned HPO policy on its well-calibrated predictions and utility performance. 

Cons:
- The main concern of this work is its unclear **meta knowledge** learned from text-based metadata. How does the proposed OptFormer indeed imitate the other HPO algorithms? Does the transformer simply memorize the choices given by different HPO algorithms? Can the proposed method adapt to more complex algorithms (e.g., hypergradient-based) and large-scale hyperparameters? 
- The learned prior seems risky to be biased on closed-set model architectures and tasks (datasets). It remains unclear if the HPO policy can be generalized to **unseen** tasks. I may miss something in the appendix; yet, it will be helpful to give more details about the split of training/test set in the RealWorldData. A non-overlapping split over the tasks or algorithms will be more convincing. 
- One major technical contribution of this work is to introduce transformers for learning HPO priors. Hence, it is expected to give an ablation study regarding this architecture choice. One baseline based on RNN (e.g., LSTMs or GRUs) will be useful to validate this point empirically. 
 
questions: - Table 4 shows more calibrated results of OptFormer than the GP-based methods. It is well known that GP could provide well-calibrated uncertainty estimates. Yet, in accordance with Eq (3-5), it seems like the proposed method just follows the standard supervised training strategy. Any insights into why OptFormer could show a better calibration result? Is the reason due to using the transformer architectures [40]? I'm also curious about the ECE comparison between OptFormer and OptFormer (TS)
- As shown in Fig. 4, the Vizier performs slightly better than OptFormer (TS) on the RealWoldData dataset. While the paper implies the reason as the GP surrogate-based test functions, it is unclear why OptFormer performs much better than GP-UCB. Also, the comparison results between RealWorldData (*mixed algorithms*) and HPO-B (*controlled algorithms*) cast a shadow on the biased issue of the proposed OptFormer. 

**Post after rebuttal**
Thanks for providing a detailed response to the questions. Due to my travel schedule, I haven't got a chance to further discuss with the authors. Yet, most of my previous concerns were well addressed. Particularly, it would be interesting to add an LSTM baseline in future work and explore more the calibration of HPO from a pre-training perspective. I would like to champion this work by upgrading my score. 
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors train a single Transformer language model (LM) on a variety of tokenized HPO trajectories from a variety of tasks and HPO algorithms. For each HPO trajectory the LM is primed on the task/algorithm (name, search space, metric, algorithm) and is then trained to predict hyper-parameters as well as the optimized metric step-by-step for each trial in order.
strengths_and_weaknesses: Strengths
- The results are strong: Reproducing the performance of most algorithms up to 100 trials as well as improving upon them. This could be the path forward for the HPO community.
- The authors provide insightful ablations.
- The method is clearly described and simple.

Weaknesses
- This paper does neither open-source its codebase (built open an open-source codebase), nor the trained model and not even the training data (built upon open datasets). Actually, even a performance number for optimization for follow-up work to compare with is missing, as scores are calculated using an undisclosed metric and the main results are only reported in the form of plots and not tables. 
- For RealWorldData (and HPO-B actually, even though less interesting there) a SOTA BO method would be an interesting baseline to add, like HEBO.
- Is the conclusion of Meta-data ablation (line 287) based on a model trained with meta-data? In that case, I would guess the worse performance stems from a train/test distribution shift, rather than from missing metadata really.


Summary:
The results are strong with some evaluation problems. It is not reproducible, though, which in this case is a particularly big problem, as this paper proposes a very new direction for a field of research in which the expertise to reproduce the results based only on descriptions (previous methods require a very different background) and the resources to reproduce the results without data (running HPO with different optimizers on millions of problems) are missing. 
questions: - Is the discretization of scalars for the input important? How would a model perform, where these numbers are normalized in some way and fed to the network directly?
- Do you have an explanation for the strong performance of using RealWorldData for training (Fig. 5 a) compared to the larger HPO-B dataset, when comparing on HPO-B? (Transfer from a smaller dataset is better than in-domain training. This is unusual.)
- What do you mean by "temporal train/test splits" in line 269?
- How do you calculate the Thompson Sampling utility function?
limitations: The listed limitations are fair, even though one limitation, I would expect there to be, is missing: Handling much longer sequences, as the Transformer is trained with a maximum sequence length.
review_title: null

Meta-review: metareview: In this work, authors investigate whether Transformers can be used for hyperparameter optimization. The work is interesting and authors outline how they frame the problem and solve practical difficulties. The resulting method is shown to be able to learn to HPO from historical HPO runs and text-based metadata. Some implementation details are missing and I would encourage the authors to add details when possible (taking into account the reviewers' feedback). The empirical evaluation of the paper contains sensible set of baselines and ablation studies. It is also appreciated the authors put extra effort into open sourcing parts of the code.
confidence: Certain
Decision: Accept