Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors establish the equivalency of hyperparameter optimization (HPO) and model-based reinforcement learning (MRL). On one hand, hyperparameter optimization is seen as an optimization of a sequence of actions (hyperparameter candidates) that improves a reward function. On the other hand, planning replaces the acquisition function of HPO. The sequence is constructed by a transition function driving the optimization towards its extremum. The transition function can be represented by a Markov Decision Process, while the reward function can be approximated by a surrogate. Within this frame, they explore the effect of planning on the performance of this approach on the task of hyperparameter optimization.
main_review: The authors provide a very good overview of HPO work and position their work well. 

In the related work section, they miss reporting other related work coming from RL, which is in turn briefly discussed in section 4. This is not very concerning, since the authors later compare to this body of work in the experiments section. However, it would be fair to acknowledge this precedent of establishing an equivalence between HPO and MRL more clearly in the related work section in order to correctly establish the novelty of the presented approach.

The authors propose and use an ensemble of probabilistic neural networks as surrogates. The authors make a case for their choice of using bootstrapped ensembles of probabilistic neural networks. However, they don't evaluate their efficiency or stability (in the context of non-transfer scenarios). In the transfer learning setting, the use of this type of surrogate is clear. However, in a non-transfer learning setting, the training data is either small or demands computationally expensive evaluation of a large number of candidates. At some point, this brings into question the utility of a surrogate at all, since it would make sense to actually train and evaluate complete models.  Shedding a light in this direction and possible limitations would be beneficial. 


summary_of_the_review: The paper is well written, the work is presented in-depth and clearly. I believe that the presented work is solid and will contribute as a point for constructive discussion between the HPO and RL communities.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for hyperparameter optimization (HO) drawing inspiration from model-based RL (MBRL). They recast the HO problem in a markov decision process (MDP) formulation. This formulation permits the use of MBRL methods to solve the HO problem. 

Concretely, having reframed HO using this MDP formulation, they propose to: (1) train an ensemble of probabilistic neural networks to model the transition function (using first order meta learning), which maps a set consisting of previous hyperparam-response value pairs and the current hyperparam setting to the associated response value (eg val loss); (2) use this transition model in Model Predictive Control (MPC) with random shootouts to obtain an effective hyperparameter setting. Given the nature of their MDP and the fact that successive states are independent of one another, they use a variant of MPC that selects the best hyperparameter setting (or action) seen (rather than use the first action, as is common in MPC).

In experiments, they demonstrate their method improves on various baselines on three different settings. They perform some ablation studies showing the importance of finetuning their model on the target domains and the importance of planning. 
main_review: ## Strengths

- Clearly described connection between HO and MDP, which was interesting (though perhaps not entirely original, given Jomaa et al, 2019).
- To my knowledge, this is the first work considering planning in HO, and results demonstrate it can be beneficial.
- Good experimental results in comparison to baselines, and a thorough set of baselines appear to be considered. 


## Weaknesses

- Related/prior work could be discussed a bit more clearly in the context of this paper. To me it was unclear how Jomaa et al (2019)'s approach differs to this paper? How does the MDP in that work compare to that presented here? The description in Section 4 was not totally clear to me. In addition, directly mentioning how this work differs to others in Section 2 Related Work would help a lot to contextualize the study.
- Would help clarity significantly to have the Algorithm description (or at least a summary) in the main text, rather than appendix. I found it hard to understand the training procedure without this. 
- Some pseudocode highlighting how the state representation evolves during MPC/planning/trials would be helpful, in the appendix. This would make it very clear how this approach differs from some existing ideas that do not use planning. 
- I understand the motivation for the use of ensembles, but it would help a lot to have an ablation over this to understand its importance empirically also.  Since this is one of the stated contributions, such an ablation would help support the claims made in the introduction. Are there other strategies that could have been tried here? Perhaps a single probabilistic NN would perform well too?
- Could you clarify why some baselines were not considered, such as Jomaa et al (2021b)? 
- Minor: Perhaps sections 5.3 and 5.2 could be swapped in order to make the narrative a bit clearer, and conclude the discussion on dynamics model training before addressing planning.
- Minor: Tables 1 and 2 could be made clearer by highlighting that 15,33,50 refer to num. of trials
summary_of_the_review: Overall this paper is interesting and has I think some original ideas. I have several questions about related works and importance of aspects of the method (e.g. the ensembles) that would be good to get answered. If the authors are able to do this, I would consider raising my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper deals with the problem of hyperparameter selection. It formulates the problem as an MDP, and solves it using model predictive control with a short lookahead. 
main_review: Detailed comments:
1. The main claimed contributions of the paper are that it formulates HPO as an MDP, and that solving it using MPC with some lookahead is better than greedy selection. The authors themselves mention in Sec 4 that Jomaa et al. (2019) and Volpp et al. (2020) have dealt with the same setting before. The fact that non-myopic selection strategies perform better is not novel - this is effectively an active search problem, and it has been shown that non-myopic strategies perform better greedy selection (e.g. see Garnett et al (2012)). Based on this, I don't find the contributions of the paper to be particularly novel.

2.  I found the method quite hard to follow. The initial part of the paper reads gives the impression that a full transition function that maps SxA -> S is being learnt. Only later do we get that only the response is being learnt here. This is what existing surrogate functions do as well. As the authors mention, Jomaa et al. (2019) also uses the same formulation, with the one difference being they use an LSTM to model it whereas the author of this paper follow the deep sets formulation. With the mention of MPC throughout, I was under the impression that the method employs some kind of non-random policy to guide the search (as I would expect from standard MPC). Only later does it become clear that the lookahead MPCis effectively a tree search where the tree is being generated picking hyperparameters (the actions in this formulation) at uniformly randomly. The authors include the the dataset in the state space. How is this being represented?

3. I am not very clear on experimental details. Particularly, what exactly is the training datasets being used - I understand that there are 80 datasets in each training split, but how many (lambda, loss) pairs are there for each dataset? How many (lambda, loss) pairs are budgeted for each dataset in the test set? In Sec 6.5 it is stated `We report ... over 5 runs for 50 trials with three different seeds per run`. What is a run and a trial here?

4. The authors compare the results of lookahead at 3 vs 5 steps and it seems like there is no clear winner. I would like to see the results for 1 step lookahead, i.e. greedy selection, and if the lookahead actually helps here at all.

5. I find some of the assertions in the paper to be quite baffling - I am not sure if this is just due to poor wording or some fundamental misconception about MBRL. For example: `In MbRL the objective is to train a transition model to approximate an underlying transition function via interactions with an environment governed by some policy`.  The objective of MBRL is the same as model free RL - its just that that it achieves it differently. By augmenting the real experiences with simulated experiences, MBRL hopes to improve on sample efficiency over model free RL. `HPO … can be seen as a special use-case of model-based reinforcement learning developed under the guise of some idiosyncratic terms.` What is the basis of this statement?

Garnett et al (2012) - Garnett, Roman, Krishnamurthy, Yamuna, Xiong, Xuehan, Schneider, Jeff, and Mann, Richard. Bayesian optimal active search and surveying. ICML, 2012.
summary_of_the_review: I found the paper quite difficult to follow in some places, with parts of the method and experimental set ups not adequately discussed. I feel that adding an algorithm of the proposed method and signposting it to each section would go a long way towards clarifying the method. I would be willing to increase my rating if these issues get addressed, but even then I think the paper lacks sufficient novelty to elicit a strong recommendation from me.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a novel transfer learning approach to hyperparameter optimization (HPO) by formulating the problem in a model-based reinforcement learning (MbRL) framework. In this setting, the transition function serves as a surrogate model for learning the validation loss of any black-box ML model which is trained via an ensemble of probabilistic neural networks. The authors further introduce a lookahead search strategy to sample trajectories for a model predictive control problem. Experiments demonstrate that the proposed method outperforms several competitors on a few meta-datasets for HPO tasks.
main_review: Strengths: 
+ The paper proposes a novel search strategy based on model predictive control for HPO by utilizing the existing meta-data.
+ Empirical study highlights promising results compared to several competitors.

Weaknesses:
- The proposed approach does not conceptually match a MbRL problem and the MDP formulation seems unnecessary.
- Some technical contents in the paper are not well supported.
- The paper lacks a clear focus and the clarity of the overall approach should be improved.
- There is not enough details on the approach and experiments to reproduce the results.


Comments:
- One of the main concerns about the paper is the MDP formulation and characterizing the approach as planning in a MbRL framework, when there is no structured way to learn a policy and the setup does not conceptually match an RL problem. It seems that the approach will work just fine without mentioning the MDP and MbRL as it only predicts the transition function, in which, the next state is a target value (e.g., validation loss for a given HP configuration) that can be learned differently (vie supervised learning), that is a bit intangible in the RL context.

- The planning strategy selects k samples without exploiting the orders between different HPs, evaluates their immediate improvement (using the trained transition function), and picks the best-performing one, which is more of a search scheme rather than planning ahead, and resembles Monte-Carlo tree search or a contextual bandit approach.

- The transition function is trained to directly learn the HPs and is not used in the planning as a dynamic model.

- The literature on HPO is not properly covered, there are several works, particularly in the bandits domain, that are not discussed: 
*Li et al., A novel bandit-based approach to hyperparameter optimization, 2018  
*Falkner et al. BOHB: Robust and efficient hyperparameter optimization at scale, 2018  
*Tavakol et al, HyperUCB: Hyperparameter Optimization using Contextual Bandits, 2019

- The authors describe the HPO as a sequential decision-making problem, while based on my knowledge, this is not very common, and they also do not provide a supporting evidence or citation for this claim. Additionally, their approach, despite mentioned otherwise, does not exploit orders in selecting the next HPs.

- It is not completely clear from the paper how aggregating the outputs of the ensemble model relates to an “excellent” estimation of uncertainty mentioned in section 5.1.

- The clarity of the paper needs improvement as the ideas are not well organized and several parts mentioned in the paper, e.g., algorithm 1 and some details about the data and experiments could be better explained. They are also not not properly referenced.

- The link to transfer learning is unclear to me.

- For a paper that is weighted more toward an empirical paper, the experiments are small-scale and the approach is not outperforming all the baselines as stated at the beginning of the paper.

- The state space is of a higher-order Markov and is growing with time t. Could you elaborate how you deal with generalization in such a large state space and why it needs to also include the loss (in addition to HP configurations)?

- The authors do not provide a detailed overview of the optimization process, e.g. in Figure 2, what is the objective function, how the loss is optimized, etc.

- It is not completely clear how the dynamic is evolved. For instance, Eq. 6 shows that the state contains the previous HPs and their respective loss, and the model predicts the next state (\hat(\lambda)). How this works for t+1? Do you compute the true validation loss for the current HP and replace it in the next state or you keep using the predictions only?

- The paper could benefit from a proofread.


Questions:
- Could you clarify why methods such as BHBO are not considered as baselines?
- Could you explain the “rank” metric in more detail?
- What is the difference of “MPC-X” and “MPC-X (vanilla)” in figure 4?
summary_of_the_review: The paper contributes to the filed of automated ML by proposing a novel approach to find optimal HP configurations. However, the ideas contains several issues in terms of clarity, significance, and correctness of the claims, and I thus vote for a reject.
review_title: null

Meta-review: This paper proposes an algorithm for hyperparameters optimization that exploits a formulation as an MDP and thus makes use of a model-based reinforcement learning approach.

The formulation of HPO as an MDP although not novel (Jomaa et al. is not the only one to have considered this case, and the connection between the two was already known in the community) is indeed an interesting topic that could be impactful for the community. Unfortunately, the current manuscript is not providing much new insight into the topic.

After carefully reading the paper, I agree with Reviewers cKwe and 8eE4 that the current manuscript has several points of concern:
1) the formulation as a sequential decision-making problem is not fully elaborated 
2) lacking comparison to look-ahead (i.e., non-myopic) HPO algorithms (there is plenty of literature on Bayesian Optimization for doing this). This also makes it difficult to understand if the performance benefits come from the look-ahead or from the MDP formulation 
3) the writing is generally understandable, but some of the important design choices and details of the algorithms are not easy to find in the manuscript -- improving the clarity of the text would be very beneficial.

I encourage the authors to incorporate the feedback from the reviewer and to polish this paper into the shiny gem that it deserves to be.

Suggestions: 
- The MDP formulation for HPO might actually prove very beneficial for hyperparameters control (i.e., dynamically adjusting parameters during the learning process) where there is a real transition function rather than hyperparameters optimization. Might be worth reading https://arxiv.org/abs/2102.13651 which attempts to do hyperparameters control in the context of MBRL. 
- Adding better visuals to explain formulation and algorithm might go a long way.
- Tables 1 and 2 could be replaced by learning curves for a more intuitive way of visualizing the results.
Decision: Reject