Improving Hyperparameter Optimization by Planning Ahead
Authors: Hadi Samer Jomaa
Jonas Falkner
Lars Schmidt-Thieme
Venue: ICLR 2022 Submitted (2022)
Keywords: model-based reinforcement learning
hyperparameter optimization
model predictive control
meta-learning
transfer learning

Abstract: Hyperparameter optimization (HPO) is generally treated as a bi-level optimization problem that involves fitting a (probabilistic) surrogate model to a set of observed hyperparameter responses, e.g. validation loss, and consequently maximizing an acquisition function using a surrogate model to identify good hyperparameter candidates for evaluation. The choice of a surrogate and/or acquisition function can be further improved via knowledge transfer across related tasks. In this paper, we propose a novel transfer learning approach, defined within the context of model-based reinforcement learning, where we represent the surrogate as an ensemble of probabilistic models that allows trajectory sampling. We further propose a new variant of model predictive control which employs a simple look-ahead strategy as a policy that optimizes a sequence of actions, representing hyperparameter candidates to expedite HPO. Our experiments on three meta-datasets comparing to state-of-the-art HPO algorithms including a model-free reinforcement learning approach show that the proposed method can outperform all baselines by exploiting a simple planning-based policy. 

Under review as a conference paper at ICLR 2022
IMPROVING HYPERPARAMETER OPTIMIZATION BY PLANNING AHEAD
Anonymous authors Paper under double-blind review
ABSTRACT
Hyperparameter optimization (HPO) is generally treated as bi-level optimization problem that involves fitting a (probabilistic) surrogate model to a set of observed hyperparameter responses, e-g. validation loss, and consequently maximizing an acquisition function using surrogate model to identify good hyperparameter can- didates for evaluation. The choice of a surrogate and/or acquisition function can be further improved via knowledge transfer across related tasks. In this paper, we propose a novel transfer learning approach, defined within the context of model- based reinforcement learning, where we represent the surrogate as an ensemble probabilistic models that allows trajectory sampling. We further propose a new variant of model predictive control which employs simple look-ahead strat- egy as a policy that optimizes a sequence of actions, representing hyperparameter candidates to expedite HPO. Our experiments to state-of-the-art HPO algorithms including a model-free reinforcement learning approach show that the proposed method can outperform all baselines by exploit- ing a simple planning-basn baaee policy.
three meta meta-datasets comparing
INTRODUCTION
Hyperparameter optimization (HPO) a ubiquitous problem within the research community and an integral aspect of tuning machine learning algorithms to ensure generalization beyond the training data. HPO is often posed as sequential Decision-making process, however, can be seen as special use-case of model-based reinforcement learning (MbRL) (Sutton 1991a Henaff et al. 2017) developed under the guise of some idiosyncratic terms.
In MbRL. the objective is to train transition model to approximate an underlying transition function via interactions with an environment governed by some policy, e.g. random shooting (Nagabandi cet al., 2018), to improve sample efficiency and learn from more useful interactions whereby the learned models are used as a simulator for sampling trajectories. An agent navigates the simu- lated environment to optimize a pre-defined reward function, while the transition model remains unchanged. Conventionally in HPO, a surrogate model is trained to estimate some black-box func- tion, e.g. validation loss of machine learning algorithm under investigation (Rasmussen 2003 Snock et al. [2015b] Springenberg et al. 2016). An acquisition function (Wilson et al.) 201 7b) in- teracts with the surrogate model propose potential hyperparameters that optimize the black-box response, viewed as a reward function. Effectively, the surrogate model is the only unknown com- ponent for transition model, that prevents HPO from being framed fully as MbRL problem.
In this paper, we present a novel formulation for HPO defined within the context of MbRL. Namely, we learn an ensemble of probabilistic neural network models [Lakshminarayanaa) aynan al., 2017) and show that using model predictive control (MPC) (Kamthe & Deisenroth 2018) and a novel look- ahead variant to navigate the simulated black-box environment, we can outperform conventional Bayesian optimization techniques with heuristic acquisition functions in both transfer and non- transfer learning settings. Thus, we claborate on the importance of explicit planning in HPO that has been largely overlooked by the community. We also formally define HPO as a Markov deci- sion process (MDP) with a simple, yet novel, state representation as the set of previously evaluated hyperparameters and their corresponding responses.
We argue that with a clearly defined transition model, we can replace the acquisition function with a simple policy that maximizes the reward across the simulated trajectories, and achieve better results

---

Under review as a conference paper at ICLR 2022
Layout Mol-2
Regularization Md
Optimization Mo-v2
2.3
2.4
2.3
2.2
2.3
2.2
2.2
2.1
2.1
2.0
1.9
1.9
1.8
Namber of Thals
Number of Thats
Number Tials
Lookatnedd NPC.5
Figure 1: Motivating the effect of planning for HPO.
through proper planning, as shown in Figure I LookAhead MPC-5 our look-ahead strategy with MPC that simulates states ahead. We focus here on the motivation with more details in Section
Our main contributions are summarized as i) formal definition of HPO as an MDP which does not depend any engineered heuristics, ii) new transfer learning surrogate model represented by an ensemble of probabilistic neural networks, iii) a novel acquisition function that implements look-ahead strategy paired with model predictive control, iv) clear motivation that highlights the impact of planning in HPO, which to the best of our knowledge, has never been addressed before.
2 RELATED WORK
Hyperparameter optimization has been extensively studied in the community within two main set- tings: single task (non-transfer learning) HPO and transfer learning HPO.
For single task HPO, solutions often involve estimating the hyperparameter response surface using (probabilistic) surrogate, such a Gaussian process (Rasmussen [003]] [Bergstra al.] (011), random forests (Hutter et al. 2011). Bayesian neural networks (Springenberg et al.) 2016)) or some hybrid approach (Snoek al. 2015a). Hyperparameter candidates are then selected via an acquisi- tion function (Wilson et al. 20177), e.g. expected improvement, that satisfies some well-motivated assumptions and utilizes the statistics from the posterior score each hyperparameter.
Transfer learning solutions on the other hand leverage available data from experiments on related tasks to expedite HPO. Simple solutions involve initializing the surrogate using hyperparameters that perform well on datasets that have similar meta-fess meta-features (Feurer et al. [0144] Jomaa et al.) 2021a), i.e. dataset statistics. The response surface can also be modeled jointly multi-task set- ting (Bardenet et al.) 2013: Yogatama & Mann 2014 Perrone et al. 2018). For example, [Salinas] al. (2020) transforms the hyperparameter response into similar distribution and learns shared Gaussian Copula. Another way to achieve transfer learning is through weighted combination of the surrogates (Wistuba et al.] 2016 Feurer et al.] 20188). More recently, it has been shown that meta-learning initialization of the surrogate by training estimate the response across the train- ing tasks, and subsequently fine-tuning to the target task improves generalization and leads to better performance (Wistuba & Grabockal |02 Jomaa et al. (2021b). Additionally one learn trans- ferable acquisition function (Wistuba et al. 2018 Jomaa et al. 2019 Volpp et al. 2020). [Jomaa et al.
2018|Jomaa 2019) propose learn a policy based on Deep Q-learning Watkins & Dayan 1992) where they
assume temporal dependency between the hyperparameters and propose an LSTM STM (Hochreiter & Schmidhuber 1997) model it. |Volpp et al. (2020) propose transferable acquisition function, MetaBO, as a policy that is meta-trained MetaBO trained by interacting with the environment and observes an engineered state represen- tation that is heavily influenced by underlying surrogates. MetaBO however is very sensitive to the number of trials (Wistuba & Grabocka 2021). We discuss the proposed MDPs and their relationship our proposed approach in Section
related tasks. As another model-free RL. approach,
Model- based reinforcement learning aims to learn model of the environment by estimating the next state given state-action pair. It is applied with great success in areas such as robotics and video games. Existing methods similarly involve learning a (probabilistic) model of the dynamics (Chual et al. 2018 Nagabandi al. 2018 Ko et al. 2007). Moreover, several methods improve the policy
2

---

Under review as a conference paper at ICLR 2022
in Dyna style approach (Sutton 1991a). by leveraging the generated trajectories from the model to mitigate the distribution shift.
PRELIMINARIES
Hyperparameter Optimization: Let D denote the space of all datasets and A E RR- the space of hyperparameters associated with unknown black-box function ((D) A + R where ((D) represents the response (X), e.g. validation loss, of certain model under investigation trained on dataset D D with hyperparameters À A. For example, we might be interested a neural network model where A == R+ x iR x N represents the learning rate, dropout rate, and the number of layers, respectively. The objective of HPO hence to find the optimal hyperparameter configuration X := arg minxeA (A)) given fixed budget of trials.
HPO is commonly treated as sequential decision makiing process, where a surrogate model é A + I is iteratively fit to the history H1!) == {),, (A:)) of evaluated hyperparameters and an acquisition function (A x R)) + is used to select the next candidate which minimizes the expected hyperparameter response:
arg min En-pp (t
where ((Ar, x=))== (1.. ((D) (2,). A; == ; (x,)) and pv is some distribution over the available datasets. Among the variety of acquisition functions, the expected improvement is widely adopted (Mockus) 1974). Although HPO algorithms can be applied on both continuous and discrete search spaces, here we focus on discrete spaces because they allow for faster training in transfer learning settings (Schilling et al. 2016 Jomaa et al. (0211).
Model-based reinforcement learning: MbRL uses interactions with the environment to learn parameterized approximation of the underlying transition function which subsequently can be used for planning. The learning task can be formulated as a discrete-time Markov decision process (MDP) defined by the tuple (S,A, r,r) with state space S, action space A, unknown transition function T : S x A + S and reward function : S A + R. The general approach RL is to learn the optimal policy maximizing the expected cumulative discounted reward given by ((T) :=
E 21=1 > 7'r(st, az) = -(5--1,, a =(s:) where s, E S and E A with discount factor 7 E [0. We note that the explicit reward function or an approximate reward model is necessary for planning actions under some learned transition model. During inference, the optimal sequence of actions is commonly presented as the solution to the following optimization problem:
arg IIxX
I f (f(st, at), ac+1).
(2)
(1 Cly
1=
which leverages the transition model to predict the next state 81+1 given a state 8, and action dz.
HYPERPARAMETER OPTIMIZATION VIA MODEL-BASED RL
In the following we formulate HPO as MDP which in turn facilitates the use of variety of RL approaches to solve the problem. As a sequential decision-making process, hyperparameters À e A iteratively selected and evaluated forming simple MDP. Particularly, the state space is defined as S = (A x RR) with
= (Ar, (4,, 4P))
(3)
where ( (2x) and action space A := A. The ground truth transition function 7 S S simply appends the new observations to the previous ones,
T( (D) A) = (AA,,)))) (PP)) (A,, (() (A, ((P)
for dataset D and ground truth reward function, r = S A - R, that returns the loss reduction of the new hyperparameters over the best loss so far,
r(
(D) (D), .)) = max(0, min (D) min (D) ,..., (D) ())
(5)
==lit
3

---

Under review as a conference paper at ICLR 2022
To estimate both, the transition function and the reward function, as done in MbRL, the only missing
piece is model for the validation loss, the called surrogate model. Given (ID) A R, the transition and reward models are just: f : S x A S where
(2, (()()),) = ( (
(,, ())())))
(h,
(6)
and f: P + with
(D) A) = max min ( minte ,....,e! F(s)
P. (P (A)))
(7)
In this sense, HPO using surrogate model can be seen as special case of MbRL, and thus all approaches researched for MbRL can be directly applied to HPO. When learning a joint surrogate across several datasets in D, the superseript is dropped and ( +
To the best of our knowledge, so far there exist two prior publications which attempt to solve HPO using RL by defining distinct MDPs. Jomaa et al. (2019) originally define the state as the dynamic history of evaluated hyperparameters and their respective responses S (A RR" and (D) as defined earlier. However, they suggest that the order by which the hyperparameters are selected impacts the decision on which action to take next, which is why they model this temporal aspect via an LSTM in their policy, which is conditioned on some engineered meta-features. Their transition function generates new states by appending the action and the observed response to the previous
state according to T(s
(D) A)
() ................................ (2( 42), (2, ((,)())())))) Volpp et al (2020)
represent the state space as S := (R with == ((H) of As, e)) where be and of are the mean and variance of the posterior distribution of an underlying surrogate and ei are some engineered attributes. The transition function simply updates the parameters of the surrogate based on the new observations. Both approaches define the action space as the discrete grid of hyperparameters, A == A, whereas the reward function is computed in terms of the regret.
In strong contrast to these methods that replace the standard acquísition function with a policy trained via model-free RL approaches, we adopt the powerful yet simple MDP proposed by Jomaa et al. 2019) to train a transition model that allows planning to improve HPO in the context of MbRL.
METAPETS ALGORITHM
In this section, we present our model- reinforcement learning algorithm. Namely, we define the transition function, our novel look-ahead acquisition function, and the associated training procedure.
5.1 PROBABILISTIC TRANSITION FUNCTION
Model-based reinforcement learning is mainly concerned with learning a parameterized model the environment which approximates the underlying dynamics, or transition function, S x A - S. such that the next state can be estimated given the current acatenaction pair. Following the standard RI. notation, we denote by actions a hyperparameters X, i.e. (P) = (A)) Accordingly, the choice of the model class plays important role. A common approach to model the environment is through Gaussian processes (GP) (Rasmussen & Kuss 2003 Ko et al.) 2007 Boedecker et al.) 2014) or mixture of GPs (Khansari-Zadch & Billard 2011 which provide uncertainty estimates for the predictions and, more importantly, for unexplored areas. However, such models suffer from the curse of dimensionality and thus difficult to scale nigh-dimensional domains. Deep neural net- works on the other hand have shown great success in handling uncertainty estimation (Springenberg al. 2016 Gal & Ghahramani 2016: Lakshminarayanan al., 2017). with success in predictive modeling of images (Watter et al. 2015 and short-horizon control tasks for high sigh-dimensional data.
In our case, the state is represented as the history of selected hyperparameters and their responses, up to a given time, while the actions consist of the available hyperparameters to be evaluated. Hence there only one missing piece of information that is required to establish the next state, and that is the response of the respective selected action (Figure 2). This missing element we ac- quire by training the transition model to directly estimate the response for given state-action pair.
4

---

Under review as a conference paper at ICLR 2022
Specifically. parameterize our transition function fo(st at): A R IR+ as a probabilistic neu- ral network with parameters 0, which takes as input the current state-action pair (s!²), a;) and outputs the defin- ing parameters of probability distribution function, e.g. Gaussian distribution. The governing assumption for this well-established approach is that the instance is de- rived from a heteroscedastic Gaussian distribution, i.e. the variance is not fixed between instances. Accordingly, the network outputs the respective mean Ai E Ry and vari- ance 03 R+ and is trained by minimizing the negative log-likelihood:
A A2
R
&
Figure 2: Architecture of the transition function
log (og + (BD) (o() a))²
DDDD
+ ct
2ō ( a)²
Thus, we define the predicted hyperparameter response as ((a) ~ N do, We then estimate s iss 5 U(a, E(a)). A simulated rollout is presented in Algorithm
The Case for Ensembles:
Probabilistic neural networks can capture and model the aleatoric uncer- tainty of the environment (e.g. observation noise) by parameterizing a suitable distribution. How- ever, single probabilistic network is not capable of capturing any epistemic uncertainties (e.g- lack of sufficient amount of data to completely determine the real dynamics). In particular, the epistemic uncertainty vanishes in the limit of infinite data but can have major impact in low data regimes like HPO. For that reason, it of major importance to account for epistemic uncertainty.
A principled approach to deal with this issue is by bootstrapping models in ensemble. An semble of deterministic models, such as standard neural networks, reduces bias and epistemic un- certainty. Therefore, capture both, aleatoric well epistemic uncertainty, bootstrapped ensemble of probabilistic neural networks, which has been shown to outperform single probabilistic networks as well as other Bayesian approaches (Chua al. 2018
employ simple
Given such ensemble of probabilistic transition models denoted as ffo, NE observe mixture of Ng distributions, i.c. {N (Ao, 6%.) NE Following Lakshminarayanan aggregate the outputs as
2017)
1 N pel. a) = NE Σ fe a,),, and (.) = NE (3 + (., (.)))))) (9)
Ng
leading to a predicted hyperparameter response in the form of E(a) N (Ma (s))),)),)) (-,a),aa a)). The explicit advantage of the ensemble approach in particular is the simple but excellent estimation of uncertainty which constitutes one of the main performance drivers in HPO in recent years.
5.2 TRAINING THE DYNAMICS MODEL
The first step in MbRL approach is generate dataset of transitions as training dataset for meta-dataset of primary
dynamics model. We denote by & = {D... (Ae, ((P.) (2:)) J.Y datasets, hyperparameters and their responses sampled from some unknown distribution of datasets PD and some unknown distribution of hyperparameters PA- We generate training dataset Erai from the meta-dataset by sampling a quadruple (Dn, (Da)
(Pa) a,, (P))(a)) of datasets D.. sets of hyperparameters and their responses as states (D.). unobserved hyperparameters actions dr, and
the missing element from the next state (Dn) as the response to o, i.e. ((D)) This is similar to the usually used approach of implementing random policy to generate the trajectories (Nagabandi et al. 2018). Although it is possible to augment the grrain with trajectories generated using MPC, noticed that it is time-consuming and ii) does not lead to significant improvement compared to using simple random trajectories. We conjecture that this is due to the nature our transition function. Specifically, because the sequence of hyperparameters selected (by any policy) does not
5

---

Under review as a conference paper at ICLR 2022
immediately affect the output of the transition model, i.e. r(s|_1, a) =((b .)) Va E A but rather the output is only affected by the selected action.
Since our state 8t is defined the collection of previously evaluated hyperparameters and their corresponding responses, we employ deep set (Zaheer et al. (2077) formulation to encode 9; into fixed-size vector representation.
contrast to the common approach of conventional model-based RL algorithms which are concerned with predicting the state difference A6 we estimate the response (a) to given action while considering the state as context information (Kim et al. 2019). Concretely, the transition model is defined as:
A. -
0=0²=
(
(0r, 1
(10)
such that := fog where g := AXR R. and f: AXRN, R X are feed-forwart neural network models, and [] represents standard concatenation. Given the variety of primary datasets, we use first-order meta-learning (Nichol al., 2018) to optimize the parameters our transition model. We summarize the training procedure in Algorithm
5.3 PLANNING WITH THE LEARNED TRANSITION FUNCTION
Given the learned dynamics model, we subsequently require approach that effectively lever- age the acquired information to predict the optimal sequence of hyperparameters to evaluate. For example, Dyna-style algorithms (Sutton| [991b) use the simulated experience mitigate the distri- bution shift (Luo et al., 2019, Kurutach et al.) 2018 Clavera et al.) 2018) by integrating trajectories that have been generated via the policy further train the transition model.
5.3.1 MODEL. PREDICTIVE CONTROL
For simplicity, here we consider the well-known approach of model- predictive control (MPC) which has been used in many complex control scenarios (Bouffard et al. [0122] Lenz et al. 2015 Amos et al. 2018). Furthermore, it is easy to implement and does not require any gradient computation.
The general idea of MPC is solve an optimization problem in specific horizon on top of the learned transition model to produce sequence of actions. In particular we employ simple random shooting (RS) (Zhou & Yan 2014) technique solve the following optimization objective:
(D) ..)).
arg max f(5 {a
(11)
Notice that since the reward is measured in terms of the regret we do not need to sum over the rewards at each simulated state, as presented in Equation but simply observe the reward at the final state in the rollout.
The RS policy generates K random action sequences with a horizon of length H from a uniform distribution and evaluates them via the learned transition function. Then usually only the first ac- tion of the best candidate sequence is executed and the procedure is repeated from the new state. Finally, also fine-tune the transition model(s) given the newly evaluated hyperparameters on the test datasets to mitigate the distribution shift, which is common practice in Bayesian-based optimization techniques.
5.3.2 LOOKAHEAD MPC
Conventional acquisition functions are agnostic to the order by which the previous hyperparameters have been selected, i.e. a(Hfe) = à a(Φ(H.)) where e is some permutation function since they are only concerned with the immediate improvement and do not account for how the optimization process evolves. MPC on the other hand normally selects the first action from the trajectory that had the highest reward at the end the rollout. This can be slightly misleading as it implies that arrive at the best hyperparameter
0.3
0.1
0.8
0.4
Figure 3: Simulated Trajec- tory rewards. MPC selects the first item. LookAhead MPC selects the third item.

---

Under review as a conference paper at ICLR 2022
configuration it is necessary sequentially evaluate bad configu- rations. However, this not true since the selection of hyperpa- rameter configurations is independent and all configurations can be selected at all times. Thus, we propose to directly select the action that provides the highest estimated reward, i.e. lowest regret, from all the actions observed across all the simulated trajectories (Figure
5.3.3 TRAJECTORY SAMPLING
The learned transition model outputs a distribution over the next state 8441. To leverage the uncer- lainty associated with the output response, create particles by randomly sampling p variations as 2(P) ~ +(s: (D) aa)) and propagate through the trajectory of every particle. At each step, the reward then estimated as the average of the rewards across all particles:
P
F(3 P) a1)
1
Σ
D)
.a.).
(12)
6 EXPERIMENTS
The experiments are designed address the following questions: 1)Does the learned transition model generalize to new unseen tasks? 2) What is the importance of planning when doing HPO?
6.1 META-DATASET
We evaluate our approach on three hyperparameter search spaces for feed-forward neural networks (Jomaa et 2021b), that includes 120 UCI classification datasets (Asuncion Newman 2007).
We refer to the meta-datasets as Layout Md-v2, Regularization Md, and Optimization Md-v2 which include the Cartesian product of the individual hyperparameters with total of 324, 288, and 432 unique configurations, and 10. 7, and 12-dimensional hyperparameters, respectively. More information presented in Appendix
6.2 TRAINING THE TRANSITION MODEL
The transition model is trained via first-order meta-learning. Each meta-dataset is divided into splits with 80 training datasets, 16 for validation and 24 for testing. We used a task batch size of 8 and 64 mini-batches per task. The number of inner iterations was set to 5. We use the ADAM (Kingma & Ba 2015) optimizer with a learning rate of 0.001. The hyperparameters were tuned on the validation set and trained for 10000 outer iterations with early stopping. We trained a total of distinctly initialized models for the ensemble using Tensorflow (Abadi et al. 2016).
6.3 BASELINE MODELS
We compare against several single tasks and transfer learning baselines for HPO designed for black- box function optimization that rely only on the hyperparameters and the corresponding response. As such, some methods were not considered, e.g. Jomaa et al. (2021b): Falkner et al. (2018). Partic-
ularly, we compare against, Random sampling (Bergstra Bengio 2012), GP (Rasmussen) (003).
SMAC (Hutter et al. 2011). BOHAMIANN (Springenberg et al. 2016), TST-R (Wistuba et al.
2016), ABLR (Perrone al. 2018). CTS (Salinas et al. 2020), and FSBO Wistuba & Grabocka,
2021 A detailed overview can be found in Appendix C
We denote by LookAhead MPC-HX, our approach, which employs LookAhead MPC by navigating the simulated environment using random shooting up to the defined horizon X and then selects the first action of the trajectory that achieves the highest reward. We sample 1000 trajectories for our method. The models of the ensemble are iteratively fine-tuned the new hyperparameters all baseline models. In Algorithm 3 we present the pseudo-code for our MetaPETS algorithm and provide the source code in the Supplementary material.
7

---

Under review as a conference paper at ICLR 2022
Layour Md-v2
Regularization
Optimization Md-v2
Methods (Trials
15
33
50
15
33
50
15
33
50
RS
6.48 0162
6.81
0.38
6.78
4: 0.37
7.11
4: 0.49
7.33 4 0.30
7.23
12330.1. 0.11
7.19
0.55
6.93 = 0.44
7.07
0.27
BOHAMIANN
6.62 0.05
6.48
0.26
6.29
0.17
6.51
6.23
0.14
5.86
6.30
GP
6.28 0.50
5.73
+ 0.19
5.22
+ 0.07
5.48
0.19
4.32.+0.33
5
5.92
5.76 0.11
5.53 4-11
SMAC
6,06 0.24
6.19
+ 0.32
6.36
:: 0.18
6.22 0:30
6.34
6.43
0.20
6.24
0
6.27 0:29
5.74
5.89
5.80
1.10
5.98
0.25
ABLR
7.11 0.50
6.78 0.36
6.55
+ 0.27
6.72 0.39
6.36
6.03
0:2
3.80
5.79
= 0.20
5.72 0:225
TST-K
5.54 0.23
5.42 + 0.17
5,49
+ 0.11
5.50 0.17
5.48 0.00
+.13
5.75
0.30
5.59
0.15
5:57 0:11
MetaBO
6,85
7.02
0.35
702
7,42
+.60+009
6.69 0.15
7.00
7.36
FSBO
5.31
0.29
5.35 +: 0.27
5.31
:: 0.15
5.50
0.14
5:45
0.17
5,33
0.17
5.39
0.18
5.38 1: 0.28
LookAhead MPC-3
5,18
0.16
5.31 d: 0.21
5.26 4: 0.38
5.10 0.15
5.13 0.12
5,15 d 0.10
5.35 à 0.15
5.36
0.22
5.36 0.24
LookAhead MPC-5
5.32
0.06
5.32 + 0.35
5,3
5.14
0.20
5.3++011
5.23
5.26 + 0.233
Table I: Average rank. We report the best results in bold and underline the second best.
Layout Md-v2
Regularization
Optimization Md-v2
Methods (Trials
50
15
33
50
RS
7.90 = 1.59
5.59
+ 0.98
4.25 + 0.64
8.54
+ 0.84
6.18 + 0.56
4.51 + 0.30
8.53
0
5 = 0.39
4.64 + 0.32
BOHAMIANN
4.78
7:59
4.30
0.40
0.48
GP
7,90 1.06
4.25 1: 0.26
3.12 1: 0.18
6.05
1: 0.27
3.46 1 0.30
2.49:00.9
6.95 0.37
4.51 0.17
3.38
008::000 0.20
SMAC
7.56 0.36
4.68
2 0:47
3.66
:: 0.28
7.31
0.77
4.81
+ 0.38
3.73 0.44
6.96
0.42
4.61
0.33
3.69
4-033
CTS
6.39
4.05
3.32
1: 0.38
6.56
0.59
3.82
19221000 0:24
0.09
6.67
4.59
0.39
3.58
ABLR
4.26
0.44
7.99
0.57
4.98
7.30+
3.64
TST-R
6.70= 0.36
3.90
2: 0.36
2,91
0.33
6.51
0.39
3.88
0:29
2.73
0.39
6.78 0.21
4.13=023
3.21 1:100
MetaBO
7.24 0.27
4.72 1: 0.32
3.86
1: 0.28
7.71
0.11
5,72
4 0.20
5.01
2:21::011
8.63
5.83 0224
5.15 0,19
FSBO
2
0.29
5.99
0.57
2.32
3.90
3.04
LookAhead MPC-3
6.15 = 0.49
3.65 4: 0.42
2.60
+ 0.32
5.50 +: 0.39
2,98 4: 0.34
1.87 0.14
6.18 0.73
4.18
0.39
3.16 0.30
LookAleed MPC-5
6,25 0.51
3.90 + 0,21
2,85
+ 0.38
5.34
+ 0.09
2,97
0.29
191
0.188
6.15 0.47
3.7
= 0.67
2,95 + 0.47
Table 2: Average normalized regret. We report the best results in bold and underline second best.
6.4 EVALUATION METRICS
We report two performance metrics to evaluate the effectiveness of the different baselines: normal- ized regret that represents the distance between the response of the evaluated hyperparameters and the optimal performance for each dataset, and the rank that measures the relative performance of each method compared to the rest of the baselines at each trial. The rank is computed at the task level and is agnostic to the heterogeneous ranges of the target response surfaces, thus can better signify the difference in performance.
6.5 RESULTS AND DISCUSSION
We report in Tables I and 2 the average rank and normalized regret, respectively, over 5 runs for 50 trials with three different seeds per run for all methods. Each transfer learning approach had access the same training datasets and all models were initialized via the same three seeds to fit the initial surrogate. Overall, our LookAhead method outperforms the baselines across all meta-datasets. We use the horizon of H3 and H5 in this section, and study the contribution of LookAhead and the importance of planning in Section 6.6
LookAhead MPC-3 and LookAhead MPC-5 demonstrate clear gains in the average rank with con- sistent performance against the baselines. Looking at the normalized regret, FSBO outperforms our model in some cases, e.g. Layout Md-v2, however, the associated average rank is still lower. This can be attributed to the fact that different tasks have heterogeneous response distributions. When there is a clear contradiction between the rank and the normalized regret, this signifies that the mar- gin of difference of the normalized regret on few tasks where FSBO shows stronger performance, is higher than where it has lost.
Comparing LookAhead MPC-3 and LookAhead MPC-5, we notice that a shorter horizon is suffi- cient to explore smaller search spaces. Albeit, early exploration in the form of longer horizons can still turn out to be more favorable, e.g in Regularization Md.
is also worth mentioning that MetaBO, the model-free RL approach, is the worst across all meta- datasets. This is attributed to the asymptotic degradation of the performance with the increasing number of trials, as pointed out in Wistuba & Grabocka (202T). Contrary to MetaBO, which relies on underlying surrogate generate the state which is bound by fixed grid, our approach is
8

---

Under review as a conference paper at ICLR 2022
Layout Md-v2
Regularization Md
Optimization Mo-v2
6.2
6.0
5.6
5.8
3..
5.4
5.4
5.0
5.0
Number Thale
Number of Thais MPC (vanila)
Number of Titals
MPC11 (vanilia)
LockAhiedd MPC33 (venila) LockAhoad MPC-3
LookAneed MPC-5 (vanilia)
MPC5 (vonlla)
M9PC1
MPC-3
LookAneed MPC-3
MPC-
Layout Mdtv2
Requlenization Md
Optimizetion Ma-v2
2.05
2.80
2.65
2.00
2.55
2.50
2:55
2:50
2.50
2:50
2.45
2.45
2.40
2.40
2.40
7.35
30
Number of Triass
Number Thills
Numbar Vials
Figure 4: (top) Investigating the impact of planning and LookAhead across different horizons: (bot- tom) Increasing the number of sampled trajectories improves planning (solid lines-> 1000 trajecto- ries vs. dashed lines 100 trajectories).
agnostic to the size of the grid and improves with more trials, leading to an improved state represen- tation.
6.6 ABLATION
We further investigate the effect of LookAhead on MPC with different horizons and trajectory sam- ples to evaluate the impact of planning under such conditions. Moreover, we compare against vanilla version, that does not involve fine-tuning the transition model to the observations of the target dataset. We summarize
performance in Figures 4 and share the following insights: Fine-tuning plays crucial role in improving performance. Contrary to existing MbRL solutions that evaluate a policy's ' performance on the environment which the transition model has been trained to estimate, we evaluate faces. We notice that with a few gradient update steps, (LookAhead) MPC-X outperforms the vanilla variant. ii) Using LookAhead MPC is better than standard MPC. This reinforces the notion that via proper planning the evaluation of bad hyperparameter configurations can be avoided. iii) Increasing the number of sampled trajectories leads to a better outcome. This is be expected, considering that for random shooting, we sample a fixed number of trajectories to evaluate. At any given trial f, the number possible trajectories for a horizon H on a grid of size N is (N--)). so for a horizon H we default to maximization over the grid.In Appendix D we further discuss the effect of various plausible dynamics by bootstrapping different sets of models as part of the ensemble.
policy
environments, i.c. new tasks, with varying response sur-
7 CONCLUSION
In this paper, we present novel solution for HPO within the context of MbRL. Specifically, we de- sign a new surrogate as an ensemble of neural networks, which is initialized through meta-learning and thus can adapt quickly to new target datasets with few observations. We propose a novel acqui- sition function based on model predictive control that utilizes simple lookahead strategy to select good hyperparameter candidates from simulated trajectories. To the best of our knowledge, the first to investigate the impact planning on HPO and present an extensive ablation study that motivates research in that direction.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenherg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Va- sudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2-4, 2016. pp. (65-283, 2016
Brandon Amos, Ivan Dario Jimenez Rodriguez, Jacob Sacks, Byron Boots, and Zico Kolter. Dif- ferentiable MPC for end-to-end planning and control. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Biance and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 31: Anmual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8299888991111 2018
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michèle Sebag. Collaborative hyperparameter tuning. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pp. 199-207, 2013.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. J. Mach. Learn. Res., 13:281-305, 2012.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parametts parameter optimization. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2. Proceedings of a meeting held 12-14 December
2011, Granada, Spain, pp. 2546-2554, 2011.
Joschka Boedecker, Jost Tobias Springenberg, Jan Wülfing, and Martin A. Riedmiller. Approximate real-time optimal control based on sparse gaussian process models. In 2014 IEEE Symposium on Adaptive Dynamic P Programming and Reinforcement Learning, ADPRL 2014, Orlando, FL, USA, December 9-12, 2014, pp. 1-8, 2014.
Patrick Bouffard, Anil Aswani, and Claire Tomlin. Learning- based model predictive control on a quadrotor: Onboard implementation and experimental results. In IEEF International Conference on Robotics and Automation, ICRA 2012, 14-18 May, 2012, St. Paul, Minnesota, USA, pp. 279- 284. IEEE, 2012.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural In- formation Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 4759-4770, 2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. In 2nd Annual Conference on Robot Learning, CoRL. 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, pp. 617- 629. 2018
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: robust and efficient hyperparameter opti- mization at scale. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockhelmsmüssan, Stockholm, Sweden, July 10-15, 2018. pp. 1436-1445, 2018.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Using meta-learning initialize bayesian optimization of hyperparameters. In Proceedings of the International Workshop on Meta-learning and Algorithm Selection co-located with 21st European Conference on Artificial Intelligence, MetaSel@ ECAI 2014, Prague, Czech Republic, August 19, 2014, pp. 3-10, 2014.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian opti- mization. CORR, abs/1 802.02219, 2018
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML. 2016, New York City, NY, USA, June 19-24, 2016, pp. 1050-1059, 2016.
10

---

Under review as a conference paper at ICLR 2022
Mikael Henaff, William F Whitney, and Yann LeCun. Model-based planning with discrete and continuous actions. arXiv preprint
2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 98)) 1780, 1997.
Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization 5thh International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers, pp. 507-523, 2011. 2011.
Hadi S. Jomaa, Josif Grabocka, and Lars Schmidt-Thieme. Hyp-rl: Hyperparameter optimization by reinforcement learning. CoRR, abs/1906. 1527, 2019.
Hadi S. Jomaa, Lars Schmidt-Thienee and Josif Grabocka. Dataset2vec: learning dataset meta-
features. Data Min. Knowl. Discov., 35(3):964 985, 2021a.
Hadi S. Jomaa, Lars Schmidt-Thieme, and Josif Grabocka. Hyperparameter optimization with dif- ferentiable metafeatures. CoRR, abs/2200.00376, 2021b.
Sanket Kamthe and Marc Peter Deisenroth. Data-efficient reinforcement learning with probabilistic model predictive control. In International Conference on Artificial Intelligence and Statistics, AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, pp- 1701 1710, 2018
Seyed Mohammad Khansari -aans and Aude Billard. Learning stable nonlinear dynamical systems with gaussian mixture models. IEEE Trans. Robotics, 27(5): 957, 2011.
Hyunjik Kim, Andriy Mnih. Jonathan Schwarz, Marta Garnelo, S. M. Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Jonathan Ko, Daniel J. Klein, Dieter Fox, and Dirk Hähnel. Gaussian processes and reinforcement learning for identification and control of autonomous blimp. In 2007 IEEE International Conference on Robotics and Automation, ICRA 2007, 10-14 April 2007, Roma, Italy, pp. 742- 747, 2007.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
Balaji Lakshminarayanam. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6402-6413, 2017.
Ian Lenz, Ross A. Knepper, and Ashutosh Saxena. Deepmpe: Learning deep latent features for model predictive control. In Lydia E. Kavraki, David Hsu, and Jonas Buchli (eds.), Robotics: Science and Systems XI, Sapienza University of Rome, Rome, Italy, July 13-17, 2015, 2015.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In 7th In- ternational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Jonas Mockus. On bayesian methods for seeking the extremum. In Optimization Techniques, IFIP Technical Conference, Novosibirsk, USSR, July 1-7, 1974, pp. 400-404, 1974.
11

---

Under review as a conference paper at ICLR 2022
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dy- namics for model-based deep reinforeement learning with model-free fine-tuning. In 2018 IEÉE International Conference Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, pp. 7559-7566, 2018.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. CoRR, 2018.
Valerio Perrone, Rodolphe Jenatton, Matthias W. Seeger, and Cédric Archambeau. Scalable hyper- parameter transfer learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréél, Canada, pp. 6846-6856, 2018.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Advanced Lectures on Ma- chine Learning, ML Summer Schools 2003, Canberra, Australia, February 2-14, 2003, Tithingen, Germany, August 4-16, 2003, Revised Lectures, pp. 63-71, 2003.
Carl Edward Rasmussen and Malte Kuss. Gaussian processes in reinforcement learning. In Ad- vances in Neural Information Processing Systems 16 [Neural Information Processing Systems, NIPS 2003, December 13, 2003, Vancouver and Whistler, British Columbia, Canada], pp. 751- 758, 2003.
David Salinas, Huibin Shen, and Valerio Perrone. quantile-based approach for hyperparameter transfer learning. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pp. 8438-8448, 2020.
Nicolas Schilling, Martin Wistuba, and Lars Schmidt-Thiemes Scalable hyperparameter optimiza- tion with products of gaussian process experts. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I, pp. 33-48, 2016.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, and Ryan P. Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 2171-2180, 2015a.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, and Ryan P. Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on Machine Learning, ICMI. 2015, Lille, France, 6-11 July 2015, pp. 2171-2180, 2015b.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 1134-4142, 2016
Richard S. Sutton. Planning by incremental dynamic programming. In Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA, pp. 353-357, 1991a.
Richard S. Sutton. Planning by incremental dynamic programming. In Proceedings of the Eighth International Workshop (ML91), Northwestern University, Evanston, Illinois, USA, pp. 353-357, 1991b.
Michael Volpp, Lukas Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in bayesian optimiza- tion. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Christopher J. C. H. Watkins and Peter Dayan. Technical note q-learning. Mach. Learn., 8:279-292, 1992.
12

---

Under review as a conference paper at ICLR 2022
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin A. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in Neu- ral Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 2746-2754, 2015.
Andrew Gordon Wilson and Zoubin Ghahramani. Copula processes. In Advances in Neural In- formation Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada, pp. -4600-2488, 2010.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning. CoRR, abs/151 L02222, 2015.
Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017,
December 4-9, 2017, Long Beach, CA, USA, pp. 4148 4158, 2017a.
Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 3148-4158, 2017b.
Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In 9th International Conference on Learming Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Martin Wistuba, Nicolas Schilling, and Lars for automatic hyperparameter optimization. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part pp. 199-214, 2016.
Two-stage transfer surrogate model
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thiemas Scalable gaussian process-based transfer surrogates for hyperparameter optimization. Mach. Learn., 107(1):43 78, 2018.
Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperparameter tuning. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, AISTATS 2014, Reykjavik, Iceland, April 22-25, 2014, pp. 1077-1085, 2014.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep sets. In Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 3391-3401, 2017.
Zhaojie Zhou and Ningning Yan. A survey of numerical methods for convection-diffusie optimal control problems. J. Num, Math., 22(1):61-88 2014.
13

---

Under review as a conference paper at ICLR 2022
A
ALGORITHM
Algorithm Simulated Rollout
(D) rollout horizon H, transition model fv. actions (a+)
1: Require: state St/
2: R R 0
3: a( < 8t
(D)
4: for - H
5:
i + to( ah)
6:
((an) N (jo,
7: R (--UU(((aa)))
8:
81th41 U(an; ((an))
9:
return Ri
Meta-learning has found resounding success in the research community as initialization scheme, which allows for fast adaption to new domains. We want to emphasize that during meta-tres veta-training the meta-test datasets are not observed and in that way remain strictly held out.
Algorithm Model-based Reinforcement Learning
1: Require: training meta-dataset errain parameters 0, learning rate 7), inner update steps v, meta- batch size n, minimum history length Tm,n, maximum history length Tma
2: while not converged do
3: to Unif [[Tiin- Tmax))
4: Di..... Dn Unif( [1, [.....,,. NJ)
5:
for to n do
(D.)
ar. en (a:) ~ Unif (gurain Di, t)
6:
7:
0.
8:
for
j to do
0, + 8, + V (s) (D.)
(Di)
ss .ai)
(P+),
9:
10:
Update + 0 + n Σi= (0, 0)
return
B META-DATASET
A meta- ireta-dataset is collection of hyperparameters, typically defined on discretized grid (Schilling al., 2016 Jomaa et al. 2019), associated with a model under investigation, that have been eval- uated offline by training the model with the mentioned hyperparameters on numerous primary datasets, and reporting some observed evaluation metric, e.g. validation loss.
We evaluate our approach three hyperparameter search spaces for feed-forward neural networks (Jomaa et al. 2021b), that includes 120 UCI classification datasets (Asuncion Newman 2007). Jomaa et al. (021b) propose a pruning strategy that climinates redundant configurations. Specifi- cally, they drop certain hyperparameter combinations, c.g. non-[ layouts with hidden layer, which here unnecessary. Therefore do prune Layout Md and Optimization Md and refer to the meta-datasets as Layout Md-y2, Regularization Md, and Optimization Md-v2 where each meta-dataset includes the Cartesian product of the individual hyperparameters with total of 324, 288, and 432 unique configurations, respectively.
The aforementioned hyperparameters encoded as follows:
C BASELINES
1. Random sampling (Bergstra & Bengio 2012
14

---

Under review as a conference paper at ICLR 2022
Algorithm 3 MetaPETS Algorithm
1: Require:
target dataset D; training meta-dataset Euain, transition model fg, parameters 0, learn- ing rate 7,, inner update steps meta-batch size minimum history length Tmin, maximum history length Tmaz. initial budget b, total budget B, planning horizon H, lookahead indicator vanilla indicator w
2: Initialize ! randomly
3: 0 -Algorithm 2 Etrain U,n,U,, n)
4: oo....,0y (n)
(D)
- ((ao, (00)
(ab, (00.400) (001)
5: 80
6: a" arg ( (a. (a:)
7: for i = to B do
8: Sample sequence of actions (a)) ) randomly
9:
Perform simulated rollout, R Algorithm so4i te. (a)))
(D)
2 = 0 (selects first action from sequence)
10:
aag min otherwise (selects best action from sequence)
a' a if (a:) < (a))
11:
a"
otherwise
12:
if w 0 do
while not converged (Comment: Fine-tuning)
14:
for t = to b
- 0 Vo (po(se st, (t+1)) bti U(a,,((a)))
(D)
(D)
16:
$6t141
return
Table 3: Hyperparameter search space for the meta-datasets.
Hyperparameter
Layout Md-v2 ReLU, SeLU 4,8, 16,32 32 1,3, 5,7
Regularization Md
Optimization Md-v2
Activation
ReLU, SeLU, LeakyReLU 4,8, 16,32 1,3,5,7
ReLU, ScLU, LeakyReLU 4,8, 16
Neurons
Layers
3,5,7
Layout
D,4,D,00A
4,p,0,A
Dropout
0.0.5 False
0, 0.2, 0.5 False, True ADAM
Normalization
False
Optimizer
ADAM
ADAM, RMSProp, GD
2. GP (Rasmussen] 2003) is hyperparameter tuning strategy that relies on a Gaussian process as a surrogate model with squared exponential kernels (Matern 5/2 kernel) with automatic relevance determination,
3. SMAC (Hutter et al. (011T) utilizes random forests instead Gaussian processes to repre- sent the surrogate model,
4. BOHAMIANN (Springenberg al. 2016) is based on Bayesian neural networks that are trained via stochastic gradient Hamiltonian Monte Carlo,
5. TST-R (Wistuba et al. 2016) is ensemble approach where the Gaussian process rogate of the target task is weighted with surrogates of the training datasets based on the ranking similarity of the evaluated hyperparameters,
6. ABLR (Perrone et al. 2018) is multi-task Bayesian linear regression approach that opti- mizes shared feature extractor across the training datasets an initialization strategy for the target task,
7. CTS (alinas et al.) 2020) trains Gaussian Copula process (Wilson & Ghahramanil 2010) jointly over the traning datasets mapped to a shared output distribution using quantile- transformations. Hyperparameter candidates are selected via Thompson Sampling.
15

---

Under review as a conference paper at ICLR 2022
Table 4: Hyperparameter Encoding
Hyperparameter
Encoding
Activation
One-hot encoding
Neurons
Scalar
Layers
Scalar
Layout
One-hot encoding
Dropout
Scalar
Normalization
Scalar
Optimizer
One-hot encoding
8. FSBO Wiistuba & Grabocka 2021 uses deep Kernel Gaussian processes (Wilson al. 2015) to estimate the response of the target dataset. The parameters are initialized via meta-learning the joint response surface over the training datasets.
D EVALUATING VARIOUS STATE PROPAGATION DYNAMICS
In the main experiment, predict plausible state trajectories using bootstrap of = models in the ensemble. We investigate here the impact of changing the models in the ensemble, i.c. boot- strapping subsets of these models, understand the stability and robustness of probabilistic neural networks. We propose as variant E.X, where we re-sample at every trial uniformly at random different subset of B X models. Effectively, we would be continually re-sampinng from the approximate marginal posterior of plausible dynamics. We summarize the results in Figure 5
Layout
Regularization
Optimization
3.70
37
3.05
3.6
2
3.5
344
3.35
10
30
40
10
20
30
40
50
300
20
60 50
of
Figure 5: Investigating the performance of MPC under various dynamics.
We notice primarily that increasing that bootstrapping more models generally leads to better perfor- mance. In Layout Md-v2, MPC-3 and MPC-3 (E3) are better than MPC-3 (E1), especially at the early stages. We also notice that increasing the horizon leads to worse results. The same behavior is noticeable with Optimization Md-v2. Although MPC-5 in Regularization Md is better than the rest, discussed in Section 6.6 we notice that with the increasing number of trials, re-sampling a single probabilistic neural network starts deteriorate.
E LIMITATIONS
The proposed MetaPETSS algorithm is designed as transfer learning solution for HPO. The transi- tion model trained on a set of correlated source tasks to approximate the joint response surface. One of the limitations of this approach is that does work properly in non-transfer learning settings due to the lack of abundant data for training the surrogate throughout the HPO process. is possible however to design another experimental protocol where a transition model trained es- timate a partially observable search space on a particular dataset ( (i... no transfer learning required), however, planning and navigation would have be done the unexplored search space.
16


Reviews:
[{"review_id": "z4j1idT_Ffn", "reviewer": "Reviewer_cKwe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Detailed comments:\n1. The main claimed contributions of the paper are that it formulates HPO as an MDP, and that solving it using MPC with some lookahead is better than greedy selection. The authors themselves mention in Sec 4 that Jomaa et al. (2019) and Volpp et al. (2020) have dealt with the same setting before. The fact that non-myopic selection strategies perform better is not novel - this is effectively an active search problem, and it has been shown that non-myopic strategies perform better greedy selection (e.g. see Garnett et al (2012)). Based on this, I don't find the contributions of the paper to be particularly novel.\n\n2.  I found the method quite hard to follow. The initial part of the paper reads gives the impression that a full transition function that maps SxA -> S is being learnt. Only later do we get that only the response is being learnt here. This is what existing surrogate functions do as well. As the authors mention, Jomaa et al. (2019) also uses the same formulation, with the one difference being they use an LSTM to model it whereas the author of this paper follow the deep sets formulation. With the mention of MPC throughout, I was under the impression that the method employs some kind of non-random policy to guide the search (as I would expect from standard MPC). Only later does it become clear that the lookahead MPCis effectively a tree search where the tree is being generated picking hyperparameters (the actions in this formulation) at uniformly randomly. The authors include the the dataset in the state space. How is this being represented?\n\n3. I am not very clear on experimental details. Particularly, what exactly is the training datasets being used - I understand that there are 80 datasets in each training split, but how many (lambda, loss) pairs are there for each dataset? How many (lambda, loss) pairs are budgeted for each dataset in the test set? In Sec 6.5 it is stated `We report ... over 5 runs for 50 trials with three different seeds per run`. What is a run and a trial here?\n\n4. The authors compare the results of lookahead at 3 vs 5 steps and it seems like there is no clear winner. I would like to see the results for 1 step lookahead, i.e. greedy selection, and if the lookahead actually helps here at all.\n\n5. I find some of the assertions in the paper to be quite baffling - I am not sure if this is just due to poor wording or some fundamental misconception about MBRL. For example: `In MbRL the objective is to train a transition model to approximate an underlying transition function via interactions with an environment governed by some policy`.  The objective of MBRL is the same as model free RL - its just that that it achieves it differently. By augmenting the real experiences with simulated experiences, MBRL hopes to improve on sample efficiency over model free RL. `HPO … can be seen as a special use-case of model-based reinforcement learning developed under the guise of some idiosyncratic terms.` What is the basis of this statement?\n\nGarnett et al (2012) - Garnett, Roman, Krishnamurthy, Yamuna, Xiong, Xuehan, Schneider, Jeff, and Mann, Richard. Bayesian optimal active search and surveying. ICML, 2012.", "summary_of_the_paper": "The paper deals with the problem of hyperparameter selection. It formulates the problem as an MDP, and solves it using model predictive control with a short lookahead. ", "summary_of_the_review": "I found the paper quite difficult to follow in some places, with parts of the method and experimental set ups not adequately discussed. I feel that adding an algorithm of the proposed method and signposting it to each section would go a long way towards clarifying the method. I would be willing to increase my rating if these issues get addressed, but even then I think the paper lacks sufficient novelty to elicit a strong recommendation from me.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tqdUBEiFZR9", "reviewer": "Reviewer_8eE4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n+ The paper proposes a novel search strategy based on model predictive control for HPO by utilizing the existing meta-data.\n+ Empirical study highlights promising results compared to several competitors.\n\nWeaknesses:\n- The proposed approach does not conceptually match a MbRL problem and the MDP formulation seems unnecessary.\n- Some technical contents in the paper are not well supported.\n- The paper lacks a clear focus and the clarity of the overall approach should be improved.\n- There is not enough details on the approach and experiments to reproduce the results.\n\n\nComments:\n- One of the main concerns about the paper is the MDP formulation and characterizing the approach as planning in a MbRL framework, when there is no structured way to learn a policy and the setup does not conceptually match an RL problem. It seems that the approach will work just fine without mentioning the MDP and MbRL as it only predicts the transition function, in which, the next state is a target value (e.g., validation loss for a given HP configuration) that can be learned differently (vie supervised learning), that is a bit intangible in the RL context.\n\n- The planning strategy selects k samples without exploiting the orders between different HPs, evaluates their immediate improvement (using the trained transition function), and picks the best-performing one, which is more of a search scheme rather than planning ahead, and resembles Monte-Carlo tree search or a contextual bandit approach.\n\n- The transition function is trained to directly learn the HPs and is not used in the planning as a dynamic model.\n\n- The literature on HPO is not properly covered, there are several works, particularly in the bandits domain, that are not discussed: \n*Li et al., A novel bandit-based approach to hyperparameter optimization, 2018  \n*Falkner et al. BOHB: Robust and efficient hyperparameter optimization at scale, 2018  \n*Tavakol et al, HyperUCB: Hyperparameter Optimization using Contextual Bandits, 2019\n\n- The authors describe the HPO as a sequential decision-making problem, while based on my knowledge, this is not very common, and they also do not provide a supporting evidence or citation for this claim. Additionally, their approach, despite mentioned otherwise, does not exploit orders in selecting the next HPs.\n\n- It is not completely clear from the paper how aggregating the outputs of the ensemble model relates to an “excellent” estimation of uncertainty mentioned in section 5.1.\n\n- The clarity of the paper needs improvement as the ideas are not well organized and several parts mentioned in the paper, e.g., algorithm 1 and some details about the data and experiments could be better explained. They are also not not properly referenced.\n\n- The link to transfer learning is unclear to me.\n\n- For a paper that is weighted more toward an empirical paper, the experiments are small-scale and the approach is not outperforming all the baselines as stated at the beginning of the paper.\n\n- The state space is of a higher-order Markov and is growing with time t. Could you elaborate how you deal with generalization in such a large state space and why it needs to also include the loss (in addition to HP configurations)?\n\n- The authors do not provide a detailed overview of the optimization process, e.g. in Figure 2, what is the objective function, how the loss is optimized, etc.\n\n- It is not completely clear how the dynamic is evolved. For instance, Eq. 6 shows that the state contains the previous HPs and their respective loss, and the model predicts the next state (\\hat(\\lambda)). How this works for t+1? Do you compute the true validation loss for the current HP and replace it in the next state or you keep using the predictions only?\n\n- The paper could benefit from a proofread.\n\n\nQuestions:\n- Could you clarify why methods such as BHBO are not considered as baselines?\n- Could you explain the “rank” metric in more detail?\n- What is the difference of “MPC-X” and “MPC-X (vanilla)” in figure 4?", "summary_of_the_paper": "The paper presents a novel transfer learning approach to hyperparameter optimization (HPO) by formulating the problem in a model-based reinforcement learning (MbRL) framework. In this setting, the transition function serves as a surrogate model for learning the validation loss of any black-box ML model which is trained via an ensemble of probabilistic neural networks. The authors further introduce a lookahead search strategy to sample trajectories for a model predictive control problem. Experiments demonstrate that the proposed method outperforms several competitors on a few meta-datasets for HPO tasks.", "summary_of_the_review": "The paper contributes to the filed of automated ML by proposing a novel approach to find optimal HP configurations. However, the ideas contains several issues in terms of clarity, significance, and correctness of the claims, and I thus vote for a reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4QU-VS57A4H", "reviewer": "Reviewer_yo76", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- Clearly described connection between HO and MDP, which was interesting (though perhaps not entirely original, given Jomaa et al, 2019).\n- To my knowledge, this is the first work considering planning in HO, and results demonstrate it can be beneficial.\n- Good experimental results in comparison to baselines, and a thorough set of baselines appear to be considered. \n\n\n## Weaknesses\n\n- Related/prior work could be discussed a bit more clearly in the context of this paper. To me it was unclear how Jomaa et al (2019)'s approach differs to this paper? How does the MDP in that work compare to that presented here? The description in Section 4 was not totally clear to me. In addition, directly mentioning how this work differs to others in Section 2 Related Work would help a lot to contextualize the study.\n- Would help clarity significantly to have the Algorithm description (or at least a summary) in the main text, rather than appendix. I found it hard to understand the training procedure without this. \n- Some pseudocode highlighting how the state representation evolves during MPC/planning/trials would be helpful, in the appendix. This would make it very clear how this approach differs from some existing ideas that do not use planning. \n- I understand the motivation for the use of ensembles, but it would help a lot to have an ablation over this to understand its importance empirically also.  Since this is one of the stated contributions, such an ablation would help support the claims made in the introduction. Are there other strategies that could have been tried here? Perhaps a single probabilistic NN would perform well too?\n- Could you clarify why some baselines were not considered, such as Jomaa et al (2021b)? \n- Minor: Perhaps sections 5.3 and 5.2 could be swapped in order to make the narrative a bit clearer, and conclude the discussion on dynamics model training before addressing planning.\n- Minor: Tables 1 and 2 could be made clearer by highlighting that 15,33,50 refer to num. of trials", "summary_of_the_paper": "This paper proposes a method for hyperparameter optimization (HO) drawing inspiration from model-based RL (MBRL). They recast the HO problem in a markov decision process (MDP) formulation. This formulation permits the use of MBRL methods to solve the HO problem. \n\nConcretely, having reframed HO using this MDP formulation, they propose to: (1) train an ensemble of probabilistic neural networks to model the transition function (using first order meta learning), which maps a set consisting of previous hyperparam-response value pairs and the current hyperparam setting to the associated response value (eg val loss); (2) use this transition model in Model Predictive Control (MPC) with random shootouts to obtain an effective hyperparameter setting. Given the nature of their MDP and the fact that successive states are independent of one another, they use a variant of MPC that selects the best hyperparameter setting (or action) seen (rather than use the first action, as is common in MPC).\n\nIn experiments, they demonstrate their method improves on various baselines on three different settings. They perform some ablation studies showing the importance of finetuning their model on the target domains and the importance of planning. ", "summary_of_the_review": "Overall this paper is interesting and has I think some original ideas. I have several questions about related works and importance of aspects of the method (e.g. the ensembles) that would be good to get answered. If the authors are able to do this, I would consider raising my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-SjcW1AwDIN", "reviewer": "Reviewer_Vs7P", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors provide a very good overview of HPO work and position their work well. \n\nIn the related work section, they miss reporting other related work coming from RL, which is in turn briefly discussed in section 4. This is not very concerning, since the authors later compare to this body of work in the experiments section. However, it would be fair to acknowledge this precedent of establishing an equivalence between HPO and MRL more clearly in the related work section in order to correctly establish the novelty of the presented approach.\n\nThe authors propose and use an ensemble of probabilistic neural networks as surrogates. The authors make a case for their choice of using bootstrapped ensembles of probabilistic neural networks. However, they don't evaluate their efficiency or stability (in the context of non-transfer scenarios). In the transfer learning setting, the use of this type of surrogate is clear. However, in a non-transfer learning setting, the training data is either small or demands computationally expensive evaluation of a large number of candidates. At some point, this brings into question the utility of a surrogate at all, since it would make sense to actually train and evaluate complete models.  Shedding a light in this direction and possible limitations would be beneficial. \n\n", "summary_of_the_paper": "The authors establish the equivalency of hyperparameter optimization (HPO) and model-based reinforcement learning (MRL). On one hand, hyperparameter optimization is seen as an optimization of a sequence of actions (hyperparameter candidates) that improves a reward function. On the other hand, planning replaces the acquisition function of HPO. The sequence is constructed by a transition function driving the optimization towards its extremum. The transition function can be represented by a Markov Decision Process, while the reward function can be approximated by a surrogate. Within this frame, they explore the effect of planning on the performance of this approach on the task of hyperparameter optimization.", "summary_of_the_review": "The paper is well written, the work is presented in-depth and clearly. I believe that the presented work is solid and will contribute as a point for constructive discussion between the HPO and RL communities.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an algorithm for hyperparameters optimization that exploits a formulation as an MDP and thus makes use of a model-based reinforcement learning approach.

The formulation of HPO as an MDP although not novel (Jomaa et al. is not the only one to have considered this case, and the connection between the two was already known in the community) is indeed an interesting topic that could be impactful for the community. Unfortunately, the current manuscript is not providing much new insight into the topic.

After carefully reading the paper, I agree with Reviewers cKwe and 8eE4 that the current manuscript has several points of concern:
1) the formulation as a sequential decision-making problem is not fully elaborated 
2) lacking comparison to look-ahead (i.e., non-myopic) HPO algorithms (there is plenty of literature on Bayesian Optimization for doing this). This also makes it difficult to understand if the performance benefits come from the look-ahead or from the MDP formulation 
3) the writing is generally understandable, but some of the important design choices and details of the algorithms are not easy to find in the manuscript -- improving the clarity of the text would be very beneficial.

I encourage the authors to incorporate the feedback from the reviewer and to polish this paper into the shiny gem that it deserves to be.

Suggestions: 
- The MDP formulation for HPO might actually prove very beneficial for hyperparameters control (i.e., dynamically adjusting parameters during the learning process) where there is a real transition function rather than hyperparameters optimization. Might be worth reading https://arxiv.org/abs/2102.13651 which attempts to do hyperparameters control in the context of MBRL. 
- Adding better visuals to explain formulation and algorithm might go a long way.
- Tables 1 and 2 could be replaced by learning curves for a more intuitive way of visualizing the results.
Decision: Reject