Multi-Source Unsupervised Hyperparameter Optimization
Authors: Masahiro Nomura
Yuta Saito
Venue:  (2021)
Keywords: Hyperparameter Optimization

Abstract: How can we conduct efficient hyperparameter optimization for a completely new task? In this work, we consider a novel setting, where we search for the optimal hyperparameters for a target task of interest using only unlabeled target task and ‘somewhat relevant’ source task datasets. In this setting, it is essential to estimate the ground-truth target task objective using only the available information. We propose estimators to unbiasedly approximate the ground-truth with a desirable variance property. Building on these estimators, we provide a general and tractable hyperparameter optimization procedure for our setting. The experimental evaluations demonstrate that the proposed framework broadens the applications of automated hyperparameter optimization.

Under review as a conference paper at ICLR 2021
MULTI-SOURCE UNSUPERVISED HYPERPARAMETER OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
How can we conduct efficient hyperparameter optimization for completely new task? In this work, we consider a novel setting, where we search for the optimal hyperparameters for a target task of interest using only unlabeled target task and 'somewhat relevant" source task datasets. In this setting, it is essential to estimate the ground -oound-truth target task objective using only the available information. We propose estimators to unbiasedly approximate the ground-truth with desirable variance property. Building on these estimators, we provide a general and tractable hyperparameter optimization procedure for our setting. The experimental eval- uations demonstrate that the proposed framework broadens the applications of automated hyperparameter optimization.
INTRODUCTION
Hyperparameter optimization (HPO) has been a pivotal part of machine learning (ML) and contributed to achieving good performance in wide range of tasks (Feurer & Hutter 2019) It is widely acknowledged that the performance of deep neural networks depends greatly on the configuration of the hyperparameters (Dacrema al. 2019 Henderson et al., 2018 Lucic al. 2018). HPO is formulated as a special case of a black-box function optimization problem, where the input is a set of hyperparameetrs, and the output is a validation score. Among the black-box optimization methods, adaptive algorithms, such as Bayesian optimization (BO) (Brochu et al. [010]] [Shahriari et al. 20155 Frazier 2018) have shown superior empirical performance compared with traditional algorithms, such grid search or random search (Bergstra & Bengio 2012).
One critical assumption in HPO is the availability of an accurate validation score. However, in reality, there are many cases where we cannot access the ground-truth of the task of interest (referred to as target task hereinafter). For example, in display advertising, predicting the effectiveness each advertisement, i.e., click-through rates (CTR), is important for showing relevant advertisements (ads) to users. Therefore, it is necessary to conduct HPO before a new ad campaign starts. However, for new ads that have not yet been displayed to users, this case, the standard HPO procedure is infeasible, as one cannot utilize the labeled target task data and the true validation score of the MI. model under consideration.
cannot
labeled data to conduct HPO. In
In this work, we address the infeasibility issue of HPO when the labels of the target task are unavailable. To formulate this situation, we introduce a novel HPO setting called multi-source unsupervised hyperparameter optimization (MSU-HPO). In MSU-HPO, it is assumed that we do have the labeled data for target task. However, we do have the data for some source tasks with a different distribution from the target task. It is natural to assume that we have access to multiple source tasks in most practical settings. In the display advertising example, several labeled datasets of old ads that have already been deployed are often available, which we can use as labeled source task datasets. To the best of our knowledge, no HPO approach that can address a situation without labeled target task data exists despite its significance and possibility for applications.
A problem with MSU-HPO is that the ground-truth truth is inaccessible, and one cannot directly apply the standard HPO procedure. Thus, it is essential to accurately approximate it using only available data. For this purpose, we propose two estimators, enabling the evaluation of the ML models without the labeled target task data. Our estimators common black-box optimization methods, such as Gaussian process-based BO (Srinivas et al.
general and can be used in combination with any
2010
Snoek et al. 2012 Hennig & Schuler, 2012 Contal et al. 2014 Hernández-LLobtto et al.
2014

---

Under review as a conference paper at ICLR 2021
Wang & Jegelka, 2017) and the tree-structured Parzen estimator (Bergstra et al. 2011 2013). In addition, we show that the proposed estimators can unbiasedly approximate the target task objective, one of which achieves desirable variance property by selecting useful source tasks based task divergence measure. We also present a general and computationally inexpensive HPO procedure for MSU-HPO building on our estimators. Finally, we demonstrate that estimators work properly through numerical experiments with synthetic and real-world datasets.
Related Work. A typical HPO setting is find better set of hyperparameters using labeled target task of interest. As faster convergence is an essential performance metric of the HPO methods, the research community is moving on to the multi-soure source or transfer settings for which there are some previously solved related source tasks. By combining the additional source task information and the labeled target task dataset, has been shown that one can improve the hyperparameter search efficiency, and thus reach a better solution with fewer evaluations (Bonilla et al.
2008 Bardenet et al.
2013; Swersky et al. 20133 Yogatama & Mann 2014 Ramachandran et al.
2018: Sprringenberg
et al. 2016; Poloczek et al. 2017, Wistuba et al. 2018 Feurer et al.) [0188] Perrone et al. 2018
2019; Salinas et al. 2019). A critical difference between the multi-source HPOs and our MSU-HPO
settings is the existence of labels for the target task. Previous studies usually assume that analysts can utilize labeled target data. However, as discussed above, this is often unavailable, and thus, most of these methods are infeasible.
One possible solution to address the unavailablity of labeled target data is to use warm starting methods (Vanschoren| 2019), which aims find good initial hyperparameters for the target task. Learning Initialization (LI) finds promising hyperparameters by minimizing a sum of loss function surrogated by a Gaussian process on each source task (Wistuba et al., 2015). While LI is effective when the source and target tasks are quite similar, it is hard to achieve a reasonable performance otherwise. In contrast, DistBO learns the similarity between the source and target tasks with a joint Gaussian process model on hyperparameters and data representations (Law et al.) 2019). However, many transfer methods including DistBO need abundant hyperparameter evaluations for the source tasks to surrogate objective function for each task well, which will be confirmed in our experiments.
Another related field is model evaluation covariate shift, whose objective is to evaluate the performance of the ML models of the target task using only relevant single source dataset (Sugiyamal eet al. 2007 You et al.) 2019; Zhong et al.) (2000). These studies build on the importance sampling (IS) method (Elvira et 2015 SSugiyama et al. (007) to obtain unbiased estimate of ground -ruth model performances. While our proposed methods are also based on IS, a major difference is that we assume that there multiple source datasets with different distributions. We demonstrate that with the multi-source setting, the previous IS method can fail, and propose an estimator satisfying the optimal variance property. Moreover, as these methods are specific model evaluation, the connection between the IS-based estimation techniques and the automated HPO methods has not yet been explored despite their possible, broad applications. Consequently, we are the first to empirically evaluate the possible combination of the IS based unbiased estimation and adaptive HPO.
Contributions. The contributions of this work can be summarized as follows: (i): We formulate a novel and highly practical HPO setting, MSU-HPO. (ii): We propose two unbiased estimators for the ground- -ruth validation score calculable with the available data. Additionally, we demonstrate that one of them achieves optimal finite variance among a reasonable class of unbiased estimators. (iii): We describe flexible and computationally tractable HPO procedure building on the proposed estimators. (iv): We empirically demonstrate that the proposed procedure works favorably in MSU-HPO setting. Furthermore, our empirical results suggest a new possible connection between the adaptive HPO and IS- unbiased estimation techniques.
2 PROBLEM SETTING
In this section, we formulate MSU-HPO. Let X C Rdd be the d-dimensional input space and y C R be the real-valued output space. We use pr (I, y) to denote the joint probability density function of the input and output variables X E X' and Y E y of the target task. The objective of this work is to find the best set of hyperparameters 0 with respect to the target distribution:
gept arg min fr (r) dE0
(1)
2

---

Under review as a conference paper at ICLR 2021
where θ is pre-defined hyperparameter search space and fr) is the target task objective, which is defined as the generalization error over the target distribution:
fr) = Px [hh((X),)))
where L 3 x y + R20 is bounded loss function such the zero-one loss. he X D is an arbitrary machine learning model that predicts the output values using the imput vectors with a set of hyperparameterr
O.
In a standard hyperparameter optimization setting (Bergstra et al. 2011 Feurer & Hutter 2019
Snoek et al., 2012), labeled i.i.d. validation samples ( yb Jijis
pr available, and can
easily estimate the target objective in Eq. (2) by the following empirical mean:
Pr L(hg(z:).
fr(0: Di Diaheted
(3)
TUT
where Diabeled is any size for of the i.i.d. labeled samples from the target task distribution. Then, hyperparameter optimization is conducted directly using the estimated target function in Eq. 3 as
reasonable replacement for the ground-trum target objective (r( in Eq. 2
In contrast, under the MSU-HPO setting, labels of the target task are assumed be unobservable; we can use only unlabeled target validation samples denoted as Dy = { hereinafter. Instead, we assume the availability of the multiple source lusk datasets which is denoted as (Dss 1 where is a source task index and N denotes the number of source tasks. Each source task data is defined as the i.idd labeled samples: Dş {z/ y J Pss where Ps: (Ir y) is joint probability density function that characterizes the source task j. Note here that marginal input distributions of the target and source tasks different, i.e., Pr (2:) + Psi (I), Vj E f1....,No Ns)
Regarding the target and source distributions, we make the following assumptions.
Assumption 1. Source tasks have support for the target task, i.e., pr(x) > 0 + pss (x) 0, Va X, Vj E f1...., Assumption 2. Conditional output distributions remain the same between the target and all of the source tasks, i.e., pr(ylz) = pss (y)z), Vj Ns,..
The above assumptions are common in the covariate shift literature Shimodaira (2000) and suggest that the input-output relation is the same, but the input distributions are different for the target and source task distributions.
One critical difficulty of the MSU-HPO setting is that the simple approximation using the empirical mean is infeasible, as the labeled target dataset is unavailable. It is thus essential to accurately estimate the target task objective using only an unlabeled target dataset and labeled multiple source datasets.
3 METHOD
In this section, we propose estimators to approximate the target task objective by applying an importance weighting technique.
3.1 UNBIASED OBJECTIVE ESTIMATOR
A natural first candidate method to approximate the target task objective function is to use importance weighting (Shimodaira 2000). To define our estimator, first introduce the density ratio between the target task distribution and the source task distribution below.
Definiton 1. (Density Ratio) For any (a, y) E X y with a positive source density ps: (I, y) the density ratio between the target and a source task distributions is
(r(x, v) Pr(x) = wss (x) SC
S w (I, u)
(4)
Pss (c. y) (x) where C a positive constant. The equalities are derived from Assumption 2
'These assumptions seem to be strict, but in fact, they relatively reasonable given that the general HPO literature implicitly assume that the train-test distributions the same.
3

---

Under review as a conference paper at ICLR 2021
Using the density ratio, we define an estimator for the target task objective function. Definiton 2. (Unbiased Estimator) For a given set of hyperparameter 8 E O, the unbiased estimator for the target task objective function is defined as
Ns Agi
fuB (D: 1ررر
wss (x.) ((ho(z))) v()
(5)
==
where UB stands for unbiased, nt - EE Rigs the total sample size of the source tasks, Das is any sample size ra of the i..d. samples from the distribution of source task j. and w (-) is the true density ratio function.
The estimator in Eq. (5) is an application of the importance weighted cross-validation (Sugiyama al. 2007) the multiple-souu task setting and can easily be shown to be statistically unbiased for the ground-truth target task objective function, i.c., for any given 0, B[fuB (0; (Dss 11-s))=11) (0).
We also characterize the variance of the unbiased estimator.
Ns
v (fuB (: (DsDs)
n2 nss (E(xY))ps)
Σ
(x) L' (ho( - (fr())))
=
(
As stated above, the unbiased estimator is valid approach for approximating target task objective because of its unbiasedness. The problem is that its variance depends on the square value of the density ratio function, which can be huge when there is a source task with distribution that is dissimilar to that of the target task.
To illustrate this variance problem. we use toy example where {21, S X, {v1 y2 S V, p(y1 (z1) = p(y2|z2) = 1, ((12(=1) = p(v1(x2) = 0. The loss values for possible tuples. and the probability densities of the target and two source tasks are presented in Table I It shows that the target task 7 is similar to the source task $2, but its distribution is significantly different from that of S For simplicity and without loss of generality, suppose there are two source task datasets such as D = {zx] (z},)})} and D2 = {<1 v;) Then from Eq. (6). the variance of the unbiased estimator about 64.27. Intuitively, this large variance a result of the large variance samples from St, In fact, by dropping the samples of S reduces the variance to 4.27. From this example, we know that the unbiased estimator fails to make the most of the source tasks, and there room to improve its variance by down-weighting the source tasks dissimilar to the target task.
Table Dropping data samples from S significantly lowers the variance of the unbiased estimator
(a1 vn))
(122.22)
loss function: L(ho(z),
10
1
larget (T) distribution: pr (xt
0.2
source task (S) distribution: Ps (2.
source task (S2) distribution: (55(2,
0.9
0.1
3.2 VARIANCE REDUCED OBJECTIVE ESTIMATOR
As illustrated with the toy example, an unbiased estimator can be unstable when there are some source tasks with a distribution significantly different from that of the target task. To address this variance issue, we define a divergence measure between the two tasks below.
Definiton 3. (Task Divergence Measure) The divergence between source task distribution Psa where j E (1,...Ns) Ns} and the target task distribution pr is defined as
Die (T (1)) = EXX
[w2. (X) L (ho - (f))
(7)
This task divergence measure is large when the corresponding source distribution deviates significantly from the target task distribution. Building on this measure, we define the following estimator for the target task objective.
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Hyperparameter optimization procedure under the MSU-HPO setting
Input: unlabeled target task dataset Dr {z; [ labeled source task datasets (Dg) Ns hyperparameter search space θ a machine learning model hø; a target task
(1)) ) ررر
objective estimator a hyperparameter optimization algorithm OPT
1: for j E (1....,NT) ] do
2: Split Dy: into three folds Dis Ddansity Dirain and D
3: Estimate density ratio ws: (-) by uLSIF with Dr and Deensity
4: end for
5: Optimize the hyperparameter 0 E e of he with OPT by setting f(;; (D Ns 1j=1) as its objective
6: (the model parameter of he is obtained by optimizing f(0; (Digr No [j=1)
7: return ha. (where 0" is the output of OPT)
Definiton 4. (Variance Reduced Estimator) For given of hyperparameters O, the variance reduced estimator for the target task objective function is defined as
No
Rgj
fVR 0: (Ds)
Σ
w(xx) L(ha(z)))
where VR stands for variance reduced, Dss any sample size nss of the i.i.d. samples from the distribution of source task j. and (-) (-) is the true density ratio function. X3 is a weight for
source task j. which is defined as X3 Div (T (1 )
fig Dio(T[|ss:)
Note that, for all
j E (1.... Nsh. X3 2 0 and Σ Xing =
The variance reduced estimator in Eq. (8) is also statistically unbiased for the ground truth target task objective in Eq. (2). i.e., for any given O, [[fVR 0; (Dss) ),²²) (r)
Then, we demonstrate that the variance reduced estimator in Eq. (8) is optimal in the sense that any other convex combination set of weights À {A1.... ANa) that satisfies the unbiasedness for the target task objective function does not provide a smaller variance.
Theorem 1. (Variance Optimality; Extension of Theorem 6.4 of (Agarwal et al. 2017 For any given
-Ns set of weights À = (A1....Aw, } that satisfies X3 2 and EØ=1 AyTlss for all j E Ns. the following inequality holds
(fvr (D: (D (
SV (0:1 (0:1ss)))))))
Die (T]
where P2 (D: } دm Ns Xj En w(xx) (ha(z)), v!). See Appendix A for the proof.
Theorem suggests that the variance reduced estimator achieves desirable finite sample variance property by weighting each source task based on its divergence to the target task.
Let return to the toy example in Table I The values of the divergence measure for S and 52 are 252.81 and 4.27, respectively. This leads to the weights of X; Rs 0.017 and >₂ R 0.983. Then, the variance of the variance reduced estimator is equal to 4.21 4.27 (variance when S is dropped.). It is obvious that the variance reduced estimator performs better than the unbiased estimator does by optimally weighting all available source tasks.
3.3 HYPERPARAMETER OPTIMIZATION PROCEDURE
We describe several detailed components of the HPO procedure in the MSU-HPO setting.
Density Ratio Estimation: general, density ratio functions between the target and source tasks unavailable and thus should be estimated beforehand. To estimate this parameter, employ the
5

---

Under review as a conference paper at ICLR 2021
unconstrained Least- Least-Squares Importance Fitting (uLSIF) procedure Kanamori et al. 22099; Yamada ct al. (201T), which suggests directly minimizing the following squared error for the true density ratio function:
8 arg min Epgr [wx(X) - (wx())- sxX))
arg min SES
Bpss [x - Epr [s]])
(9)
SES
where S is class of measurable functions. It should be noted that the empirical version of Eq. calculable with unlabeled target and source task datasets.
is
Task Divergence Estimation: To utilize the variance reduced estimator, the task divergence measure Dic (T || 5") in Eq. (7) needs to be estimated from the available data. This can be done using the following empirical mean.
How to train h To evaluate the validation score of 8 O the model parameters of hg should be optimized by the supervised learning procedure. However, in the MSU-HPO setting, the labeled target task dataset is unavailable, and direct training of ha is infeasible. Therefore, splitting the labeled source task datasets (Dss J into the training (Dzrain) and validation {D sets. Then, we train he using the training set by hg arg min,, EHo f(0; (Digrin ( } where f is an estimator for the target task objective function such as the unbiased and variance reduced estimators, and Hg is a hypothesis space defined by set of hyperparameters E O. This training procedure enables us obtain the model parameters of he as if it were trained on the labeled target task dataset. In addition, it is sufficient to train hig only once to evaluate E O; the proposed procedure is computationally inexpensive.
suggest
Building on the above details, Algorithm summarizes the high-level hyper- parameter optimization procedure under the MSU-HPO setting We also provide the regret bound of our HPO procedure in the MSU-HPO setting in Appendix C
Overall Procedure:
4
EXPERIMENTS
We investigate the behavior of our proposed HPO procedure in MSU-HPO using a synthetic problem
in Section 4. and real-world datasets in Section We compare the following methods as possible baselines
Learning Initialization (LJ) (Wistuba et al. 2015), (ii) DisiBO (Law et al. 2019). the concatenation of source tasks validation
(iii): Naive method, which uses the performance score, (iv) Oracle method, which uses the labeled target task for HPO. Thus, the oracle method is infeasible in MSU-HPO, and we regard the performance of the oracle method as an upper bound to which other methods can reach. In all the experiments, we set the number of evaluations B = 50 and use GP-UCB (Srinivas al. 2010) as a hyperparameter optimization algorithm
4.1 TOY PROBLEM
We consider 1-dimensional regression problem with the MSU-HPO setting. The generative process of the toy dataset is as follows:
~ u(-o, ci), {ai)i_1 nd i.i.d. Nop', 1), (i (a))_ Lid [A(00.7)) 1)
where is the uniform distribution, N denotes the normal distribution, and c; E IR is a prior parameter that characterizes the marginal input distribution (p(x)) of task i. The objective function is given by:
[(0: = L(,,)), L(O, 9) (0 2/2
(10)
2 Die - Mi (w(x)) (ha(
Σ'i w(x)) ((ha(a)). vt)
"g
WWe deseribe the specific hyperparameter optimization procedure when BO is used as OPT in Appendix
We describe the details of the haseline methods in Appendix D
We deseribe detailed settings in our experiments Appendix

---

Under review as a conference paper at ICLR 2021
unbissed veriance reduced
* U
DuBC
unbised
reduad
1.2
1.0
0.5
1.0
1.0
2,0
1.0
200
3.0
4.0
source bound
source c bound
(a) Comparing all methods
(b) Comapring unbiased and variance reduced
Figure 1: Results of the experiment on synthetic toy problems over 30 runs.
Note: The horizontal axis represents the prior parameters of the source tasks of E {1.0, 5.0), (i [1, J.))). ]- (a) The vertical axis represents the mean and standard eroor of the evaluated loss for each estimator. (b) The vertical axis represents the ratio of the mean loss of the unbiased estimator to the variance reduced estimator.
Similar to the toy experiment in (Law et al. 2019). A E [-8, 8.888 is hypothetical "hyperparameter" we would like to optimize. The optimal solution for this experiment is thus 0 = n n E'E', Vt.
We described in Section 3 that when p(z) the source task and the target task differs significantly, the performance of the variance reduced estimator is better than that of the unbiased estimator. To demonstrate this, we set separately for the source (c.² E {1.0, 2.0. 5.0), E {1, ,Ns)) and the target tasks (c" = 1.0). That is, the source and target distributions are similar when cc 1.0(= ef in contrast, the source and target distributions are quite different when of 5.0.
Finally, we set N 2 and
1000.
Figure Tigure[showss shows the results of the experiment on the toy problem over runs with different random seeds. First, Figure [ (a) indicates that the proposed unbiased and variance reduced estimators significantly outperform the naive method and LI in all settings. This is because our estimators unbiasedly approximate the target task objective by considering the distributional shift, while the naive method and LI cannot. Moreover, this figure shows the advantage of unbiasedness is highlighted when the distributions of the target and source tasks diverge largely (i.e., when of is large.). DistBO also shows relatively good performance despite the lack of unbiasedness. Next, we compare the performance of the unbiased and variance reduced estimator in Figure (b)) This reports the performance of the unbiased estimator relative to the variance reduced one with varying values of The result indicates that the advantages of using the variance reduced estimator over the unbiased one are further strengthened when there is large divergence between the target and source task distributions, which is consistent with our theoretical analysis. Finally, as shown Figure (a), the variance reduced estimator achieves almost the same performance as the upper bound without using the labels of the target task, suggesting its powerful HPO performance on an unlabeled target task.
4.2 HYPERPARAMETHE OPTIMIZATION ON REAL--WORLD DATASETS
Datasets: We use Parkinson's telemonitoring (Parkinson) Tsanas et al. 2009 and Craft-versus- host disease (GvHD) datasets (Brinkman et al.) 2007) to evaluate our methods on real-world problems.
Parkinson data consists voice measurements of patients with the early-stage Parkinson disease collected by using a telemonitoring device in remote symptom progression monitoring. Fach patient has about 150 recordings characterized by feature vector with 17 dimensions. The goal to predict the Parkinson disease symptom score for each recording from the recordings.
GvHD is an important medical problem in the allogeneic blood transplantation field (Brinkman al. 2007). The issue occurs in allogeneic hematopoietic stem cell transplant recipients when conor-immune cells in the graft recognize the recipient as foreign and initiate an attack on several tissues. The GvHD dataset contains weekly peripheral blood samples obtained from 31 patients characterized by a feature vector with dimensions. Following (Muandet et al.) (203)), we omit one patient who has insufficient data, and subsample data of each patient to have 1000 data points each.
7

---

Under review as a conference paper at ICLR 2021
The goal is to classify CD3+CD4+CDS+ cells, which have high correlation with the development of the disease (Brinkman et al. (007).
Experimental Procedure: To create the MSU-HPO setting, for both datasets, treat each patient as a task. We select one patient as a target task and regard the remaining patients as multiple source tasks. Then, ML model by an HPO method using the unlabeled target task and labeled source tasks, (2) Split the original target task data into 70% training set and 30% test set, (3) Train an ML model tuned by an MSU-HPO method using the training set of the target task, (4) Predict target variables (symptom scores for Parkinson and CD3+CD4+CD8+ cells for GvHD) on the test set of the target patient, (5) Calculate target task objective of the prediction and regard it as the performance of the MSU-HPO method under consideration, (6) Repeat the above steps 10 times with different seeds and report the mean and standard error over the simulations.
use the following experimental procedure: (1) Tune hyperparameters of an
As for an ML model and a target task objective, we use support vector machine (SVM) implemented in scikit-learn (Pedregosa et al.] 2011) and mean absolute error (MAE) for Parkinson. In contrast, we use LightGBM (Ke et al. 2011 7) as an ML model and binary cruss-entropy (BCE) as a target task objective for GvHD.
Table 2: Comparing different MSU-HPO methods (Mean +StdEm)
Estimators
Parkinson (MAE)
GvHD (BCE)
LI
0.41507 10.1669
0.19695 10.0468
DISTBO
1.54202 10.1006
0.33015 +0.0600
NAIVE
1.10334 10.9908
0,02121 #0.0052
UNBIASED (ours)
1. 1.08283 10.1981
0.02141 +.000552
VARIANCE REDUCED (ours)
0.40455 10.1755
0.01791 1..0039
ORACLE (reference)
0.06862 10.0011
0.01584 +0.0043
Note:
The red fonts represent the best performance among estimators using only the unlabeled target task and labeled source task datascts. The mean and standard error (StdErr) are induced by running 10 simulations with different random seeds.
Table 2 presents the results the experiments over 10 runs with different random seeds. In contrast to the results in Section 4.1 the performance of DistBO has deteriorated significantly. While DistBO requires a reasonable number of hyperparameter evaluations per source task, our setting allows only very small number of evaluations per source task, which may lead to learning inaccurate surrogate m The unbiased estimator performs almost the same with naive in Parkinson given their standard errors. Moreover, it slightly underperforms the naive in GvHD, although the unbiased estimator satisfies the unbiasedness. This is because the number of data for each task is small, and the variance issue of the unbiased estimator highlighted in these data. Therefore, pursuing only unbiasedness in the approximation of the target task objective is not sufficient in MSU-HPO. On the other hand, the variance reduced estimator alleviates the instability issue of the unbiased estimator and performs best in both datasets. The results also suggest that the variance reduced estimator works well on both regression (Parkinson) and classification (GvHD) tasks. Therefore, we conclude from its variance optimality and empirical performance that using the variance reduced estimator is the best choice for MSU-HPO.
Results:
CONCLUSION
We studied novel problem setting, MSU-HPO, with the goal of enabling effective HPO with only an unlabeled target task and multiple labeled source task datasets. We proposed two estimators to approximate the target task objective from available data. Empirical evaluations demonstrated that the proposed HPO procedure helps to determine useful hyperparameters without the labels of the target task.
A detailed discussion is provided in Appendix D.3
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Aman Agarwal, Soumya Basu, Tobias Schnabel, and Thorsten Joachims. Effective evaluation using logged bandit feedback from multiple loggers. Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 687-696, 2017.
Rémi Bardenet, Matyás Brendel, Balázs Kégl, and Michele Sebag. Collaborative hyperparameter tuning. In International conference machine learning, pp. 199-207, 2013.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(Febb:281-300 2012.
James Bergstra, Daniel Yamins, and David Daniel Cox. Making a science of model search: Hyperpa- rameter optimization hundreds of dimensions for vision architectures. 2013.
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for Hyper- Parameer Optimization. In Advances neural information processing systems, pp. 2546-2554, 2011.
Ilija Bogunovic, Jonathan Scarlett, Stefanie Jegelka, and Volkan Cevher. Adversarially robust optimization with gaussian processes. In Advances neural information processing systems, pp. 5760-5770, 2018
Edwin V Bonilla, Kian M Chai, and Christopher Williams. Multi-task Gaussian Process Prediction. In Advances neural information processing systems, pp. 153-160, 2008.
Ryan Remy Brinkman, Maura Gasparetto, Shang-Jung Jessica Lee, Albert J Ribickas, Janelle Perkins, William Janssen, Renee Smiley, and Clay Smith. High-content flow cytometry and temporal data analysis for defining cellular signature of graft-versus-hot disease. Biology of Blood and Marrow Transplantation, 13(6):691-700 2007.
Eric Brochu, Vlad M Cora, and Nando De Freitas. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hicrarchical reinforcement learning. arXiv preprint arXiv: 1012::011 2010.
Emile Contal, Vianney Perchet, and Nicolas Vayatis. Gaussian process optimization with mutual information. In International Conference Machine Learning, pp. 253 2014.
Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are we really making much progress? worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 101-109, 2019.
Víctor Elvira, Luca Martino, David Luengo, and Mónica F Bugallo. Efficient multiple importance sampling estimators. IEEE Signal Processing Letters, 22(10):1777-177 2015.
Matthias Feurer and Frank Hutter. Hyperparameter Optimization. Automated Machine Learning, pp. 3-33. 2019.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable Meta-Learning for Bayesian Optimization using Ranking-Weigee Gaussian Process Ensembles. In AutoML Workshop at ICML, 2018.
Peter Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv: 2018.
Javier González, Zhenwen Dai, Philipp Hennig, and Neil Lawrence. Batch bayesian optimization via local penalization. Artificial intelligence and statistics, pp. 648-657, 2016.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep Reinforcement Learning that Matters. In Thirly-Second AAAI Conference on Artificial Intelligence, 2018.
Philipp Hennig and Christian J Schuler. Entropy search for information-efficienn global optimization. Journal of Machine Learning Research, 13(Jun):
1899-1837, (80J0 2012.

---

Under review as a conference paper at ICLR 2021
José Miguel Hemándezz-Lbbtto Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. In Advances in neural information processing systems, pp. 918-926, 2014.
Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. A least-squares approach to direct importance estimation. Journal of Machine Learning Research, 10(Jul): 2009.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in neural information processing systems, pp. 3146-3154, 2017.
Ho Chung Law, Peilin Zhao, Leung Sing Chan, Junzhou Huang, and Dino Sejdinovic. Hyperparameter learning via distributional transfer. In Advances in Neural Information Processing Systems, pp. 6801-6812, 2019.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are Gans Created Equal? A Large-Scale Study. In Advances in neural information processing systems, pp. 700-709, 2018.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18, 2013.
Vu Nguyen, Santu Rana, Sunil Gupta, Cheng Li, and Svetha Venkatesh. Budgeted batch bayesian optimization. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 1107- 1112. IEEE, 2016.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, al. Scikit-leam: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cédric Archambeau. Scalable Hyperpa- rameter Transfer Learning. In Advances in Neural Information Processing Systems, pp. 6845 6845-6855,
2018.
Valerio Perrone, Huibin Shen, Matthias W Seeger, Cedric Archambeau, and Rodolphe Jenatton. Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning. Advances in Neural Information Processing Systems, pp. 12751-12761, 2019.
Matthias Poloczek, Jialei Wang, and Peter Frazier. Multi-information source optimization. In
Advances in Neural Information Processing Systems, pp. 4288 4298, 2017.
Anil Ramachandran, Sunil Gupta, Santu Rana, and Svetha Venkatesh. Information-theoreet Transfer Learning framework for Bayesian Optimisation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 827-842. 2018.
David Salinas, Huibin Shen, and Valerio Perrone. A copula approach for hyperparameter transfer learning. arXiv preprint arXiv:1909. 13595, 2019.
Bobak Shahriari. Kevin Swersky. Ziyu Wang, Ryan Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 2015.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp. 2951-2959, 2012.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian Optimization with Robust Bayesian Neural Networks. In Advances in Neural Information Processing Systems, pp. 41444414, 2016.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process opti- mization in the bandit setting: no regret and experimental design. In Proceedings of the 27th International Conference International Conference Machine Learning, pp. 1015-1022, 2010.
10

---

Under review as a conference paper at ICLR 2021
Masashi Sugiyama, Matthias Krauledat, and Klaus Robert MÅäller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(May):985-100) 2007.
Kevin Swersky, Jasper Snoek, and Ryan Adams. Multi-Task Bayesian Optimization. In Advances in neural information processing systems, pp. 2004-2012, 2013.
Athanasios Tsanas, Max A Little, Patrick E McSharry, and Lorraine O Ramig. Accurate telemon- itoring of parkinson's disease progression by noninvasive speech tests. IEEE transactions on Biomedical Engineering, 57(4):884-893, 2009,
Joaquin Vanschoren. Meta-Learning. In Automated Machine Learning, pp. 35-61 2019.
Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient bayesian optimization. In Proceedings of the 34th International Conference on Machine Learning-Vor Volume 70, pp. 3627-3635.
JMLR. org, 2017.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Learning hyperparameter optimization initializations. In 2015 IEEE international conference data science and advanced analytics (DSAA), pp. 1-10. IEEE, 2015.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Scalable gaussian process- based transfer surrogates for hyperparameter optimization. Machine Learning. 107(():43-78, 2018.
Makoto Yamada, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Masashi Sugiyama.
Relative density-ratio estimation for robust distribution comparison. In Advances in neural information processing systems, pp. 594 602, 2011.
Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperparameter tuning. In Artificial intelligence and statistics, pp. 1077-1085, 2014.
Kaichao You, Ximei Wang. Mingsheng Long, and Michael Jordan. Towards accurate model selection in deep unsupervised domain adaptation. In International Conference on Machine Learning, pp. 7124-7133. 2019.
Erheng Zhong, Wei Fan, Qiang Yang. Olivier Verscheure, and Jiangtao Ren. Cross validation framework to choose amongst models and datasets for transfer learning. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, pp. 547-562. Springer, 2010.

---

Under review as a conference paper at ICLR 2021
A
OMITED PROOFS
A.1 DERIVATION OF UNBIASEDNESS
We first define general class of unhiased estimators called X-unbiased estimator that includes the unbiased and variance reduced estimators as special cases.
Definiton 5. (X-unbiased Estimator) When sel of weights À -A .Ng) that satisfies X 2
and Σj= Xynss = for all j E [I.....Ns) is given, the X-unbiased estimator for the target task objective function is
No
Bgg
A (D: No
Σ
X
Uga
(xt) [(ho(z)) (xx)) (x()) v.))
(11)
When X, = ni:: /N, it is the unbiased estimator in Eq. estimator in Eq. (8) when X3 >}
5 In contrast, it is the variance reduced
Then we show that the X-unbiased estimator statistically unbiased for the target task function.
Proof. By the linearity of the expectation operator,
Ns
Rig
Aa (Ds
No
B B (X):(X) L(ho (I)
j=1
Ng
Rgi
pr
L(ho(X),))
Pss (X,Y)
Σ
E B(x [L(ho(X), YY
Σ fr)
Y=1
Xyngs
fr())
(0)
Thus, the unbiased estimator in Eq. 5 and the variance reduced estimator in Eq. statistically unbiased for the ground truth target task objective function Eq. (2)
( are both
A.2 DERIVATION OF EQ. (
Proof. The variance can be represented follows because samples are independent
Ns ngi
v (fuB (0: (D:())))
(wsi (X) [(ho(X), V))
n
j=
Ns V (wss: (Xho(X), Y))
ng:
V (ws: (X) (X), is decomposed as
V (wss (X) L(hg (X), E(x.Yi<pe/ [wS (X) (h)b - (Exx,r><ps [wss (X) L(ho(X),,
From the unbiasedness property, ((x.Y) [wss (X) L(ho(X), f0().. Then, we now have
V (wss (X) T((ho()) (X), Y)) Ex,,))~ps (x) (ho Y)] (Fr(0))²
12

---

Under review as a conference paper at ICLR 2021
A.3 PROOF OF THEOREM 1
By following the same logic flow as in Section A.2, the variance of the X-unbiased estimator in Eq.
N₂
V (fx (D:(Ds) 1 =
Xnnss
(Exxy))ppr [x) L2 ( -x)
j=1
N₂
Σ
Xing Die (T)
(12)
Thus, by replacing X, for (Div (T (1) Ej=ı
No
ngt
we have
Dio
No
No
V (fx 0: (Dsi 1
Σ
nts
nig Div (T|1St)
Div (T Si)
ng: Div (T)
Div (T) (1 -
.ss
Die[T
-2
T1:s Diu (T ()))
Die (T (1))
Ns
ng:
Diu (T) SS)
Moreover, for any set of weights X = {A1,.. ANgh h. we obtain the following variance optimality using the Cauchy-Schwarr inequality.
No Xnss Div (T(15')
Ne
Ns
nlgs
X sss
=1
Div (T ST)
Ns
N₂
Xnns Div (T(15t)
M
Σ
nss
Div (TII
j=1
(fx (0: (D:(ss)) 2 V (fva (0: (D=
B BAYESIAN OPTIMIZATION UNDER THE MSU-HPO SETTING
In Algorithm we described the abstracted hyperparameter optimization procedure which allows any black-bor box optimization method to be used. Here, in Algorithm 2, we describe the hyperparameter optimization procedure under the MSU-HPO setting with the popular Bayesian optimization method.
C REGRET ANALYSIS
In this section, we analyze the regret bound under the MSU-HPO setting. We define a regret as rB f f(0)),
NNs where f θ R is the ground-truth target task objective, Σ nsy is the total sample size among source tasks, B is the total number of evaluations, 0- = arg mingee f()), and 0; = arg mingg(e).....9) fa (0) where fa O R is estimated target task objective by any estimator
13

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Bayesian Optimization under the MSU-HPO setting
Input: unlabeled target task dataset Dr { labeled source task datasets (Dg) (s, 1 J hyperparameter search space θ a machine learning model ho; a target task objective estiamtor f. number of evaluations B, acquísition function ax-))
Output: the optimized set of hyperparameters 0* E θ
1: Set Ag +
2: for j E {1, [[....,Ns}} do
3: Split Dss into three folds Dss
Ddansity Dtroin and De
4: Estimate density ratio wss (.) by uLSIF with Dr and Ddensity
5: end for
6: for t- 1,.,..,. do
7: Select 0g by optimizing a(0 At-1)
8: Train hg, by optimizing f(0; (Diggin )イ=1) and obtain a trained model hi
9: Evaluate h and obtain a validation score 2t f(0; (D5y )
10: A - Ar-1 {(0,, x
11: end for
12: t" = arg min, (21,... ZB
13: return hg. (where 0* 0,-)
(e.g., the unbiased estimator and the variance reduced estimator). Note that each of (01. the hyperparameter selected B evaluations in the optimization.
033 is
To bound the regret above, first decompose it into the following terms:
rB -(0z) f(0") (S(0%) (0x)) fm(0)) + -fx(0') ((0)) (0)
+ (f((0)) - + - + -
(13)
(A)
(B)
(C)
where 0- = arg mingee (0).
The term (A) represents the regret obtained by optimizing the estimated target task objective fu. The term (B) represents the difference function value between the true objective f and the estimated objective fn at 0z. which is the solution obtained by the optimization for the estimated objective. The term (C) represents the difference between the minimum value for the estimated objective f.. and that of the true objective f.
We first show the following two lemmas which is used to bound the regret.
Lemma 2. The following inequality holds with probability of at least - 6, 8 E (0,
(f(0%) fa(())) & (v(fx(07))66
Proof. By Chebyshev's inequality, we have
P(f(()) fa 2 - of
{V(F2(6)))
Putting the RHS as 8 and solving it for completes the proof.
Lemma 3. The following inequality holds with probability of at least 1 - 6, 8 E (0,1)
f,((0)) 5(00) = (VV
14

---

Under review as a conference paper at ICLR 2021
Proof. By Chebyshev's inequality, we have
P( f(0") 2 c)
$ PPll (6*) - f(0)) cf PPllfn(0 - f(0)) -0) S Pf (6*) - f(()) 2 c) +PlIf f(0))
E (V((. + ((f
Putting the RHS as and solving it for e completes the proof.
Theorem 4. (Regret Bound on the MSU-HIPO setting) When the X-unbiased estimator with an
arbitrary set of weights X is used as f(0,; (D 1 1==1), the following regret bound holds with probability of at least
8 (0,1).
-
rB - Rm +
(14)
where R. = fm (01) - (ō*).
Proof. Putting Lemma to the term (B) in Eq. 13 and Lemma to the term (C) Eq. complete the proof.
When an estimator is the proposed unbiased estimator or variance reduced esitmator, the variance (-)) is o(n); the second term and third term in Eq. (44 is to be no-regret (Srinivas et al. 2010
with respect to i.e., lim (v (f + 2((((f + ((fx( 0. This means that, if the optimization method is no-regret with respect to the number of evaluations B, the overall regret approaches as and B are increased.
D BASELINES
In this section, we first give brief description of Learning Initialization (LI) (Wistuba al. 2015) and DisBBO (Law et al., 2019) used as baseline methods in the experiments. Then, we discuss the application of these methods MSU-HPO.
D.1 LEARNING INITIALIZATION
Learning Initialization (LI) by Wistuba et al. 2015) suggests promising hyperparameters by minimiz- ing a meta-loss function. The meta-loss function defined by the sum of a surrogated loss function on each source task. Intuitively, by minimizing the meta-loss function, LI can obtain hyperparameters that show good performance on average for source tasks.
D.2 DISTBO
DistBO transfers knowledge across tasks using learnt representations of training datasets (Law et al. 2019). To measure similarity between these tasks, DistB uses a distributional kernel, which learns the relationship between source and target tasks by joint Gaussian process model on hyperparameters and data representation. At the first iteration of optimization for the target task, DistBO uses LCB as the acquisition function to quickly select good hyperparameter<"
DistBO models a joint distribution p(z, y) of the training data for each task. However, in the MSU- HPO setting, modeling p(x, y) for the target task is not possible because the labels the target task not available. Therefore, in our experiments, DistBO models the marginal distribution p(x)), not the joint distribution p(æ, y). for each task. This setting also used in the original paper DistBO (Section 5.1 in (Law et al. 2019)). If the covariate shift assumption holds, then from the optimization point of view, it sufficient model p(x), not p(r,y).
Note that while our goal is to minimize the objective function, DistBO aims to maximize the objective function.
15

---

Under review as a conference paper at ICLR 2021
D.3 DISCUSSION ON APPLICATION TO MSU-HPO
The major difference from the proposed method lies in the method of evaluating hyperparameters. While the proposed method evaluates one hyperparameter using all source tasks, LI and DistB consider the situation where hyperparameters are evaluated for any one source task. The effect originated by this difference will become significant when the number of source tasks is large. For example, let us consider scenario of optimization by DistBO where the available evaluation budget B 100. If the number of source tasks is 2, we can optimize each source task with evaluation budget B = 50 to assign a uniform evaluation budget to each source task. On the other hand, if the number of source tasks is 50, the evaluation budget for each source task is only B 2, which makes optimization very difficult. Actually, in the Appendix C.5 in (Law et al., 2019). the existence of 1230 hyperparameter evaluations on source tasks is assumed on Parkinson's ' dataset. In contrast, our method works properly even if there is limited evaluation budget for source tasks. In experiment on Parkinson's is dataset (in Section 4.2 we assume that only 50 hyperparameter evaluations on source tasks available.
E EXPERIMENTAL SETTING
E. SETTINGS
For fair comparison, we used Gaussian Process Upper Confidence Bound (GP-UCB) (Srinivas et al., 2010 as a hyperparameter optimization algorithm for source tasks in all the optimization methods. For example of the proposed method, this corresponds to OPT in Algorithm 1. In experiments. the confidence parameter in GP-UCB is fixed to 2 (following the setting used in (Nguyen et al. 2016
González al. 2016 [Bogunovic al. 2018)).
We set the number of evaluations (i.e., B in Algorithm and 2) to 50. the beginning of optimization, we sample 5 initial points uniform randomly. A Matérn 5/2 kernel used in the implementation of GP-UCB. Note that our study is formulated as a minimization, so we actually use LCB (Lower Confidence Bound) instead of UCB as an acquisition function. We used densratio _acattio________ to estimate the density ratio by uLSIF (Kanamori et al. (009). All the experiments were conducted on Google Cloud Platform (n atl-standard-4) or MacBook Pro (2.2 GHz Intel Core 17, 16 GB).
For LI. which utilizes the gradient descent algorithm to optimize a meta-loss function defined by source tasks, we need to set a learning rate 7 and a number of epochs Following (Wistuba et 2015), we set = 10-3 and F 10 Note that we need one hyperparameter to be evaluated for the target task in MSU-HPO, so the number of hyperparameters we obtain by LI is one. To obtain the results of DistBO in our experiments, we use the authors' implementation 9 We use Matérn 5/2 kernel for a fair comparison with other methods, whereas the default kernel in the code implementation is Matérn 3/2 kernel.
E.2 HYPERPARAMETER OPTIMIZATION
Table and show the hyperparameter search spaces (O) of SVM on the Parkinson's telemonitoring dataset and LightGBM on the GvHD dataset. We treat integer-valued hyperparameters as a continuous variable and rounded off before evaluations. For SVM, used Radial Basis Function kernel in scikit-learn (Pedregosa et al. 2011). We used microsoft/LightGGBBM to implement LightGBM. For the details of these parameters, please refer to the seikit-learn documentation and the LightGBMM documentation [2
We normalized the feature vectors of the GvHD dataset preprocessing procedure. In contrast, in the Parkinson dataset, we did not apply standardization or normalization, because we have confirmed that these operations cause performance degradation. In the Parkinson dataset, we selected task (patient) with the maximum number of data as the target task. In contrast, the tasks in the GvHD
http:////tttbcccc////oooooo/e/ee "htps://github.co https///gghuub..ooo/cllww//w/a//
LighGGMM
wtt.://aaaaae..aaaa..ssssmmsee SVR.htmR.
16

---

Under review as a conference paper at ICLR 2021
dataset all have the same number of data, thus selected the task (patient) with the first task index the target task. For statistical correctness, in the proposed unbiased and variance reduced estimators, used different sources of data for the density ratio estimation and for the training of ML models. Specifically, we used 30% of the training set for the density ratio estimation and the remaining 70% for the learning of the MI. models.
Table 3: Details of hyperparameters of SVM Parkinson's telemonitoring dataset.
Hyperparameters
Type
Scale
Search Space
Kernel Coefficient
float
log
5.0 10 10-5 5 5.0
10
L2 Regularization Parameter
float
log
5.0 x 10 5,5.0 x 10
Table 4: Details of hyperparameters of LightGBM on GvHD dataset.
Hyperparameters
Type
Scale
Search Space
Max Depth for Tree Feature Fraction
int
linear
2, 6]
float
linear
0.0,
Learning Rate
float log
1.0 x 10 1.0 x 10-1)
L2 Regularization Parameter
float log
5.0 x 10-5. 5,5.0 10
17


Reviews:
[{"review_id": "ytpXKyoxcSs", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces multi-source unsupervised hyperparameter optimization (MSU-HPO), a novel BO framework where a range of related tasks are available but labels cannot be accessed for the target task. As ground truth on the target task is unavailable, the work introduces two estimators to approximate the target task objective. This enables HPO to be run to optimize the hyperparameters on the target task, converging faster to a good hyperparameter configuration.\n\n\nPositive\n\n1. **Significance.** To my knowledge, this is the first paper to investigate a transfer learning setting for HPO where labels are unavailable for the target task. This is of interest in several practical applications (e.g., advertising, as the authors discuss). The exploration of a new problem together with the introduction of principled estimators make both the paper's goal and methodology significant. \n2. **Clarity.** The paper was a pleasure to read. Very clear and well structured. I am also confident about reproducibility as enough details about the algorithm and experimental setup are provided. \n3. **Rigorous evaluation.** I appreciated the solid theoretical analysis paired with fully-controlled synthetic experiments where the degree of task similarity can be regulated and results compared against ground truth. All figures are based on multiple runs and have clearly detailed error bars. \n\nNegative\n\n1. **Easy experiments.** The experiments are run on synthetic data and on real-data with only two ML models (SVM and LightGBM). This is secondary considering that the theoretical analysis is solid, but tuning a wider range of ML algorithms (such as neural networks/NAS) would have made the case even stronger by showing that transfer learning is possible across a diverse class of models. The dimensionality of the optimized hyperparameter spaces is also very small, with respectively two and four tuned hyperparameters for SVM and LightGBM. Applying the method to more challenging scenarios would further demonstrate the benefits of the proposed approach. \n2. **Not many baselines.** The main baselines the method is compared against are LI and DISTBO. While many transfer learning baselines are inapplicable as most assume target labels to be available (as discussed in the related work), this is not the case for all of them. An example is Feurer, et al.: Initializing Bayesian hyperparameter optimization via meta-learning, AAAI, 2015. This is not referenced but should be discussed, as it only uses hyperparameter configurations from previous related tasks to warm-start the new optimization. As this does not look at the labels of the target task, it could be compared against. This is also the case for Perrone et al., 2019, which learn a search space purely based on previous tasks and does not require target labels. Another search space pruning method that is not compared against nor discussed is Wistuba, et al.: Hyperparameter search space pruning–a new component for sequential model-based hyperparameter optimization, in Machine Learning and Knowledge Discovery in Databases, 2015.\n\nOverall, I am inclined towards accepting this paper due to the rigorous theoretical evaluation, the significance of the problem, and the fairly satisfactory experimental evaluation (although more baselines and more challenging experiments would be needed to make a stronger case).\n\nAdditional questions\n\na. Related to the point above, how does the proposed approach compare to the transfer learning baselines references above that are purely based on learning an initialization or a good search space?\n\nb. The naive method performs on part with LI in the synthetic experiments and clearly better than DISTBO in the real-world ones. Is this expected? Does this indicate that the chosen baselines (LI and DISTBO) may be too weak? In appendix D3 the authors discuss that this is probably due to the fact that only 50 hyperparameter evaluations on source tasks are available. Would it then not be useful to re-run the comparison under a varying number of hyperparameter evaluations available from source tasks? If the proposed method only outperforms baselines when a small number of prior evaluations are available, this should be clearly stated and shown. \n\nc. Considering the relatively poor results of the unbiased estimator, would it not be better to re-frame the narrative to focus on the variance-reduced estimator? The unbiased one is an interesting ablation study, but given the results it might be better to clearly state from the beginning that the variance reduced estimator is the recommended choice. \n\nd. Will code be made available? \n\nTypos:\n1. Page 12: \"omited\" ---> \"omitted\"\n2. Figure 1b: \"Comapring\" ----> \"Comparing\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "y4DJCew5DUk", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nIn the situation where a given objective is computed with samples from a distribution, e.g. loss on validation data in hyperparameter optimization, this paper proposes a method to construct a surrogate objective using objectives computed on sets of samples each of which is from a different distribution. Basic idea is to use a linear combination of importance sampling estimators. Moreover, the optimal linear combination coefficients are identified in a sense of being optimal in a certain family of convex combination coefficients. This approach has an interesting application that hyperparameters of machine learning deployed on an unseen dataset, importantly, without labels(output) can be optimized as long as the distribution of the input of the unseen dataset is samplable.\n\n\n**Strengths**\n1. The paper provides an algorithm that can estimate an objective computed on data without requiring access to labels in some optimal sense.\n2. Based on the proposed estimator, a transfer hyperparameter optimization algorithm to solve interesting problems is introduced.\n\n\n**Weaknesses**\n1. Maybe the novelty of the paper mainly lies in the combination not on inventing something new, which is, however, not certain since my coverage of relevant literature was not extensive.\n2. I can imagine that someone may ask for more experiments of a large scale or of the type exemplified in the intro. On the other hand, the experiments can be regarded as designed concisely to demonstrate the authors' main points.\n\n\n**Recommendation**\nOverall, I am willing to defend the acceptance of this paper. This combination of transfer HPO and importance sampling estimator seem novel, interesting, and well-demonstrated, with which many interesting applications can be imagined. \n\n\n**Questions**\n- On the line right below eq.(2), the loss function L is assumed to be bounded. Is this condition is necessary in proofs of any theoretical ones? It seems that, in all experiments, all losses are unbounded.\n\n\n**Additional feedback**\n- Explicitly emphasizing that Def 3 is motivated by eq.(6) may guide readers better.\n- While reading the paper, the questions arose were mostly answered after a few lines. The reading was pleasant for me and the paper is well-structured.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hXwEejlzBsM", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors describe a method for training and tuning a machine learning model for a prediction task where no labels are available, and where thus no model can be fit in the standard supervised manner. Instead labels are estimated based on related tasks that do have labels. After this a predictor can be trained on those estimated labels, and can be tuned using a standard Bayesian optimization algorithm.\n\nThe main contribution consist in the description of an estimator for the labels. The paper also provides basic experimental results, both in the form of a naive toy example, and of results on two machine learning datasets. \n\n### Questions / Comments\nThe paper is described (and titled) as a HP tuning paper. But to me it appears to mainly be concerned with unsupervised training, when there are related labeled datasets available, and should thus be described, analysed and tested mainly as such, and compared to other unsupervised learning techniques. The fact that it can be combined with any supervised learning method and can incorporate hyperparameter (HP) tuning is interesting, but the end results is still a unsupervised prediction model.\n\nIn this framing, the comparison only HP tuning algorithms not made for the specific setting does not seem to be the right comparison. Thus I argue that the results are not sufficient to show that the technique proposed can be useful in practice.\n\nThe results are compared to existing HP Tuning warm start algorithms by getting a single suggested HP configuration. One detail that I do not understand from the text is how the HPs found by the baseline are evaluated, if no available labels are assumed. Can the authors shed more light on this aspect?\n\nAs the authors themselves note, the baselines are not well suited for the setting, where we have a very low transfer budget to be split among many source tasks. I would expect much simpler baselines to perform much better, for example: 1) just using the default HPs of the given algorithm 2) running on a single arbitrary source task the whole budget as a simple BO task, and use the best found HPs of this source task. (But it is still not clear how finding HPs is useful if we have no labels, so I might be missing something major here).\n\nThe description of the baseline they call \"Naive\" is also not very clear. If would be good to have more details on this baseline.\n\nIn Experimental Procedure, What does (1) do? How is the ML model tuned if not by MSU-HPO, which seems to be done later. It is not clear to me from the text.\n\nTo summarize, while the method is interesting, it is insufficiently motivated, either as a special type of unsupervised learning, or if it is something different a stronger motivation of why this is worthwhile (you can plug in arbitrary models, use arbitrary HP, tuning algorithms or other reasons). Additionally, the experiments would benefit from clearer descriptions and stronger baselines.\n\n### Typos\nFigure 1b:\nComapring -> Comparing\n\nTable 1 caption:\nperforms almost the same with naive in Parkinson given\ntheir standard errors -> grammar should be improved", "extra_scores": {}, "extra_text": {}}, {"review_id": "bT9_nAK8G2I", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strong points:\n- Well-written and easy to read and follow.\n- Results show the method does what it promises to.\n\nWeaknesses:\n- While it was easy to follow the paper's rationale, I found it difficult to motivate. Until the final experiment, I was left wondering what kind of application would have these constraints but also where the assumptions would be reasonable. This point I think should be easy to address given a small rework of the intro or perhaps a running example to periodically come back to.\n- After reading about the last experiment, I found myself wondering why this solution is billed as a hyperparameter optimization solution; it sounds to me that the parameters of the model are also being optimized along the way. Again, this is a question of clarification and can easily be addressed.\n- The fact that the Naive method beats the other two baselines and performs comparably to the unbiased proposed method makes me wonder whether (a) these are challenging enough tasks, or (b) those are competitive baselines.\n- The density estimator and the divergence estimator are moving parts the errors of which perhaps warrant quantifying. For example, in both experiments the true labels were known and the authors can measure the error in the divergence estimate.\n\nI'd be happy to increase my score if the above points are addressed.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has been actively discussed in the light of the authors’ response. Even though the paper was, overall, found quite clear with a solid theoretical support, the reviewers listed several concerns that remained unsolved after the rebuttal, e.g.,

* The proposed approach may not be properly scoped/positioned and evaluated as an HPO method, a concern unanimously shared across the reviewers. Although this is not a concern impossible to overcome, the reviewers believed it could not be achieved as part of a simple revision of the paper.
* The lack of challenging baselines to fully assess the performance of the proposed method (e.g., see the list suggested by Reviewer 1) 
* Along the lines of the previous point, the experiments focus on small-scale settings, which does not make it possible to completely assess the performance of the approach
* Some discrepancy between the theoretical analysis and the actual experimental settings (e.g., the assumption about bounded losses not valid with the squared loss) 

As illustrated by its scores, the paper is extremely borderline. Given the mixed perspectives of pros and cons, the paper is eventually recommended for rejection. 
This list, together with the detailed comments of the reviewers, highlight opportunities to improve the manuscript for a future resubmission.

Decision: Reject