A Copula approach for hyperparameter transfer learning

Bayesian optimization ( BO ) is a popular methodology to tune the hyperparameters of expensive black-box functions . Despite its success , standard BO focuses on a single task at a time and is not designed to leverage information from related functions , such as tuning performance objectives of the same algorithm across multiple datasets . In this work , we introduce a novel approach to achieve transfer learning across different datasets as well as different objectives . The main idea is to regress the mapping from hyperparameter to objective quantiles with a semi-parametric Gaussian Copula distribution , which provides robustness against different scales or outliers that can occur in different tasks . We introduce two methods to leverage this estimation : a Thompson sampling strategy as well as a Gaussian Copula process using such quantile estimate as a prior . We show that these strategies can combine the estimation of multiple objectives such as runtime and accuracy , steering the optimization toward cheaper hyperparameters for the same level of accuracy . Experiments on an extensive set of hyperparameter tuning tasks demonstrate significant improvements over state-of-the-art methods . 1 INTRODUCTION . Tuning complex machine learning models such as deep neural networks can be a daunting task . Object detection or language understanding models often rely on deep neural networks with many tunable hyperparameters , and automatic hyperparameter optimization ( HPO ) techniques such as Bayesian optimization ( BO ) are critical to find the good hyperparameters in short time . BO addresses the black-box optimization problem by placing a probabilistic model on the function to minimize ( e.g. , the mapping of neural network hyperparameters to a validation loss ) , and determine which hyperparameters to evaluate next by trading off exploration and exploitation through an acquisition function . While traditional BO focuses on each problem in isolation , recent years have seen a surge of interest in transfer learning for HPO . The key idea is to exploit evaluations from previous , related tasks ( e.g. , the same neural network tuned on multiple datasets ) to further speed up the hyperparameter search . A central challenge of hyperparameter transfer learning is that different tasks typically have different scales , varying noise levels , and possibly contain outliers , making it hard to learn a joint model . In this work , we show how a semi-parametric Gaussian Copula can be leveraged to learn a joint prior across datasets in such a way that scale issues vanish . We then demonstrate how such prior estimate can be used to transfer information across tasks and objectives . We propose two HPO strategies : a Copula Thompson Sampling and a Gaussian Copula Process . We show that these approaches can jointly model several objectives with potentially different scales , such as validation error and compute time , without requiring processing . We demonstrate significant speed-ups over a number of baselines in extensive experiments . The paper is organized as follows . Section 2 reviews related work on transfer learning for HPO . Section 3 introduces Copula regression , the building block for the HPO strategies we propose in Section 4 . Specifically , we show how Copula regression can be applied to design two HPO strategies , one based on Thompson sampling and an alternative GP-based approach . Experimental results are given in Section 5 where we evaluate both approaches against state-of-the-art methods on three algorithms . Finally , Section 6 outlines conclusions and further developments . 2 RELATED WORK . A variety of methods have been developed to induce transfer learning in HPO . The most common approach is to model tasks jointly or via a conditional independence structure , which has been been explored through multi-output GPs ( Swersky et al. , 2013 ) , weighted combination of GPs ( Schilling et al. , 2016 ; Wistuba et al. , 2018 ; Feurer et al. , 2018 ) , and neural networks , either fully Bayesian ( Springenberg et al. , 2016 ) or hybrid ( Snoek et al. , 2015 ; Perrone et al. , 2018 ; Law et al. , 2018 ) . A different line of research has focused on the setting where tasks come over time as a sequence and models need to be updated online as new problems accrue . A way to achieve this is to fit a sequence of surrogate models to the residuals relative to predictions of the previously fitted model ( Golovin et al. , 2017 ; Poloczek et al. , 2016 ) . Specifically , the GP over the new task is centered on the predictive mean of the previously learned GP . Finally , rather than fitting a surrogate model to all past data , some transfer can be achieved by warm-starting BO with the solutions to the previous BO problems ( Feurer et al. , 2015 ; Wistuba et al. , 2015b ) . A key challenge for joint models is that different black-boxes can exhibit heterogeneous scale and noise levels ( Bardenet et al. , 2013 ; Feurer et al. , 2018 ) . To address this , some methods have instead focused on search-space level , aiming to prune it to focus on regions of the hyperparameter space where good configurations are likely to lie . An example is Wistuba et al . ( 2015a ) , where related tasks are used to learn a promising search space during HPO , defining task similarity in terms of the distance of the respective data set meta-features . A more recent alternative that does not require meta-features was introduced in Perrone et al . ( 2019 ) , where a restricted search space in the form of a low-volume hyper-rectangle or hyper-ellipsoid is learned from the optimal hyperparameters of related tasks . Rank estimation can be used to alleviate scales issues however the difficulty of feeding back rank information to GP leads to restricting assumptions , for instance ( Bardenet et al. , 2013 ) does not model the rank estimation uncertainty while ( Feurer et al. , 2018 ) uses independent GPs removing the adaptivity of the GP to the current task . Gaussian Copula Process ( GCP ) ( Wilson & Ghahramani , 2010 ) can also be used to alleviate scale issues on a single task at the extra cost of estimating the CDF of the data . Using GCP for HPO was proposed in Anderson et al . ( 2017 ) to handle potentially non-Gaussian data , albeit only considering non-parametric homoskedastic priors for the single-task and single objective case . 3 GAUSSIAN COPULA REGRESSION . For each task denote with f j : Rp → R the error function one wishes to minimize , and with D = { xi , yi } Ni=1 the evaluations available for an arbitrary task . Given the evaluations on M tasks DM = ⋃ 1≤j≤M { x j i , y j i } Nj i=1 , we are interested in speeding up the optimization of an arbitrary new task f , namely in finding arg minx∈Rp f ( x ) in the least number of evaluations . One possible approach to speed-up the optimization of f is to build a surrogate model f̂ ( x ) . While using a Gaussian process is possible , scaling such an approach to the large number of evaluations available in a transfer learning setting is challenging . Instead , we propose fitting a parametric estimate of f̂θ ( x ) distribution which can be later used in HPO strategies as a prior of a Gaussian Copula Process . A key requirement here is to learn a joint model , e.g. , we would like to find θ which fits well on all observed tasks f j . We show how this can be achieved with a semi-parametric Gaussian Copula in two steps : first we map all evaluations to quantiles with the empirical CDF , and then we fit a parametric Gaussian distribution on quantiles mapped through the Gaussian inverse CDF . First , observe that as every yi comes from the same distribution for a given task , the probability integral transform results in ui = F ( yi ) , where F is the cumulative distribution function of y . We then model the CDF of ( u1 , . . . , uN ) with a Gaussian Copula : C ( u1 , . . . , uN ) = φµ , Σ ( Φ −1 ( F ( y1 ) ) , . . . , Φ −1 ( F ( yN ) ) ) , where Φ is the standard normal CDF and φµ , Σ is the CDF of a normal distribution parametrized by µ and Σ . Assuming F to be invertible , we define the change of variable z = Φ−1 ◦ F ( y ) = ψ ( y ) and g = ψ ◦ f .1 We regress the marginal distribution of P ( z|x ) with a Gaussian distribution whose 1Note that if z is regressed perfectly , then finding the minimum of f is solved as a parameter x minimizing ψ ( f ( x ) ) also minimizes f ( x ) since ψ is monotonically increasing . mean and variance are two deterministic parametric functions given by P ( z|x ) ∼ N ( µθ ( x ) , σθ ( x ) ) = N ( wTµhwh ( x ) + bµ , Ψ ( wTσ hwh ( x ) + bσ ) ) , where hwh ( x ) ∈ Rd is the output of a multi-layer perceptron ( MLP ) where wh , wµ ∈ Rd , bµ ∈ R , wσ ∈ Rd , bσ ∈ R are projection parameters and Ψ ( t ) = log ( 1 + exp t ) is an activation mapping to positive numbers . The parameters θ = { wh , wµ , bµ , wσ , bσ } together with the parameters in MLP are learned by minimizing the Gaussian negative log-likelihood on the available evaluations DM = ⋃ 1≤j≤M { x j i , z j i } Nj i=1 , e.g. , by minimizing∑ ( x , z ) ∈DM 1 2 log 2πσ ( x ) 2 + 1 2 ( z − µ ( x ) σ ( x ) ) 2 + ψ′ ( ψ−1 ( z ) ) , ( 1 ) with SGD . Here , the term ψ′ ( ψ−1 ( z ) ) accounts for the change of variable z = ψ ( y ) . Due to the term ψ′ ( ψ−1 ( z ) ) , errors committed where the quantile function changes rapidly have larger gradient than when the quantile function is flat . Note that while we weight evaluations of each tasks equally , one may alternatively normalize gradient contributions per number of task evaluations . The transformation ψ requires F , which needs to be estimated . Rather than using a parametric or density estimation approach , we use the empirical CDF F̃ ( t ) = 1N ∑N i=1 1yi≤t . While this estimator has the advantage of being non-parametric , it leads to infinite value when evaluating ψ at the minimum of maximum of y . To avoid this issue , we use the Winsorized cut-off estimator F ( t ) ≈ δN if F̃ ( t ) < δN F̃ ( t ) if δN ≤ F̃ ( t ) ≤ 1− δN 1− δN if F̃ ( t ) > 1− δN whereN is the number of observations of y and choosing δN = 14N1/4√π logN strikes a bias-variance trade-off ( Liu et al. , 2009 ) . This approach is semi-parametric in that the CDF is estimated with a non-parametric estimator and the Gaussian Copula is estimated with a parametric approach . The benefit of using a non-parametric estimator for the CDF is that it allows us to map the observations of each task to comparable distributions as zj ∼ N ( 0 , 1 ) for all tasks j . This is critical to allow the joint learning of the parametric estimates µθ and σθ , which share their parameter θ across all tasks . As our experiments will show , one can regress a parametric estimate that has a standard error lower than 1 . This means that information can be leveraged from the evaluations obtained on related tasks , whereas a trivial predictor for z would predict 0 and yield a standard error of 1 . In the next section we show how this estimator can be leveraged to design two novel HPO strategies .

Summary: This paper proposes a new Bayesian optimization (BO) based hyperparameter searching method that can transfer across different datasets and different metrics. The method is to build a regression model based on Gaussian Copula distribution, which maps from hyperparameter to metric quantiles. The paper shows that by leveraging this estimation using some specific (sampling) strategies, it is able to improve over other BO methods.