Zero-shot Transfer Learning for Gray-box Hyper-parameter Optimization

1 INTRODUCTION . Within the research community , the concentration of efforts towards solving the problem of hyperparameter optimization ( HPO ) has been mainly through sequential model-based optimization ( SMBO ) , i.e . iteratively fitting a probabilistic response model , typically a Gaussian process ( Rasmussen ( 2003 ) ) , to a history of observations of losses of the target response , and suggesting the next hyper-parameters via a policy , acquisition function , that balances exploration and exploitation by leveraging the uncertainty in the posterior distribution ( Jones et al . ( 1998 ) ; Wistuba et al . ( 2018 ) ; Snoek et al . ( 2012 ) ) . However , even when solutions are defined in conjunction with transfer learning techniques ( Bardenet et al . ( 2013 ) ; Wistuba et al . ( 2016 ) ; Feurer et al . ( 2015 ) ) , the performance of SMBO is heavily affected by the choice of the initial hyper-parameters . Furthermore , SMBO is sequential by design and additional acceleration by parallelization is not possible . In this paper , we present the problem of zero-shot hyper-parameter optimization as a meta-learning objective that exploits dataset information as part of the surrogate model . Instead of treating HPO as a black-box function , operating blindly on the response of the hyper-parameters alone , we treat it as a gray-box function ( Whitley et al . ( 2016 ) ) , by capturing the relationship among the dataset meta-features and hyper-parameters to approximate the response model . In this paper , we propose a novel formulation of HPO as a conditional gray-box function optimization problem , Section 4 , that allows us to regress from the dataset/hyper-parameter pair directly onto the response . Driven by the assumption that similar datasets should have similar response approximations , we introduce an additional data-driven similarity regularization objective to penalize the difference between the predicted response of similar datasets . In Section 5 , we perform an extensive battery of experiments that highlight the capacity of our universal model to serve as a solution for : ( 1 ) zero-shot HPO as a stand-alone task , ( 2 ) zero-shot as an initialization strategy for Bayesian Optimization ( BO ) , ( 3 ) transferable sequential model-based optimization . A summary of our contributions is : • a formulation of the zero-shot hyper-parameter optimization problem in which our response model predicts upfront the full set of hyper-parameter configurations to try , without access to observations of losses of the target response ; • a novel multi-task optimization objective that models the inherent similarity between datasets and their respective responses ; • three new meta-datasets with different search spaces and cardinalities to facilitate the experiments and serve as a benchmark for future work ; • an empirical demonstration of the performance of our approach through a battery of experiments that address the aforementioned research aspects , and a comparison against state-of-the-art HPO solutions for transfer-learning . 2 RELATED WORK . The straightforward zero-shot approaches for HPO consist of random search ( Bergstra & Bengio ( 2012 ) ) , or simply selecting hyper-parameters that perform well on general tasks ( Brazdil et al . ( 2003 ) ) . Some recent work has also shown that simply selecting random hyper-parameters from a restricted search space significantly outperforms existing solutions , and improves the performance of conventional SMBO approaches ( Perrone et al . ( 2019 ) ) . The restricted search space is created by eliminating regions that are further away from the best hyper-parameters of the training tasks . Another prominent direction for zero-shot HPO depends heavily on engineered meta-features , i.e . dataset characteristics ( Vanschoren ( 2018 ) ) , to measure the similarity of datasets . Following the assumption that the responses of similar datasets behave similarly to the hyper-parameters , it has been shown that even the simplest of meta-features ( Bardenet et al . ( 2013 ) ) improve the performance of single task BO algorithms ( Feurer et al . ( 2014 ; 2015 ) ) . The target response is initialized with the top-performing hyper-parameters of the dataset nearest neighbor in the meta-feature space . The shortcomings of using engineered meta-features are that they are hard to define ( Leite & Brazdil ( 2005 ) ) , and are often selected through trial-and-error or expert domain knowledge . As a remedy , replacing engineered meta-features with learned meta-features ( Jomaa et al . ( 2019 ) ) compensates for such limitations , by producing expressive meta-features agnostic to any meta-task , such as HPO . Zero-shot HPO is also posed as an optimization problem that aims to minimize the meta-loss over a collection of datasets ( Wistuba et al . ( 2015a ) ) by replacing the discrete minimum function with a differentiable softmin function as an approximation . The initial configurations boost the single task BO without any meta-features . In ( Wistuba et al . ( 2015b ) ) , hyper-parameter combinations are assigned a static ranking based on the cumulative average normalized error , and dataset similarity is estimated based on the relative ranking of these combinations . Winkelmolen et al . ( 2020 ) introduce a Bayesian Optimization solution for zero-shot HPO by iteratively fitting a surrogate model over the observed responses of different tasks , and selecting the next hyper-parameters and datasets that minimize the aggregated observed loss . Aside from zero-shot HPO , transfer learning is employed by learning better response models ( Wistuba et al . ( 2016 ) ) based on the similarity of the response . Feurer et al . ( 2018 ) propose an ensemble model for BO by building the target response model as a weighted sum of the predictions of base models as well as the target model . In addition to the transferable response models , Volpp et al . ( 2019 ) design a transferable acquisition function as a policy for hyper-parameter optimization defined in a reinforcement learning framework . As a replacement to the standard Gaussian process , Perrone et al . ( 2018 ) train a multi-task adaptive Bayesian linear regression model with a shared feature extractor that provides context information for each independent task . In contrast to the literature , we formulate the problem of zero-shot HPO as a gray-box function optimization problem , by designing a universal response model defined over the combined domain of datasets and hyper-parameters . We rely on the embeddings to estimate the similarities across datasets and design a novel multi-task optimization objective to regress directly on the response . This allows us to delineate from the complexity paired with Bayesian uncertainty , as well as the trouble of engineering similarity measures . 3 HYPER-PARAMETER OPTIMIZATION . Consider a dataset D = { ( x ( Train ) , y ( Train ) ) , ( x ( Val ) , y ( Val ) ) , ( x ( Test ) , y ( Test ) ) } for a supervised learning task , with training , validation and test splits of predictors x ∈ X and targets y ∈ Y . We aim at training a parametric approximation of the target using ŷ : = f ( θ , λ ) : X → Y , where θ ∈ Θ denotes the parameters and λ ∈ Λ its hyper-parameters , by minimizing a loss function L : Y × Y → R as : λ∗ = arg min λ∈Λ L ( y ( Val ) , f ( x ( Val ) ; θ∗ , λ ) ) s.t . θ∗ = arg min θ∈Θ L ( y ( Train ) , f ( x ( Train ) ; θ , λ ) ) ( 1 ) We hereafter denote the validation error as the response ` ( λ ) : = L ( y ( Val ) , f ( x ( Val ) ; θ∗ , λ ) ) . Unfortunately , a direct optimization of the response ` ( λ ) in terms of λ is not trivial , because θ∗ is the result of the minimization problem and its gradients with respect to λ are not easy to compute . Instead , in order to learn the optimal hyper-parameters λ we train a probabilistic surrogate ˆ̀ ( λ ; β ) : Λ× B → R parameterized by β ∈ B , with B as the space of response model parameters , that minimizes the log-likelihood of approximating the response ` ( λ ) over a set of K evaluations S : = { ( λ1 , ` ( λ1 ) ) , . . . , ( λK , ` ( λK ) ) } . We denote P as the probability of estimating the response given a surrogate model . Given the surrogate , the next hyper-parameter to be evaluated λ ( next ) is computed by maximizing an acquisition function A ( e.g . EI ( Močkus ( 1975 ) ) as : λ ( next ) : = arg max λ∈Λ A ( ˆ̀ ( λ ; β∗ ) ) s.t . β∗ : = arg min β∈B K∑ k=1 lnP ( ` ( λk ) , ˆ̀ ( λk ; β ) ) ( 2 ) 4 META-LEARNING OF CONDITIONAL GRAY-BOX SURROGATES . Let us define a collection of T datasets as { D ( 1 ) , . . . , D ( T ) } and let ` ( t ) ( λ ) measure the response of the hyper-parameter λ on the t-th dataset D ( t ) . Furthermore , assume we have previously evaluated K ( t ) many hyper-parameters λ ( t ) k , k ∈ { 1 , . . . , K ( t ) } on that particular dataset . We condition the surrogate ˆ̀ to capture the characteristics of the t-th dataset , by taking as input the meta-features representation of the dataset as φ ( t ) . Therefore , a dataset-aware surrogate can be trained using meta-learning over a cumulative objective function O ( β ) as : O ( β ) : = T∑ t=1 K ( t ) ∑ k=1 ( ` ( t ) ( λ ( t ) k ) − ˆ̀ ( λ ( t ) k , φ ( t ) ; β ) ) 2 ( 3 ) 4.1 THE META-FEATURE EXTRACTOR . Introducing engineered meta-features has had a significant impact on hyper-parameter optimization . However , learning meta-features across datasets of varying schema in a task-agnostic setting provides more representative characteristics than to rely on hard-to-tune empirical estimates . The meta-feature extractor is a set-based function ( Zaheer et al . ( 2017 ) ) that presents itself as an extended derivation of the Kolmogorov-Arnold representation theorem ( Krková ( 1992 ) ) , which states that a multi-variate function φ can be defined as an aggregation of univariate functions over single variables , Appendix B . Each supervised ( tabular ) dataset D ( t ) : = ( x ( t ) , y ( t ) ) consists of instances x ( t ) ∈ X ∈ RN×M and targets y ( t ) ∈ Y ∈ RN×C such that N , M and C represent the number of instances , predictors and targets respectively . The dataset can be further represented as a set of smaller components , set of sets , D ( t ) = { ( x ( t ) i , m , y ( t ) i , c ) | m ∈ { 1 , . . . , M } , i ∈ { 1 , . . . , N } , c ∈ { 1 , . . . , C } } . A tabular dataset composed of columns ( predictors , targets ) and rows ( instances ) is reduced to single predictor-target pairs instead of an instance-target pairs . Based on this representation , a meta-feature extractor parameterized as a neural network ( Jomaa et al . ( 2019 ) ) , is formulated in Equation 4 . For simplicity of notation , we drop the superscript ( t ) unless needed . φ ( D ) = h ( 1 MC M∑ m=1 C∑ c=1 g ( 1 N N∑ i=1 f ( xi , m , yi , c ) ) ) ( 4 ) with f : R2 → RKf , g : RKf → RKg and h : RKg → RK represented by neural networks with Kf , Kg , and K output units , respectively . This set-based formulation captures the correlation between each variable ( predictor ) and its assigned target and is permutation-invariant , i.e . the output is unaffected by the ordering of the pairs in the set . Other set-based functions such as ( Edwards & Storkey ( 2016 ) ; Lee et al . ( 2019 ) ) can also be used for meta-feature extraction , however , we focus on this deep-set formulation ( Jomaa et al . ( 2019 ) ) because it is proven to work properly for hyper-parameter optimization .

Summary: In this papar, the authors formulated a new objective function for HBO, which included an additional regularization term based on the dataset similarity. The authors used the distance between the meta-features of selected datasets to measure this dataset similarity and assumpted that similar datasets should have similar hyper-paprameters.  The experiments are complete and demonstrate the advantages of using this new optimization formulation.