Zero-shot Transfer Learning for Gray-box Hyper-parameter Optimization

1 INTRODUCTION . Within the research community , the concentration of efforts towards solving the problem of hyperparameter optimization ( HPO ) has been mainly through sequential model-based optimization ( SMBO ) , i.e . iteratively fitting a probabilistic response model , typically a Gaussian process ( Rasmussen ( 2003 ) ) , to a history of observations of losses of the target response , and suggesting the next hyper-parameters via a policy , acquisition function , that balances exploration and exploitation by leveraging the uncertainty in the posterior distribution ( Jones et al . ( 1998 ) ; Wistuba et al . ( 2018 ) ; Snoek et al . ( 2012 ) ) . However , even when solutions are defined in conjunction with transfer learning techniques ( Bardenet et al . ( 2013 ) ; Wistuba et al . ( 2016 ) ; Feurer et al . ( 2015 ) ) , the performance of SMBO is heavily affected by the choice of the initial hyper-parameters . Furthermore , SMBO is sequential by design and additional acceleration by parallelization is not possible . In this paper , we present the problem of zero-shot hyper-parameter optimization as a meta-learning objective that exploits dataset information as part of the surrogate model . Instead of treating HPO as a black-box function , operating blindly on the response of the hyper-parameters alone , we treat it as a gray-box function ( Whitley et al . ( 2016 ) ) , by capturing the relationship among the dataset meta-features and hyper-parameters to approximate the response model . In this paper , we propose a novel formulation of HPO as a conditional gray-box function optimization problem , Section 4 , that allows us to regress from the dataset/hyper-parameter pair directly onto the response . Driven by the assumption that similar datasets should have similar response approximations , we introduce an additional data-driven similarity regularization objective to penalize the difference between the predicted response of similar datasets . In Section 5 , we perform an extensive battery of experiments that highlight the capacity of our universal model to serve as a solution for : ( 1 ) zero-shot HPO as a stand-alone task , ( 2 ) zero-shot as an initialization strategy for Bayesian Optimization ( BO ) , ( 3 ) transferable sequential model-based optimization . A summary of our contributions is : • a formulation of the zero-shot hyper-parameter optimization problem in which our response model predicts upfront the full set of hyper-parameter configurations to try , without access to observations of losses of the target response ; • a novel multi-task optimization objective that models the inherent similarity between datasets and their respective responses ; • three new meta-datasets with different search spaces and cardinalities to facilitate the experiments and serve as a benchmark for future work ; • an empirical demonstration of the performance of our approach through a battery of experiments that address the aforementioned research aspects , and a comparison against state-of-the-art HPO solutions for transfer-learning . 2 RELATED WORK . The straightforward zero-shot approaches for HPO consist of random search ( Bergstra & Bengio ( 2012 ) ) , or simply selecting hyper-parameters that perform well on general tasks ( Brazdil et al . ( 2003 ) ) . Some recent work has also shown that simply selecting random hyper-parameters from a restricted search space significantly outperforms existing solutions , and improves the performance of conventional SMBO approaches ( Perrone et al . ( 2019 ) ) . The restricted search space is created by eliminating regions that are further away from the best hyper-parameters of the training tasks . Another prominent direction for zero-shot HPO depends heavily on engineered meta-features , i.e . dataset characteristics ( Vanschoren ( 2018 ) ) , to measure the similarity of datasets . Following the assumption that the responses of similar datasets behave similarly to the hyper-parameters , it has been shown that even the simplest of meta-features ( Bardenet et al . ( 2013 ) ) improve the performance of single task BO algorithms ( Feurer et al . ( 2014 ; 2015 ) ) . The target response is initialized with the top-performing hyper-parameters of the dataset nearest neighbor in the meta-feature space . The shortcomings of using engineered meta-features are that they are hard to define ( Leite & Brazdil ( 2005 ) ) , and are often selected through trial-and-error or expert domain knowledge . As a remedy , replacing engineered meta-features with learned meta-features ( Jomaa et al . ( 2019 ) ) compensates for such limitations , by producing expressive meta-features agnostic to any meta-task , such as HPO . Zero-shot HPO is also posed as an optimization problem that aims to minimize the meta-loss over a collection of datasets ( Wistuba et al . ( 2015a ) ) by replacing the discrete minimum function with a differentiable softmin function as an approximation . The initial configurations boost the single task BO without any meta-features . In ( Wistuba et al . ( 2015b ) ) , hyper-parameter combinations are assigned a static ranking based on the cumulative average normalized error , and dataset similarity is estimated based on the relative ranking of these combinations . Winkelmolen et al . ( 2020 ) introduce a Bayesian Optimization solution for zero-shot HPO by iteratively fitting a surrogate model over the observed responses of different tasks , and selecting the next hyper-parameters and datasets that minimize the aggregated observed loss . Aside from zero-shot HPO , transfer learning is employed by learning better response models ( Wistuba et al . ( 2016 ) ) based on the similarity of the response . Feurer et al . ( 2018 ) propose an ensemble model for BO by building the target response model as a weighted sum of the predictions of base models as well as the target model . In addition to the transferable response models , Volpp et al . ( 2019 ) design a transferable acquisition function as a policy for hyper-parameter optimization defined in a reinforcement learning framework . As a replacement to the standard Gaussian process , Perrone et al . ( 2018 ) train a multi-task adaptive Bayesian linear regression model with a shared feature extractor that provides context information for each independent task . In contrast to the literature , we formulate the problem of zero-shot HPO as a gray-box function optimization problem , by designing a universal response model defined over the combined domain of datasets and hyper-parameters . We rely on the embeddings to estimate the similarities across datasets and design a novel multi-task optimization objective to regress directly on the response . This allows us to delineate from the complexity paired with Bayesian uncertainty , as well as the trouble of engineering similarity measures . 3 HYPER-PARAMETER OPTIMIZATION . Consider a dataset D = { ( x ( Train ) , y ( Train ) ) , ( x ( Val ) , y ( Val ) ) , ( x ( Test ) , y ( Test ) ) } for a supervised learning task , with training , validation and test splits of predictors x ∈ X and targets y ∈ Y . We aim at training a parametric approximation of the target using ŷ : = f ( θ , λ ) : X → Y , where θ ∈ Θ denotes the parameters and λ ∈ Λ its hyper-parameters , by minimizing a loss function L : Y × Y → R as : λ∗ = arg min λ∈Λ L ( y ( Val ) , f ( x ( Val ) ; θ∗ , λ ) ) s.t . θ∗ = arg min θ∈Θ L ( y ( Train ) , f ( x ( Train ) ; θ , λ ) ) ( 1 ) We hereafter denote the validation error as the response ` ( λ ) : = L ( y ( Val ) , f ( x ( Val ) ; θ∗ , λ ) ) . Unfortunately , a direct optimization of the response ` ( λ ) in terms of λ is not trivial , because θ∗ is the result of the minimization problem and its gradients with respect to λ are not easy to compute . Instead , in order to learn the optimal hyper-parameters λ we train a probabilistic surrogate ˆ̀ ( λ ; β ) : Λ× B → R parameterized by β ∈ B , with B as the space of response model parameters , that minimizes the log-likelihood of approximating the response ` ( λ ) over a set of K evaluations S : = { ( λ1 , ` ( λ1 ) ) , . . . , ( λK , ` ( λK ) ) } . We denote P as the probability of estimating the response given a surrogate model . Given the surrogate , the next hyper-parameter to be evaluated λ ( next ) is computed by maximizing an acquisition function A ( e.g . EI ( Močkus ( 1975 ) ) as : λ ( next ) : = arg max λ∈Λ A ( ˆ̀ ( λ ; β∗ ) ) s.t . β∗ : = arg min β∈B K∑ k=1 lnP ( ` ( λk ) , ˆ̀ ( λk ; β ) ) ( 2 ) 4 META-LEARNING OF CONDITIONAL GRAY-BOX SURROGATES . Let us define a collection of T datasets as { D ( 1 ) , . . . , D ( T ) } and let ` ( t ) ( λ ) measure the response of the hyper-parameter λ on the t-th dataset D ( t ) . Furthermore , assume we have previously evaluated K ( t ) many hyper-parameters λ ( t ) k , k ∈ { 1 , . . . , K ( t ) } on that particular dataset . We condition the surrogate ˆ̀ to capture the characteristics of the t-th dataset , by taking as input the meta-features representation of the dataset as φ ( t ) . Therefore , a dataset-aware surrogate can be trained using meta-learning over a cumulative objective function O ( β ) as : O ( β ) : = T∑ t=1 K ( t ) ∑ k=1 ( ` ( t ) ( λ ( t ) k ) − ˆ̀ ( λ ( t ) k , φ ( t ) ; β ) ) 2 ( 3 ) 4.1 THE META-FEATURE EXTRACTOR . Introducing engineered meta-features has had a significant impact on hyper-parameter optimization . However , learning meta-features across datasets of varying schema in a task-agnostic setting provides more representative characteristics than to rely on hard-to-tune empirical estimates . The meta-feature extractor is a set-based function ( Zaheer et al . ( 2017 ) ) that presents itself as an extended derivation of the Kolmogorov-Arnold representation theorem ( Krková ( 1992 ) ) , which states that a multi-variate function φ can be defined as an aggregation of univariate functions over single variables , Appendix B . Each supervised ( tabular ) dataset D ( t ) : = ( x ( t ) , y ( t ) ) consists of instances x ( t ) ∈ X ∈ RN×M and targets y ( t ) ∈ Y ∈ RN×C such that N , M and C represent the number of instances , predictors and targets respectively . The dataset can be further represented as a set of smaller components , set of sets , D ( t ) = { ( x ( t ) i , m , y ( t ) i , c ) | m ∈ { 1 , . . . , M } , i ∈ { 1 , . . . , N } , c ∈ { 1 , . . . , C } } . A tabular dataset composed of columns ( predictors , targets ) and rows ( instances ) is reduced to single predictor-target pairs instead of an instance-target pairs . Based on this representation , a meta-feature extractor parameterized as a neural network ( Jomaa et al . ( 2019 ) ) , is formulated in Equation 4 . For simplicity of notation , we drop the superscript ( t ) unless needed . φ ( D ) = h ( 1 MC M∑ m=1 C∑ c=1 g ( 1 N N∑ i=1 f ( xi , m , yi , c ) ) ) ( 4 ) with f : R2 → RKf , g : RKf → RKg and h : RKg → RK represented by neural networks with Kf , Kg , and K output units , respectively . This set-based formulation captures the correlation between each variable ( predictor ) and its assigned target and is permutation-invariant , i.e . the output is unaffected by the ordering of the pairs in the set . Other set-based functions such as ( Edwards & Storkey ( 2016 ) ; Lee et al . ( 2019 ) ) can also be used for meta-feature extraction , however , we focus on this deep-set formulation ( Jomaa et al . ( 2019 ) ) because it is proven to work properly for hyper-parameter optimization .

Summary: The authors propose a new zero-shot hyper-parameter optimization method based-on the meta-learning framework. The proposed method incorporates two ideas from the meta-learning framework namely the task similarity based on the meta-features and the dataset identification. The former idea is used to achieve the requirement that responses to similar data sets should be similar.The latter idea has the role of preventing data of dissimilar tasks from being embedded in close proximity in the meta-feature space.