Learning Representation for Bayesian Optimization with Collision-free Regularization

Bayesian Optimization has been challenged by the large-scale and highdimensional datasets , which are common in real-world scenarios . Recent works attempt to handle such input by applying neural networks ahead of the classical Gaussian process to learn a ( low-dimensional ) latent representation . We show that even with proper network design , such learned representation often leads to collision in the latent space : two points with significantly different observations collide in the learned latent space , leading to degraded optimization performance . To address this issue , we propose LOCO , an efficient deep Bayesian optimization framework which employs a novel regularizer to reduce the collision in the learned latent space and encourage the mapping from the latent space to the objective value to be Lipschitz continuous . LOCO takes in pairs of data points and penalizes those too close in the latent space compared to their target space distance . We provide a rigorous theoretical justification for LOCO by inspecting the regret of this dynamic-embedding-based Bayesian optimization algorithm , where the neural network is iteratively retrained with the regularizer . Our empirical results further demonstrate the effectiveness of LOCO on several synthetic and realworld benchmark Bayesian optimization tasks . 1 INTRODUCTION . Bayesian optimization is a classical sequential optimization method and is widely used in various fields in science and engineering , including recommender systems ( Galuzzi et al. , 2019 ) , medical trials ( Sui et al. , 2018 ) , robotic controller optimization ( Berkenkamp et al. , 2016 ) , scientific experimental design ( Yang et al. , 2019 ) , and hyper-parameter tuning ( Snoek et al. , 2012 ) , among many others . Many of these applications involve evaluating an expensive blackbox function ; therefore , the number of queries should be minimized . A common way to model the unknown function is via Gaussian processes ( GPs ) ( Rasmussen and Williams , 2006 ) , which have been extensively studied under the bandit setting , as an effective surrogate model of the unknown objective function in a broad class of blackbox function optimization problems ( Srinivas et al. , 2010 ; Djolonga et al. , 2013 ) . A key computational challenge for learning with GPs lies in optimizing specific kernels used for modeling the covariance structures . Such an optimization task depends on the dimension of the input space . For high-dimensional data , it is often prohibitive to train a GP model . Meanwhile , local kernel machines are known to suffer from the curse of dimensionality ( Bengio et al. , 2005 ) , while the required number of training samples could grow exponentially with the dimensionality of the data . Therefore , dimensionality reduction and representation learning algorithms are needed to optimize the learning process . Recently , Gaussian process optimization has been investigated in the context of latent space models . For example , deep kernel learning ( Wilson et al. , 2016 ) learns a ( low-dimensional ) data representation and a scalable kernel simultaneously via an end-to-end trainable deep neural network . In general , the neural network is trained to learn a simpler latent representation with reduced dimension and has the structure information already embedded for the GP . Combining the representation learned via a neural network with GP could improve the scalability and extensibility of classical Bayesian optimization , but it also poses new challenges for the optimization task , such as dealing with the tradeoff between representation learning and function optimization ( Tripp et al. , 2020 ) . As we later demonstrate , a critical challenge brought by introducing representation learning into Bayesian optimization is that the latent representation is prone to collisions : two points with significantly different observations can get too close , and therefore collide in the latent space . The collision effect in latent space models for Bayesian optimization is especially evident when information is lost during dimensionality reduction and/or when the training data is limited in size . As illustrated in Figure 1 , when passed through the neural network , data points with drastically different observations are mapped to close positions in the latent space . Such collisions could be regarded as additional noise introduced by the neural network . Although Bayesian optimization is known to be robust to mild noisy observations ( Bogunovic et al. , 2018 ) , the collision in latent space could be harmful to the optimization performance , as it is non-trivial to model the collision into the acquisition function explicitly . Also , the additional noise induced by the collision effect will further loosen the regret bound for classical Bayesian optimization algorithms ( Srinivas et al. , 2010 ) . Overview of main results To mitigate the collision effect , we propose a novel regularization scheme that can be applied as a simple plugin amendment for the latent space based Bayesian optimization models . The proposed algorithm , namely Latent Space Optimization via Collision-free regularization ( LOCO ) , leverages a regularized regression loss function to optimize the latent space for Bayesian optimization periodically . Concretely , our collision-free regularizer is encoded by a novel pairwise collision penalty function defined jointly on the latent space and the output domain . In order to mitigate the risk of collision in the latent space ( and consequently boost the optimization performance ) , LOCO applies the regularizer to minimize the collisions uniformly in the latent space . We further note that for Bayesian optimization tasks , collisions in regions close to the optimum are more likely to mislead the optimization algorithm . Based on this insight , we propose an optimization-aware regularization scheme that assigns higher weight to the collision penalty on those pairs of points closer to the optimum region in the latent space . This algorithm , which we refer to as Dynamically-Weighted LOCO , is designed to dynamically assess the importance of a collision during optimiza- tion . Compared with the uniform collision penalty in the latent space , the dynamic weighting mechanism has demonstrated drastic improvement over the state-of-the-art latent space based Bayesian optimization models . We summarize our key contributions as follows : I . We investigate latent space based Bayesian optimization , and expose the limitations of existing latent space optimization approaches due to the collision effect on the latent space ( Section 3 ) . II . We propose a novel regularization scheme as a simple plugin amendment for latent space based Bayesian optimization models . Our regularizer penalizes collisions in the latent space and effectively reduces the collision effect . Furthermore , we propose an optimization-aware dynamic weighting mechanism for adjusting the collision penalty to improve the effectiveness of regularization for Bayesian optimization ( Section 4 ) . III . We provide theoretical analysis for the performance of Bayesian optimization on regularized latent space ( Section 5 ) . IV . We conducted an extensive empirical study on several synthetic and real-world datasets , including a real-world case study for cosmic experimental design , and demonstrate the promising empirical performance for our algorithm ( Section 6 ) . 2 RELATED WORK . This section provides a short survey on recent work in Bayesian learning , which was designed to overcome the high-dimensionality challenge for Gaussian process regression tasks and Bayesian optimization . Different surrogate models with internal latent space Some alternative surrogate models have been proposed to replace classical GP in Bayesian optimization to overcome the challenge of highdimensional and highly-structured input in BO . Deep Network for Global Optimization ( DNGO ) Snoek et al . ( 2015 ) uses a pre-trained deep neural network with a Bayesian linear regressor at the last hidden layer of the network as the surrogate model . More generally , Deep Kernel Learning ( DKL ) combines the power of the Gaussian process and neural network by introducing a deep neural network g to learn a mapping g : X → Z from the input domain X to a latent space Z ( Wilson et al. , 2016 ) . It uses the latent representation z ∈ Z as the input of the base Gaussian process . The neural network g and a spectral mixture-based kernel k form a scalable expressive closed-form deep covariance kernel , denoted by kDK ( xi , xj ) → k ( g ( xi ) , g ( xj ) ) . The deep kernel allows end-to-end learning and Bayesian optimization on the original input space . Representation learning and latent space optimization Instead of reducing the dimensionality and performing optimization in an end-to-end process , other methods aim to optimize in a related low-dimensional space first and then map the solution back to the original input space . Djolonga et al . ( 2013 ) assume that only a subset of input dimensions varies , and the kernel is smooth ( i.e . with bounded RKHS norm ) . Under these assumptions , the underlying subspace is learned via low-rank matrix recovery . Random feature is another solution under this setting ( Rahimi et al. , 2007 ; Letham et al. , 2020 ; Binois et al. , 2015 ; Nayebi et al. , 2019 ; Wang et al. , 2016 ) . It is known that a random representation space of sufficiently large dimension is guaranteed to contain the optima with high probability . Mutnỳ and Krause ( 2019 ) consider Quadrature Fourier Features ( QFF ) —as opposed to Random Fourier Feature ( RFF ) in Rahimi et al . ( 2007 ) —to overcome the variance starvation problem , and proved that Thompson sampling and GP-UCB achieve no-regret with squared exponential kernel in optimization tasks . However , both RFF and QFF methods rely on a key assumption that the function to be optimized has a low effective dimension . In contrast , as discussed in Section 6 and the supplemental materials , we show that LOCO performs well for challenging high-dimensional BO problems where algorithms relying on the low effective dimension assumption may fail . Another line of work on latent space optimization uses autoencoders to learn low-dimensional representations of the inputs to improve the scalability and capability to leverage the structural information ( Mathieu et al. , 2019 ) , ( Ding et al. , 2020 ) , ( Gómez-Bombarelli et al. , 2018 ; Huang et al. , 2015 ; Tripp et al. , 2020 ; Lu et al. , 2018 ) . Mathieu et al . ( 2019 ) , Ding et al . ( 2020 ) focus on disentangled representation learning that breaks down , or disentangles , each feature into narrowly defined variables and encodes them as separate dimensions.Tripp et al . ( 2020 ) iteratively train the autoencoder with a dynamic weighting scheme when performing optimization to improve the embedding . Griffiths and Hernández-Lobato ( 2020 ) and Letham et al . ( 2020 ) enforce certain properties on the representation space to improve the optimization performance . To the best of the authors ’ knowledge , collision of the embeddings has not been explicitly studied . Binois et al . ( 2015 ) propose a warped kernel to guarantee the injectivity in the random linear embedding , which is not applicable in neural network-based methods . A common challenge in applying these techniques to generic optimization tasks lies in the assumption on the accessibility of training data : Bayesian optimization often assumes limited access to labeled data , while surrogate models built on deep neural networks often rely on abundant access to data for pretraining . Another problem lies in the training objective : During the training phase , these surrogate models typically focus on improving the regression performance , and do not explicitly address the artifact caused by collisions of the learned embeddings , which—as we later demonstrate in Section 3.3—could be harmful to sequential decision-making tasks .

Summary: Despite its success, Gaussian process based Bayesian optimization still struggles in high dimensional search spaces. Current approaches aim to learn an embedding to optimize the objective in a low dimensional continuous latent space. This paper provides evidence that with current approaches, different data points in the input space can be mapped to same point in the latent space. To avoid these collisions, the paper proposes a new regularization technique based on pairs of observed datapoints.