Learning Representation for Bayesian Optimization with Collision-free Regularization

Bayesian Optimization has been challenged by the large-scale and highdimensional datasets , which are common in real-world scenarios . Recent works attempt to handle such input by applying neural networks ahead of the classical Gaussian process to learn a ( low-dimensional ) latent representation . We show that even with proper network design , such learned representation often leads to collision in the latent space : two points with significantly different observations collide in the learned latent space , leading to degraded optimization performance . To address this issue , we propose LOCO , an efficient deep Bayesian optimization framework which employs a novel regularizer to reduce the collision in the learned latent space and encourage the mapping from the latent space to the objective value to be Lipschitz continuous . LOCO takes in pairs of data points and penalizes those too close in the latent space compared to their target space distance . We provide a rigorous theoretical justification for LOCO by inspecting the regret of this dynamic-embedding-based Bayesian optimization algorithm , where the neural network is iteratively retrained with the regularizer . Our empirical results further demonstrate the effectiveness of LOCO on several synthetic and realworld benchmark Bayesian optimization tasks . 1 INTRODUCTION . Bayesian optimization is a classical sequential optimization method and is widely used in various fields in science and engineering , including recommender systems ( Galuzzi et al. , 2019 ) , medical trials ( Sui et al. , 2018 ) , robotic controller optimization ( Berkenkamp et al. , 2016 ) , scientific experimental design ( Yang et al. , 2019 ) , and hyper-parameter tuning ( Snoek et al. , 2012 ) , among many others . Many of these applications involve evaluating an expensive blackbox function ; therefore , the number of queries should be minimized . A common way to model the unknown function is via Gaussian processes ( GPs ) ( Rasmussen and Williams , 2006 ) , which have been extensively studied under the bandit setting , as an effective surrogate model of the unknown objective function in a broad class of blackbox function optimization problems ( Srinivas et al. , 2010 ; Djolonga et al. , 2013 ) . A key computational challenge for learning with GPs lies in optimizing specific kernels used for modeling the covariance structures . Such an optimization task depends on the dimension of the input space . For high-dimensional data , it is often prohibitive to train a GP model . Meanwhile , local kernel machines are known to suffer from the curse of dimensionality ( Bengio et al. , 2005 ) , while the required number of training samples could grow exponentially with the dimensionality of the data . Therefore , dimensionality reduction and representation learning algorithms are needed to optimize the learning process . Recently , Gaussian process optimization has been investigated in the context of latent space models . For example , deep kernel learning ( Wilson et al. , 2016 ) learns a ( low-dimensional ) data representation and a scalable kernel simultaneously via an end-to-end trainable deep neural network . In general , the neural network is trained to learn a simpler latent representation with reduced dimension and has the structure information already embedded for the GP . Combining the representation learned via a neural network with GP could improve the scalability and extensibility of classical Bayesian optimization , but it also poses new challenges for the optimization task , such as dealing with the tradeoff between representation learning and function optimization ( Tripp et al. , 2020 ) . As we later demonstrate , a critical challenge brought by introducing representation learning into Bayesian optimization is that the latent representation is prone to collisions : two points with significantly different observations can get too close , and therefore collide in the latent space . The collision effect in latent space models for Bayesian optimization is especially evident when information is lost during dimensionality reduction and/or when the training data is limited in size . As illustrated in Figure 1 , when passed through the neural network , data points with drastically different observations are mapped to close positions in the latent space . Such collisions could be regarded as additional noise introduced by the neural network . Although Bayesian optimization is known to be robust to mild noisy observations ( Bogunovic et al. , 2018 ) , the collision in latent space could be harmful to the optimization performance , as it is non-trivial to model the collision into the acquisition function explicitly . Also , the additional noise induced by the collision effect will further loosen the regret bound for classical Bayesian optimization algorithms ( Srinivas et al. , 2010 ) . Overview of main results To mitigate the collision effect , we propose a novel regularization scheme that can be applied as a simple plugin amendment for the latent space based Bayesian optimization models . The proposed algorithm , namely Latent Space Optimization via Collision-free regularization ( LOCO ) , leverages a regularized regression loss function to optimize the latent space for Bayesian optimization periodically . Concretely , our collision-free regularizer is encoded by a novel pairwise collision penalty function defined jointly on the latent space and the output domain . In order to mitigate the risk of collision in the latent space ( and consequently boost the optimization performance ) , LOCO applies the regularizer to minimize the collisions uniformly in the latent space . We further note that for Bayesian optimization tasks , collisions in regions close to the optimum are more likely to mislead the optimization algorithm . Based on this insight , we propose an optimization-aware regularization scheme that assigns higher weight to the collision penalty on those pairs of points closer to the optimum region in the latent space . This algorithm , which we refer to as Dynamically-Weighted LOCO , is designed to dynamically assess the importance of a collision during optimiza- tion . Compared with the uniform collision penalty in the latent space , the dynamic weighting mechanism has demonstrated drastic improvement over the state-of-the-art latent space based Bayesian optimization models . We summarize our key contributions as follows : I . We investigate latent space based Bayesian optimization , and expose the limitations of existing latent space optimization approaches due to the collision effect on the latent space ( Section 3 ) . II . We propose a novel regularization scheme as a simple plugin amendment for latent space based Bayesian optimization models . Our regularizer penalizes collisions in the latent space and effectively reduces the collision effect . Furthermore , we propose an optimization-aware dynamic weighting mechanism for adjusting the collision penalty to improve the effectiveness of regularization for Bayesian optimization ( Section 4 ) . III . We provide theoretical analysis for the performance of Bayesian optimization on regularized latent space ( Section 5 ) . IV . We conducted an extensive empirical study on several synthetic and real-world datasets , including a real-world case study for cosmic experimental design , and demonstrate the promising empirical performance for our algorithm ( Section 6 ) . 2 RELATED WORK . This section provides a short survey on recent work in Bayesian learning , which was designed to overcome the high-dimensionality challenge for Gaussian process regression tasks and Bayesian optimization . Different surrogate models with internal latent space Some alternative surrogate models have been proposed to replace classical GP in Bayesian optimization to overcome the challenge of highdimensional and highly-structured input in BO . Deep Network for Global Optimization ( DNGO ) Snoek et al . ( 2015 ) uses a pre-trained deep neural network with a Bayesian linear regressor at the last hidden layer of the network as the surrogate model . More generally , Deep Kernel Learning ( DKL ) combines the power of the Gaussian process and neural network by introducing a deep neural network g to learn a mapping g : X → Z from the input domain X to a latent space Z ( Wilson et al. , 2016 ) . It uses the latent representation z ∈ Z as the input of the base Gaussian process . The neural network g and a spectral mixture-based kernel k form a scalable expressive closed-form deep covariance kernel , denoted by kDK ( xi , xj ) → k ( g ( xi ) , g ( xj ) ) . The deep kernel allows end-to-end learning and Bayesian optimization on the original input space . Representation learning and latent space optimization Instead of reducing the dimensionality and performing optimization in an end-to-end process , other methods aim to optimize in a related low-dimensional space first and then map the solution back to the original input space . Djolonga et al . ( 2013 ) assume that only a subset of input dimensions varies , and the kernel is smooth ( i.e . with bounded RKHS norm ) . Under these assumptions , the underlying subspace is learned via low-rank matrix recovery . Random feature is another solution under this setting ( Rahimi et al. , 2007 ; Letham et al. , 2020 ; Binois et al. , 2015 ; Nayebi et al. , 2019 ; Wang et al. , 2016 ) . It is known that a random representation space of sufficiently large dimension is guaranteed to contain the optima with high probability . Mutnỳ and Krause ( 2019 ) consider Quadrature Fourier Features ( QFF ) —as opposed to Random Fourier Feature ( RFF ) in Rahimi et al . ( 2007 ) —to overcome the variance starvation problem , and proved that Thompson sampling and GP-UCB achieve no-regret with squared exponential kernel in optimization tasks . However , both RFF and QFF methods rely on a key assumption that the function to be optimized has a low effective dimension . In contrast , as discussed in Section 6 and the supplemental materials , we show that LOCO performs well for challenging high-dimensional BO problems where algorithms relying on the low effective dimension assumption may fail . Another line of work on latent space optimization uses autoencoders to learn low-dimensional representations of the inputs to improve the scalability and capability to leverage the structural information ( Mathieu et al. , 2019 ) , ( Ding et al. , 2020 ) , ( Gómez-Bombarelli et al. , 2018 ; Huang et al. , 2015 ; Tripp et al. , 2020 ; Lu et al. , 2018 ) . Mathieu et al . ( 2019 ) , Ding et al . ( 2020 ) focus on disentangled representation learning that breaks down , or disentangles , each feature into narrowly defined variables and encodes them as separate dimensions.Tripp et al . ( 2020 ) iteratively train the autoencoder with a dynamic weighting scheme when performing optimization to improve the embedding . Griffiths and Hernández-Lobato ( 2020 ) and Letham et al . ( 2020 ) enforce certain properties on the representation space to improve the optimization performance . To the best of the authors ’ knowledge , collision of the embeddings has not been explicitly studied . Binois et al . ( 2015 ) propose a warped kernel to guarantee the injectivity in the random linear embedding , which is not applicable in neural network-based methods . A common challenge in applying these techniques to generic optimization tasks lies in the assumption on the accessibility of training data : Bayesian optimization often assumes limited access to labeled data , while surrogate models built on deep neural networks often rely on abundant access to data for pretraining . Another problem lies in the training objective : During the training phase , these surrogate models typically focus on improving the regression performance , and do not explicitly address the artifact caused by collisions of the learned embeddings , which—as we later demonstrate in Section 3.3—could be harmful to sequential decision-making tasks .

Summary: This paper is concerned with latent space Bayesian optimisation that typically involves a step of learning a lower-dimensional latent representation. The authors focus on non-linear embeddings generated through neural networks. They observe a collision problem in the latent space and attempt to resolve it by introducing a regulariser based on Lipschitz continuity. In a set of experiments, they demonstrate that such a method is effective in various benchmarks. Although interesting, I still find this paper lacking as presented in the next section.