Fourier Representations for Black-Box Optimization over Categorical Variables

1 INTRODUCTION . A plethora of practical optimization problems involve black-box functions , with no simple analytical closed forms , that can be evaluated at any arbitrary point in the domain . Optimization of such black-box functions poses a unique challenge due to restrictions on the number of possible function evaluations , as evaluating functions of real-world complex processes is often expensive and time consuming . Efficient algorithms for global optimization of expensive black-box functions take past queries into account in order to select the next query to the black-box function more intelligently . While black-box optimization of real-world functions defined over integer , continuous , and mixed variables has been studied extensively in the literature , limited work has addressed incorporation of purely categorical type input variables . Categorical type variables are particularly challenging when compared to integer or continuous variables , as they do not have a natural ordering . However , many real-world functions are defined over categorical variables . One such problem , which is of wide interest , is the design of optimal chemical or biological ( protein , RNA , and DNA ) molecule sequences , which are constructed using a vocabulary of fixed size , e.g . 4 for DNA/RNA . Designing optimal molecular sequences with improved or novel structures and/or functionalities is of paramount importance in material science , drug and vaccine design , synthetic biology and many other applications ( see Dixon et al . ( 2010 ) ; Ng et al . ( 2019 ) ; Hoshika et al . ( 2019 ) ; Yamagami et al . ( 2019 ) ) . Design of optimal sequences is a difficult black-box optimization problem over a combinatorially large search space ( Stephens et al . ( 2015 ) ) , in which function evaluations often rely on either wet-lab experiments , physics-inspired simulators , or knowledge-based computational algorithms , which are slow and expensive in practice . Another problem of interest is the constrained design problem , e.g . find a sequence given a specific structure ( or property ) , which is inverse of the well-known folding problem discussed in Dill & MacCallum ( 2012 ) . This problem is complex due to the strict structural constraints imposed on the sequence . In fact one of the ways to represent such a complex structural constraint is to constrain the next choice sequentially based on the sequence elements that have been chosen a priori . Therefore , we divide the black box optimization problem into two settings , depending on the constraint set : ( i ) Generic Black Box Optimization ( BBO ) problem referring to the unconstrained case and ( ii ) Design Problem that refers to the case with complex sequential constraints . Let xt be the t-th sequence evaluated by the black box function f . The key question in both settings is the following : Given prior queries x1 , x2 . . . xt and their evaluations f ( x1 ) . . . f ( xt ) , how to choose the next query xt+1 ? This acquisition must be devised so that over a finite budget of black-box evaluations , one is closest to the minimizer in an expected sense over the acquisition randomness . In the literature , for design problems with sequential constraints , MCTS ( Monte Carlo Tree Search ) based acquisitions are often used with real function evaluations f ( xt ) . In the generic BBO problems in the unconstrained scenario , Simulated Annealing ( SA ) based techniques are typically used as acquisition functions . A key missing ingredient in the categorical domain is a surrogate model for the black-box evaluations that can interpolate between such evaluations and use cost-free approximate evaluations from the surrogate model internally ( in acquisition functions ) in order to reduce the need for frequently accessing real evaluations . This leads to improved sample efficiency in acquisition functions . Due to the lack of efficient interpolators in the categorical domains , existing acquisition functions suffer under a finite budget constraint , due to reliance on only real black-box evaluations . Contributions : We address the above problem in our work . Our main contributions are as follows : 1 . We present two representations for modeling real-valued combinatorial functions over categorical variables , which we then use in order to learn a surrogate model for the generic BBO problem and the design problem . The surrogate model is updated via a hedge algorithm where the basis functions in our representations act as experts . The latter update happens once for every real black-box evaluation . To the best of our knowledge , the representations and/or their use in black-box optimization of functions over categorical variables are novel to this work . 2 . In the BBO problem , the proposed method uses a version of simulated annealing that utilizes the current surrogate model for many internal cost-free evaluations before producing the next black-box query . 3 . In the design problem , the proposed method uses a version of MCTS in conjunction with the current surrogate model as reward function of the terminal states during intermediate tree traversals/backups in order to improve the sample efficiency of the search algorithm . 4 . Numerical results , over synthetic benchmarks as well as real-world biological ( RNA ) sequence optimization and design problems demonstrate the competitive or superior performance of the proposed methods over state-of-the-art counterparts , while substantially reducing the computation time and sample efficiency , respectively . 2 RELATED WORK . Hutter et al . ( 2011 ) suggests a surrogate model based on random forests to address optimization problems over categorical variables . The proposed SMAC algorithm uses a randomized local search under the expected improvement acquisition criterion in order to obtain candidate points for black-box evaluations . Bergstra et al . ( 2011 ) suggests a tree-structured Parzen estimator ( TPE ) for approximating the surrogate model , and maximizes the expected improvement criterion to find candidate points for evaluation . For optimization problems over Boolean variables , multilinear polynomials Ricardo Baptista ( 2018 ) ; Dadkhahi et al . ( 2020 ) and Walsh functions Leprêtre et al . ( 2019 ) have been used in the literature . Bayesian Optimization ( BO ) is a commonly used approach for optimization of black-box functions ( Shahriari et al . ( 2015 ) ) . However , limited work has addressed incorporation of categorical variables in BO . Early attempts based on converting the black-box optimization problem over categorical variables to that of continuous variables have not been very successful ( Gómez-Bombarelli et al . ( 2018 ) ; Golovin et al . ( 2017 ) ; Garrido-Merchán & Hernández-Lobato ( 2020 ) ) . A few BO algorithms have been specifically designed for black-box functions over combinatorial domains . In particular , the BOCS algorithm Ricardo Baptista ( 2018 ) , primarily devised for Boolean functions , employs a sparse monomial representation to model the interactions among different variables , and uses a sparse Bayesian linear regression method to learn the model coefficients . The COMBO algorithm of Oh et al . ( 2019 ) uses Graph Fourier Transform ( GFT ) over a combinatorial graph , constructed via graph cartesian product of variable subgraphs , to gauge the smoothness of the black-box function . However , both BOCS and COMBO are hindered by associated high computational complexities , which grow polynomially with both the number of variables and the number of function evaluations . More recently , a computationally efficient black-box optimization algorithm ( COMEX ) ( Dadkhahi et al . ( 2020 ) ) was introduced to address the computational impediments of its Bayesian counterparts . COMEX adopts a Boolean Fourier representation as its surrogate model , which is updated via an exponential weight update rule . Nevertheless , COMEX is limited to functions over the Boolean hypercube . We generalize COMEX to handle functions over categorical variables by proposing two representations for modeling functions over categorical variables : an abridged one-hot encoded Boolean Fourier representation and Fourier representation on finite Abelian groups . The utilization of the latter representation as a surrogate model in combinatorial optimization algorithms is novel to this work . Factorizations based on one-hot encoding has been previously ( albeit briefly ) suggested in Ricardo Baptista ( 2018 ) to enable black-box optimization algorithms designed for Boolean variables to address problems over categorical variables . Different from Ricardo Baptista ( 2018 ) , we show that we can significantly reduce the number of additional variables introduced upon one-hot encoding , and that such a reduced representation is in fact complete and unique . For design problems , we focus on the RNA sequence design problem ( RNA inverse folding ) . The goal is to find an RNA sequence consistent with a given secondary structure , as the functional state of the RNA molecule is determined by the latter structure ( Hofacker et al . ( 1994 ) ) . Earlier RNA design methods explore the search space by trial and error and use classic cost function minimization approaches such as adaptive random walk ( Hofacker ( 2003 ) ) , probabilistic sampling ( Zadeh et al . ( 2011 ) ) , and genetic algorithms ( Taneda ( 2015 ) ) . Recent efforts employ more advanced machine learning methods such as different Monte Carlo Tree Search ( MCTS ) algorithms , e.g . MCTS-RNA ( Yang et al . ( 2017 ) ) or Nested MCTS ( Portela ( 2018 ) ) , and reinforcement learning that either performs a local search as in Eastman et al . ( 2018 ) or learns complete candidate solutions from scratch ( Runge et al . ( 2018 ) ) . In all these approaches , the assumption is that the algorithm has access to a large number of function evaluations , whereas we are interested in sample efficiency of each algorithm . As an alternative to parameter free search methods ( such as SA ) , Swersky et al . ( 2020 ) suggests to use a parameterized policy to generate candidates that maximize the acquisition function in Bayesian optimization over discrete search spaces . Our MCTS acquisition method is similar in concept to Swersky et al . ( 2020 ) in the sense that the tabular value functions are constructed and maintained over different time steps . However , we are maintaining value functions rather than a policy network . 3 BLACK-BOX OPTIMIZATION OVER CATEGORICAL VARIABLES . Problem Setting : Given the combinatorial categorical domain X = [ k ] n and a constraint set C ⊆ X , with n variables each of cardinality k , the objective is to find x∗ = arg min x∈C f ( x ) ( 1 ) where f : X 7→ R is a real-valued combinatorial function . We assume that f is a black-box function , which is computationally expensive to evaluate . As such , we are interested in finding x∗ in as few evaluations as possible . We consider two variations of the problem depending on how the constraint set C is specified . Generic BBO Problem : In this case , the constraint set C = X . For example , RNA sequence optimization problem that searches for an RNA sequence with a specific property optimized lies within this category . A score for every RNA sequence , reflecting the property we wish to optimize , is evaluated by a black box function . Design Problem : The constraint set is complex and is only sequentially specified . For every sequence of x1x2 . . . xi consisting of i characters from the alphabet [ k ] , the choice of the next character xi+1 ∈ C ( x1x2 . . . xi ) ⊆ [ k ] is specified by a constraint set function C ( x1 . . . xi ) . The RNA inverse folding problem in Runge et al . ( 2018 ) falls into this category , where the constraints on the RNA sequence are determined by the sequential choices one makes during the sequence design . The goal is to find the sequence that is optimal with respect to a pre-specified structure that also obeys complex sequential constraints . Our Techniques : In order to address this problem , we adopt a surrogate model-acquisition function based learning framework , where an estimate for the black-box function f̂ ( i.e . the surrogate model ) is updated sequentially via black-box function evaluations observed until time step t. The selection of candidate points for black-box function evaluation is carried out via an acquisition function , which uses the surrogate model f̂ as an inexpensive proxy ( to make many internal calls ) for the black-box function and produces the next candidate point to be evaluated . The sequence proceeds as follows : Surrogate model updated on ( xt , f ( xt ) ) → Acquisition function makes ( many ) calls to Surrogate model to propose xt+1 → Surrogate model updated on ( xt+1 , f ( xt+1 ) ) In the sequel , we propose two representations that can be used as surrogate models for black-box combinatorial functions over categorical variables . These representations serve as direct generalizations of the Boolean surrogate model based on Fourier expansion proposed in Dadkhahi et al . ( 2020 ) in the sense that they reduce to the Fourier representation for real-valued Boolean functions when the cardinality of the categorical variables is two . In addition , both approaches can be modified to address the more general case where different variables are of different cardinalities . However , for ease of exposition , we assume that all the variables are of the same cardinality . Finally , we introduce two popular acquisition function to be used in conjunction with the proposed surrogate models in order to propose new queries for subsequent black-box function evaluations .

Summary: The paper proposes two representations, namely one-hot encoded Boolean expansion and group-theoretical Fourier expansion, for the surrogate model used for the black-box evaluations on purely categorical variables. With the two surrogate models, the authors tackle both the black-box optimization problem and the design problem. Two forms of acquisition functions are applied for query selection – simulated annealing and Monte Carlo Tree Search. The new algorithms are compared with the existing methods in simulations and have advantage for the objective value and speed-up.