Fourier Representations for Black-Box Optimization over Categorical Variables

1 INTRODUCTION . A plethora of practical optimization problems involve black-box functions , with no simple analytical closed forms , that can be evaluated at any arbitrary point in the domain . Optimization of such black-box functions poses a unique challenge due to restrictions on the number of possible function evaluations , as evaluating functions of real-world complex processes is often expensive and time consuming . Efficient algorithms for global optimization of expensive black-box functions take past queries into account in order to select the next query to the black-box function more intelligently . While black-box optimization of real-world functions defined over integer , continuous , and mixed variables has been studied extensively in the literature , limited work has addressed incorporation of purely categorical type input variables . Categorical type variables are particularly challenging when compared to integer or continuous variables , as they do not have a natural ordering . However , many real-world functions are defined over categorical variables . One such problem , which is of wide interest , is the design of optimal chemical or biological ( protein , RNA , and DNA ) molecule sequences , which are constructed using a vocabulary of fixed size , e.g . 4 for DNA/RNA . Designing optimal molecular sequences with improved or novel structures and/or functionalities is of paramount importance in material science , drug and vaccine design , synthetic biology and many other applications ( see Dixon et al . ( 2010 ) ; Ng et al . ( 2019 ) ; Hoshika et al . ( 2019 ) ; Yamagami et al . ( 2019 ) ) . Design of optimal sequences is a difficult black-box optimization problem over a combinatorially large search space ( Stephens et al . ( 2015 ) ) , in which function evaluations often rely on either wet-lab experiments , physics-inspired simulators , or knowledge-based computational algorithms , which are slow and expensive in practice . Another problem of interest is the constrained design problem , e.g . find a sequence given a specific structure ( or property ) , which is inverse of the well-known folding problem discussed in Dill & MacCallum ( 2012 ) . This problem is complex due to the strict structural constraints imposed on the sequence . In fact one of the ways to represent such a complex structural constraint is to constrain the next choice sequentially based on the sequence elements that have been chosen a priori . Therefore , we divide the black box optimization problem into two settings , depending on the constraint set : ( i ) Generic Black Box Optimization ( BBO ) problem referring to the unconstrained case and ( ii ) Design Problem that refers to the case with complex sequential constraints . Let xt be the t-th sequence evaluated by the black box function f . The key question in both settings is the following : Given prior queries x1 , x2 . . . xt and their evaluations f ( x1 ) . . . f ( xt ) , how to choose the next query xt+1 ? This acquisition must be devised so that over a finite budget of black-box evaluations , one is closest to the minimizer in an expected sense over the acquisition randomness . In the literature , for design problems with sequential constraints , MCTS ( Monte Carlo Tree Search ) based acquisitions are often used with real function evaluations f ( xt ) . In the generic BBO problems in the unconstrained scenario , Simulated Annealing ( SA ) based techniques are typically used as acquisition functions . A key missing ingredient in the categorical domain is a surrogate model for the black-box evaluations that can interpolate between such evaluations and use cost-free approximate evaluations from the surrogate model internally ( in acquisition functions ) in order to reduce the need for frequently accessing real evaluations . This leads to improved sample efficiency in acquisition functions . Due to the lack of efficient interpolators in the categorical domains , existing acquisition functions suffer under a finite budget constraint , due to reliance on only real black-box evaluations . Contributions : We address the above problem in our work . Our main contributions are as follows : 1 . We present two representations for modeling real-valued combinatorial functions over categorical variables , which we then use in order to learn a surrogate model for the generic BBO problem and the design problem . The surrogate model is updated via a hedge algorithm where the basis functions in our representations act as experts . The latter update happens once for every real black-box evaluation . To the best of our knowledge , the representations and/or their use in black-box optimization of functions over categorical variables are novel to this work . 2 . In the BBO problem , the proposed method uses a version of simulated annealing that utilizes the current surrogate model for many internal cost-free evaluations before producing the next black-box query . 3 . In the design problem , the proposed method uses a version of MCTS in conjunction with the current surrogate model as reward function of the terminal states during intermediate tree traversals/backups in order to improve the sample efficiency of the search algorithm . 4 . Numerical results , over synthetic benchmarks as well as real-world biological ( RNA ) sequence optimization and design problems demonstrate the competitive or superior performance of the proposed methods over state-of-the-art counterparts , while substantially reducing the computation time and sample efficiency , respectively . 2 RELATED WORK . Hutter et al . ( 2011 ) suggests a surrogate model based on random forests to address optimization problems over categorical variables . The proposed SMAC algorithm uses a randomized local search under the expected improvement acquisition criterion in order to obtain candidate points for black-box evaluations . Bergstra et al . ( 2011 ) suggests a tree-structured Parzen estimator ( TPE ) for approximating the surrogate model , and maximizes the expected improvement criterion to find candidate points for evaluation . For optimization problems over Boolean variables , multilinear polynomials Ricardo Baptista ( 2018 ) ; Dadkhahi et al . ( 2020 ) and Walsh functions Leprêtre et al . ( 2019 ) have been used in the literature . Bayesian Optimization ( BO ) is a commonly used approach for optimization of black-box functions ( Shahriari et al . ( 2015 ) ) . However , limited work has addressed incorporation of categorical variables in BO . Early attempts based on converting the black-box optimization problem over categorical variables to that of continuous variables have not been very successful ( Gómez-Bombarelli et al . ( 2018 ) ; Golovin et al . ( 2017 ) ; Garrido-Merchán & Hernández-Lobato ( 2020 ) ) . A few BO algorithms have been specifically designed for black-box functions over combinatorial domains . In particular , the BOCS algorithm Ricardo Baptista ( 2018 ) , primarily devised for Boolean functions , employs a sparse monomial representation to model the interactions among different variables , and uses a sparse Bayesian linear regression method to learn the model coefficients . The COMBO algorithm of Oh et al . ( 2019 ) uses Graph Fourier Transform ( GFT ) over a combinatorial graph , constructed via graph cartesian product of variable subgraphs , to gauge the smoothness of the black-box function . However , both BOCS and COMBO are hindered by associated high computational complexities , which grow polynomially with both the number of variables and the number of function evaluations . More recently , a computationally efficient black-box optimization algorithm ( COMEX ) ( Dadkhahi et al . ( 2020 ) ) was introduced to address the computational impediments of its Bayesian counterparts . COMEX adopts a Boolean Fourier representation as its surrogate model , which is updated via an exponential weight update rule . Nevertheless , COMEX is limited to functions over the Boolean hypercube . We generalize COMEX to handle functions over categorical variables by proposing two representations for modeling functions over categorical variables : an abridged one-hot encoded Boolean Fourier representation and Fourier representation on finite Abelian groups . The utilization of the latter representation as a surrogate model in combinatorial optimization algorithms is novel to this work . Factorizations based on one-hot encoding has been previously ( albeit briefly ) suggested in Ricardo Baptista ( 2018 ) to enable black-box optimization algorithms designed for Boolean variables to address problems over categorical variables . Different from Ricardo Baptista ( 2018 ) , we show that we can significantly reduce the number of additional variables introduced upon one-hot encoding , and that such a reduced representation is in fact complete and unique . For design problems , we focus on the RNA sequence design problem ( RNA inverse folding ) . The goal is to find an RNA sequence consistent with a given secondary structure , as the functional state of the RNA molecule is determined by the latter structure ( Hofacker et al . ( 1994 ) ) . Earlier RNA design methods explore the search space by trial and error and use classic cost function minimization approaches such as adaptive random walk ( Hofacker ( 2003 ) ) , probabilistic sampling ( Zadeh et al . ( 2011 ) ) , and genetic algorithms ( Taneda ( 2015 ) ) . Recent efforts employ more advanced machine learning methods such as different Monte Carlo Tree Search ( MCTS ) algorithms , e.g . MCTS-RNA ( Yang et al . ( 2017 ) ) or Nested MCTS ( Portela ( 2018 ) ) , and reinforcement learning that either performs a local search as in Eastman et al . ( 2018 ) or learns complete candidate solutions from scratch ( Runge et al . ( 2018 ) ) . In all these approaches , the assumption is that the algorithm has access to a large number of function evaluations , whereas we are interested in sample efficiency of each algorithm . As an alternative to parameter free search methods ( such as SA ) , Swersky et al . ( 2020 ) suggests to use a parameterized policy to generate candidates that maximize the acquisition function in Bayesian optimization over discrete search spaces . Our MCTS acquisition method is similar in concept to Swersky et al . ( 2020 ) in the sense that the tabular value functions are constructed and maintained over different time steps . However , we are maintaining value functions rather than a policy network . 3 BLACK-BOX OPTIMIZATION OVER CATEGORICAL VARIABLES . Problem Setting : Given the combinatorial categorical domain X = [ k ] n and a constraint set C ⊆ X , with n variables each of cardinality k , the objective is to find x∗ = arg min x∈C f ( x ) ( 1 ) where f : X 7→ R is a real-valued combinatorial function . We assume that f is a black-box function , which is computationally expensive to evaluate . As such , we are interested in finding x∗ in as few evaluations as possible . We consider two variations of the problem depending on how the constraint set C is specified . Generic BBO Problem : In this case , the constraint set C = X . For example , RNA sequence optimization problem that searches for an RNA sequence with a specific property optimized lies within this category . A score for every RNA sequence , reflecting the property we wish to optimize , is evaluated by a black box function . Design Problem : The constraint set is complex and is only sequentially specified . For every sequence of x1x2 . . . xi consisting of i characters from the alphabet [ k ] , the choice of the next character xi+1 ∈ C ( x1x2 . . . xi ) ⊆ [ k ] is specified by a constraint set function C ( x1 . . . xi ) . The RNA inverse folding problem in Runge et al . ( 2018 ) falls into this category , where the constraints on the RNA sequence are determined by the sequential choices one makes during the sequence design . The goal is to find the sequence that is optimal with respect to a pre-specified structure that also obeys complex sequential constraints . Our Techniques : In order to address this problem , we adopt a surrogate model-acquisition function based learning framework , where an estimate for the black-box function f̂ ( i.e . the surrogate model ) is updated sequentially via black-box function evaluations observed until time step t. The selection of candidate points for black-box function evaluation is carried out via an acquisition function , which uses the surrogate model f̂ as an inexpensive proxy ( to make many internal calls ) for the black-box function and produces the next candidate point to be evaluated . The sequence proceeds as follows : Surrogate model updated on ( xt , f ( xt ) ) → Acquisition function makes ( many ) calls to Surrogate model to propose xt+1 → Surrogate model updated on ( xt+1 , f ( xt+1 ) ) In the sequel , we propose two representations that can be used as surrogate models for black-box combinatorial functions over categorical variables . These representations serve as direct generalizations of the Boolean surrogate model based on Fourier expansion proposed in Dadkhahi et al . ( 2020 ) in the sense that they reduce to the Fourier representation for real-valued Boolean functions when the cardinality of the categorical variables is two . In addition , both approaches can be modified to address the more general case where different variables are of different cardinalities . However , for ease of exposition , we assume that all the variables are of the same cardinality . Finally , we introduce two popular acquisition function to be used in conjunction with the proposed surrogate models in order to propose new queries for subsequent black-box function evaluations .

Summary: The paper considers the problem of black-box optimization of expensive functions defined over categorical variables. A surrogate model-based optimization approach is proposed to tackle this problem. Fourier representations are proposed as surrogate model by treating the categorical input as the direct sum of cyclic groups Z/kZ (k is the arity/category size). The coefficients of this representation are learned via exponentially-weighted update rule. The selection of each subsequent input for evaluation is performed via direct optimization of the surrogate model built over inputs collected previously. Simulated Annealing and Monte Carlo Tree Search is proposed as the acquisition function optimization procedure for unconstrained and constrained problems respectively. Experiments are performed on two synthetic problems and RNA-sequence optimization.