Unifying Likelihood-free Inference with Black-box Sequence Design and Beyond

1 INTRODUCTION . Discovering new drugs to fulfill specific criteria , such as binding affinity towards a given molecular target , is a fundamental problem in chemistry and the pharmaceutical industry ( Hughes et al. , 2011 ) . In this work , we focus on an important subdomain : de novo biological sequence design . This task is challenging for two reasons : ( 1 ) the exploration space for sequences is combinatorially large ; and ( 2 ) sequence usefulness is evaluated via a complicated process which usually involves time-consuming and expensive wet-lab experiments . Despite the difficulty of this task , many approaches have been developed over the past few decades thanks to recent advances in biochemistry and machine learning . The Nobel Prize wining paradigm , directed evolution ( Chen & Arnold , 1991 ) , which conducts local evolutionary search under human guidance , is one of the popular techniques . Unfortunately , it is limited by its sample inefficiency and reliance on strong prior knowledge , e.g. , about where to mutate ( Ahn et al. , 2020 ) . Furthermore , to compete with other machine learning methods ( Gottipati et al. , 2020 ) , guided evolution ( Yoshikawa et al. , 2018 ; Jensen , 2019 ; Nigam et al. , 2019 ) heavily relies on human intuition for designing domain-specific evolutionary operators , which may not always apply to tasks at hand . In this work , we deem sequence design to be a black-box optimization problem , tasked with maximizing an unknown oracle function . We assume that oracle queries are limited due to the constraint on resources , such as the budgets for evaluating queries in a wet-lab . Thus , sample efficiency is crucial . We develop a probabilistic framework by reformulating the aforementioned black-box optimization target as a posterior modeling problem . With this framework , we draw a surprising connection between likelihood-free inference and sequence design , and thus linking two fields which are previously considered as unrelated . The key observation we leverage here for establishing this connection is that both settings share similar elements and targets which will be elaborated in Section 2.2 . This connection facilitates our understanding of both fields and provides a recipe for developing sequence design algorithms . Going beyond , we also combine different probabilistic modeling insights and develop three novel composite probabilistic algorithms . We point out that our framework could actually be applied to any black-box optimization settings , but in this work we focus on its application to biological sequence design . To demonstrate the empirical effectiveness of our methods , we conduct systematical experiments to evaluate their performance on four in-silico sequence design benchmarks . Our proposed meth- ∗Corresponding Author ods achieve at least comparable results to existing baselines , and the proposed composite methods behave consistently better than all other ones across various sequence design tasks . We summarize our contribution as follows : • We develop a probabilistic framework that unifies likelihood-free inference and black-box optimization . • Based on this framework , we provide a recipe for designing algorithms for black-box problems . We apply these ideas to propose a series of composite design algorithms . • We perform systematical evaluation on a series of black-box sequence design benchmarks , and find that these algorithms achieves consistently comparable or better results compared to previous ones , thus illustrating the benefit of the proposed unified framework . 2 A UNIFYING PROBABILISTIC FRAMEWORK . 2.1 BACKGROUND . Likelihood-free inference ( LFI ) . We use θ ∈ Θ and x ∈ X to separately denote the parameters and the data generated via the mechanism x ∼ p ( x|θ ) . In this scenario , LFI refers to a special kind of Bayesian inference setting where the likelihood function is not tractable but sampling ( by simulation ) from the likelihood is feasible . Consider the objective of modeling the Bayesian posterior when we can not compute the likelihood p ( xo|θ ) : p ( θ|xo ) ∝ p ( θ ) p ( xo|θ ) ︸ ︷︷ ︸ ? , ( 1 ) where xo is the observed data , p ( θ ) is the ( given ) prior over the model parameters θ , p ( x|θ ) is the intractable likelihood function and p ( θ|x ) is the desired posterior over θ . While we do not have access to the exact likelihood , we can still simulate ( sample ) data x from the model simulator : x ∼ p ( x|θ ) . Instead of trying to obtain a numerical value of the generic posterior p ( θ|x ) for arbitrary x , LFI only tries to obtain an approximation of p ( θ|xo ) for the given xo . During the inference process , we can take advantage of the sampled data : D = { ( θi , xi ) } ni=1 where xi ∼ p ( x|θi ) for selected values of θi . Biological black-box sequence design . We consider biological sequence design as a black-box optimization problem : m∗ = arg max m∈M f ( m ) , where f ( · ) is the oracle score function , and we would like to discover values of m for which f ( m ) is large . In real-world situations , a query of this oracle f could represent a series of wet-lab experiments to measure specific chemical properties or specificity for a given binding site target . In general , these experiments are time- and cost-consuming . As a result , the total number of queries is limited . In our setting , we useM = VL to denote the search space for sequences with fixed length L , where V is the vocabulary for each entry of the sequence : for DNA nucleotides |V| = 4 , and for protein amino acids |V| = 20 . For variable length setting , we haveM = ∪L∈ [ Lmin , Lmax ] VL , where Lmin and Lmax are the minimal and maximal length , respectively . 2.2 CONNECTING LFI AND BLACK-BOX OPTIMIZATION . In order to draw a connection to LFI , we require a probabilistic formulation of the black-box sequence design problem . To this end , we relax the goal of searching for a single maximum of the oracle / score function f to a posterior modeling problem , i.e. , finding a representative sample of the configurations of m sampled with probability related to some target posterior . Think of C is the set of sequences with these desirable configurations , E is a Boolean event about whether a sequence m belongs to C , and our goal is to characterize the posterior distribution p ( m|E ) from which we obtain the desired sequences . Below , we consider two specific ways of doing this : Example A . We explicitly define C ( and E accordingly ) as all the sequences whose scores are larger than a given threshold s : C = { m|f ( m ) ≥ s } . ( 2 ) Here s could be any fixed value , or a certain quantile of a particular score distribution . In this way , we have p ( E|m ) = p ( m ∈ C|m ) = 1 { f ( m ) ≥ s } where 1 { } is the indicator function . Example B . In a softer version of E and C , we can define its conditional probability of being true to follow a Boltzmann distribution : p ( E|m ) = p ( m ∈ C|m ) ∝ exp ( f ( m ) /τ ) . ( 3 ) where τ is a temperature parameter . We introduce the exponential because f ( · ) does not necessarily take positive values . Any monotone transformation of f ( · ) to non-negative reals could be used , so that sequences with larger oracle scores have a greater probability of making E true . With this posterior objective , our goal now becomes effectively modeling and sampling from the posterior p ( m|E ) . It is thus natural to resort to the tools of Bayesian inference for this task . In order to examine this possibility , we draw a detailed comparison between the settings of black-box sequence design problem and likelihood-free Bayesian inference in Table 1 . It can be observed that both tasks share similar elements and targets . The two settings also share similar limitations on the allowed queries , which are too time-consuming and / or cost-intensive . Notice that in sequence design , the oracle could be either exact or noisy , thus we use the more general s ∼ f ( m ) formulation rather than s = f ( m ) . We will further present several concrete examples as demonstrations of this correspondence in the following section . Another way to understand this correspondence is to consider the following mapping T : T : Θ×X →M× R ( θ , x ) 7→ ( m , s ) , s.t . s = −‖x− xo‖ . Here we can see the score value s as a quantitative metric for how close the generated data x ( given θ ) is to the target observed data xo . In addition , querying the oracle in the sequence design setting can also be thought of as follows : ( 1 ) sample x ∼ p ( ·|θ ) and then ( 2 ) calculate s = −‖x− xo‖ under some distance ‖·‖ . In this manner , T could conceptually transform any LFI problem into a black-box optimization task . In this work , we only focus on the application of sequence design . 3 METHODOLOGY . We provide a recipe for designing new sequence design algorithms based on the correspondence in Section 2.2 . The recipe induces different approaches by modeling different probabilistic component of the Bayesian inference problem . We begin with common algorithm restrictions under this setting . Common constraint for algorithms . Due to the restriction of simulation / query in our setting , we constrain our algorithms to act in a sequential / iterative way , gradually achieving the desired posterior round by round . Every algorithm starts with an empty dataset D = ∅ and an initial proposal p1 ( · ) = p ( · ) , where p ( · ) is the prior given by the task . In the r-th round of this multi-round setting , the algorithm would use the proposal pr ( · ) of this round to sample a batch of data ( θ / m ) for simulation / query , and augment the current datasetD with the newly obtained batch of data . We use n to denote the batch size for each round ’ s simulation / query . Afterwards , the algorithm updates the proposal to pr+1 ( · ) . The outcomes for the two settings we discuss may be slightly different : an algorithm for likelihood-free inference would return the posterior , while a sequence design method would return the dataset of all the sequences it has queried , which hopefully contains desired high scored sequences . On the other hand , a sequence design method could produce as an intermediate result a generative model for sampling queries , which then completely fits with the LFI framework . 3.1 BACKWARD MODELING OF THE MECHANISM . Approximate Bayesian Computation ( ABC ) ( Beaumont et al. , 2002 ) is a standard method for tackling LFI problems . In Algorithm 1 , we display one of the most popular variants : Sequential Monte Carlo-Approximate Bayesian Computation ( SMC-ABC ) ( Beaumont et al. , 2009 ) . In each round , parameters θ are sampled from the current proposal distribution pr ( θ ) for simulation . A rejection step is then involved to remove the θi whose simulation outcomes xi can not reproduce the observed data xo with sufficient accuracy . The remaining accepted { θi } i are adopted to update the next round ’ s proposal pr+1 ( · ) towards the target posterior , i.e. , by refitting qφ with the modified data . We defer more details of this approach to Section A.1 in Appendix . Algorithm 1 SMC-ABC p1 ( θ ) ← p ( θ ) ; for r in 1 to R do repeat sample θi ∼ pr ( θ ) ; simulate xi ∼ p ( x|θi ) ; until n samples are obtained D ← D ∪ { ( θi , xi ) } ni=1 sort D according to −‖xi − xo‖ ; fit qφ ( θ ) with top { θi } i in D ; pr+1 ( θ ) ← qφ ( θ ) ; end for return p̂ ( θ|xo ) = pR+1 ( θ ) Algorithm 2 FB-VAE p1 ( m ) ← p ( m ) ; for r in 1 to R do repeat sample mi ∼ pr ( m ) ; query the oracle : si ← f ( mi ) ; until n samples are obtained D ← D ∪ { ( mi , si ) } ni=1 ; sort D according to si fit qφ ( m ) with top { mi } i in D ; pr+1 ( m ) ← qφ ( m ) ; end for return { m : ( m , s ) ∈ D } It would then be natural to construct an analogical sequence design algorithm using top scored entities { mi } i to guide the update of a certain sequence distribution , see Algorithm 2 . Interestingly , this is the proposed sequence design algorithm in Gupta & Zou ( 2019 ) , where the authors name this kind of updating “ feedback ” because training of the parametric generator qφ ( m ) exploits feedback signals from the oracle . In this paper , we follow Brookes & Listgarten ( 2018 ) to crystallize qφ ( m ) to be a variational autoencoder ( Kingma & Welling , 2014 ) , and use the term Feedback-Variational AutoEncoder ( FB-VAE ) to refer to Algorithm 2 . We place Algorithm 1 & 2 side-by-side to highlight their correspondence . We also make the same arrangement for the following Algorithm 3 & 4 , Algorithm 5 & 6 and Algorithm 7 & 8 . Algorithm 3 Sequential Neural Posterior p1 ( θ ) ← p ( θ ) ; for r in 1 to R do repeat sample θi ∼ pr ( θ ) ; simulate xi ∼ p ( x|θi ) ; until n samples are obtained D ← D ∪ { ( θi , xi ) } ni=1 ; qφ ← arg minq Ex [ DKL ( p ( θ|x ) ‖q ) ] ; pr+1 ( θ ) ← qφ ( θ|xo ) ; end for return p̂ ( θ|xo ) = pR+1 ( θ ) Algorithm 4 Design by Adaptive Sampling p1 ( m ) ← p ( m ) ; for r in 1 to R do repeat sample mi ∼ pr ( m ) ; query the oracle : si ← f ( mi ) ; until n samples are obtained D ← D ∪ { ( mi , si ) } ni=1 ; qφ ← arg minqDKL ( p ( m|E ) ‖q ) ; pr+1 ( m ) ← qφ ( m ) ; end for return { m : ( m , s ) ∈ D } In comparison with SMC-ABC , the Sequential Neural Posterior ( SNP ) method ( Papamakarios & Murray , 2016 ; Lueckmann et al. , 2017 ; Greenberg et al. , 2019 ) for likelihood-free inference adopts a more flexible approach , taking the power of conditional neural density estimator ( e.g. , Papamakarios et al . ( 2017 ) ) to model the general posterior p ( θ|x ) , which takes arbitrary θ and x as two inputs and outputs a distribution . This neural estimator is trained via approximately minimizing the Kullback–Leibler ( KL ) divergence between qφ ( θ|x ) and the true posterior p ( θ|x ) . We defer more training details to Section A.1 in Appendix . Under the connection viewpoint , one similar algorithm for sequence design is the Design by Adaptive Sampling ( DbAS ) proposed in Brookes & Listgarten ( 2018 ) which is characterized in Algorithm 4 , fitting qφ ( m ) through minimizing the KL divergence with the posterior p ( m|E ) . Based on the difference in specific implementations , both algorithms have more than one variant , whose details are deferred to Section A.1 in Appendix . We refer to the above algorithms as “ backward modeling ” because the trained generative network qφ ( going from x/E to θ/m ) is a sort of reverse model of the simulation mechanism ( which goes from θ/m to x/s ) .

Summary: The paper draws on connections between likelihood-free inference and black-box optimization to propose new black-box optimization methods. In general, the goal here is to not find the exact optimum of the black-box objective, but to sample from a set of sequences with high-quality objective. This is akin to the problem of collecting posterior samples in a likelihood-free inference problem.   The paper provides a number of proposed methods and compares them on some benchmark sequence optimization problems that have appeared in recent literature.