A generative nonparametric Bayesian model for whole genomes

1 Introduction . Measuring and making DNA is central to modern biology and biomedicine . Generative probabilistic modeling o ers a framework for learning from sequencing data and forming experimentally testable predictions of unobserved or future sequences that can be synthesized in the laboratory [ 19 , 31 , 63 ] . Existing approaches to genome modeling typically preprocess the data to build a matrix of genetic variants such as single nucleotide polymorphisms [ 25 , 57 ] . However , most modes of sequence variation are more complex . Structural variation occurs widely within individuals ( e.g . in cancer ) , between individuals ( e.g . in domesticated plant populations ) and between species ( e.g . in the human microbiome ) , and methods for detecting and classifying structural variants are heuristic and designed only for predefined types of sequence variation such as repeats [ 12 , 44 , 50 , 69 , 78 ] . Ideally , we would be able to directly model genome sequencing data and/or assembled genome sequences . However , building generative models that work with raw nucleotides , not matrices of alleles , raises the úThese authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . . . .GTATCC A T G C A T G C A T G A A T G C AT ATATA AG GC AC CA CT TAC ACA CAC ACT CTA TAG AGC GCA B 2 Bayesian embedded autoregressive models . We first briefly review autoregressive ( AR ) models as applied to sequences of discrete characters . Let f ( ◊ ) denote an autoregressive function with parameter ◊ and let L denote the lag of the autoregressive model ; then the AR model generates data as Xi|Xi≠L : i≠1 ≥ Categorical ( fXi≠L : i≠1 ( ◊ ) ) , ( 1 ) where i indexes position in the sequence X and Xi≠L : i≠1 consists of the previous L letters in the sequence . Since sequence length as well as nucleotide or amino acid content is relevant to biological applications , we use a start symbol ÿ at the beginning and a stop symbol $ at the end of each sequence ; letters Xi are sampled sequentially starting from the start symbol and continuing until a stop symbol is drawn . We propose the Bayesian embedded autoregressive ( BEAR ) model , a Bayesian Markov model that embeds an AR model into its prior . The BEAR model takes the form , L ≥ fi ( l ) , vk ≥ Dirichlet ! 1 h fk ( ◊ ) '' for all k , Xi|Xi≠L : i≠1 ≥ Categorical ( vXi≠L : i≠1 ) , ( 2 ) where fi ( l ) is a prior on the lag with support up to infinity , h > 0 is a concentration hyperparameter , and k is a length L kmer . The BEAR model has three key properties ( Fig . 1 ) . First , the unrestricted transition parameter v and lag L allow the model to capture exact conditional distributions of pú to arbitrarily high order : pú ( Xi|Xi≠1 ) at L = 1 , then p ú ( Xi|Xi≠2 , Xi≠1 ) at L = 2 , etc .. This property allows the BEAR model to be used for nonparametric density estimation ( Section 3 ) . Second , in the limit where h æ 0 , the BEAR model reduces to the embedded AR model ( Eqn . 1 ) . The optimal h provides a measurement of the amount of misspecification in the AR model ( Section 4 ) . Third , the choice of the conjugate Dirichlet prior allows the conditional marginals p ( ( Xn ) Nn=1|L , h , ◊ ) to be computed analytically , and ( since L is one-dimensional ) the total marginal likelihood p ( ( Xn ) Nn=1|h , ◊ ) to be estimated tractably . This allows BEAR models to be used for hypothesis testing ( Section 5 ) . There are a variety of ways of performing inference in BEAR models , but for most applications we will focus on empirical Bayes methods that optimize point estimates of L , h and ◊ . Let # ( k , b ) denote the number of times the length L kmer k is seen followed by the letter or stop symbol b in the dataset ( Xn ) Nn=1 . Using a high-performance kmer counter optimized for nucleotide data , KMC , we can compute the count matrix # ( · , · ) for all observed kmers k in terabyte-scale datasets , even when the matrix does not fit in main memory ( Section J.2 ) [ 39 ] . To optimize h and ◊ , we take advantage of the fact that the log conditional marginal likelihood can be written as a sum over observed kmers , log p ( ( Xn ) Nn=1|L , h , ◊ ) = ÿ k : # k > 0 log 5 ( q b 1 h fkb ( ◊ ) ) r b ( 1h fkb ( ◊ ) ) r b ( 1h fkb ( ◊ ) + # ( k , b ) ) ( q b 1 h fkb ( ◊ ) + # ( k , b ) ) 6 . ( 3 ) This decomposition lets us construct unbiased stochastic estimates of the gradient with respect to h and ◊ by subsampling rows of the count matrix ( Section J.1 ) . Empirical Bayes in the BEAR model therefore costs little extra time as compared to standard stochastic gradient-based optimization of the original AR model . Code is available at https : //github . com/debbiemarkslab/BEAR . Toy example We next briefly illustrate the properties and advantages of the BEAR model in simulation . We generated samples from an AR model in which fk ( ◊ ) depends on k linearly as a function of both individual positions and pairwise interactions between positions , with the strength of the pairwise interaction weighted by a parameter —ú ( Section I.1 ) . We first fit ( using maximum likelihood ) a linear AR model that lacks pairwise terms and is thus misspecified when —ú > 0 . Since the AR model is misspecified , it does not asymptotically approach the true data-generating distribution pú ( Fig . 2A , gray ) . We next computed the posterior of a vanilla BEAR model without the embedded AR in its prior , instead using the Je reys prior vk ≥iid Dirichlet ( 1/2 , . . . , 1/2 ) . The vanilla BEAR model asymptotically approaches the true data generating distribution , since it is a nonparametric model ; however , it underperforms the AR model in the low data regime ( Fig . 2A , black ) . Finally , we fit a BEAR model with the misspecified linear AR model embedded , using our empirical Bayes procedure . The BEAR model performs just as well as its embedded AR model in the low data regime , just as well as the vanilla model in the high data regime , and better than both at intermediate values ( Fig . 2A , blue and yellow ) . When the AR model is well-specified , the empirical Bayes estimates of the parameters ◊ under the BEAR model match the maximum likelihood estimates of ◊ under the AR model nearly exactly ( Fig . S7 ) . When the AR model is misspecified , however , the BEAR model provides a warning : the empirical Bayes estimate of h converges to a non-zero value , rather than zero ( Fig . 2B ) . This warning emerges early : h converges well before the vanilla model starts outperforming the misspecified AR model . Related Work The key idea behind BEAR models is to nonparametrically perturb a parametric model [ 48 ] , following a similar strategy to the Polya tree method proposed by Berger and Guglielmi [ 6 ] . As in Berger and Guglielmi [ 6 ] , we use Dirichlet priors centered at the parametric model ’ s predictions , and construct tractable goodness-of-fit tests by exploiting Dirichlet-categorical conjugacy . BEAR models extend these ideas from one-dimensional continuous data to finite-length sequences of discrete characters . Markov and AR models have a long history and wide range of applications in biological sequence analysis [ 21 , 51 , 58 ] . Compression methods , in particular , often rely on accurate density estimation and use Markov or AR models to achieve it [ 17 , 55 , 56 , 67 ] . We establish theoretical guarantees for density estimation with fully Bayesian Markov models ( Section 3 ) . AR models used for compression , like other AR models , can be embedded into BEAR models for improved statistical performance and to measure misspecification . BEAR models are closely linked to non-generative genome analysis methods . Assembly algorithms and variant callers often analyze paths in the de Bruijn graph of a sequence dataset ; in the limit h æ Œ , samples from the posterior predictive distribution of the BEAR model , conditional on L , correspond to paths through the L-mer de Bruijn graph of the data [ 11 , 33 ] . Comparisons between genomes and other sequences are often made on the basis of kmer counts ; our two-sample test provides a generative perspective on this idea [ 3 , 16 , 78 ] . BEAR models are also connected to ideas in natural language processing , where kmers are referred to as ngrams . Under the vanilla BEAR model , the mean of the posterior predictive distribution conditional on L corresponds to an ngram additive smoothing model [ 9 ] . Comparisons between datasets using their ngram counts are also common in model evaluation metrics such as the BLEU score [ 53 ] . 3 Density estimation . The density estimation problem is that of estimating pú given data ( Xn ) Nn=1 drawn i.i.d . from pú . Density estimation is particularly crucial for biological sequence analysis due to its connections to fitness estimation [ 31 , 65 ] . State-of-the-art mutation e ect prediction methods and clinical variant interpretation methods rely on density estimates of evolutionary sequence data [ 20 , 60 ] . Density estimation with generative models is particularly useful for protein design , as samples from accurate density estimates are likely to be functional and can be synthesized in the laboratory [ 63 , 66 ] . Despite all these applications , existing density estimation methods for biological sequences lack theoretical guarantees on their accuracy and are often limited in their scale , being restricted to relatively short sequences [ 79 ] . Here , we show that the posterior distribution of the BEAR model is consistent and will concentrate on pú as N æ Œ , regardless of what pú actually is , so long as pú generates finite length sequences almost surely ( a.s. ) . We first study the expressiveness of BEAR models . Let ML be the set of Markov models pv with transition probabilities v and lag L that generate finite length strings a.s .. Note that M1 µ M2 µ . . .. Define the union M = fiŒL=1ML . We can compare M to the set of distributions over finite strings S , of which pú is a member . In Section D we prove that , Summary of Propositions 1-4 Not all possible distributions over S are in M. However , M is dense on the space of probability distributions over S with the total variation metric . The implication of this result is that although BEAR models can not exactly match arbitrary data-generating distributions , they can approximate pú arbitrarily well as L increases . This makes asymptotic consistency possible . We now show that the posterior of the BEAR will in fact asymptotically concentrate on the true pú , i.e . it is consistent . For tractability , we assume in this section that the prior is fixed ( we do not use empirical Bayes ) . The result relies on the tools for understanding convergence rates of posteriors developed in Ghosal et al . [ 23 ] . The most important assumption is that p ú is subexponential , meaning that for some t > 0 , Epú exp ( t|X| ) < Œ where |X| is the sequence length . Let ( ·| ( Xn ) Nn=1 ) denote the posterior over sequence distributions . Let B ( pú , ” ) denote a ball of radius ” centered at pú , using the Hellinger distance . Summary of Theorem 35 Given M > 0 large enough and ‘ œ ( 0 , 1 ) small enough , we have ( B ( pú , MN≠ 12 ‘ ) | ( Xn ) Nn=1 ) æ 1 in probability . A proof is in Section H and simulations in Section I.2 . This result states that the posterior distribution of the model converges to a delta function at the true distribution pú regardless of what pú is . It also provides a rate of convergence : in a parametric model , the uncertainty would shrink as N≠ 12 , but here the rate is slower , N≠ 12 ‘ , a price paid for the nonparametric model ’ s expressivity [ 23 , 29 , 35 ] . The proof includes a variety of new theoretical constructions and algorithms that are used to approximate subexponential sequence distributions .

Summary: The authors propose a generative sequence model, Bayesian embedded autoregressive (BEAR), which is non-parametric as opposed to parametric autoregressive models to address the misspecification problem in neural networks. The BEAR model is a Bayesian Markov model with an embedded autoregressive model into its prior. Using parametric model output as a prior allows BEAR to be used for non-parametric density estimation which makes asymptotic consistency possible. They also show that the learned concentration hyperparameter for a prior dirichlet distribution represents the amount of misspecification in the embedded autoregressive model (or other neural network that was embedded) that can be used as a warning when the model can not be trusted. Then, they prove that absolute performance of the distribution predicted by some model can be measured through testing whether it matches the optimal BEAR model.   They theoretically prove that the BEAR model is more expressive than the naive AR model and that it converges to the true distribution as the size of the dataset increases. BEAR models are scalable and uncertainty-aware, which make them a useful generative model for full biological sequences.