Absorb & Escape: Overcoming Single Model Limitations in Generating Heterogeneous Genomic Sequences

Authors: zehui  li
yuhao  ni
guoxuan  xia
william  beardall
akashaditya  das
guy-bart  stan
yiren  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 04:16:00

Abstract: Recent advances in immunology and synthetic biology have accelerated the development of deep generative methods for DNA sequence design. Two dominant approaches in this field are AutoRegressive (AR) models and Diffusion Models (DMs). However, genomic sequences are functionally heterogeneous, consisting of multiple connected regions (e.g., Promoter Regions, Exons, and Introns) where elements within each region come from the same probability distribution, but the overall sequence is non-homogeneous. This heterogeneous nature presents challenges for a single model to accurately generate genomic sequences. In this paper, we analyze the properties of AR models and DMs in heterogeneous genomic sequence generation, pointing out crucial limitations in both methods: (i) AR models capture the underlying distribution of data by factorizing and learning the transition probability but fail to capture the global property of DNA sequences. (ii) DMs learn to recover the global distribution but tend to produce errors at the base pair level. To overcome the limitations of both approaches, we propose a post-training sampling method, termed Absorb & Escape (A&E) to perform compositional generation from AR models and DMs. This approach starts with samples generated by DMs and refines the sample quality using an AR model through the alternation of the Absorb and Escape steps.  To assess the quality of generated sequences, we conduct extensive experiments on 15 species for conditional and unconditional DNA generation. The experiment results from motif distribution, diversity checks, and genome integration tests unequivocally show that A&E outperforms state-of-the-art AR models and DMs in genomic sequence generation. A&E does not suffer from the slowness of traditional MCMC to sample from composed distributions with Energy-Based Models whilst it obtains higher quality samples than single models. Our research sheds light on the limitations of current single-model approaches in DNA generation and provides a simple but effective solution for heterogeneous sequence generation. Code is available at the [Github Repo](https://github.com/Zehui127/Absorb-Escape).

Main paper: 
 introduction: 
 DNA sequences, as the blueprint of life, encode proteins and RNAs and directly interact with these molecules to regulate biological activities within cells. The success of deep generative models in image [28] , text [27] , and protein design [35] has drawn the attention of deep learning researchers to the problem of DNA design, i.e. applying these models to genomic sequence generation [4, 34, 38] . However, one rarely explored issue is how well existing methods can handle the unique property of DNA sequences: heterogeneity. DNA sequences are highly heterogeneous, consisting of multiple The DM and AR models jointly optimize a given sequence by alternating between the A-step and the E-step. See Section 4 for a detailed explanation. connected functional regions (e.g. Promoter Regions, Exons, and Introns) in sequential order. While elements within each functional region might be homogeneous (coming from the same distribution), the overall sequence is non-homogeneous. This heterogeneity, along with the discrete nature of genomic sequences, poses challenges to popular deep generative methods.Limitations of Existing Single-Model Approaches in Generating Genomic Sequences AutoRegressive Models (AR) [8, 17, 24] are one of the most dominant approaches for discrete sequence generation. To model the data distribution of a sequence x of length T , the probability of x is factorized as:p AR (x) = T i=1p θ (x i |x 1 , x 2 , . . . , x i-1 ).(EQUATIONAn issue arises when modeling heterogeneous data, where the value of θ may vary significantly from one segment to another. Additionally, AR models assume a dependency between the current element and previous elements; this assumption may not hold true for heterogeneous sequences, potentially hindering the learning process (see Section 3 for details).(b) We design the theoretical framework Absorb & Escape (A&E) to sample from the compositional distribution of an AR model and a DM (Section 4.1). Furthermore, as shown in Figure 1b , we propose an efficient post-training sampling algorithm termed Fast A&E to sample from the composed model, requiring at most one forward pass through the pretrained DM and AR model (Section 4.2). (c) We design a comprehensive evaluation workflow for DNA generation, assessing the sequence composition, diversity, and functional properties of generated genomic sequences. Extensive experiments (15 species, 6 recent DMs, 1 AR model, and 3 types of evaluations) reveal: 1) the limitations of existing models in DNA generation (Section 5.3), and 2) that the proposed algorithm Fast A&E consistently outperforms state-of-the-art models as measured by motif distribution and functional property similarity to natural DNA sequences (Sections 4.2 and 5.3).2 Preliminaries and Related Work2.1 Problem Formulation: DNA Generation DNA generation aims to produce synthetic sequences that functionally approximate real DNA sequences. Formally, let N 4 = {1, 2, 3, 4}, where each element represents one of the four nucleotides: adenine (A), thymine (T), guanine (G), and cytosine (C). A DNA sequence of length L can be represented as x ∈ N L 4 , with each element/nucleotide denoted by x 1 , x 2 , • • • , x L . Unconditional Generation: Given a dataset of real-world DNA sequences X = {x (n) } N n=1 collected from some distribution p(x), where each sequence x (n) ∈ N L 4 represents a chain of nucleotides, the objective is to develop a generative model p θ (x) of the data distribution p(x) from which we can sample novel sequences x ∼ p θ (x). These sequences should be structured arrangements of A, T, G, and C, reflecting the complex patterns found in actual DNA. Earlier works applying Generative Adversarial Networks (GANs) [13] to generate protein-encoding sequences [14] and functional elements [33, 34, 38] fall into this category.Conditional Generation: In this task, the dataset of DNA sequences X = {x (n) , c (n) } N n=1 is sampled from the joint distribution p(x, c), where c represents the condition associated with each sequence. The objective is to develop a model p θ (x|c) that generates new DNA sequences x given condition c. Recently, a discrete diffusion model DDSM [4] and an AutoRegressive model RegML [20] have used expression level as the condition, while DNADiffusion [26] has used cell type as the condition. 
homogeneous vs. heterogeneous sequences: 
 Homogeneous Generation Process In the context of sequence generation, a homogeneous Markov Chain is characterized by constant probabilistic rules for generating the sequence at each time step t. More generally, a process is defined as homogeneous if the transition probabilities are independent of time t. This means there exists a constant P c,j such that:EQUATIONholds for all times t, where P c,j is a constant, and c represents a specific sequence of past values, i.e., c = (c 1 , c 2 , . . . , c t-1 ).Heterogeneous Generation Process Assuming homogeneous properties simplifies modeling but can be overly restrictive for certain modalities, leading to inaccuracies. For example, DNA sequences consist of various functionally distinct regions, such as promoters, enhancers, regulatory regions, and protein-coding regions scattered across the genome [11] . Each region may be assumed to be homogeneous, but the overall sequence is non-homogeneous due to the differing properties of these elements.For sequences like DNA, which consist of locally homogeneous segments, we define them as heterogeneous sequences. Formally, a heterogeneous sequence is defined as follows: Suppose a sequence x is divided into segments S 1 , S 2 , . . . , S m , where each segment S i is homogeneous. For each segment S i = (x t , x t-1 , x t-2 , . . . , x t-k ), there exists a constant P ci,j such that:EQUATIONholds for all times t within segment S i , where P ci,j is a constant, and c i = (c i,t-1 , c i,t-2 , . . . , c i,t-k ) represents values characterizing the history within that segment. While segment S i is homogeneous, the entire sequence is non-homogeneous due to the varying properties across different segments.  
single model limitations in heterogeneous sequence generation: 
 How powerful are AR models and DMs in modelling heterogeneous sequences? We first provide a theoretical analysis, and then perform experiments on synthetic sequences to validate our assumption.AutoRegressive (AR) Models Suppose a heterogeneous sequence x consist of two homogeneous segments of length k, thenx = {{x 1 , x 2 , • • • , x k }, {x k+1 , x k+2 , • • • , x 2k }}.AR models factorize p(x) into conditional probability in eq. ( 4); consider the case where the true factorisation of p(x) follows eq. ( 5).EQUATIONEQUATIONAR factorisation allows the accurate modelling of the first homogeneous segment; however, it may struggle to disassociate the elements of the second segment from the first segment. More precisely, sufficient data is needed for AR model to learn that p θ (x k+1 ), p θ (x k+2 ), • • • , p θ (x 2k ) should be independent to the elements {x 1 , x 2 , • • • , x k } in the first segments. Secondly, when the context length of the AR model is shorter than the sequence length 2k, it could struggle to capture the difference between p 1 and p 2 with a single set of parameters θ.Diffusion Models (DMs) On the other hand, DMs estimate the overall probability distribution p(x) without factorization. The elements of x are usually generated in parallel. Thus, they do not suffer from the conditional dependence assumption. However, the removal of the conditional dependence assumption may also decrease the accuracy of generation within each homogeneous segment compared to AR models, as DMs do not explicitly consider previous elements.To evaluate the performance of Autoregressive (AR) models and Diffusion Models (DMs) in generating heterogeneous sequences, we consider a toy example with 50,000 heterogeneous sequences X = {x (n) } 50000 n=1 . Each sequence contains 16 segments, as illustrated in Figure 2 (a), and each segment comprises 16 elements, resulting in a total sequence length of 256 (x ∈ N 256 4 ). A simple Hidden Markov Model (HMM) is used to generate each segment, as shown in Figure 2(b) , with deterministic transition and emission probabilities that ensure homogeneity within each segment. The emitted tokens differ from one segment to another, mimicking the properties of real DNA sequences. Whilst it is possible to use more complex distributions for each segment, doing so could complicate the evaluation of the generated sequences.Evaluation Under our toy HMM setup, a generative model could make two types of mistakes within each segment: 1) Illegal Start Token: The generated sequence starts with a token which has zero emission probability. E.g. in Figure 2 (b), the starting token could only be {A, T }. {G, C} at the beginning of the sequence are classified as illegal start tokens. 2) Incorrect Transition: The generated sequence contains tokens with zero transition probability. E.g. in Figure 2 (b) given the start of the sequence is (A, T, A, T ), the next token must be A, any other tokens such as {T, G, C} are classified as incorrect transitions. We use the number of incorrect tokens as the metric for evaluation. x′ j ∼ p AR θ (x j |x 0:i ) 7:while p AR (x ′ j ) > p DM (x j ) do 8:Increment j = j + 1 Absorb-Step Inspired by Gibbs sampling [12] , which iteratively refines each dimension of a single sample and moves to higher density areas, our algorithm starts with a sequence x 0 ∼ p DM (x), generated by the diffusion model and then refines the samples through the Absorb step and Escape step. By exploiting the heterogeneous nature of the sequence, we assume that x 0 can be factorized into multiple segments {s 1 , s 2 , . . . , s n }. For each segment s k , we set i and j as the start and end indices, respectively. During the Absorb step, we sample a subset of segments S ⊆ {s 1 , s 2 , . . . , s n } and refine each segment s k by sampling x′ i:j ∼ p(x i:j |x 0:i-1 , x j+1:L ) ≈ p AR θ (x i:j |x 0:i-1 ), using the autoregressive model to approximate the conditional probability.Escape-Step After refining the segment in the Absorb step, we proceed with the Escape step where we update the refined segment xt i:j to x′ i:j . This iterative process continues for each selected segment s k , with t incrementing after each update. By leveraging the ability of the diffusion model to capture the overall data distribution and the autoregressive model to refine homogeneous sequences within each segment, our algorithm efficiently improves the quality of the generated samples. The final output xt is hereby closer to the true data distribution p(x) compared to the initial sample x0 . A proof for the convergence in Proposition 1 is provided in Appendix C.Proposition 1. The Absorb & Escape (A&E) algorithm converges to the target distribution p C θ,β (x) = p AR θ (x) • p DM β (x), under the assumptions that both models are properly trained, the segments of x are homogeneous, the subset of segments is chosen randomly, and the conditional distribution p(x i:j |x 0:i-1 , x j+1:L ) is accurately approximated by p AR θ (x i:j |x 0:i-1 ). 
practical implementation: fast a&e: 
 While A&E offers a method to sample from a compositional distribution, two practical issues remain unresolved. Firstly, the algorithm may take a considerable amount of time to converge. Secondly, a crucial step in Line 3 of Algorithm 1 involves splitting x into homogeneous segments {s 1 , s 2 , • • • , s n } and then sampling a subset of these segments. Segmentation is straightforward when the boundaries of functional regions of the DNA sequence are known, such as protein-coding regions, exons, or introns, where each region naturally forms a homogeneous segment. However, this information is often unavailable in practice.To address these challenges, we propose a practical implementation termed Fast A&E. For generating a sequence x ∈ N L 4 , it requires at most L forward passes through the AR model. As shown in Algorithm 2 and Figure 1b , Fast A&E adopts a heuristic-based approach to select segments for refinement. It scans the sequence from left to right, identifying low-quality segments through a thresholding mechanism. Tokens with predicted probabilities smaller than the T absorb threshold trigger the absorb action, while the autoregressive process terminates once the probability of a token generated by the AR model p DM (x ′ j ) is smaller than that of the diffusion model p AR (x j ). In this manner, Fast A&E corrects errors made by the diffusion model with a maximum running time of O(T DM + T AR ), where T DM and T AR are the times required for generating a single sequence from the diffusion model and autoregressive model, respectively. 
transcription profile (tp) conditioned promoter design: 
 We first evaluate Fast A&E in the task of TP-conditioned promoter design, following the same evaluation procedures and metrics as used by DDSM [4] and Dirichlet Flow Matching (DFM) [32] . c ), the generated sequence x and the ground truth sequence are processed through the genomic neural network Sei [37] . The performance metric is the mean squared error (MSE) between Sei(x) and Sei(x).Results: As shown in Table 2 , we ran Fast A&E on this task with a default threshold of T absorb = 0.85, using the pretrained model as both the autoregressive (AR) model and the denoising model (DM) component. The evaluation followed the same procedure as described in the DFM repository. The Fast A&E model, comprising AR and DM components (i.e., the language model and the distilled DFM checkpoints provided in the DFM repository), achieved state-of-the-art results with an MSE of 0.0262 on the test split.Dataset Construction Prior efforts in DNA generation have been constrained by small, single-species datasets [19, 34] . To better evaluate the capability of various generative algorithms in DNA generation, we construct a dataset with 15 species from the Eukaryotic Promoter Database (EPDnew) [23] . Table 3 compares our EPD dataset with those used in previous studies, including DDSM [4] , ExpGAN [38] ,and EnhancerDesign [33] . The key advantage of EPD is its diversity in both species types and DNA sequence types. Additionally, although the number of sequences in EPD is on a similar scale to that of DDSM, EPD offers greater uniqueness: each sequence corresponds to a unique promoter-gene combination, a guarantee not provided by the other datasets. Model Training All the models are implemented in Pytorch and trained on a NVIDIA A100-PCIE-40GB with a maximum wall time of 48 GPU hours per model; most of the models converged within the given time. Adam optimizer [7] is used together with the CosineAnnealingLR [22] scheduler. The learning rate of each model are detailed in Appendix D. For the evaluation of various diffusion models in unconditional generation (see Section 5.2.2), we sample 50,000 sequences from each model. For the conditional generation across 15 species (see Section 5.3), we generate 4,000 sequences. In both cases, we use the DDPM sampler [31] with 1,000 sequential denoising steps. Pearson correlation between the motif distributions of generated and natural DNA sequences for motifs like TATA-box, GC-box, Initiator, and CCAAT-box. MSE M is the average squared differences between these motif distributions. 2. S-FID (Sei Fréchet Inception Distance): Measures the distance between distributions of generated and natural DNA sequences in latent space similar to the FID metric [15] for images, replacing the encoder with the pre-trained genomic neural network, Sei [37] .The results are presented in Table 4 , indicating that most existing models perform worse than the simple baseline DiscDiff proposed here, as measured by S-FID, Cor TATA , and MSE TATA . TATA is one of the most fundamental motifs for gene transcription -a special type of protein called transcription factors binds to TATA tokens on the DNA as shown in Figure 1a . It changes the shape of DNA and then enables the gene transcription. The failure of existing diffusion models to capture the TATA-box distribution indicates a potential gap in existing research. In the following, we hereby use DiscDiff as the p DM β to initialize the A&E algorithm.We compare our model Fast A&E with Hyena [24] and the best-performing diffusion model from Section 5.2.2, DiscDiff, on the task of generating species-specific DNA sequences.We consider four types of motifs closely related to promoter activities {TATA-box, GC-box, Initiator, CCAAT-box}. We calculate 4 types of motif distributions for 15 species across 3 models, resulting in 180 frequency distributions. Sequence Diversity To assess the diversity of the generated sequences, we applied BLASTN [18] to check (1) the similarity between the training set (Natural DNA) and generated sequences from three models, and (2) the similarity within the generated sequences. BLASTN takes a query DNA sequence and compares it with a database of sequences, returning all the aligned sequences in the database that are similar to the query. For each alignment, an alignment score, the alignment length (AlignLen), and statistical significance (eValue) are provided to indicate the quality of the alignment, where a larger alignment score, a smaller statistical significance (eValue), and a longer alignment sequence (AlignLen) indicate a higher similarity between the query sequence and the database sequences. Ideally, when using generated sequences to query the training dataset, a good generative sequence should align better than a random sequence, but not replicate the sequences in the training set. Table 5 shows the results of the BLASTN algorithm. From the table, DiscDiff, Hyena, and A&E all satisfied the mentioned criteria. In terms of the diversity within the groups, none of the algorithms generated repetitive sequences. Furthermore, A&E tends to lie between Hyena and DiscDiff in terms of the diversity of the generated sequences, implying that A&E may combine the properties of AR and DM models. One notable fact is that the alignment scores are very high within the natural sequences, potentially indicating that natural sequences naturally have repetitive properties (conservative motifs), while the generative sequences do not have this characteristic.Genome Integration with Promoter Sequences As shown in Figure 5 , to evaluate the functional properties of sequences generated by Hyena, DiscDiff, and A&E, we inserted the generated promoter sequences of length 128 bp upstream (5') of three commonly studied genes in oncology: TP53, We use Enformer [5] to predict transcription profiles. Enformer takes a DNA sequence of 200k bps as input and outputs a matrix P ∈ R 896×638 , representing a multi-cell type transcription profile. We sampled 300 promoter sequences from each source: Natural DNA promoters, Hyena, DiscDiff, and A&E. For each set, we calculated the average transcription profile across the sequences. The Sum of Squared Errors (SSE) between these average transcription profiles of the generated sequences and those of natural promoters are shown in Table 6 . The results indicate that A&E produces the smallest SSE, suggesting it best captures the properties of natural DNA. This finding highlights the potential of generative algorithms to create promoter sequences that effectively regulate gene expression, with applications in bioproduct manufacturing and gene therapy.We perform a sensitivity analysis of A&E algorithm over hyperparameter T absorb . As shown in Figure 6 , with a small T absorb , the sequences generated by A&E are dominated by the diffusion model. As T absorb increases, the AR helps to correct the errors made by the DM. Finally, when T absorb is larger than 0.7, the correlation flattens and fluctuates. In conclusion, A&E is robust under different values of T absorb , and it is best to use the validation dataset to choose the optimal value. However, a wide range of T absorb can still be used with improved performance. The choice of VAE architecture in LDMs is critical and often domain-specific. We find that mapping the input data to a higher dimension space can help to learn a better denoising network, generating more realistic DNA sequences. We hereby propose to use a two-stage VAE architecture as shown in Figure 7 .E ϕ 1 : N L 4 → R K×M maps s ¯∈ N L 4 to a 2D latent space z ¯1 ∈ R K×M, where K is the number of channels and M is the length of the latent representation. The second stage encoder E ϕ 2 : R 1×K×M → R C×K ′ ×M ′ first adds a dummy dimension to z ¯1 such that z 1 ∈ R 1×K×M and then maps it to 3d latent space z ∈ R C×K ′ ×M ′ , where C is the number of channels, K ′ and M ′ are the reduced dimensions of K and M respectively. The decoder in the first and second stage are D θ 1 and D θ 2 respectively. Which are symmetric to the encoders. Overall, we have z = E ϕ (s) = E ϕ 2 (E ϕ 1 (s)), and the reconstruction is s = D θ (z) = D θ 1 (D θ 2 (z)).When training the VAE, we propose to use Cross Entropy (CE) as reconstruction loss. The loss function is given by:L θ,ϕ = E p(s) E q ϕ (z|s) - L l=1 4 i=1 δ is l log p θ (s l |z) Reconstruction Loss + β • E p(s) [KL(q ϕ (z|s) || N (z; µ, Σ))] KL Divergencewhere δ ij is the Kronecker delta, p θ (s|z) is the probabilistic decoder output from D θ ; q ϕ (z|s) is the probabilistic output from encoder E ϕ that represents the approximate posterior of the latent variable z given the input s; N (z; µ, Σ) is the prior on z. Here we use a simple isotropic. β is a mixing hyperparameter. β is set to 10 -5 in the experiments used in this paper.Once D θ and E ϕ are trained in the first step, we train a noise prediction ε θ in the latent space z = E ϕ (s) with Equation ( 6).E z,t∼U [1,T ],ε∼N (0,I) ∥ε -ε θ (z t , t)∥ 2 2 (6)We train both HyenaDNA and DiscDiff on an NVIDIA A100-PCIE-40GB using the Adam optimizer. For HyenaDNA, the learning rate is set to 0.0001, and we use the model heyenadna-large-1m-seqlen for this task. The maximum number of epochs is set to 100, with early stopping enabled to facilitate early convergence.For DiscDiff, the VAE is trained with a learning rate of 0.0001, while the UNet is trained with a learning rate of 0.00005. DiscDiff is trained for 600 epoches; during the inference time, we use DDPM [16] sampler with 1000 denoising steps.In this section, we provide a proof for the convergence of the Absorb & Escape (A&E) algorithm under certain assumptions. The convergence proof will demonstrate that the sequence generated by the A&E algorithm converges to a sample from the target distribution p C θ,β (x).We make the following assumptions for the convergence proof:1. The autoregressive model p AR θ (x) and the diffusion model p DM β (x) are both properly trained and provide accurate approximations of the underlying data distribution. 2. The initial sample x 0 ∼ p DM β (x) is a valid sample from the diffusion model.3. The segments {s 1 , s 2 , . . . , s n } of the sequence x are chosen such that each segment is homogeneous. 4. The subset of segments S is chosen randomly and includes all segments over multiple iterations. 5. The conditional distribution p(x i:j |x 0:i-1 , x j+1:L ) approximated by p AR θ (x i:j |x 0:i-1 ) is accurate.We aim to show that the A&E algorithm produces samples from the target distribution p C θ,β (x). We do this by showing that the Markov chain defined by the A&E algorithm has p C θ,β (x) as its stationary distribution.Step 1: Initialization The initial sample x 0 ∼ p DM β (x) is drawn from the diffusion model. This ensures that x 0 is a valid sample from p DM β (x).Step 2: Absorb Step For each segment s k in the subset S, the Absorb step samples x′ i:j from the conditional distribution p(x i:j |x 0:i-1 , x j+1:L ). Under the assumption that p AR θ (x i:j |x 0:i-1 ) accurately approximates this conditional distribution, the refined segment x′ i:j is a valid sample from the target conditional distribution.Step 3: Escape Step The Escape step updates the segment xt i:j to x′ i:j . This ensures that the updated sequence xt incorporates the refinement from the autoregressive model.Step 4: Stationary Distribution To show that the Markov chain defined by the A&E algorithm converges to p C θ,β (x), we need to show that p C θ,β (x) is the stationary distribution of this Markov chain. The transition probability for the A&E algorithm is given by the product of the probabilities of the Absorb and Escape steps:P (x → x ′ ) = P Absorb (x → x ′ )P Escape (x ′ → x).Given that p DM β (x) captures the overall structure and p AR θ (x) refines the segments, the composed distribution p C θ,β (x) is achieved by iteratively applying the Absorb and Escape steps. We need to show that the stationary distribution satisfies:p C θ,β (x)P (x → x ′ )dx = p C θ,β (x ′ ).Using the detailed balance condition for the Markov chain:p C θ,β (x)P (x → x ′ ) = p C θ,β (x ′ )P (x ′ → x).Given that the Markov chain is ergodic, the detailed balance condition implies that p C θ,β (x) is the stationary distribution.Step 5: Ergodicity Ergodicity ensures that the Markov chain will visit all possible states given sufficient iterations. The random selection of segments S and the iterative updates in the A&E algorithm guarantee that all parts of the sequence are refined over time.Conclusion By satisfying the detailed balance condition and ergodicity, we have shown that the Markov chain defined by the A&E algorithm converges to the target distribution p C θ,β (x). Therefore, the A&E algorithm produces samples from the composed distribution p C θ,β (x) as the number of iterations t approaches infinity.Baselines The details about the architecture and implementation of the baseline models are as below:• DNADiffusion [26] : We enhance the current DNADiffusion implementation for DNA synthesis, originally from the DNA-Diffusion project2 , by expanding the models to encompass 380 million parameters. This network is composed of Convolutional Neural Networks (CNNs), interspersed with layers of cross-attention and self-attention. The learning rate is set to 0.0001. • DDSM [4] : We scale up the original implementation of the denoising network used for promoter design in DDSM 3 to what is the corresponding size of the network given 470 million parameters. It is a convolution-based architecture with dilated convolution layers. The learning rate is set to 0.00001.• D3PM [3] : We take the implementation of D3PM for biological sequence generation from EvoDiff [2] 4 , adopting the algorithm for DNA generation. We use the original implementation of the denoising network, which has two versions: with sizes of 38M and 640M. We hereby have D3PM (small) and D3PM (big), respectively. The learning rate for both D3PM (small) and D3PM (large) are set to 0.0001. • Hyena [24] : We modify the RegLM [20] 5 , a existing work uses hyena for DNA generation.Four pretrained Hyena models of different sizes (hyenadna-large-1m-seqlen, hyenadnamedium-160k0seqlen, heynadna-small-32k-seqlen, and hyenaana-tiny-16k-seqlen-d128) are downloaded from HuggingFace6 and used for full-size fine-tuning, we apply the fine-tuned models for generations on EPD-GenDNA. The learning rate for fine-tuning is set to 0.0001. • DiscDiff: A 2D-UNet of 500 Million parameters are used as the denoising network. See Appendix A for the implementation details. The learning rate is set to 0.00005 for UNet training, 0.0001 for VAE training.For the Fast A&B algorithm, we set the T absorb to 0.80.Our code folder includes the following sub-folders, within each folder there is a readme file, detailing the steps to run the code. The below is the list of sub-folders.Include the source code to reproduce Section 3. No external package is required to run the code except for pythonA implementation of the discdiff baseline in Appendix A. Please follow the readme for running the code.This includes an implementation of the proposed algorithm in Section 4.2; however, it depends on the external AR models, please adopt it accordingly to AR models which you want to try.We include the training dataset used for producing the main results, which includes 160K DNA sequences from EPD, each sequence has a length of 256 bp.We plot TATA-box, GC content, Initiator, and CCAAT-box for 15 speces as below. Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Mainly in the introduction part Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All the code repos -The MIT License (MIT) eukaryotic promoter database -CC-BY 4.0Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.https://github.com/Zehui127/Absorb-Escapehttps://github.com/pinellolab/DNA-Diffusionhttps://github.com/jzhoulab/ddsmhttps://github.com/microsoft/evodiffhttps://github.com/Genentech/regLMhttps://huggingface.co/LongSafari 
 
 Appendix:  


Decision: The paper shed light on important weaknesses of AR and DM-based models for genomic sequence design and proposes an intuitive and promising solution combining both models. 

The authors have provided convincing additional experiments and clarification that alleviate initial concerns on the evaluation and the methodology.  We strongly urge them to incorporate these into the manuscript as these significantly strengthen the quality of the submission.

In particular it is critical to add the benchmarking experiments, sensitivity analysis, as well as the insightful comments on the pertinence of promoter generation, and the rationale for considering the EPD dataset.

Reviews: review: paper_summary: paper_summary: This paper presents a sampling approach called Absorb and Escape (A&E) that combines the strengths of diffusion models (DMs) and autoregressive models (AR models) for generating DNA sequences. The authors rightly point out that DNA sequences are generally composed of segments that do not follow the same distribution (i.e. they are heterogeneous sequences). Then, using a toy example, they clearly illustrate the shortcomings of DMs and AR models for modelling heterogeneous sequences but point out that they have complementary strengths, motivating them to combine these models. Their proposed A&E algorithm (and the practically useful Fast A&E algorithm) can be used at sampling time to leverage these complementary strengths - it first samples a sequence from a DM before iteratively refining it using an AR model. After training DMs and AR models on a new dataset of DNA sequences, the authors show that using Fast A&E leads to better samples when compared to using DMs or AR models alone. They also show that the sequences are sufficiently diverse with Fast A&E samples being less diverse than DM samples while being more diverse than AR model samples.
strength_weakness: strength_weakness: ### Reservations about the main evaluation
- The authors use gene sequences from the Eukaryotic Promoter Database (EPD) for training and testing various models. I do not see any details about how exactly these sequences are constructed. From Table 2 and Figure 4, I assume that the authors are extracting sequences of various lengths (either 256bp for 2048bp) centered at the TSS, but the paper needs a more comprehensive description of the data processing since this data is used in all of the main experiments.

- I am not convinced of the utility of modelling these DNA sequences as presented in the paper. The authors train conditional diffusion models where the conditions are various species. I cannot think of a use case for sequences generated using this modelling strategy - there is no way to tune the promoter strength or target gene. Other DNA generation models use arguably better tasks. For example, DNADiffusion uses diffusion models to generate cell-type-specific regulatory elements which could be useful for synthetic biology applications. I would like to understand the authors' motivation for using this task. 

- The paper could be made much stronger by using the same evaluations as the previous papers to show that the usage of A&E with the DMs proposed in those papers (maybe in combination with HyenaDNA for AR modelling) leads to improved performance.

- Furthermore, the authors somewhat abandon their focus on the heterogeneity of the DNA sequences being modelled in the main results. Although it is clear from Figure 4 that the motifs are located in the promoter as expected, one could perform two more simple analyses to show that A&E actually helps in modelling heterogenous sequences:
  - If half the training sequence is promoter-like and the other half is protein-like, the authors could show that this is true in the sampled sequences as well (maybe by BLASTing the two halves against the promoter-like and protein-like segments from the training set).
  - An even simpler analysis would be to look at the protein-like segment of the samples and show that it obeys the rules of protein coding sequences - starts with a start codon and there are no premature stop codons.
questions: questions: In addition to the questions/suggestions above, I have the following minor ones below:
- Since the absorb step is reliant on using an AR model to refine certain segments, is it better to use a masked language model instead of an AR model? This way, context from both sides of the segment can be incorporated and the probability approximation will be better.
- How are probabilities for a sequence/sequence segment extracted from the DM?
- There are discrepancies between the model names mentioned in lines 222-223 and those in Table 3.
limitations: limitations: Apart from the assumptions in Appendix C, I could not find any other discussion of the limitations. I can think of the following other limitations:
- The evaluation scheme is based on a single dataset, it is unclear how A&E will generalize to other datasets/settings.
- Tuning $T_{absorb}$ seems non-trivial.

I do not foresee any negative societal impacts.
paper_summary: paper_summary: The authors introduce a novel approach, called Absorb & Escape (A&E), for generating DNA sequences by combining the strengths of autoregressive (AR) and diffusion models (DMs). The authors argue that existing single-model approaches struggle with the heterogeneous nature of genomic sequences, which consist of multiple functionally distinct regions. Their method initially generates a sequencing using a DM, and then refines sequence segments using an AR model. They evaluate their approach on various design tasks. The authors claim improved performance over existing methods, particularly in generating sequences that satisfy complex constraints and exhibit properties similar to natural genomic sequences.
strength_weakness: strength_weakness: The biological application lacks compelling justification. The authors do not clearly explain why generating synthetic promoter sequences is necessary or advantageous compared to using and optimizing existing genomic promoters. A comparison with established promoter optimization methods is missing, which is crucial for demonstrating the practical value of this approach in genomic applications.

The paper suggests broader applicability of the A&E method beyond DNA design, but this is not sufficiently supported by the presented experiments. Additional tasks and datasets would be necessary to make a convincing case for the method's general effectiveness.

The paper lacks an analysis of how the various hyperparameters, such as the Absorb Threshold (T_Absorb), affect the model's performance. A sensitivity analysis would provide valuable insights into the robustness of the method.

The use of Sum of Squared Errors (SSE) to evaluate generated promoters against natural promoters is questionable. Given that Enformer makes predictions for a large region of sequence (896 consecutive 128 bp bins), this metric may not accurately reflect the quality or functionality of the generated promoters, which cover 1-2 bins.
questions: questions: Algorithm 2 assumes access to token-level probabilities from the diffusion model (p_DM), which is not a standard output for most diffusion models. The authors do not explain how they obtain these probabilities, which is a critical missing piece of information for understanding and implementing their method.

There is insufficient exploration of how the model learns to differentiate between different genomic regions (e.g., coding sequences, promoters, introns). Does the AR model produce different generations in these different regions? This analysis would strengthen the claim that the method effectively handles heterogeneous sequences.

The description of the Eukaryotic Promoter Database (EPD) content is vague. E.g. why would a promoter database contain coding sequence. A more specific explanation of the types of sequences it contains would help readers understand the dataset's composition and relevance.

In Algorithm 2 line 3, I cannot determine exactly what p^DM refers to. I.e. is this a global probability of the entire sequence or specific to a local region. It’s presence within the line 2 loop suggests locality.

The convergence criteria for Algorithm 1 are not specified, which makes it difficult to assess the algorithm.
limitations: limitations: Yes
paper_summary: paper_summary: This paper addresses limitations in existing methods for generating genomic sequences, which struggle to capture the heterogeneous nature of DNA. The authors propose a new framework called Absorb & Escape (A&E) that combines the strengths of autoregressive (AR) models and diffusion models (DMs) to generate more realistic DNA sequences. They first analyze the shortcomings of AR models and DMs when used individually for heterogeneous sequence generation. Then they introduce A&E, which alternates between refining segments using an AR model (Absorb step) and updating the full sequence (Escape step). A practical implementation called Fast A&E is also presented. The authors evaluate their method on a large dataset of promoter sequences from 15 species. They compare Fast A&E against state-of-the-art baselines on metrics like motif distribution similarity, sequence diversity, and functional properties when inserted into genomes.
strength_weakness: strength_weakness: - If trained on original DDSM dataset, how would DiscDiff perform? I ask because the DDSM seems to perform quite poorly in terms of MSE in Table 3, but its performance was reasonable even when compared with Dirichlet Flow Matching (Stark et al, arxiv 2023).
- The explanation of Autoregressive model struggling to capture difference between p1 and p2 with a single set of parameters could be true, but hierarchical AR models coUnuld learn latent segment states. So, unclear why deep AR models not learning the heterogeneous structure is challenging in practice. 
- Regulatory sequences can be quite diverged across species where a promoter in one species may not be a strong promoter in another species. So, what is the benefit of multi-species promoter pretraining? Perhaps comparing vs just on human promoters (i.e. DDSM dataset using CAGE-seq) could help appreciate the nuances. 
- In 5.1, under baseline models, DNADiffusion is cited but not explored in Table 3. HyenaDNA is not benchmarked in Table 3. It would be good to get a genomic language model baseline since the purpose of DiscDiff is to resolve issues of diffusion models and autoregressive models. 
- It's not clear how metrics were calculated. How were motif distributions calculated? ALso, what about MSE of the conditional generation of the model (using Sei as an oracle). THis was done in the DDSM paper as well as DFM. 
- Dirichlet flow matching performed better than DDSM and so it would be worth benchmarking on this (but may be outside scope for this paper). At least mention the new diffusion models that beat out DDSM and the open question remains whether they require absorb and escape or whether they can also benefit.
- Fig 3 doesn't show any comparisons with other diffusion models (eg. DDSM)
- Table 5 should show distribution of enformers predictions to see if generated distribution are even close to natural sequences. The presented metrics are difficult to assess beyond relative comparisons.
questions: questions: Questions are integrated within Weaknesses (above).
limitations: limitations: While the method is very interesting, potentially bridging the limitations of autoregressive models and diffusion models, the evaluation is a bit on the weak side. This makes it difficult to assess any advance. Better evaluations could help improve this work to demonstrate the true gains. It would also be worthwile to explore absorb & escape using other SOTA diffusion models (eg. DFM).
paper_summary: paper_summary: The authors propose to generate DNA sequences using pre-trained DMs and then modifying the segments (randomly selected), through autoregressive models.
strength_weakness: strength_weakness: The main weakness of the paper is that the authors didn't validate the proposed approach in real world long sequences, and the simulated data is not convincing. Apart from that it is not clear how they train the baselines for multiple tissues, and the fact that the authors generalized the claims about all the DMs but only consider a version that they proposed by themself is somehow making the claims not acceptable in some sense.
questions: questions: - How did you train the model for DDSM in the case of multiple species? How did you incorporate species information?

- How do the results look in the same dataset that DDSM and DFM used for evaluation, specifically in promoter design and enhancer design in a single species? It is unclear how the proposed DMs function initially in such scenarios.

- What are the benefits of the proposed DMs and why can't one use previous pre-trained DMs, such as DDSM?

- What is the effect of absorption and escape in unconditional generation?

- How did you choose the length of segments in real-world and synthetic cases?

- What is the effect of the segment selection?

- Why one cannot use evolutionary based models for each segment? At least as a baseline.

- Can you show couple of generated samples for each method and showing how the segments look like in real and generated data?
limitations: limitations: The main limitation of the paper is that it is very dependent to the pre-trained models, as well as selections of segments in a write manner.

Comments: title: Additional Clarification About Enformer Prediction
Further Clarification about promoter generation task
Satisfactory response
comment: Thank you so much for your insightful feedback on our paper! We hope you found our responses useful. As the discussion period is coming to a close, please feel free to ask any remaining questions you may have. We're happy to provide further clarification!
We sincerely appreciate your valuable feedback. Below, we would like to provide additional clarification on how to interpret the output from Enformer.

### Additional Clarification About Enformer Prediction

We understand that the CAGE assay displays aligned reads at transcription start sites (TSSs). However, given that many genes, including TP53, EGFR, and AKT1, possess multiple TSSs. While we can only take the bin value corresponding to one TSS, the exact number of TSSs can vary depending on the specific cell type and experimental conditions. Therefore, focusing solely on the bin corresponding to the TSS immediately following the promoter might overlook other important TSSs. This is why we consider the entire prediction range corresponding to the gene. The output from Enformer reflects the CAGE track values across all TSS positions, ensuring more complete coverage of transcription start sites.
Thank you for the further clarification! Since this paper proposes interesting methods and the benchmarking argument proposed by the authors is valid, I am increasing my score.
We sincerely appreciate your suggestions, which are indeed very helpful. However, we would like to offer some additional clarification regarding the purpose of the promoter generation task.

### Further Clarification about promoter generation task
We agree that incorporating promoter strength into the conditioning is a logical extension of our work and something we plan to explore in future iterations. However, we believe that the current approach provides a necessary first step in mapping the regulatory landscape across species, which could lead to new insights that are not apparent when focusing solely on promoter strength.

In contexts such as synthetic biology or evolutionary studies, understanding the full range of available promoters within a species can provide valuable insights into the species’ inherent regulatory potential, independent of expression strength. This foundational knowledge can then inform subsequent research where specific promoter strengths are prioritized. For example, a recent study published in Science [1] demonstrated that human promoter sequences might follow relatively simple syntax. Such studies require a thorough examination of native promoter sequences, and generative models can offer significant insights in these scenarios.

Additionally, from a benchmarking perspective, if a generative algorithm struggles with species-wise generation, it would likely face challenges in other settings where only class information is used for conditioning. Conversely, strong performance in species-specific class conditioning suggests the algorithm may also perform well when conditioning on other factors, such as expression strength classes conditioning.

[1] Dudnyk, K., Cai, D., Shi, C., Xu, J. and Zhou, J., 2024. Sequence basis of transcription initiation in the human genome. Science, 384(6694), p.eadj0116.
Thank you for the detailed response! Most of my questions have been answered but I keep my original score as I am still unconvinced by the main evaluation task in the paper although the additional evaluations are useful. I have a few suggestions:

1. Regarding Q3: I am still unconvinced of the utility of generating new promoter sequences as presented in the paper. I cannot think of a promoter generation scenario that would require us to only condition on the species (or even cell type) without conditioning on promoter strength. I understand that each species can have a different distribution of promoters but each promoter in the distribution has a different strength (i.e. how much expression it can drive). In most generation scenarios, we want a promoter that drives a certain level of expression and not just a random promoter from the distribution of promoters for that species. Therefore, I encourage the authors to consider incorporating expression strength into their conditioning. Alternatively, using an evaluation task similar to DNADiffusion where they only model cell-type-specific regulatory elements could be more meaningful since any random sequence from the distribution being modeled could be potentially useful.

2. Regarding Q5: Given the length of sequences being modeled, I doubt that the efficiency improvements would be very significant when using a masked language model vs. an AR. The improvement in probability estimation might be worth the slight increase in runtime.
Thanks for your reply. I would like to further clarify the following questions:

### I’m unclear about how you conditioned your model on the dense CAGE profile from the DDSM paper.

Our evaluation was conducted using the pretrained checkpoint from the [DFM repository](https://github.com/HannesStark/dirichlet-flow-matching). The A&E algorithm is a general approach that can be applied to various autoregressive (AR) models and diffusion models (DM). In the additional experiment we presented, A&E utilized the **language model** and **distilled DFM** checkpoints as the AR and DM components. Specifically, A&E can be implemented by modifying the  [PromoterModule](https://github.com/HannesStark/dirichlet-flow-matching/blob/6f360612da8f69ecf4860c075f74ed52b37ce64b/lightning_modules/promoter_module.py#L21C7-L21C21) class in the original DFM repository. No further changes were made to the underlying models or the score network.

### Regarding the Motif Distribution as the Evaluation metric
We have indeed provided additional evaluation metrics, such as FID-based metrics, for human promoters. Although we have some reservations about the benchmark provided in the DDSM, we agree with the reviewers that it is still important to use it, as it better contextualizes our proposed approach within the existing literature. For other species, using motif distribution is a straightforward approach to assess the quality of generated sequences. For example, if a generative algorithm cannot correctly place a TATA-box in the appropriate position, it is generally unlikely to produce a valid promoter sequence.

PS:  I wonder if the reviewer's comment might have been placed in the wrong thread.
I have read the authors’ rebuttal. I hope you’ll include these clarifications and additional information in your revision.

I don’t understand how you conditioned your model on the dense CAGE profile from the DDSM paper. 

I appreciate the difficult in evaluating generated DNA sequence quality, but motif composition is so simple that it’s really not very compelling.

I think you’re misunderstanding Enformer predictions. The length of the target gene is irrelevant; Enformer predicts many assays but the closest to gene expression from a promoter is CAGE, and CAGE will produce aligned reads at the TSS only, unaffected by gene length.

Altogether, I’ll maintain my current scores.
I thank the authors for their clarifications and additional experiments. I believe this is now a solid contribution and therefore will maintain my score of accept.