Absorb & Escape: Overcoming Single Model Limitations in Generating Heterogeneous Genomic Sequences
Authors: Zehui Li
Yuhao Ni
Guoxuan Xia
William Beardall
Akashaditya Das
Guy-Bart Stan
Yiren Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Computational Biology
Genomics
Deep Learning
Generative Model
Primary area: generative_models
TL;DR: This paper proposes a post-training sampling method to perform compositional generation from autoRegressive mdoels and diffusion models for DNA generation.

Abstract: Recent advances in immunology and synthetic biology have accelerated the development of deep generative methods for DNA sequence design. Two dominant approaches in this field are AutoRegressive (AR) models and Diffusion Models (DMs). However, genomic sequences are functionally heterogeneous, consisting of multiple connected regions (e.g., Promoter Regions, Exons, and Introns) where elements within each region come from the same probability distribution, but the overall sequence is non-homogeneous. This heterogeneous nature presents challenges for a single model to accurately generate genomic sequences. In this paper, we analyze the properties of AR models and DMs in heterogeneous genomic sequence generation, pointing out crucial limitations in both methods: (i) AR models capture the underlying distribution of data by factorizing and learning the transition probability but fail to capture the global property of DNA sequences. (ii) DMs learn to recover the global distribution but tend to produce errors at the base pair level. To overcome the limitations of both approaches, we propose a post-training sampling method, termed Absorb & Escape (A&E) to perform compositional generation from AR models and DMs. This approach starts with samples generated by DMs and refines the sample quality using an AR model through the alternation of the Absorb and Escape steps.  To assess the quality of generated sequences, we conduct extensive experiments on 15 species for conditional and unconditional DNA generation. The experiment results from motif distribution, diversity checks, and genome integration tests unequivocally show that A&E outperforms state-of-the-art AR models and DMs in genomic sequence generation. A&E does not suffer from the slowness of traditional MCMC to sample from composed distributions with Energy-Based Models whilst it obtains higher quality samples than single models. Our research sheds light on the limitations of current single-model approaches in DNA generation and provides a simple but effective solution for heterogeneous sequence generation. Code is available at the [Github Repo](https://github.com/Zehui127/Absorb-Escape).

Absorb & Escape: Overcoming Single Model Limitations in Generating Genomic Sequences
Zehui Li",, Yuhao Ni!, Guoxuan Xia William Beardall', Akashaditya Das', Guy-Bart Stan Yiren Zhao
Imperial College London, {zehui 1122,, harry. g-xia21, william beardal115, akashadi tya. w.a. das13, g. stan, a. zmmo))@ieerraal. "Correspondence: {zehui 1122, g. g.stan, @.mphr)) ac.
Abstract
Recent advances in immunology and synthetic biology have accelerated the development of deep generative methods for DNA sequence design. Two dominant approaches in this field AutoRegressive (AR) models and Diffusion Models (DMs). However, genomic sequences are functionally heterogencous, consisting of multiple connected regions (c.g., Promoter Regions, Exons, and Introns) where elements within each region come from the same probability distribution, but the overall sequence is non-homogeneous. This heterogeneous nature presents chal- lenges for a single model to accurately generate genomic sequences. In this paper, analyze the properties of AR models and DMs in heterogeneous genomic quence generation, pointing out crucial limitations in both methods: (i) AR models capture the underlying distribution of data by factorizing and learning the transition probability but fail to capture the global property of DNA sequences. (ii) DMs learn to recover the global distribution but tend produce errors the base pair level. To overcome the limitations of both approaches, we propose post-training sampling method, termed Absorb & Escape (A&E) perform compositional generation from AR models and DMs. This approach starts with samples generated by DMs and refines the sample quality using an AR model through the alternation of the Absorb and Escape steps. To assess the quality of generated sequences, we conduct extensive experiments on 15 species for conditional and unconditional DNA generation. The experiment results from motif distribution, diversity checks, and genome integration tests unequivocally show that A&E outperforms state-of-the-art AR models and DMs in genomic sequence generation. A&E does not suffer from the slowness of traditional MCMC to sample from composed distributions with Energy-Based Models whilst it obtains higher quality samples than single models. Our research sheds light on the limitations of current single-model approaches in DNA generation and provides simple but effective solution for heterogeneous sequence generation. Code is available at the Github Repo'.
Introduction
DNA sequences, as the blueprint of life, encode proteins and RNAs and directly interact with these molecules to regulate biological activities within cells. The success of deep generative models in image [28], text [27], and protein design [35] has drawn the attention of deep learning researchers to
the problem of DNA design, i.e. applying these models genomic sequence generation [4, 34, 38]. However, one rarely explored issue is how well existing methods can handle the unique property of DNA sequences: heterogeneity. DNA sequences highly heterogeneous, consisting of multiple
httg:://gthuubcoo//zzuu////////
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Escape Slep. Update Subsequence with
X1y
P(Xe Kau- b- p (X., (oa-
pARy
Absorb Step. Use Model refine
X0xi--
Homogeneous Subsequence Xit
Heterogeneous Sequence
AutoRegressive
(a) A DNA generated by A&E, interact- ing with TATA-binding protein. The DNA sequences highlighted in magenta are the TATA-hox motif. The confirma- tion is predicted by AlphaFold 3 [I]: the DNA bends at the TATA-box position.
Diffusion Model
(AR) Model
(h) The proposed framework Fast Absorh & Escape (Fast A&E): The DM and AR models jointly optimize a given sequence by alternating between the A-step and the E-step. See Section 4 for detailed explanation.
Figure I: (a) Generated DNA interacting with TATA-binding protein. (b) Proposed A&E framework.
connected functional regions (e.g. Promoter Regions, Exons, and Introns) in sequential order. While elements within each functional region might be homogeneous (coming from the same distribution), the overall sequence is non- non-homogeneous, This heterogeneity, along with the discrete nature of genomic sequences, poses challenges to popular deep generative methods.
Limitations Existing Single-Model Approaches in Generating Genomic Sequences AutoRe- gressive Models (AR) [8, 17, 24] are one of the most dominant approaches for discrete sequence gener- ation. To model the data distribution sequence x of length T, the probability is factorized as:
p AR (x)
II po (= T.,
An issue arises when modeling heterogeneous data, where the value of may vary significantly from one segment to another. Additionally, AR models assume a dependency between the current element and previous elements; this assumption may not hold true for heterogeneous sequences, potentially hindering the learning process (see Section 3 for details).
On the other hand, Diffusion Models (DMs), initially proposed by [30], have been dominant in image generation. In the probabilistic denoising view [16], DMs gradually add noise to the input data Ty. and a reverse diffusion (generative) process trained gradually remove the noise from the perturbed data z₁. DMs directly model the data distribution without AutoRegressive factorization, thereby avoiding the issues associated with AR models. However, it has been shown that DMs are less competent than AR models for discrete data generation [21, 36]. When it comes to modeling heterogeneous genomic sequences, remains unclear how the performance of DMs compares to AR models within each homogeneous segment.
Model Composition As a Solution Balancing the ability of generative algorithms to capture both local and global properties of the data distribution central the problem. An obvious solution could be to combine these two types of models and perform generation using the composed models. However, this typically requires converting these models into an Energy-Based Model and then sampling using Markov Chain Monte Carlo (MCMC), which can be inherently slow due to the sam- pling nature of the algorithm [10], and the potential long inference time of individual models. With the goal of accurate and efficient DNA generation, we aim to investigate two key questions in this work: (i) How well does single AR model or DM perform in DNA generation, given the heterogeneous nature of genomic sequences? (ii) Is there an efficient algorithm to combine the benefits of AR models and DMs, outperforming a single model? In answering these two questions, our contribution is three-fold:
(a) We study the properties of AR models and DMs in heterogeneous sequence generation through theoretical and empirical analysis (Section 3).

---

(b) We design the theoretical framework Absorb & Escape (A&E) to sample from the compositional distribution of an AR model and DM (Section 4.1). Furthermore, as shown in Figure Ib, we propose an efficient post-training sampling algorithm termed Fast A&E sample from the composed model, requiring at most one forward pass through the pretrained DM and AR model (Section 4.2).
(c) We design a comprchensive evaluation workflow for DNA generation, assessing the sequence composition, diversity, and functional properties of generated genomic sequences. Extensive experiments (15 species, 6 recent DMs, AR model, and 3 types of evaluations) reveal: 1) the limitations of existing models in DNA generation (Section 5.3), and 2) that the proposed algo- rithm Fast A&E consistently outperforms state-of-the-- art models as measured by motif dis- tribution and functional property similarity to natural DNA sequences (Sections 4.2 and 5.3).
2 Preliminaries and Related Work
2.1 Problem Formulation: DNA Generation
DNA generation aims to produce synthetic sequences that functionally approximate real DNA sequences. Formally, let N4 = 2,, 2, 3,4, where each element represents one of the four nucleotides: adenine (A), thymine (T), guanine (G), and cytosine (C). A DNA sequence of length L can be represented as x N4, with each element/nncleotide denoted by X1, X2,X2,... , XLL
Unconditional Generation:
Given dataset of real-world DNA sequences X'
(x(")) w==1
collected from some distribution p(x), where each sequence (n) E N/ represents chain of nucleotides, the objective is to develop generative model (x( of the data distribution p(x) from which we can sample novel sequences arrangements of A, T, G, and C, reflecting the complex patterns found in actual DNA. Earlier works applying Generative Adversarial Networks (GANs) [13] generate protein-encoding sequences [14] and functional elements [33, 34, 38] fall into this category.
(x). These sequences should be structured
n== is sampled from the joint distribution p(x, c). where represents the condition associated with each sequence. The objective is to develop a model pe [x] c) that generates new DNA sequences X given condition Recently, discrete diffusion model DDSM [4] and an RegoRegees model RegML [20] have used expression level as the condition, while DNADiffusion [26] has used cell type as the condition.
Conditional Generation: this task, the dataset of DNA sequences X' = [xx(0, (x(x))(()))4) )
2.2 Homogeneous vs. Heterogeneous Sequences
Homogeneous Generation Process In the context of sequence generation, a homogeneous Markov Chain is characterized by constant probabilistic rules for generating the sequence at each time step t. More generally, a process is defined as homogeneous if the transition probabilities are independent of time This means there exists a constant Pe,j such that:
Poj Pr|x; = X cl
(2)
holds for all times f, where Po.j a constant, and e represents specific sequence of past values, i... c (c1, (cI,ceg...,.....
Heterogeneous Generation Process Assuming homogeneous properties simplifies modeling but can be overly restrictive for certain modalities, leading inaccuracies. For example, DNA sequences consist of various functionally distinct regions, such as promoters, enhancers, regulatory regions, and protein-coding regions scattered across the genome [11]. Each region may be assumed to be homoge- neous, but the overall sequence is honohooggss due to the differing properties these elements.
For sequences like DNA, which consist of locally homogeneous segments, we define them as heterogeneous sequences. Formally, heterogeneous sequence is defined as follows: Suppose sequence x is divided into segments S S2, ... Sm. where each segment S, is homogeneous, For each segment S, = (x,, X; 1, X , x(),), there exists a constant Po..j such that:
Per. Pr[x, j X =c]
holds for all times within segment S, where Pe..j is constant, and = (c,,-1, Ci.t+) represents values characterizing the history within that segment. While segment S, is homogeneous, the entire sequence is non-homogencous due to the varying properties across different segments.
3

---

(a) Toy Example of Heterogenecus Sequences
(b) Zoom-in Segment1 HMM
Initial Prob.
Segment1
Segment2
Segment16
Start
Transition Prob.
0.6
0.4
Emission Prob.
Seq:
ATATATA
GCGCGC
TGTGTGTG
Seg:
TATATAT
CGCGCG
TGTGTGTG
State
State2
Seqx ATATATA
(GGGGGG
GIGIGIGT
Figure 2: A toy example with heterogeneous sequences: (a) The overall training set consists of N = 50, 000 heterogeneous sequences, where each sequence further consists of 16 homogeneous segments. We apply an autoregressive and a diffusion model to learn the underlying distribution. (b) Within each segment, the sequences are generated with a simple Hidden Markov Model (HMM), with deterministic transition probability and emission probability.
Single Model Limitations in Heterogeneous Sequence Generation
How powerful are AR models and DMs in modelling heterogeneous sequences? We first provide theoretical analysis, and then perform experiments on synthetic sequences to validate our assumption.
AutoRegressive (AR) Models Suppose a heterogeneous sequence x consist of two homogeneous segments of length k, then x = {(11, 322 { ... FA (I 1 2k+22 ... 522 }} AR models factorize p(x) into conditional probability in eq. (4); consider the case where the true factorisation of p(x) follows eq. (5).
(x) po (o022//1)) -p ( po( |X ( (118+))))))))))))))))))))))1))))
dato (22( .. (Fx (1:kk11
P2 (2 122 xx+1)
P2 (Z XK+1:2k-1
Segment
Segment 2
AR factorisation allows the accurate modelling of the first homogeneous segment; however, it may struggle to disassociate the elements of the second segment from the first segment. More precisely. sufficient data is needed for AR model learn that pa po(**+1), (o(xx)))) -...---polll (PF2))) should be independent to the elements { ... ] in the first segments. Secondly, when the context length of the AR model is shorter than the sequence length 2k, it could struggle to capture the difference between p1 and with single set of parameters 0.
Diffusion Models (DMs) On the other hand, DMs estimate the overall probability distribution p(x) without factorization. The elements x are usually generated in parallel. Thus, they do not suffer from the conditional dependence assumption. However, the removal of the conditional dependence assumption may also decrease the accuracy of generation within each homogeneous segment compared AR models, DMs do not explicitly consider previous elements.
3.1 A Toy Example
To evaluate the performance of Autoregressive (AR) models and Diffusion Models (DMs) in generating heterogeneous sequences, we consider toy example with 50,000 heterogeneous
sequences X' = {x(") Jn=1 (=000) 50000, Each sequence contains 16 segments, as illustrated in Figure 2(a). and each segment comprises 16 elements, resulting
total sequence length of 256 (x N256). A simple Hidden Markov Model (HMM) is used to generate each segment, as shown in Figure 2(b), with deterministic transition and emission probabilities that ensure homogeneity within each segment. The emitted tokens differ from segment another, mimicking the properties of real DNA

---

sequences. Whilst it is possible to use more complex distributions for each segment, doing so could complicate the evaluation of the generated sequences.
Evaluation Under our toy HMM setup, a generative model could make two types of mistakes within each segment: 1) Illeaal Start Token: The generated sequence starts with token which has zero emission probability. E.g. in Figure 2(b), the starting token could only be { T.. {G, C) the beginning of the sequence are classified as illegal start tokens. 2) Incorrect Transition: The generated sequence contains tokens with zero transition probability. E.g. in Figure 2(b) given the start of the sequence is (A,T,A, the next token must be A, any other tokens such as (7, G, C) are classified as incorrect transitions. We use the number of incorrect tokens as the metric for evaluation.
Experiment We naDNA [20, 24] as the repre- sentative autoregressive (AR) model. For the diffusion model, we develop simple latent Discrete Diffusion model termed DiscDiff. It resembles the design of StableDiffusion [28]. a latent diffusion model for image generation. DiscDiff consists of CNN-based Variational Encoder-decoder. trained with cross entropy, map the discrete DNA data into a latent space, and standard 2-D) UNet as the denoising network (detailed in appendix A). The training dataset consists of X' = (x(*)) 3 For detailed training procedures see Appendix B. generate 4,000 sequences from each model and present the evaluation results in Table 1. As expected, the diffusion model DiscDiff makes fewer errors regarding Illegal Start (IS) tokens but tends to generate more Incorrect Transition (IT) tokens. Conversely, while the AR model NyenaDNA generates some IS token errors, it produces significantly fewer IT token errors. This motivates the question: can we combine the strengths of both algorithms to achieve better sequence generation?
Hye-
Table I: Number of Incorret Tokens on Synthetic Dataset. The performance metrics used are the number of Illegal Start (IS) Tokens and Incorrect Transition (IT) Tokens. Note that there are total of 4, 000 x 256 1024, 000 tokens.
HYENADNA DISCDIFF
# IS TOKENS +
812
# IT TOKENS J
3,586
110,192
Method
Algorithm Absorb & Escape Algorithm
Algorithm 2 Fast Absorb & Escape Algorithm
Require: Pretrained AutoRegressive model
Require: Absorb Threshold T Ahaarbb Pretrained
Po pAR(xx (x) and pretrained Diffusion Model
AutoRegressive model p and pre-
DM (x)
trained Diffusion Model pg (x)
1: Initialize X DM
(x)
1: Initialize x P/ D. (x)
2: Set t =0
2: for i in (xn(X) do if DM < Tbbsorb then
3: Assume = {si, 82,...... ) where each
3:
-xx; Xi+1. .x,} is segment 4: while not converged do
4:
Absorb step:
5:
j=i+1
5: Sample segment S {si, S2..... ..,............................. )p) start index of Sk. j
X) ~ P0 AR ( (x0:s)
6: Set
end index of sæ
7:
while / (x)) > DM PM(x) do
8:
Increment =
7: Absorb step:
9:
X Pe All (x, Xo:ss Xij-1) //effine Inaccurate region of the sequence to-
8:
X
p( Np: X,+1:L)
pe (x (D:s-1) 1) //Refine segment Sy. us-
ken by token end while
ing the AR model
10:
9:
Escape step:
11:
Escape step:
10: X X,,y //Upatte X
12:
S;yj = Kn, //Update x
Increment
13:
Increment i
12:
end while
14:
end
Output: Final sample X with improved quality
15:
end for
16:
Output: with improved quality
5

---

4.1 The Absorb & Escape Framework
Given a pretrained AutoRegressive model p3 (x) and a Diffusion Model P3 DM (xx, we aim to generate a higher quality example & from the composed distribution pi, (x) P/ "x) P (x). However, directly computing p s (x) is generally intractable, as both the autoregressive factorizations from pAR and score functions from rrompei pPM are not directly composable [9, 10]. We propose the Absorb & Escape (A&E) framework, as shown in Algorithm
efficiently sample from pf (x))
Absorb-Step Inspired by Gibbs sampling [12), which iteratively refines each dimension of single sample and moves to higher density areas, our algorithm starts with sequence x" p pDM (x), generated by the diffusion model and then refines the samples through the Absorb step and Escape step. By exploiting the heterogeneous nature of the sequence, we assume that x" can be factorized into multiple segments {s1, $2,.... ,..,.... S. For each segment 5g, set and j as the start and end indices, respectively. During the Absorb step, we sample a subset of segments S C (2,, s, and refine each segment Sę by sampling K p(X::j No:i-1 1x Rs Pg R. (x:: Xua-1), using the autoregressive model to approximate the conditional probability.
Escape-Step After refining the segment in the Absorb step, we proceed with the Escape step where we update the refined segment X to X,y. This iterative process continues for each selected segment Sk, with / incrementing after each update. By leveraging the ability of the diffusion model to capture the overall data distribution and the autoregressive model to refine homogeneous sequences within each segment,
algorithm efficiently improves the quality of the generated samples. The final output X is hereby closer to the true data distribution p(x) compared to the initial sample x². A proof for the convergence in Proposition 1 is provided in Appendix C.
Proposition 1. The Absorb & Escape (A&E) algorithm converges to the target distribution po,s(x) Pe
Al (x) o p3 p (x), under the assumptions that both models are properly trained, the segments of
x are homogeneous, the subset of segments is chosen randomly, and the conditional distribution p(x,y Mooj|ooo-it Xj+1:L) is accurately approximated by p (xx: (0:--1).
4.2 Practical Implementation: Fast A&E
While A&E offers a method to sample from a compositional distribution, two practical issues remain unresolved. Firstly, the algorithm may take considerable amount of time to converge. Secondly, a crucial step in Line 3 of Algorithm involves splitting x into homogeneous segments S2, } and then sampling subset of these segments. Segmentation is straightforward when the boundaries of functional regions of the DNA sequence are known, such as protein-coding regions, exons, introns, where each region naturally forms homogeneous segment. However, this information often unavailable in practice.
To address these challenges, propose practical implementation termed Fast A&E. For generating a sequence x E N4. it requires at most L forward passes through the AR model. As shown in Algorithm 2 and Figure lb, Fast A&E adopts heuristic-basel approach to select segments for refinement. It scans the sequence from left to right, identifying low-quality segments through a thresholding mechanism. Tokens with predicted probabilities smaller than the Tabeorb threshold trigger the absorb action, while the autoregressive process terminates once the probability of a token generated by the AR model p DM (x)) is smaller than that of the diffusion model pAf (x;). In this manner, Fast A&E corrects errors made by the diffusion model with maximum running time of (TT + T AA), where TDm and TAR are the times required for generating a single sequence from the diffusion model and autoregressive model, respectively.
5 Experiment
5.1 Transcription Profile (TP) conditioned Promoter Design
We first evaluate Fast A&E in the task of TP-conditioned promoter design, following the same evaluation procedures and metrics used by DDSM [4] and Dirichlet Flow Matching (DFM) [32].

---

Data Format & Evaluation Metrie:
Table 2: Evaluation of transeription profile conditioned promoter sequence design. A&E achieves the smallest MSE with Language Model and DFM distilled being the AR and DM components.
Each data point in this task is represented as (DNA, signal) pair, where signal corresponds to the CAGE values for given DNA sequence, providing quanti- tative measure of gene expression around the transcription start site (TSS). Both the DNA sequence and the signal have length of 1024. The goal of this task is to generate DNA sequences conditioned on specified signals. During evaluation, for a given test set data point (2; c), the gen- erated sequence x and the ground truth sequence are processed through the ge- nomic neural network Sei [37]. The per- formance metric is the mean squared er- ror (MSE) between Sei(z) and (xi(
Method
MSE
Bit Diffusion (bit-eneoding)"
.0414
Bit Diffusion (one-hot encoding) D3PM-uniform*
.0355
.0375
DDSM
.0334
Language Model
.0333
Linear FM Dirichlet FM (DFM))
.0281
.0269
Dirichlet FM distilled (DFM distilled)*
.0278
A&E (Language Model+Dirichlet FM distilled)
.0222
As shown in Table 2, we ran Fast A&E on this task with default threshold of absorb = 0.85, using the pretrained model as both the autoregressive (AR) model and the denoising model (DM) component. The evaluation followed the same procedure as described in the DFM repository. The Fast A&E model, comprising AR and DM components (i.e., the language model and the distilled DFM checkpoints provided in the DFM repository), achieved state-of-the-art results with an MSE of 0.0262 the test split.
Results:
5.2
Multi-species Promoter Generation
5.2.1
Experimental Setup
Dataset Construction Prior efforts in DNA generation have been constrained by small, single-species datasets [19, 34]. To better evaluate the capability of various generative algorithms DNA generation,
we construct dataset with 15 species from the Eukaryotic Promoter Database (EPDnew) [23]. Table 3 compares our EPD dataset with those used previous studies, including DDSM[4], ExpGAN [38], and EnhancerDesign [33]. The key advantage of EPD is its diversity in both species types and DNA sequence types. Additionally, although the number of sequences in EPD is on a similar scale to that of DDSM, EPD offers greater uniqueness: each sequence corresponds to unique promoter-gene combination, a guarantee not provided by the other datasets.
Baseline Model We evaluate the state- of-the-art diffusion models for DNA sequence generation: DDSM [4], DNADiffusion [26], DDPM [2, and AR model Hyena [20, 24]. In ad- dition, we implement a VAE with a CNN-based encoder-decoder architec- ture. Adding UNet as the denoising network to VAE results in another baseline latent diffusion model termed DiscDiff. For fair evaluation, maximally scale up the denoising net- works of each diffusion model fit into an 40GB NVIDIA A100. Additionally, we adapt four pretrained Hyena models from Hugging- Face for comprehensive fine-tune -unint. The additional details of the network architectures are shown in Appendix D.
Table 3: A comparison of DNA generation datasets. EPD used in this work is significantly larger in size and contains fifteen species. Reg. represents the regulatory regions, and Prot. represents the protein encoding region.
Dataset
# DNA
Multi Species
DNA Regions
EPD (Ours)
160,000
Reg. & Prot.
DDSM [4]
100,000
Reg. & Prot. Reg.
ExpGAN [38]
4238
EnhancerDDesign [33]
7770
X
Reg.
Model Training All the models are implemented in Pytorch and trained on NVIDIA A100-PCIE- 40GB with a maximum wall time 48 GPU hours per model; most the models converged within the given time. Adam optimizer [7] is used together with the CosineAnnealingl... [22] scheduler. The learning rate of each model are detailed in Appendix For the evaluation of various diffusion models in unconditional generation (see Section 5.2.2), we sample 50,000 sequences from each model. For
7

---

the conditional generation across 15 species (see Section 5.3), we generate 4,000 sequences. In both cases, we use the DDPM sampler [31] with 1,000 sequential denoising steps.
5.2.2 Evaluating Diffusion Models on Mammalian Model Organisms
Table 4: Comparison of diffusion models on unconditional generation evaluated on EPD (256 bp) and EPD (2048 bp). Metrics include S-FID, COrYATA, and MSETNTA. The best and second-best scores are highlighted in bold and underlined, respectively.
EPD (256 bp)
EPD (2048 bp)
Model
S-FID
CorAAA t
MSETAIA &
S-FID 4
COTIAA
MSETAΤ
Random (Reference)
119.0
-0.241
8.21
106.0
0.030
1.86
Sample from Training Set
0.509
1.0
0.100
0.999
0
VAE
295.0
-0.167
26.5
250.0
0.007
9.40
BitDiffusion
405
0.058
5.29
100.0
0.066
5.91
D3PM (small)
97.4
0,0964
4.97
94.5
0.363
1,50
D3PM (large)
161.0
-0.208
4.75
224.0
0.307
8.49
DDSM (Time Dilation)
504.0
0.897
13.4
1113.0
0.839
2673.7
DiscDiff (Ours)
57.4
0.973
0.669
45.2
0.858
1.74
One prerequisite of Fast A&E (Algorithm 2) is that the diffusion model P should be properly trained and provide accurate approximations of underlying data distribution. We first evaluate existing Diffusion Models on subset of EPD datasets. This subset includes sequences from four mammalians H. Sapiens (human), Rattus Norvegicus (rat), Macaca mulatta, and Mus musculus (mouse), which collectively represent 50% of the total dataset. Training this subset allows for more precise assessment of the generative algorithm's 's accuracy in a unconditional generation setting.
Metrics
Motif Distribution Correlation (Cors) and Mean Square Error (MSFM): Corи the Pearson correlation between the motif distributions of generated and natural DNA sequences for motifs like TATA-box, GC-box, Initiator, and CCAAT-box. MSEM is the average squared differences between these motif distributions.
2. S-FID (Sei Fréchet Inception Distance): Mcasures the distance between distributions of generated and natural DNA sequences in latent space similar the FID metric [15] for images, replacing the encoder with the pre-trained genomic neural network, Sei [37].
The results are presented in Table 4, indicating that most existing models perform worse than the simple baseline DiscDiff proposed here, as measured by S-FID, COTTATA, and MSETATA- TATA is one of the most fundamental motifs for gene transcription a special type of protein called transcription factors binds to TATA tokens on the DNA as shown in Figure la. It changes the shape of DNA and then enables the gene transcription. The failure of existing diffusion models capture the TATA-box distribution indicates potential gap in existing research. In the following, we hereby use DiscDiff as the p DM to initialize the A&E algorithm.
5.3 Multi-species DNA Sequences Generation
We compare model Fast A&E with Hyena [24] and the best- pest-performing diffusion model from Section 5.2.2, DiscDiff. on the task of generating species-speciff DNA sequences.
Motif-centric Evaluation We consider four types of motifs closely related to promoter activities (TxTA- GC-box, Initiator, CCAAT-box].. We calculate types of motif' distributions for 15 species across 3 models, resulting in 180 frequency distributions.
Figure shows the average MSE and Correlation between generated and natural DNA distributions for each model and motif type across 15 species. Fast A&E improves upon Hyena and Disc generating the most realistic sequences across almost species and motif types. achieves the lowest MSE and highest Correlation across all four motifs. This pattern is consistent across all 15 species. The motif plots for all species are provided in Appendix As an example, Figure shows the motif distributions of sequences generated by the three models versus real DNA sequences

---

Comparison Squared Error (MSE)
Comparison Correlation
10a
discdiff
dlecdfff
Mean Squared Eiror (MSE)
hryena
hyynna
_eccpe
0.0
107
0.4
104
10
COAAT
TAIA
Inilator
GC DOX
CEAAT
TATA Box
Inttator
Figure 3: The average MSE and Correlation between generated and natural DNA distributions for each model and motif type across 15 species. Fast A&E outperforms Hyena and DiscDiff, generating the most realistic sequences with the lowest MSE and highest Correlation across four motif types. This pattern is consistent across all 15 species.
for macaque. Fast A&E closely resembles the natural motif distribution, especially for the TATA and GC box motifs, while Hyena and DiscDiff fail capture the values trends accurately.
Sequence Diversity To assess the diver- sity of the generated sequences, we applied BLASTN [18] to check (1) the similarity be- tween the training set (Natural DNA) and gen- erated sequences from three models, and (2)
75
Natural DNA
Lisceiff
Hyena
Abworl Excape 400
-100
-50
50
GC
rrequency (%)
the similarity within the generated sequences. BLASTN takes a query DNA sequence and compares it with a database of sequences, returning all the aligned sequences in the database that are similar the query. For each alignment, an alignment score, the align- ment length (AlignLen). and statistical signif- icance (e Value) are provided to indicate the quality of the alignment, where a larger align- ment score, a smaller statistical significance (e Value), and a longer alignment sequence (AlignLen) indicate higher similarity be- tween the query sequence and the database sequences. Ideally, when using generated sequences query the training dataset, good generative sequence should align better than a random sequence, but not replicate the sequences the training set.
100
50
100
TATA box
-100
-50
0
50
100
OCA4T box
2.5
0.0
-100
Relative Position to TSS 100
-50
Figure 4: Motif distributions in macaque DNA com- pared across natural DNA, FAST A&E, DiscDiff, and Hyena. FAST A&E closely aligns with natural DNA, especially for the TATA and GC motifs.
Table shows the results of the BLASTN algorithm. From the table, DiscDiff, Hyena, and A&E all satisfied the mentioned crite- ria. In terms of the diversity within the groups, none of the algorithms generated repetitive sequences. Furthermore, A&E tends to lie between Hyena and DiscDiff terms of the diversity of the generated se- quences, implying that A&E may combine the properties of AR and DM models. One notable fact is that the alignment scores are very high within the natural sequences, po- tentially indicating that natural sequences naturally have repetitive properties (conservative motifs), while the generative sequences do not have this characteristic.
Table 5: BLASTN Results
Query vs. Database
Scoret
eValue!
AlignLent
Random vs. Natural DNA
17.78
1.1769
24.2
A&E vs. Natural
21.39
0.1695
35,9
Hyena vs. Natural
22.89
0.2895
40.1
DiscDiff vs. Natural
20.25
0.2098
31.4
A&E vs. A&E Hyena Hyena
20.14
0.0968
33.69
19.57
0.0843
28.7
DiscDiff vs. DiscDiff
20.95
0.1029
37.6
Natural vs. Natural DNA
57.06
0.0633
77.6
Genome Integration with Promoter Sequences As shown in Figure 5, to evaluate the functional properties of sequences generated by Hyena, DiscDiff, and A&F, we inserted the generated promoter sequences of length 128 bp upstream (5') of three commonly studied genes in oncology: TP53,

---

Enformer
Upstream
Downstream context
Inserted Promoter
TP53/GGFR/AKT1
context
Target Gene
Transcription Profiles
Figure 5: Evaluation of Generated Promoters for gene regulation through Genome Integration EGER, and AKTI [6, 25,29], which are closely related to tumor activities. Our goal was to determine which model generates promoter sequences that produce gene expression levels closest to those of natural promoters when reinserted into the human genome.
We use Enformer [5] to predict transcription profiles. Enformer takes a DNA sequence of 200k bps as input and outputs matrix P E R$96x638 representing a multi-cell type transcription profile. We sampled 300 promoter sequences from each source: Natural DNA promoters, Hyena, DiscDiff, and A&E. For each set, calculated the average transcription profile across the sequences. The Sum of Squared Errors (SSE) between these average transcription profiles of the generated sequences and those natural promoters shown in Table 6. The results indicate that A&E produces the smallest SSE, suggesting it best captures the properties of natural DNA. This finding highlights the potential of generative algorithms create promoter sequences that effectively regulate gene expression, with applications in bioproduct manufacturing and gene therapy.
Table 6: Sum of Squared Errors (SSE) of Transcription Profiles between Real and Generated Sequences
TP53
EGFR
AKTIL
Random
278.18
8.09
65.70
A&E
17.21
0.28
1.65
Hyena
36,25
0.89
2.88
DiscDiff
124.03
2.17
25.50
5.3.1 Sensitivity Analysis of T Absorb
We perform sensitivity analysis of A&E algorithm over hyperparameter Tahaub- As shown in Figure 6, with small Tabsorb. the sequences generated by A&E are dominated by the diffusion model. As absorb increases, the AR helps to correct the errors made by the DM. Finally, when Tabsorb is larger than 0.7, the correlation flattens and fluctuates, In conclusion, A&E is robust under different values of T abborr, and it is best to use the validation dataset to choose the optimal value. However, wide range of T Tbb can still be used with improved performance.
Mutif Correiation by Absorb Threshald for Fruit Fly Initiutor
Modil Correlsion
8825
2922
3.50
2825
3.30
04 0.5 3.5 0.7
3.5
1.0
Absorb Threshold
Figure 6: Sensitivity of A&E under various Tahsorb. For each value of Tahsorb. 3000 sequences are generated and compared with natural DNA. The correlation between the generated sequences and natural DNA increases initially as T Tasseb increases, and then it flattens. An optimal Tabsorb can be selected based on the validation set, or default value of 0.85 can be used.
6 Conclusion
This paper demonstrates that (i) both the RegrRsgees (AR) model and Diffusion Models (DMs) fail accurately model DNA sequences due to the heterogeneous nature of DNA sequences when used separately, and (ii) this limitation can be overcome by introducing A&E. a novel sampling algorithm that combines AR models and DMs. Additionally, algorithm, Fast A&E, which enables efficient generation of realistic DNA sequences without the repetitive function evaluations required by conventional sampling algorithms. Experimental results across 15 species show that Fast A&E consistently outperforms single models in generating DNA sequences with functional and structural similarities to natural DNA, as evidenced by metrics such as Motif Distribution, Sequence Diversity, and Genome Integration. Regarding the future work, the generated DNA sequences still require validation through wet-lab experiments before they can be directly used in clinical settings.
developed a fast implementation the proposed
10

---

Acknowledgments and Disclosure of Funding
This research project has benefitted from the Microsoft Accelerate Foundation Models Research (AFMR) grant program. Zchui Li is grateful to his for their support.
Li, Huiqing Wang, and Yiqiu Sun
References
[1] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1-3, 2024.
[2] Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex Xijie Lu, Nicolo Fusi, Ava Pardis Amini, and Kevin K Yang. Protein generation with evolutionary diffusion: sequence is all you need. bioRxiv, pages 2023-09, 2023.
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 14:17981-1799 2021.
[4] Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. arXiv preprint arXiv:2305. 10699, 2023. [5] Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska- Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley, Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10): 196-119622 1203, 2021.
[6] Gillian Bethune, Drew Bethune, Neale Ridgway, and Zhaolin Xu. Epidermal growth factor receptor (egfr) in lung cancer: an overview and update. Journal of thoracic disease, 2(1):48, 2010.
[7] Kingma Da. method for stochastic optimization. arXiv preprint arXiv: 22800 2014.
[8] Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Reviile Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, et al. The nuclcotide transformer: Building and evaluating robust foundation models for human genomics. bioRxiv, pages 2023-01, 2023.
[9] Yilun Du Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. In International conference on machine learning, pages 8489-83 8489-8510. PMLR, 2023.
[10] Yilun Du and Leslie Kaelbling. Compositional generative modeling: A single model is not all you need. arXiv preprint arXiv:2402.01103, 2024.
[11] Gökcen Eraslan, Ziga Avsec, Julien Gagneur, and Fabian J Theis. Deep learning: new computa-
tional modelling techniques for genomics. Nature Reviews Genetics, 20(7):389-403, 2019.
[12] Alan E Gelfand. Gibbs sampling. Journal of the American statistical Association, 95445)):1300
1304, 2000.
[13] Ian Goodfellow, Jean Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil
[14] Anvita Gupta and James Zou. Feedback gan for dna optimizes protein functions. Nature Machine Intelligence, 1(2):105-111 2019.
[15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems. 30, 2017.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
[17] Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana Davuluri. Dnabert: pre-trained bidirectional
encoder representations from transformers model for dna-language in genome. Bioinformatics, 37(15):211222120, 2021.
[18] W James Kent. Blat-the the blast-like alignment tool. Genome research, 12(4))656-664, 2002.
[19] Nathan Killoran, Leo J Lee, Andrew Delong, David Duvenaud, and Brendan Frey. Generating and designing dna with deep generative models. arXiv preprint arXiv: 2017.
Avantika Lal, David Garfield, Tommaso Biancalani, and Gokcen Eraslan. regim: Designing realistic regulatory dna with autoregressive language models. bioRriv, pages 2024-02, 2024. [21] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- lm improves controllable text generation. Advances in Neural Information Processing
[20]
11

---

Systems, 35:4328-4433 4343, 2022.
[22] Ily Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv
preprint arXiv: 03983, 2016.
[23]
Patrick Meylan, René Dreos, Giovanna Ambrosini, Romain Groux, and Philipp Bucher. Epd in 2020: enhanced data visualization and extension to nemn promoters. Nucleic Acids Research, 48(D1):D65-DD) 2020.
[24] Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow Callum Birch- Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequenee modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024.
[25] Magali Olivier, Monica Hollstein, and Pierre Hainaut. Tp53 mutations in human cancers: origins, consequences, and clinical use. Cold Spring Harbor perspectives in biology. 2(1):a001 1008,
2010.
[26] Luca Pinello. Dna-diffusion: Leveraging generative models for controlling chromatin acces- sibility and gene expression via synthetic regulatory elements. In ICLR 2024 Workshop on Machine Learning for Genomics Explorations.
[27] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog. 188): 2019.
[28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF
conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[29] Marion Rudolph, Tobias Anzeneder, Anke Schulz, Georg Beckmann, Annette T Byrne, Michael Jeffers, Carol Pena, Oliver Politz, Karl Köchert, Richardus Vonk, et al. Akt1 e17k mutation profiling in breast cancer: prevalence, concurrent oncogenic alterations, and blood-based detection. BMC cancer, 16:1-12, 2016.
[30] Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265 PMLR, 2015.
[31] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint XrXiv:2000..02 020222, 2020.
[32] Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnic Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. arXiv preprint arXiv:2402. 2402054100, 2024.
[33] Ibrahim Taskiran. Katina I Spanier, Hannah Dickmänken, Niklas Kempynck, Alexandra Pančíková, Eren Can Ekşi, Gert Hulselmans, Joy N Ismail, Koen Theunis, Roel Vandepoel, et al. Cell type directed design of synthetic enhancers. Nature, pages 1-3, 2023.
[34] Ye Wang, Haochen Wang. Lei Wei, Shuailin Li, Liyang Liu, and Xiaowo Wang. Synthetic promoter design in escherichia coli based deep generative network. Nucleic Acids Research, (8(12):6403-6442 2020.
[35] Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas Milles, al. De novo design of protein structure and function with rfdiffusion. Nature, pages 1-3, 2023.
[36] Yizhe Zhang, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Joshua Susskind, and Navdeep Jaitly, Planner: Generating diversified paragraph via latent language diffusion model. Advances in Neural Information Processing Systems, 36, 2024.
[37] Jian Zhou and Olga G Trooyanskaya Predicting effects noncoding variants with deep learning- based sequence model. Nature methods. 12(10)9931994) 2015.
[38] Jan Zrimec, Xiaozhi Fu, Azam Sheikh Muhammad, Christos Skrekas, Vykintas Jauniskis, Nora K Speicher, Christoph S Börlin, Vilhelm Verendel, Morteza Haghir Chehreghani, Devdatt Dubhashi, et al. Controlling gene expression with deep generative design of regulatory dna. Nature communications, 13(1):5099, 2022.
12

---

Step
VAE Training
1D/Enoider
2D Encoder
20 Deccder
TD Decoder
A. P
#f (4.7,6, (4, cysuya
Step 2: Denoising Network Training
Specioa
Selt Altention
(optonal
E Gaussan Datt
Cross Atlantion
L-NN
Locked Decader
Figure 7: DiscDiff Model: A two-step process for DNA sequence generation. Step 1: VAE Training: sequence (A, G, C) encoded via a ID-Encoder to a 2D-Encoder. The latent space representation Z with parameters or is then decoded back to through 2D-Deccder and ID-Decoder. Step Denoising Network Training: The latent representation denoising network comprising ResNet Block, optional Self-Attention, and Cross Attention, with species and time information. The network outputs a Gaussian distribution N N(;; μl, ))) A U-Net architecture takes this distribution to produce various 2g representations, which Locked Decoder (fronzen parameters) used to generate the final DNA sequences.
processed through
A simple baseline latent diffusion model for discrete data: DiscDiff
A.1 The DiscDiff Model
We design DiscDifff a Latent Discrete Diffusion model for DNA generation tasks. As shown in Figure 7, this model is structured into two main components: Variationall-Ato--nnn a denoising model. The VAE consists of an encoder E S + Z, which maps discrete input sequence s to continuous latent variable and decoder D z S. which reverts z back to in the discrete space. The denoising model Ep (Zz, I) is trained to predict the added noise in the latent space.
(VAE) and
A.1.1 VAE Architecture
The choice of VAE architecture LDMs is critical and often domain-specific. We find that mapping the input data a higher dimension space can help to learn a better denoising network, generating more realistic DNA sequences. We hereby propose to use a two-stage VAE architecture as shown in Figure 7.
The first stage encoder Eg, N + KxX maps E N/- to a 2D latent space Zı E RKxM where K is the number of channels and M is the length of the latent representation. The second stage encoder EØ² RI*KXM
CCCK's x.M' first adds dummy dimension Zı such that Z1 E R KxM M and then maps to 3d latent space z E RC*K''M' where C' is the number of channels, K' and M' the reduced dimensions of K and M respectively. The decoder in the first and second stage are Dø1 and Dø2 respectively. Which are symmetric to the encoders. Overall, we have (s) Exg (Es, (s)), and the reconstruction is $ Dz(z) Doy (Do2(2))
A.1.2 VAE Loss
When training the VAE, we propose to use Cross Entropy (CE) as reconstruction loss. The loss function is given by:
13

---

Lo.ø = Ep(s)
(ao()
disr log po (s/|z)
1=
Reconstruction
B Pp(s) [KL(ga (z]s) [|AV p4, 2)1
KT. Divergence
where dig is the Kronecker delta, (s) is the probabilistic decoder output from Dg: (z|s) the probabilistic output from encoder Eg that represents the approximate posterior of the latent variable z given the input s; (Z μ., Σ) is the prior on z. Here we use a simple isotropic. a mixing hyperparameter. 3 is set to 10 in the experiments used in this paper.
A.1.3 Denoising Network Training
Once Dy and Eg are trained the first step, we train noise prediction Ey in the latent space (s) with Equation (6).
Ee [||s =0,
(6)
B Toy Experiment Training Details
We train both HyenaDNA and DiscDiff optimizer. For HyenaDNA, the learning rate is set to 0,0001, and we use the model -arradaa--arge-------
NVIDIA A100-PCIE-40GB using the Adam
for this task. The maximum number of epochs is set to 100, with early stopping enabled to facilitate early convergence.
For DiscDiff, the VAE is trained with learning rate of 0.0001, while the UNet is trained with learning rate 0.00005. DiscDiff is trained for 600 epoches; during the inference time, we use DDPM [16] sampler with 1000 denoising steps.
C Convergence Proof of the Absorb & Escape Algorithm
In this section, we provide a proof for the convergence of the Absorb & Escape (A&E) algorithm under certain assumptions. The convergence proof will demonstrate that the sequence generated by the A&E algorithm converges to sample from the target distribution (x).
C.1 Assumptions
We make the following assumptions for the convergence proof:
1. The autoregressive model AR (xx and the diffusion model p3 DM (x) are both properly trained and provide accurate approximations of the underlying data distribution.
2. The initial sample x p/3
DM (x) valid sample from the diffusion model.
3. The segments (si, S., of the sequence x are chosen such that each segment homogeneous.
4. The subset of segments S is chosen randomly and includes all segments over multiple iterations.
5. The conditional distribution p(x., 1o.- approximated by pa (xa(x (xix]|xxxi11 is accurate.
C.2 Proof
We aim to show that the A&E algorithm produces samples from the target distribution P (x). We do this by showing that the Markov chain defined by the A&E algorithm has px) as its stationary distribution.
14

---

Step 1: Initialization The imitial sample x" ~ (//M(x) is drawn from the diffusion model. This ensures that x2 is valid sample from p (x).
Step 2: Absorb Step For each segment sy in the subset 8, the Absorb step samples X/y from conditional distribution p(Xaz) (oos|Xo0-111 x,+1:1). Under the assumption that pa AAR (Xag) (
accurately approximates this conditional distribution, the refined segment Kig is valid sample from target conditional distribution.
Step 3: Escape Step The Escape step updates the segment K,y to Kgy- This ensures that the updated sequence x incorporates the refinement from the autoregressive model.
Step 4: Stationary Distribution To show that the Markov chain defined by the A&E algorithm converges to pb,s (x), we need to show that Pa.a (x) is the stationary distribution of this Markov chain.
The transition probability for the A&E algorithm is given by the product of the probabilities of the Absorb and Escape steps:
P(x xx) PAbsorb (x + x') Esscape (x' + x).
Given that pOM (x) captures the overall structure and p/ (x) refines the segments, the composed
distribution (x) is achieved by iteratively applying the Absorb and Escape steps. We need to show that the stationary distribution satisfies:
J
Using the detailed balance condition for the Markov chain: p (x) + x') px,'(x'')(('' x).
Given that the Markov chain is ergodic, the detailed balance condition implies that P stationary distribution.
px, (x) is the
Step 5: Ergodicity Ergodicity ensures that the Markov chain will visit all possible states given sufficient iterations. The random selection of segments S and the iterative updates in the A&E algorithm guarantee that all parts of the sequence refined over time.
Conclusion By satisfying the detailed balance condition and ergodicity, we have shown that the Markov chain defined by the A&E algorithm converges to the target distribution (x). Therefore, the A&E algorithm produces samples from the composed distribution (x, as the number of iterations approaches infinity.
D Experiment Details
Baselines
The details about the architecture and implementation of the baseline models are as
below:
We enhance the current DNADiffusion implementation for DNA syn- thesis, originally from the -Nf-Diifusion project", by expanding the models to encompass 380 million parameters. This network composed of Convolutional Neural Networks (CNNs), interspersed with layers of cross-attention and self-attention. The learning rate is set 0.0001.
DNADiffusion [26]:
We scale up the original implementation of the denoising network used for promoter design in 3 to what is the corresponding size of the network given 470 million parameters. It is convolution-basee architecture with dilated convolution layers. The learning rate is set to 0.00001.
DDSM [4]:
15

---

D3PM [3]: We take the implementation of D3PM for biological sequence generation from EvoDiff [2]', adopting the algorithm for DNA generation. We use the original implementa- tion of the denoising network, which has two versions: with sizes of 38M and 640M. We hereby have D3PM (small) and D3PM (big), respectively. The learning rate for both D3PM (small) and D3PM (large) are set to 0.0001.
Hyena [24]: We modify the RegLM [20]*, a existing work uses hyena for DNA generation. Four pretrained Hyena models of different sizes (hyenadna-large- Im--gg-----sqeenn hyenadna- medium- 160kOseglen, heynadna-small- -ml---m2--l----llllen and hyenaana-tiny- -inn-------ellen-de 128) are downloaded from HuggingFace" and used for full-size fine-tuning, we apply the fine-tuned models for generations on DeD.GeeDNA. The learning rate for fine-tuning is set to 0.0001.
DiscDiff:
2D-UNNt of 500 Million parameters are used as the denoising network. See Appendix A for the implementation details. The learning rate is set to 0.00005 for UNet training, 0.0001 for VAE training.
For the Fast A&B algorithm, we set the Tabsorb to 0.80.
E
Content List of Supplementary Code and Data
Our code folder includes the following sub-folders, within each folder there is readme file, detailing the steps to run the code. The below is the list of sub-folders.
E.1 Toy Experiment
Include the source code to reproduce Section 3. No external package is required to run the code except for python
E.2
DiseDiff
implementation of the discdiff baseline in Appendix A. Please follow the readme for running the code.
E.3
AbsorbEscape
This includes an implementation of the proposed algorithm in Section 4.2; however, it depends on the external AR models, please adopt accordingly to AR models which you want try.
E.4
EPD Data
We include the training dataset used for producing the main results, which includes 160K DNA sequences from EPD, each sequence has a length of 256 bp.
F
Motif Distributions for 15 species
We plot ToTA-box, GC content, Initiator, and CCAAT-box for 15 speces as below.
http:////gtttaoooce/ccccr//////s "tctpe://uugginngaccccoo http:///gggc..ccooooosssattttt
16

---

Baking Yeast
discdit!
hyana
1.5
COAA box
Frequency:
0.5
TATA BOX
GC box
50
100
200
Sequence Index
Figure 8: Baking Yeast
Chicken
TATA DOX
discdiff
absort__sccape
Frequency
40
CCART box
50
100
150
200
Sequence Index
Figure 9: Chicken
17

---

Chicken
discditt
hyena
absorby pecape
Frequency
40
SCAAT 2004
box
100
200
Sequence Index
Figure 10: Chicken
Corn
discdiff
natural
COMAT box
Frequency
GC box
20
TAXTA box
50
100
150
200
Sequence Index
Figure 11: Corn
18

---

GC box
hyena
natural
COATT box
Frequency
TATA BOX
0
60
30
100
200
Sequence Index
Figure 12: dog
Elegans
CCAAT DOL
absorb escape
natural
discdiff
Frequency/
2
DOX
50
100
150
200
Sequence Index
Figure 13: elegans
19

---

Fission Yeast
CCART DOX
discditt
hyena
0.5
absorb. peccaee
MOA VANAA
Frequency
200 39
0
60
100
200
Sequence Index
Figure 14: fission yeast
Fruit Fly
discdiff
absorb,
hyana
0.8
0.6
6C 800
0.4
Frequency
0.2
1.5
CCART
50
100
150
200
Sequence Index
Figure 15: fruit fly
20

---

Honey Bee
GC
natural
hyana
discditf
20
15
Frequency
10
OCKAT DOX
60
100
200
Sequence Index
Figure 16: honey bee
Human
disddiff
hyena
absorb__escoo
NOB VAVA
Frequency/
Invistor
CCAAT box
1.5
50
100
150
200
Sequence Index
Figure 17: human
21

---

Plasmodium
hyena
30
absorty secape
20
discdiff
TATTA BOX
Frequency
0.4
OCANT DOX
0.15
GC box
50
100
200
Sequence Index
Figure 18: plasmodium
Rat
CCATT DOX
natural
absorb,
hyyna
discditf
Tritator
Prequendy
TATA BoX
GC DOX
50
100
150
200
Sequence Index
Figure 19: rat
22

---

Thale Cress
CCART DON
hyena
discdilf
TATTA BO0
Frequency
0.6
100 C9
0-
50
100
200
Sequence Index
Figure 20: thale cress
Macaque
absorb escape
discdiff
Frequency
Investor
CCAAT box
50
100
150
200
Sequence Index
Figure 21: macaque
23

---

Zebrafish
200 COA
absorb. esccpe
natural
X00 VAYA
Frequency
1002 10
50
100
150
200
Sequence Index
Figure 22: zebrafish
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The sections related to the contributions are listed at the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We talked about the assumption required by the algorithm the Appendix C, we also talked about other limitations through the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification:
Can be found in Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: For the Toy Example experiment details see Appendix B, for the baseline and main result set up see Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
See Appendix for the instructions to access the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.c cc/
mublic/guides httee//iidee//oduuuboeee Sowps///seeuuuuiiiiieeeeeem wwwssssiooooooooooo oniolicyy ooiicylyyy))ol for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //wpps..cppuu..a///// ctcccc/pp.. wwtpccccpppccc.eee.ce.......ceet gttp///wiee//eeeeeeeeee httgg/gddede//euubiiiii
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 5.3 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For the main result which compares the motif distributions across 15 species, we compute the error bar. For some of the other tasks requiring heavy compute, we didn't repeat the experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix and Section 5.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp:/:///uui...//eeeee wttppppp.ccpppp/eateeeee httpppulcc/hhhcccceeeeeeee..eeee ctwuidelelllllleeccccccc ines?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Mainly in the introduction part
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the code repos The MIT License (MIT) eukaryotic promoter database
CC-BY 4.0
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See Appendix E
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No Crowdsourcing Involved
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No Crowdsourcing Involved
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "AfuuVJNCeQ", "reviewer": "Reviewer_crFN", "summary": "This paper presents a sampling approach called Absorb and Escape (A&E) that combines the strengths of diffusion models (DMs) and autoregressive models (AR models) for generating DNA sequences. The authors rightly point out that DNA sequences are generally composed of segments that do not follow the same distribution (i.e. they are heterogeneous sequences). Then, using a toy example, they clearly illustrate the shortcomings of DMs and AR models for modelling heterogeneous sequences but point out that they have complementary strengths, motivating them to combine these models. Their proposed A&E algorithm (and the practically useful Fast A&E algorithm) can be used at sampling time to leverage these complementary strengths - it first samples a sequence from a DM before iteratively refining it using an AR model. After training DMs and AR models on a new dataset of DNA sequences, the authors show that using Fast A&E leads to better samples when compared to using DMs or AR models alone. They also show that the sequences are sufficiently diverse with Fast A&E samples being less diverse than DM samples while being more diverse than AR model samples.", "questions": "In addition to the questions/suggestions above, I have the following minor ones below:\n- Since the absorb step is reliant on using an AR model to refine certain segments, is it better to use a masked language model instead of an AR model? This way, context from both sides of the segment can be incorporated and the probability approximation will be better.\n- How are probabilities for a sequence/sequence segment extracted from the DM?\n- There are discrepancies between the model names mentioned in lines 222-223 and those in Table 3.", "limitations": "Apart from the assumptions in Appendix C, I could not find any other discussion of the limitations. I can think of the following other limitations:\n- The evaluation scheme is based on a single dataset, it is unclear how A&E will generalize to other datasets/settings.\n- Tuning $T_{absorb}$ seems non-trivial.\n\nI do not foresee any negative societal impacts.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Originality: The main novel contribution of this paper is the A&E algorithm for combining the strengths of DMs and AR models. The algorithm is very well-motivated, both based on prior work and the toy example. It is also simple to implement and understand. Combining DMs and AR models for generating DNA sequences has also not been attempted by previous work to the best of my knowledge.\n- Quality: The toy example is very useful and clearly illustrates the problems with current generative approaches for DNA generation. However, I have reservations about the dataset and task being used for the main evaluation although I am generally convinced of the A&E algorithm's usefulness. My questions and suggestions are listed in the next section.\n- Clarity: The paper is very well-written and easy to understand. However, a few details about the evaluations are missing, I have listed them in the next section.\n- Significance: The A&E algorithm is likely going to be useful for many computational genomics researchers. Even beyond DNA generation, I think the algorithm is an interesting and intuitive way to blend DMs and AR models and it could lead to more research in this area.", "weaknesses": "### Reservations about the main evaluation\n- The authors use gene sequences from the Eukaryotic Promoter Database (EPD) for training and testing various models. I do not see any details about how exactly these sequences are constructed. From Table 2 and Figure 4, I assume that the authors are extracting sequences of various lengths (either 256bp for 2048bp) centered at the TSS, but the paper needs a more comprehensive description of the data processing since this data is used in all of the main experiments.\n\n- I am not convinced of the utility of modelling these DNA sequences as presented in the paper. The authors train conditional diffusion models where the conditions are various species. I cannot think of a use case for sequences generated using this modelling strategy - there is no way to tune the promoter strength or target gene. Other DNA generation models use arguably better tasks. For example, DNADiffusion uses diffusion models to generate cell-type-specific regulatory elements which could be useful for synthetic biology applications. I would like to understand the authors' motivation for using this task. \n\n- The paper could be made much stronger by using the same evaluations as the previous papers to show that the usage of A&E with the DMs proposed in those papers (maybe in combination with HyenaDNA for AR modelling) leads to improved performance.\n\n- Furthermore, the authors somewhat abandon their focus on the heterogeneity of the DNA sequences being modelled in the main results. Although it is clear from Figure 4 that the motifs are located in the promoter as expected, one could perform two more simple analyses to show that A&E actually helps in modelling heterogenous sequences:\n  - If half the training sequence is promoter-like and the other half is protein-like, the authors could show that this is true in the sampled sequences as well (maybe by BLASTing the two halves against the promoter-like and protein-like segments from the training set).\n  - An even simpler analysis would be to look at the protein-like segment of the samples and show that it obeys the rules of protein coding sequences - starts with a start codon and there are no premature stop codons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful reviews. Following your suggestions, we have presented additional experiments on DDSM dataset and more detailed description on the dataset in the general response and uploaded PDF. Hopefully our general reply addresses your concerns regarding the selection of $T_{absorb}$ and the extraction of $p^{DM}(\\mathbf{x}_i)$. Below, we provide detailed responses to each of your questions.\n\n### Q1. Using the same evaluations as previous papers\n \nWe have hereby presented additional experiment results on the *transcription profile conditioned promoter sequence design* task presented in DDSM paper with A\\&E. As shown in Table 6 of the uploaded PDF, A&E with the Language Model as the AR component and DFM distilled as the DM component achieves the smallest MSE of 0.0262, outperforming DFM and DDSM. This confirms the effectiveness of A&E across various tasks.\n \n|Method| MSE $\\downarrow$|\n|-------------------------------------|-------|\n|Bit Diffusion (bit-encoding)| 0.0414 |\n|Bit Diffusion (one-hot encoding)| 0.0395 |\n|D3PM-uniform | 0.0375 |\n|DDSM| 0.0334 |\n|Language Model| 0.0333 |\n|Linear FM | 0.0281 |\n|Dirichlet FM| 0.0269 |\n|Dirichlet FM distilled| 0.0278 |\n|**A\\&E** (Language Model+Dirichlet FM distilled)          |**0.0262**|\n\n### Q2:  Details about how exactly these sequences are constructed\nWe have detailed the construction process of the EPD dataset in the global response. Briefly, we downloaded all the promoter records (30 million) from EPD datasets. Each record is a triple of (sequence, species, cell types, expression levels). Sequences could be duplicated in these records, so we aggregated these records by sequence, producing 160K unique sequences (as mentioned in the line 218 of original paper). Each sequence is a promoter-downstream sequence centered around the transcription start site (TSS) of length 256 or 2048.\n\n### Q3: Why species-wise generation and not cell-type conditioning?\n \nThere are two reasons for setting species conditioning as a task. First, promoter sequences from different species follow different distributions, making it an excellent testbed for benchmarking various generative algorithms. In contrast, cell-type conditioning is less distinct since all cell types share the same genome, and regulatory elements can bind to specific cell types only under certain conditions. However, if enough samples are provided, each sequence will appear in all cell types. In theory, the optimal task format should be (species, cell type, expression) co-conditioning, and this paper presents the first step towards that goal.\n\nAdditionally, while we can train a single model for each species to generate promoters, limited data is available for certain species (e.g., P. falciparum in EPD), which is insufficient for training a generative model. By training on cross-species data, our model leverages genomic similarities across different species, enabling promoter sequence generation for species with limited data. One use case of unconditional promoter generation is expanding existing promoter libraries, crucial for synthetic biology. We expect future foundation models for genomics to make use of all available data, just like how LLMs are making use of the web-scale data, and being able to perform downstream tasks with contextual prompts.\n\n\n### Q4. Heterogeneity of the DNA sequences in the main results\n \nFollowing your suggestion, we performed BLASTN on the two halves of the sequences sampled by A&E against promoter-like and protein-like segments from the training set. A larger score indicates higher similarity. We will add these additional results to our Appendix.\n \n| Blasting Score     \t| Nature Promoter | Nature Protein |\n|------------------------|-----------------|----------------|\n| Absorb Escape Promoter | 19.4   \t|   18.88    \t|\n| Absorb Escape Protein  | 18.89       \t| 19         \t|\n \nThe results confirm the existence of heterogeneity in the generated and training sequences.\n\n### Q5. Is it better to use a masked language model instead of an AR model?\nWe recognize this as a valuable suggestion. A masked language model could potentially improve the quality of generated sequences by considering the bidirectional context $P(x_{i:j}|x_{0:i-1},x_{j+1:L})$. However, AR model with caches (storing intermediate results for previous generated tokens) could potentially be computationally efficient as it requires at most sequence length number of function evaluations. In contrast, a masked language model needs to consider the whole context each time.\n\n### Q6. How are probabilities for a sequence/sequence segment extracted from the DM?\n \n$p^{DM}(\\mathbf{x}_i)$ can be retrieved for most discrete diffusion algorithms (such as DDSM, DFM, DiscDiff). In DiscDiff, logits are available after the second stage decoder layer (Appendix A). In general,  $p^{DM}(\\mathbf{x})$ are usually stored as a variable **logits** $\\in  \\mathbb{R}^{L\\times4}$, where each row  $p^{DM}(\\mathbf{x}_i)$ represents probability distribution over \\{A,T,G,C\\} token.  More details are available in our general response.\n \n### Q7. Discrepancies between the model names mentioned in lines 222-223 and those in Table 3.\nWe thank the reviewer for pointing it out. This is a typo, the BitDiffusion in Table 3 should be *DNADiffusion*. We retrain the *DNADiffusion* on our dataset and reported the results in Table 3.\n \n### Q8. How A&E generalises to other datasets/settings and evaluations\nThis is addressed with the additional experiment and more clarification about metrics and datasets in the general response. We believe motif distribution is essential to DNA generation, as it directly measures how DNA functions.\n \n### Q9. Tuning $T_{absorb}$ seems non-trivial.\nWe have found a default value of 0.85 effective in many settings. A sensitivity analysis of the algorithm on  $T_{absorb}$  is provided in the general response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QqG007z66r", "reviewer": "Reviewer_Y7is", "summary": "The authors introduce a novel approach, called Absorb & Escape (A&E), for generating DNA sequences by combining the strengths of autoregressive (AR) and diffusion models (DMs). The authors argue that existing single-model approaches struggle with the heterogeneous nature of genomic sequences, which consist of multiple functionally distinct regions. Their method initially generates a sequencing using a DM, and then refines sequence segments using an AR model. They evaluate their approach on various design tasks. The authors claim improved performance over existing methods, particularly in generating sequences that satisfy complex constraints and exhibit properties similar to natural genomic sequences.", "questions": "Algorithm 2 assumes access to token-level probabilities from the diffusion model (p_DM), which is not a standard output for most diffusion models. The authors do not explain how they obtain these probabilities, which is a critical missing piece of information for understanding and implementing their method.\n\nThere is insufficient exploration of how the model learns to differentiate between different genomic regions (e.g., coding sequences, promoters, introns). Does the AR model produce different generations in these different regions? This analysis would strengthen the claim that the method effectively handles heterogeneous sequences.\n\nThe description of the Eukaryotic Promoter Database (EPD) content is vague. E.g. why would a promoter database contain coding sequence. A more specific explanation of the types of sequences it contains would help readers understand the dataset's composition and relevance.\n\nIn Algorithm 2 line 3, I cannot determine exactly what p^DM refers to. I.e. is this a global probability of the entire sequence or specific to a local region. It’s presence within the line 2 loop suggests locality.\n\nThe convergence criteria for Algorithm 1 are not specified, which makes it difficult to assess the algorithm.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The combination of AR and DM is innovative, and these approaches do seem to excel and struggle at different aspects of the problem.\n\nBiological sequence design is generally an interesting research area with potentially valuable applications.", "weaknesses": "The biological application lacks compelling justification. The authors do not clearly explain why generating synthetic promoter sequences is necessary or advantageous compared to using and optimizing existing genomic promoters. A comparison with established promoter optimization methods is missing, which is crucial for demonstrating the practical value of this approach in genomic applications.\n\nThe paper suggests broader applicability of the A&E method beyond DNA design, but this is not sufficiently supported by the presented experiments. Additional tasks and datasets would be necessary to make a convincing case for the method's general effectiveness.\n\nThe paper lacks an analysis of how the various hyperparameters, such as the Absorb Threshold (T_Absorb), affect the model's performance. A sensitivity analysis would provide valuable insights into the robustness of the method.\n\nThe use of Sum of Squared Errors (SSE) to evaluate generated promoters against natural promoters is questionable. Given that Enformer makes predictions for a large region of sequence (896 consecutive 128 bp bins), this metric may not accurately reflect the quality or functionality of the generated promoters, which cover 1-2 bins.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback on the paper.  In our general response, we included additional results comparing A&E with other methods on additional datasets, a sensitivity analysis of the hyperparameter $T_{absorb}$, an explanation of how to define token-level probability from the diffusion model, and other common questions proposed by reviewers. We suggest reviewing the global response first. Below, we provide detailed responses to each of your questions.\n\n### Q1. Sensitivity Analysis over $T_{absorb}$\n \nFigure 23 of the global response provides a sensitivity analysis of the A&E algorithm over  $T_{absorb}$. As  $T_{absorb}$  increases, the motif plot correlation between generated DNA and natural DNA first increases and then flattens. A larger  $T_{absorb}$  encourages more frequent corrections by the AR model, improving the quality of generated sequences. However, a smaller  $T_{absorb}$  is more computationally efficient as it requires fewer function evaluations. A&E is robust over a wide range of  $T_{absorb}$. While it is best to use the validation dataset to choose the optimal value, we found that a value of 0.85 is generally appropriate for different tasks and scenarios.\n \n \n### Q2. Additional Task and Dataset\nFollowing your suggestion, we have compared **A&E** with other SoTA discrete generation algorithm on the *transcription profile conditioned promoter sequence design task* from DDSM paper. As shown in Table 6 of the uploaded PDF, A\\&E with Language Model as the AR component and DFM distilled as the DM component achieves the smallest MSE of 0.0262, outperforming DFM and DDSM. This confirms the effectiveness of A&E across various tasks. Please see the global response for more details about the additional experiment.\n \n|Method| MSE $\\downarrow$|\n|-------------------------------------|-------|\n|Bit Diffusion (bit-encoding)| 0.0414 |\n|Bit Diffusion (one-hot encoding)| 0.0395 |\n|D3PM-uniform | 0.0375 |\n|DDSM| 0.0334 |\n|Language Model| 0.0333 |\n|Linear FM | 0.0281 |\n|Dirichlet FM| 0.0269 |\n|Dirichlet FM distilled| 0.0278 |\n|**A\\&E** (Language Model+Dirichlet FM distilled)          |**0.0262**|\n\n### Q3. Motivation for Promoter Generation Task\n \nSimilar to prior work (ExpGAN [1]), conventional promoter design methods based on existing promoter libraries span only a short section of the promoter. In contrast, deep generative models for promoter design explore the distribution properties of promoter sequences, enabling the generation of new promoters that classical methods cannot achieve. This motivation is shared by the DDSM and DFM papers and other DNA generation tasks.\n \n### Q4. How Enformer predictions are used for Evaluation\n \nThe Enformer outputs are used to show how the promoter sequence influences the downstream target gene instead of the bins corresponding to the promoter sequence itself. We inserted the generated promoter before the target gene, e.g., TP53, and checked how the promoter changed the expression level of the target gene. While the promoter sequence is only 128 bp, accounting for 1 bin in the prediction, the target gene is more than 10,000 bp long. For example, TP53 is about 25,000 bp, accounting for 195 bins in the Enformer output. Using SSE, we aggregate the change in expression levels across all the cell lines.\n \n### Q5. Token-Level Probabilities from the Diffusion Model ($p^{DM}$)\n \n$p^{DM}$ at line 3 of Algorithm 2  should be $p^{DM}(\\mathbf{x}_i) \\in  \\mathbb{R}^{L\\times4}$, representing the token level emission probability from the Diffusion Model.  $p^{DM}(\\mathbf{x}_i)$ can be retrieved for most of the discrete diffusion algorithms such as DDSM, DFM, and DiscDiff.  For DNA generation,  $p^{DM}(\\mathbf{x})$ are usually stored as a variable **logits** $\\in  \\mathbb{R}^{L\\times4}$, where each row  $p^{DM}(\\mathbf{x}_i)$ represents token level probability distribution over \\{A,T,G,C\\}.  $p^{DM}(\\mathbf{x})$ are produced after the last sampling steps. In the latent diffusion model used in this paper, it is produced by the second-stage decoder (Appendix A), while in DDSM and DFM, it is produced by a 1D-CNN (output_channel=4) in the score net.\n\n### Q6. Description of EPD Dataset\nWe detailed the construction process of the DPE dataset in the global response. Briefly, we downloaded all the promoter records (30 million) from EPD datasets. Each record is a triple of (sequence, species, cell types, expression levels). Sequences could be duplicated in these records, so we aggregated these records by sequence, producing 160K unique sequences. Each sequence is a promoter-downstream sequence centered around the TSS of length 256 or 2048.\n \n### Q7. How the Model Learns to Differentiate Between Different Genomic Regions\nFollowing the suggestion from reviewer crFN, we performed BLASTN on the two halves of sampled sequences against promoter-like and protein-like segments from the training set. A larger score indicates higher similarity. The results confirm that A&E better handles heterogeneity than the AR model.\n \n| Blasting Score | Nature Promoter | Nature Protein |\n|------------------------|-----------------|----------------|\n| A\\&E Promoter | 19.40|18.88|\n| A\\&E Protein  | 18.89| 19.00|\n \n \n| Blasting Score  | Nature Promoter | Nature Protein |\n|------------------------|-----------------|----------------|\n|Hyena Promoter | 18.95|18.98|\n|Hyena Protein  | 18.86|18.88|\n \n \n### Q8.  Convergence criteria for Algorithm 1\nThe convergence criteria of Alg 1 is $\\tilde{\\mathbf{x}}^t$ converge to a fixed distribution, which can be achieved through  $|\\tilde{\\mathbf{x}}^{t+1} - \\tilde{\\mathbf{x}}^t\\| < \\delta$.\n \n \n[1] Zrimec, J., et al., 2022. Controlling gene expression with deep generative design of regulatory DNA", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I0qSu6XUR1", "reviewer": "Reviewer_m7HE", "summary": "This paper addresses limitations in existing methods for generating genomic sequences, which struggle to capture the heterogeneous nature of DNA. The authors propose a new framework called Absorb & Escape (A&E) that combines the strengths of autoregressive (AR) models and diffusion models (DMs) to generate more realistic DNA sequences. They first analyze the shortcomings of AR models and DMs when used individually for heterogeneous sequence generation. Then they introduce A&E, which alternates between refining segments using an AR model (Absorb step) and updating the full sequence (Escape step). A practical implementation called Fast A&E is also presented. The authors evaluate their method on a large dataset of promoter sequences from 15 species. They compare Fast A&E against state-of-the-art baselines on metrics like motif distribution similarity, sequence diversity, and functional properties when inserted into genomes.", "questions": "Questions are integrated within Weaknesses (above).", "limitations": "While the method is very interesting, potentially bridging the limitations of autoregressive models and diffusion models, the evaluation is a bit on the weak side. This makes it difficult to assess any advance. Better evaluations could help improve this work to demonstrate the true gains. It would also be worthwile to explore absorb & escape using other SOTA diffusion models (eg. DFM).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Absorb & Escape is a plausible solution to addressing the limitations of autoregressive models and diffusion models\n- theoretical analysis of their method -- convergence proof\n- comparisons against previous diffusion models on promoter generation\n- comparison across multiple species DNA generation\n- deploys various metrics to assess generated sequences\n- new training data that considers multispecies", "weaknesses": "- If trained on original DDSM dataset, how would DiscDiff perform? I ask because the DDSM seems to perform quite poorly in terms of MSE in Table 3, but its performance was reasonable even when compared with Dirichlet Flow Matching (Stark et al, arxiv 2023).\n- The explanation of Autoregressive model struggling to capture difference between p1 and p2 with a single set of parameters could be true, but hierarchical AR models coUnuld learn latent segment states. So, unclear why deep AR models not learning the heterogeneous structure is challenging in practice. \n- Regulatory sequences can be quite diverged across species where a promoter in one species may not be a strong promoter in another species. So, what is the benefit of multi-species promoter pretraining? Perhaps comparing vs just on human promoters (i.e. DDSM dataset using CAGE-seq) could help appreciate the nuances. \n- In 5.1, under baseline models, DNADiffusion is cited but not explored in Table 3. HyenaDNA is not benchmarked in Table 3. It would be good to get a genomic language model baseline since the purpose of DiscDiff is to resolve issues of diffusion models and autoregressive models. \n- It's not clear how metrics were calculated. How were motif distributions calculated? ALso, what about MSE of the conditional generation of the model (using Sei as an oracle). THis was done in the DDSM paper as well as DFM. \n- Dirichlet flow matching performed better than DDSM and so it would be worth benchmarking on this (but may be outside scope for this paper). At least mention the new diffusion models that beat out DDSM and the open question remains whether they require absorb and escape or whether they can also benefit.\n- Fig 3 doesn't show any comparisons with other diffusion models (eg. DDSM)\n- Table 5 should show distribution of enformers predictions to see if generated distribution are even close to natural sequences. The presented metrics are difficult to assess beyond relative comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback. Following your suggestions, we have provided additional results comparing A&E with DDSM and DFM in the global response, along with clarifications to address common questions from other reviewers. Below, we provide detailed responses to each of your questions.\n\n### Q1. If trained on the original DDSM dataset, how would DiscDiff perform?\nWe believe the DDSM task is easier than unconditional and class-conditioning generation. The DDSM task uses a transcription profile, a real-value array of the same length as the DNA sequence, as the condition, allowing the generative model to learn a direct mapping from condition to sequence.\n \nThis is evident when applying the DDSM model to the unconditional generation task proposed in this paper during the exploration stage. Initially, using the original score net design from DDSM for unconditional generation, it could not learn the training set distribution. We scaled up the original score net by 10 times, which allowed DDSM to perform reasonably in unconditional sequence generation.\n \nDue to time constraints, we could not adapt the DiscDiff architecture to the DDSM task. However, we provide additional experiments with the A&E algorithm on the DDSM task by using the pretrained language model as the AR component and the DFM distilled model as the DM component. As shown below, A&E achieved a new state-of-the-art on this task.\n \n \n| Method                          \t| MSE $\\downarrow$  |\n|-------------------------------------|-------|\n| Bit Diffusion (bit-encoding)| 0.0414 |\n| Bit Diffusion (one-hot encoding)| 0.0395 |\n| D3PM-uniform | 0.0375 |\n| DDSM | 0.0334 |\n| Language Model | 0.0333 |\n| Linear FM| 0.0281 |\n| Dirichlet FM| 0.0269 |\n| Dirichlet FM distilled | 0.0278 |\n| **A\\&E** (Language Model+Dirichlet FM distilled)| **0.0262** |\n\n### Q2. Why deep AR models do not learn the heterogeneous structure\nAs detailed in line 120 section 3 of the original paper, given the assumption that a sequence consists of independent segments $seg_1$ and $seg_2$, AR models struggles to learn the independence between two elements $x_k \\in seg_1$ and $x_{2k} \\in seg_2$. This issue would be more challenging when the training data is insufficient. As AR models have intrinsic assumptions about token dependencies, overcoming this challenge could be difficult. \n\n### Q3.  Benefit of multi-species promoter pretraining and Additional Experiment on DDSM dataset\nWe can train a single model for each species for promoter generation. However, for certain species (e.g., P. falciparum in EPD), we have limited data, which is insufficient for training a generative model. By training on cross-species data, our model leverages genomic similarities across different species, enabling promoter sequence generation for species with limited data. We expect future foundation models for genomics to make use of all available data, just like how LLMs are making use of the web-scale data, and being able to perform downstream tasks with contextual prompts.\n\nSee the response to Q1 for the additional exp. results.\n\n### Q4. DNADiffusion is cited but not explored in Table 3\n \nWe thank the reviewer for pointing it out. This is a typo. We mislabeled DNADiffusion as BitDiffusion in Table 3. We ran the training code from the DNADiffusion repository and benchmarked it on the EPD datasets. The details of the experiments are shown in Appendix D.\n\n### Q5 a)  How motifs distribution were calculated\nWe use [EPDnew analysis tools - motif distribution](https://epd.expasy.org/epd/EPDnew_study.php) for plotting the motif distribution and get the motif distribution data. \n \n### Q5 b) How MSE is calculated in conditional generation\nWe clarify the definition of metric in the final section of global response. Note that Sei model is used for evaluating unconditional generation as Sei is for human genome, For conditional generation, the MSE is the MSE between the **motif distribution** of the generated sequence and nature sequence. As detailed in line 250 section 5.2.\n \n### Q6. Benchmarking against DFM\nSee the response to Q1.\n\n### Q7. Fig 3 doesn’t show any comparisons with other diffusion models (e.g., DDSM)\nOne reason is that training on the whole datasets is expensive. We first select the best-performing DM and AR components with unconditional generation in section 5.1 of the original paper. Secondly, section 5.2 demonstrates the effectiveness of the A&E algorithm by showing A&E outperforms the single model.\n \n### Q8. Distribution of enformers predictions\nWe are happy to include the single track expression level prediction results in the appendix of final draft, but due to the limitation of one page PDF, we don’t have additional space to include it here.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jEu5cikYvm", "reviewer": "Reviewer_hZwW", "summary": "The authors propose to generate DNA sequences using pre-trained DMs and then modifying the segments (randomly selected), through autoregressive models.", "questions": "- How did you train the model for DDSM in the case of multiple species? How did you incorporate species information?\n\n- How do the results look in the same dataset that DDSM and DFM used for evaluation, specifically in promoter design and enhancer design in a single species? It is unclear how the proposed DMs function initially in such scenarios.\n\n- What are the benefits of the proposed DMs and why can't one use previous pre-trained DMs, such as DDSM?\n\n- What is the effect of absorption and escape in unconditional generation?\n\n- How did you choose the length of segments in real-world and synthetic cases?\n\n- What is the effect of the segment selection?\n\n- Why one cannot use evolutionary based models for each segment? At least as a baseline.\n\n- Can you show couple of generated samples for each method and showing how the segments look like in real and generated data?", "limitations": "The main limitation of the paper is that it is very dependent to the pre-trained models, as well as selections of segments in a write manner.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The problem that the authors are trying to address, i.e. heterogeneity of the sequences due to existing multiple different element is valid question in drug discovery, specifically for very long DNA sequences.", "weaknesses": "The main weakness of the paper is that the authors didn't validate the proposed approach in real world long sequences, and the simulated data is not convincing. Apart from that it is not clear how they train the baselines for multiple tissues, and the fact that the authors generalized the claims about all the DMs but only consider a version that they proposed by themself is somehow making the claims not acceptable in some sense.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciated for your feedback. We have provided additional experiment results in response to your questions, this includes applying A\\&E algorithm on the Dirichlet Flow Matching in the new task and answers to common questions from other reviewers. In the following, we respond to your questions about this paper.\n \n### Q1. A&E performance on the DDSM task\nWe ran **A\\&E** on this task with a default threshold $T_{\\text{absorb}}=0.85$. The same dataloader and random seed as the [DFM repository](https://github.com/HannesStark/dirichlet-flow-matching?tab=readme-ov-file) were used. The **A\\&E** model comprises AR and DM components, specifically the **language model** and **distilled DFM** checkpoints provided in the DFM repository. As shown in the table below, **A\\&E** achieved state-of-the-art results of **0.0262** on the test split, despite using the second best DM, **distilled DFM**.\n \n| Method                          \t| MSE $\\downarrow$  |\n|-------------------------------------|-------|\n| Bit Diffusion (bit-encoding)\t| 0.0414 |\n| Bit Diffusion (one-hot encoding)| 0.0395 |\n| D3PM-uniform                \t| 0.0375 |\n| DDSM                        \t| 0.0334 |\n| Language Model              \t| 0.0333 |\n| Linear FM                   \t| 0.0281 |\n| Dirichlet FM                \t| 0.0269 |\n| Dirichlet FM distilled      \t| 0.0278 |\n| **A\\&E** (Language Model+Dirichlet FM distilled)      \t| **0.0262** |\n| |. |\n\n### Q2. Validate the proposed approach in real world long sequences\n \nThe evaluation in Section 5 is based on real-world DNA sequences from the EPD database. For details on the dataset construction process, please refer to the global response. The only simulated data used is in Section 3, which illustrates the limitations of AR models and DM in handling heterogeneous data. Additionally, we have provided results on the performance of A&E in the *transcription profile conditioned promoter sequence design task* from the DDSM paper (see global response and Answer to Q3 below).\n \n### Q3. How did you train the model for DDSM in the case of multiple species?\nWe did not train DDSM for multi-species conditioning. The DDSM is evaluated in an unconditional setting in Section 5.1, where the training data are promoter sequences from {human, rat, Macaca mulatta, mouse}.\n \n \n### Q4. The benefits of the proposed DMs and why can’t one use previous pre-trained DMs, such as DDSM?\n \nThe Latent Diffusion Model (LDM) proposed in this paper serves as a baseline model. We found that this simple baseline outperformed existing models such as DDSM in the unconditional generation scenario (shown in Table 3 of Section 5.2). Our aim is not to prove that the proposed baseline LDM outperforms other discrete DMs. Instead, the focus of this paper is to demonstrate the effectiveness of the proposed sampling algorithm A&E by showing that the composed model can outperform single models.\n \n### Q5. What is the effect of absorption and escape in unconditional generation?\n \nWe have included additional results below. As you can see, the A&E algorithm achieves the best performance in unconditional generation.\n|Model|EPD(256bp)|||EPD(2048bp)|||\n|-------------------------------------|--------------|----------|-----------|---------------|---------|-----------|\n|Model|S-FID↓|Cor_TATA↑|MSE_TATA↓|S-FID↓|Cor_TATA↑|MSE_TATA↓|\n|VAE|295.0|-0.167|26.5|250.0|0.007|9.40|\n|BitDiffusion|405|0.058|5.29|100.0|0.066|5.91|\n|D3PM(small)|_97.4_|0.0964|4.97|_94.5_|0.363|1.50|\n|D3PM(large)|161.0|-0.208|_4.75_|224.0|0.307|8.49|\n|DDSM(TimeDilation)|504.0|_0.897_|13.4|1113.0|_0.839_|2673.7|\n|DiscDiff(Ours)|57.4|0.973|0.669|45.2|0.858|_1.74_|\n|A&E(Ours)|**3.21**|**0.975**|**0.379**|**4.38**|**0.892**|**0.528**|\n \n### Q6 How did you choose the length of segments in real-world and synthetic cases?\nAlgorithm 1 requires segment annotations (start and end positions of each segment) to be available. However, this information is not available in most use cases. To overcome this issue, Algorithm 2 is proposed to run without knowing the true segment annotations, as detailed in line 193, Section 4.2. In both cases, the user does not need to choose the segment lengths.\n \nFor synthetic data used in the toy example, we already know the segment annotations by construction.\n \n### Q7 What is the effect of the segment selection.\nSimilar to Q6, both algorithm 1 and algorithm 2 don't need to select segment. For line 5 in alg1, the segments could be sampled randomly. \n\n\n### Q8 Why one cannot use evolutionary based models for each segment?\nIn practical DNA generation, where the user only has DNA sequences without annotations, Algorithm 2 handles this scenario. In addition, conventional promoter design methods based on existing promoter libraries span only a short section of the promoter.\n \n### Q9 Examples of how the segments look like in real and generated data\nThe examples of generated sequences are available from the following [Anonymous repo](https://anonymous.4open.science/r/dna_examples-3248/).  Note that no segment annotations are available for the training data; we apply Algorithm 2 for generation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciated the valuable feedback from all reviewers. Overall, the reviewers agree that our proposed algorithm, A&E, is well-motivated and clearly illustrates the limitations of AutoRegressive (AR) models and Diffusion Models (DMs). 

Most reviewers (crFN, m7HE, hZwW) recognize the significant real-world impact of the task of DNA generation. Both reviewers m7HE and crFN believe that the proposed method, A&E, is a plausible solution to bridge the limitations of existing AR models and DMs. Additionally, reviewer crFN suggests that A&E “could lead to more research beyond DNA generation.” While there are some concerns about the evaluation presented in the paper, the additional results below show that A&E achieves state-of-the-art performance on both existing DDSM datasets and the more challenging EPD datasets.
### Benchmarking A\&E against DFM and DDSM
Reviewers suggest benchmarking **A&E** against DDSM[1] and DFM[2] on the *transcription profile conditioned promoter sequence design task* used in DDSM paper. We ran **A&E** on this task with a default threshold $T_{absorb}=0.85$. The same evaluation procedure as the [DFM repository](https://github.com/HannesStark/dirichlet-flow-matching) were used. The A&E model comprises AR and DM components, specifically the **language model** and **distilled DFM** checkpoints provided in the DFM repository. As shown in the table below (this is also Table 6 in the uploaded PDF file), **A&E** achieved state-of-the-art results of **0.0262** on the test split. We are happy to include the results of A&E with different combinations of DM and AR components in the final draft of the paper.

|Method| MSE $\downarrow$|
|-------------------------------------|-------|
|Bit Diffusion (bit-encoding)| 0.0414 |
|Bit Diffusion (one-hot encoding)| 0.0395 |
|D3PM-uniform | 0.0375 |
|DDSM| 0.0334 |
|Language Model| 0.0333 |
|Linear FM | 0.0281 |
|Dirichlet FM| 0.0269 |
|Dirichlet FM distilled| 0.0278 |
|**A\&E** (Language Model+Dirichlet FM distilled)          |**0.0262**|
### Sensitivity analysis of $T_{Absorb}$
In response to reviewers Y7is and crFN's request, we include sensitivity analysis results to show the **influence of the hyperparameter $T_{\text{Absorb}}$** of the A&E algorithm in Figure 23 (uploaded as a one-page PDF).  
As $T_{\text{Absorb}}$ increases, the motif plot correlation between generated DNA and natural DNA first increases and then flattens. This is because a larger $T_{\text{Absorb}}$ encourages more frequent corrections made by the AR model, which generally improves the quality of generated sequences. However, a smaller value of $T_{\text{Absorb}}$ is more computationally efficient as it requires fewer function evaluations of the AR model. In practice, we found that a value of 0.85 is generally appropriate for different tasks and scenarios, and will add this sensitivity analysis to the appendix of our paper.
### Clarification about $p^{DM}(\mathbf{x}_i)$ from Diffusion Model
$p^{DM}$ at line 3 of Algorithm 2  should be $p^{DM}(\mathbf{x}_i) \in  \mathbb{R}^{L\times4}$, representing the token level emission probability from the Diffusion Model. To make it clearer, while $p^{DM}(\mathbf{x}_i)$ is difficult to obtain for continuous DMs, such as DDPM.  $p^{DM}(\mathbf{x}_i)$ can be retrieved for most of the discrete diffusion algorithms (such as DDSM, DFM, DiscDiff). For DNA generation,  $p^{DM}(\mathbf{x})$ are usually stored as a variable **logits** $\in  \mathbb{R}^{L\times4}$, where each row  $p^{DM}(\mathbf{x}_i)$ represents probability distribution over \{A,T,G,C\} token. A concrete example is DFM’s official implementation.
### EPD dataset construction process
**Issues with the DDSM dataset** While the DDSM dataset [1] has brought attention to the task of DNA generation, we want to caution against using it as a general benchmark. This is because, in the DDSM task formulation, the condition *CAGE expression value* is an array of real-valued signals with the same length as the sequence to be generated. This differs from most existing generation tasks, where the condition is typically a class. In fact, rather than a generation task, it is more akin to a machine translation task, borrowing a metaphor from NLP. This motivated us to develop a more challenging generation task using the EPD dataset.

**EPD Dataset** Figure 22 in the uploaded PDF illustrates the construction process of the EPD dataset. Promoter-downstream sequence pairs are curated from EPD. While the EPD database contains 30 million samples, we aggregated the sequences to avoid repetition. This differs from the DDSM dataset, where the same promoter could appear in different instances. Two sets of datasets were created: 1)EPD (256): A 256bp window centered on a gene’s transcription start site (TSS), split into an upstream promoter and a downstream segment. 2)EPD (2048): A 2048bp window centered on TSS, covering broader genetic regions.

**Metrics: Motif Distribution** Motif plots have been widely used in prior DNA generation works([3,4,5]). In this work, the motif plots are obtained using [EPDnew analysis tools](https://epd.expasy.org/ssa/oprof.php?series=epdnew&species=hg38). To improve upon prior work, we compute the MSE and the Correlation between two motif frequency distributions $y_1, y_2 \in \mathbb{R}^L$, with the following formulas to qualitatively evaluate the differences between two motif plots. The MSE evaluates the distance of motif distributions, while correlation evaluates the change in the shape of motif distributions.
  $\text{MSE} = \frac{1}{L} \sum_{i=1}^L (y_{1}^i - y_{2}^i)^2$
  
$r = \frac{\sum_{i=1}^L (y_{1}^i - \overline{y_1})(y_{2}^i - \overline{y_2})}{\sqrt{\sum_{i=1}^L (y_{1}^i - \overline{y_1})^2 \sum_{i=1}^L (y_{2}^i - \overline{y_2})^2}}$

[1] Avdeyev, P., et al. DDSM [2] Stark, H., et al. DFM [3] Wang, Y., et al. promoter design [4] Taskiran, I.I., et al. synthetic enhancers. [5] Zrimec, J., et al. ExpGAN

Meta-review: The paper shed light on important weaknesses of AR and DM-based models for genomic sequence design and proposes an intuitive and promising solution combining both models. 

The authors have provided convincing additional experiments and clarification that alleviate initial concerns on the evaluation and the methodology.  We strongly urge them to incorporate these into the manuscript as these significantly strengthen the quality of the submission.

In particular it is critical to add the benchmarking experiments, sensitivity analysis, as well as the insightful comments on the pertinence of promoter generation, and the rationale for considering the EPD dataset.
Decision: Accept (poster)