Language Models for Controllable DNA Sequence Design
Authors: Xingyu Su
Xiner Li
Yuchao Lin
Ziqian Xie
Degui Zhi
Shuiwang Ji
Venue: Accepted by TMLR ()

Abstract: We consider controllable DNA sequence design, where sequences are generated by conditioning on specific biological properties. While language models (LMs) such as GPT and BERT have achieved remarkable success in natural language generation, their application to DNA sequence generation remains largely underexplored. In this work, we introduce ATGC-Gen, an Automated Transformer Generator for Controllable Generation, which leverages cross-modal encoding to integrate diverse biological signals. ATGC-Gen is instantiated with both decoder-only and encoder-only transformer architectures, allowing flexible training and generation under either autoregressive or masked recovery objectives. We evaluate ATGC-Gen on representative tasks including promoter and enhancer sequence design, and further introduce a new dataset based on ChIP-Seq experiments for modeling protein binding specificity. Our experiments demonstrate that ATGC-Gen can generate fluent, diverse, and biologically relevant sequences aligned with the desired properties. Compared to prior methods, our model achieves notable improvements in controllability and functional relevance, highlighting the potential of language models in advancing programmable genomic design.

Shuiwang Ji Published in Transactions on Machine Learning Research (12/2025)
Language Models for Controllable DNA Sequence Design
Xingyu Su
aingyu. Claraa edu
Texas A&M University
Xiner Li Tezas ABM University
Ize@tama edu
Yuchao Lin
kruskallón Glama edu
Tezas A&M University
Ziqian Xie
ziqian.rie wiw.ua..iie@uu..mm..... edu
University of Texas Health Science Center Houston
Dogui Zhi
degui. zhi Buth. Inc. eduu
University of Texos Health Science Center Houston
sji Gta@ma edu
Texas A&M University
Reviewed OpenReview: https: https://openreviee
openreview. net/ forum? id= twmoBu605
Abstract
We consider controllable DNA sequence design, where sequences are generated by condition- ing on specific biological propertics. While language models (LMs) such as GPT and BERT have achieved remarkable success in natural language generation, their application DNA sequence generation remains largely underexplored. Tn this work, introduce ATGC-Gen, an Automated Transformer Generator for Controllable Generation, which leverages cross- modal encoding to integrate diverse biological signals. ATGC-Gen is instantiated with both decoder-only and encoder-only transformer architectures, allowing flexible training and gen- eration under either autoregressive or masked recovery objectives. We evaluate ATGC-Gen on representative lasks including promoter and enhancer sequence design, and further intro- duce new dataset based on ChIP-Seq experiments for modeling protein binding specificity. Our experiments demonstrate that ITGC-Gen can generate fluent, diverse, and biologically relevant sequences aligned with the desired properties. Compared to prior methods, our model achieves notable improvements in controllability and functional relevance, highlight- ing the potential of language models in advancing programmable genomic design. The source Gen)
code is relcased at http://ht/...ooeeee/////////////
1 Introduction
DNA sequence design is transformative scientific endeavor that revolutionizes our understanding of biology and catalyzes major advances across healthcare, agriculture, and environmental conservation, elc (Zrimec et al. 2022: Killoran et al. 2017). In these tasks, it is commonly expected that the generated DNA sequence achieves some biological outcomes (Uehara et al. 2025 Li et al. 2025). Thus, controllable generation is of practical significance (Nguyen al. (0224a) in which the sequence generation is guided by various properties, e.g-, binding to specific proteins exhibiting particular transcription activation likelihoods.
Recent advances in generative models have triggered widespread interests in DNA sequence design using diffusion (Avdeyev et al. 2023 Li et al. 2024 Sarkar et al. 2024) and flow matching (Stark et al. 2024
*Corresponding author

---

Published in Transactions on Machine Learning Research (12/2025)
generative methods. These methods have shown promise, particularly in modeling global structure and optimizing over continuous latent spaces. However, they are not inherently tailored to discrete, symbolic sequence generation properties that are central to DNA sequences. While those methods are in general effective in gencration task, they are not naturally designed for generating discrete and variable-length sequences. In contrast, Language models (LMs) offer an alternative perspective: they are naturally suited for discrete, variable-length generation and have achieved remarkable success in analogous domains like natural language. In this work, we explore the potential of LMs for controllable DNA sequence design, proposing them complementary approach to existing generative methods.
In this paper, we aim to explore the use of transformer-baseed language models for DNA sequence design conditioned on specific biological properties (Vaswani et al. 2017; Stark et al. 2024 Sarkar et al. 2024 Su et al. 2025 To this end, we develop ATGC-Gen, an Automated Transformer Generator for Controllable Generation. We instantiate ATGC-Gen with both decoder-only (e.g., GPT) and encoder-only (e.g., BERT) transformer architectures to examine their respective capabilities for controllable generation. It is designed to encode and integrate helerogeneous biological information, such as cell lypes, protein sequences, and transcription activation signals, into the sequence generation process. This unified framework enables ATGC- Gen to capture complex relationships between DNA sequences and their biological contexts, facilitating the generation of sequences with specific desirable properties.
To evaluate the effectiveness of our proposed approach, we apply ATGC-Gen, a transformer-basee language model that incorporates cross-modal biological information for controllable DNA sequence generation. By conditioning on diverse properties such as cell types, transcription factors, and regulatory signals, ATGC-Gen enables flexible and biologically grounded sequence design. We first assess its performance established promoter (Avdeyev et al., 2023 Stark et al. 2024 Sarkar et al. 2024 and enhancer (Taskiran et al. 2024) generation tasks. To further explore its capability in handling complex biological contexts, we introduce new dataset based on ChiP-Seq experiments, which involves generating sequences that bind to specific proteins in specific coll types. Experimental results across different tasks, evaluated in terms functionality, fluency, and diversily, demonstrate that ATGC-Gen achieves strong and consistent performance, highlighting the promise of language models for advancing controllable and property-aware DNA sequence design.
In summary, our main contributions are follows:
We propose ATGC-Gen, a language model framework for controllable DNA sequence generation. ATGC- Gen supports flexible conditioning on diverse biological modalities, enabling biologically meaningful se- quence design under complex control constraints.
We introduce a new dataset for controllable DNA generation based on ChIP-Seq experiments, which captures protein-DNA binding patterns. The benchmark includes well-structured evaluation metries assess functionality, fluency, and diversity.
We conduct extensive experiments on promoter, enhancer, and ChIP-Seq-basee tasks. Results demon- strate that ATGC- outperforms strong baselines in generating accurale, coherent, and diverse se- quences under various biological conditions.
2 Background and Related Work
DNA Generation. We study the problem of DNA sequence generation. In the field of synthetic biology and genetic engineering, unconditional or random DNA generation has limited applications, since DNA sequences are designed edited to achieve specific outcomes, e.g., producing protein or altering metabolic pathway. An unconditionally generated DNA sequence lacks the necessary context to be meaningful useful in biological systems. Hence, in this work, we focus on studying the task of DNA generation given specific properties, such as a particular organism, cell lype, or functional requirement. Formally, we define a DNA dataset as U = S1), (U2, S2), ... (UN, SN)}, where U₁ is a DNA sequence and S, is its corresponding biological property. We aim to learn a conditional generative model po(-(S,) on U so that the model Fa. parameterized with 6, can generate DNA sequences fulfilling S, Previous studies have made several initial attempts to apply deep learning hoth for controllable DNA generation. The Dirichlet Diffusion Score
2

---

Published in Transactions on Machine Learning Research (12/2025)
Model (DDSM) (Avdeyyy et al., 2023 uses Dirichlet distribution to discretize the diffusion process, effectively generating DNA sequences. In Stark et al.
(2024) the Dirichlet distribution is used in the flow matching process, enhancing the quality of generated DNA sequences. DiscDiff (Li et al. 2024) employs Latent Discrete Diffusion model on the DNA gencration task and proposes a post-training refinement algorithm to improve the generation quality. D3 (Sarkar et al. 2024 uses score entropy for discrete diffusion on the
conditional DNA sequence generation.
DNA Language Model. Recent rescarch has demonstrated the power of language models for modeling
biomolecular discrete sequences (Wang et al., 2024 Sgarbossa et al. 2025 Xu et al. 2024 which is ideally suited for modeling DNA sequences. A few recent studics have employed language models for encoding DNA sequences in prediction or representation learning tasks. Early works adopt Transformers to encode DNA sequence data, and notable examples include DNABERT, DNABERT-2 and Nucleotide Transformer (Ji 2023 More recently, state space models such as Hye- naDNA Nguyen et al. 2024b) and Mamba (Gu & Dao (023) have been applied long DNA sequences, with
ct al. 2021 Zhou al. 2023b Dalla-Torre et al.
Caduceus [Schill et al. 2024) further improving Mamnb with bi-directional encoding and reverse complement awareness. Evo (Nguyen et al.) 2024a] extends this line of work by pretraining DNA, RNA, and protein modalities. In parallel, Bio-XISTM (Schmidinger et al. 2024 explores xLSTM architectures for biological sequences, which, similar to state space models, achieve a linear runtime dependency on sequence length and provide efficient sequence modeling. These approaches primarily propose efficient modeling architectures for biological sequences and have achieved strong results in prediction benchmarks. While powerful for encod- ing and prediction, these works do not directly address explicit controllable sequence design. In practice, although uni-modal models can achieve limited coltaollayi by conditioning sequences with desirable propertics, generating condition-specific DNA sequences typically requires ro-training fine-tuning a sepa- rale model for each condition. By contrast, our framework integrates heterogeneous biological properties conditioning inputs, providing a unified and scalable way to support diverse design criteria.
CrdaltMo Reproscntation Learning in Biology. Many works combine heterogencous modalities in latent spaces biomolecular modeling. For example, some works (Schimunek el. al. 2023 Fifty el al., 2023 Zhou et al. 2025) fuse molecular structure with other modality information, achieving strong perfor- mance molecular property prediction benchmarks. Other efforts integrate different modalities of protein information to improve representation learning (Gligorijević et al., 2021 Brandes et al. 2022 Hayes et al. 2025). However, these approaches focus primarily on prediction tasks rather than sequence generation, and the modalities considered are typically molecular or protein structure and activity labels. In contrast, our work extends cross-modal conditioning to the setting of controllable DNA sequence generation, where di- verse biological attributes are incorporated as conditioning signals to guide the design of functional DNA sequences.
3 The Proposed ATGC-Gen
In this section, we introduce the overall framework of ATGC-Gen for controllable DNA sequence generation, as illustrated in Figure [ The framework encodes various biological property modalities and employs a language model to generale DNA sequences that aligned with specified properties. We begin by describing how the property encoder is used to transform biological property inputs into dense representations. Next, we explain how these property representations are integrated with the DNA sequence embeddings to condition the language model. Finally, we present the training objective used to optimize the model.
3.1 Representation Encoding
In this section, we illustrate encoding method for target propertics from different modalitics.
Sequencc-level Integration. Sequenco-level integration provides a general mochanism for incorporating biological properties in a global form. This approach encodes the property as a set of global representations that summarize the desired biological context- such as cell types or pooled statisties.
3

---

Published in Transactions on Machine Learning Research (12/2025)
DNA-Protein Binding
DNA Transcription Likelihood
Various Cell Types
(a) DNA Datasets based on Various Properties
TCCGTAGC...
AATCGG...
TCCTCTAGCTATCG... CCTTCAAGCTATCO...TTTGACCCOO CTATGGATCOGC
Biological Property
DNA Sequence
DNA Generation
(b) Generation Model Architecture
Cross Attention
H,
AGCCCGTATCG... TCGTAACCGGC.. AAACAAGGATG...
Property Encoder 9p
Language
Model
Figure 1: Overview of the proposed ATGC-Gen framework. (a) Different biological properties used for generating DNA sequences. (b) The architecture of the generation model.
Given a property matrix E R4x4, where lp is the number global property tokens and dp is the input feature dimension, we transform into the model's hidden space using a learnable linear projection H, PW + bp, where W, and by are learnable paramcters.
Let the DNA embedding sequence be H. = (be.1.... ham] E Raxd, where each h is obtained by mapping the i-th DNA token through a learnable embodding layer that projects discrete nucleotides into a dy-dimensional continuous space. We prepend the property embedding to the DNA sequence to form the final transformer input:
Hipput (Hp:H.) He) E (((xt))xdn.
(1)
This design allows the transformer treat the property tokens as global control inputs, enabling them to influence generation via self-attention across the full input sequence.
Feature-level Integration. Feature-level integration can be applied when the property sequence is aligned position-wise with the DNA sequence, i.e., both have the same length. This setting applicable to base pair resolution properties, such as transcriptional signals defined at each nucleotide position.
Let the DNA sequence be represented a token sequence representing the specific nucleotide. We convert it to one-hot embeddings X, (Xe,1,"' Xea), where Xe, E R is the one-hot encoding of ug Let the aligned property sequence be S = {si, (.) with each S; E I
{w,,-.. un where u, (A,C,G,T)
For each position i, we concatenate the DNA one-hot and property vectors:
x concat((x:;....... (x., s;) E (4+d,)xx
We then apply a shared linear transformation to map the concatenated vector to the model's hidden dimen- sion:
he. = Wy + by,
(2)
where Wy 1d and by E Rd, The resulting transformer input is
H Himput (he.).... hen)) xadd
Feature-level integration enables the model condition on fino-grained biological signals each position in the DNA sequence.
4

---

Published in Transactions on Machine Learning Research (12/2025)
3.2 Training Objectives
We consider two alternative training paradigms for controllable DNA sequence generation: autoregressive training objective based decoder-only transformers (e.g,, GPTT) and masked language model objective based on encoder-only transformers (o.g., BERT). Both approaches condition on external biological properties as described in the previous section.
Autoregressive Training. In this setting, train decoder-only language model Fg to generate DNA sequences autoregressieely To unify notation, let X; denote the model input at step i: (i) under sequence- level integration, X; = {Hp; ......, .......,u u.y, where the prepended property tokens act as a global prefix; (ii) under feature-level integration, X, = (he.1...., he, where he.j defined in equation
Following standard practice [Zhou et al. 2023b Schiff et al. 2024 Radford al. 2018 we use a single- nucleotide tokenizer, where each base (A, C, G, T) treated as discrete token. 'hhe model is trained using the next-token prediction objective with enssopy loss
1--1 Σ
((Fo(Xi), unn))
min Evore
(4)
where U = fuu,.... d is a DNA sequence from the dataset U and é is the cross-entropy loss over the predicted mucleotide distribution.
This autoregressive formulation naturally supports variable-lnngth sequence generation, making it suitable for DNA design tasks with flexible output lengths.
Masked Language Modeling. In parallel, we consider an aucoder-based training approach using masked language modeling (MLM), similar to BERT (Devlin et al. 2019). This formulation requires sequences of fixed length and trained to recover randomly masked tokens in the input. Specifically, given an input embedding sequence Hipput (constructed from both property and DNA tokens described in Section 3.1 randomly select a subset of positions M C {I,-..,, n) to mask, and train the model F& to reconstruct the original nucleotides at those positions:
Σ e (Fo (Himput), ); .i)) LiEM
min Eu-u
(5)
where Hinput is the full sequence embedding (property and DNA sequence with some tokens masked), and Fa(+); denotes the model prediction at position i.
Although the masked language model objective requires fixed-length sequences and not naturally suited for generation, enables efficient parallel training and leverages bidirectional context, making it expressive and effective in some certain cases.
3.3 Controllable Generation
Given trained model and specified biological properties, perform controllable DNA sequence generation by conditioning on the encoded property representations described earlier. We support two generation strategies, corresponding to the two training objectives: autoregressive decoding with a decoder-only (GPT- style) model, and masked language modeling with an encoder-only (BERT-style) model.
Autoregressive Generation. In the GPT-style setup, generation proceeds from left to right. At each time step i, the model predicts the next nucleotide u, conditioned on the property representation H, and previously generated tokens U:-1
po H ". .. (.-1))
5

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: Examples of the ChIP-Seq generation dataset.
Chrom
ChromStart
ChromEnd
Transcription Factor (TF)
Cell Type
Score
chrl
26677454
26677790
TCF7
K562
405
chr4
64158376
64159457
CTCF
mcdulloblastona
1000
chr2 190070290
190070667
RAD21
GM12878
1000
chr11 86800296
86800780
MXT1
SK-N-SH
277
We initialize generation with special start token and stop when designated end-of-sequence token is generated or maximum sequence length is reached.
To promote diversity and controllability, we sample tokens using temperature sampling (Ackley et al. 1985
Ficler & Goldberg 2017).
Masked Recovery Generation. For the BERT-style model, generation is performed by iterative parallel masked token prediction. We initialize the entire sequence with masked tokens: {[MASK], [MASK]] and jointly condition on the property representation Hp. The model then predicts the nucleotide for each masked position randomly.
There are two generation modes this setting:
One-shot unmasking: all positions are predicted in parallel from fully masked input in single forward pass.
Iterative unmasking: a subset of masked positions is predicted at each step, and the predictions are fed back in; the remaining positions stay masked and are updated progressively.
This formulation enables parallel generation and full bidirectional conditioning, but requires fixing the se- quence length advance.
Remark. This formulation shares structural similarities with recent masked denoising approaches, such masked discrete diffusion (Sahoo al. 2024 Nie et al. 2025). where corrupted input is progressively reconstructed through itcrative masked token prediction.
4 Controllable DNA Generation with ChiP-Seq
While existing DNA datasets are sufficient to benchmark the DNA generation tasks, they do not capture the variable-length nature of many real biological sequences. We therefore curate variable-length dataset highlight the adaptability of our framework more realistic and flexible generation scenarios. Therefore, we develop a dataset derived from ChIP-Seq experiments for evaluating controllable DNA generation problems. ChIP-Seq experiments aim to identify the binding sites of DNA-associated proteins on the genome, specif- ically the DNA sequences where proteins physically attach or bind. In our setting, we focus on predicting potential binding DNA sequences of variable lengths based on given protein sequences across different cell types. This prediction can advocate our understanding of the complex networks of gene regulation and aid in designing DNA sequences with specific functionalities for genetic engineering.
Our proposed dataset is obtainod from comprehensive collection of ChIP-Soq experiments generated by the ENCODE project (Consortium et. al., 2012),, which identifies specific DNA sequences to which proteins bind. We filter the raw data by considering the highest binding scores, length properties and other criteria, and detailed steps are provided in the Appendix A
Dataset Description. Table I shows several example rows of the ChIP-Seq generation dataset. The raw dataset contains approximately
million rows, with each row describing a binding relationship between DNA sequence and a protein for specific cell type(s). This dataset contains 340 transcription factors (proteins) and 129 cell types. In addition to the (DNA, protein, cell type) triplets, each row presents the binding scores to indicate the binding strength.

---

Published in Transactions on Machine Learning Research (12/2025)
Protein and Cell Type Property
Generated DNA Sequence
21,907 Chromatin Profile
Caco-2, CTCF ENCODE
Fetal Heart, DNaase Roadmap
Y
Generative Model
Chromatin Profile Prediction
Figure 2: Illustration of the ChIP-Seq generation task. The objective is to generate DNA sequences based on specific proteins and cell types. The functionality of the generated DNA sequences is evaluated using chromatin profile prediction.
The DNA sequences are from GRCh38 (HG38) reference human genome assembly, used for mapping and aligning genetic data. Each base pair in the DNA sequence is one of the four muclcotids:: adenine (A), cylosine (C), guanine (G), thymine (T). Given the protein name, we can query protein databases (Berman et al. 2000 to find the corresponding protein sequences and structures.
Task and Evaluation Metrics. The controlled generation task involves generating DNA sequences basod on a given cell type and binding protein. Figure 2 illustrates the generation task. It is important to note that for the DNA-protein binding, single protein may bind with different DNA sequences, and the same DNA sequence can bind to different proteins.
We propose three quantitative metrics to evaluate the generated sequences.
This metric measures how well the generated DNA sequence binds with the given protein under a specific cell type. We use the SEI framework (Chen et al. 2022), which employs a deep learning model to predict 21,907 chromatin profiles based on DNA sequences. shown in Figure each chromatin profile defined by cell type, protein name and experiment. The prediction value indicates the probability of binding between the DNA and the protein. We focus exclusively on chromatin profiles derived from ENCODE experiments, our ChIP-Seq dataset sourced from the ENCODE project.
Functionality:
Fluency: Similar to the fluency used in the text generation task (Li et al. 2022 Zhou et al. 2023a
we adopt it to assesses how smooth and natural the generated DNA sequence is. Fluency defined as the perplexity of the generated DNA a pretrained language model, with lower value indicating better fluency. We use the HyenaDNA pretrained model as the base model Fb (Nguyen et al.) (0224b). Given the cross-entropy loss E, the fluency of single DNA sequence U {ur." und is the exponential of the loss
Flu = exp() Σ E (Fo(11, ... ,4),, u4+1)).
(6)
The total fluency is then calculated the mean fluency across all DNA sequences.
Diversity: In a specific cell type, given protein can bind to many different DNA sequences. This metric measures the diversity of the generated sequences. Following Li al. (2024). the diversity score is defined
(n, Un) II n=10 count(n, U)
Diversity >
(7)
where count(n, U) is the total number of n-grams in the generated DNA sequence U, and [count(n, U) is the number of unique n-grams. Given the variability in the number of samples each category e, the weight of diversity in each category is denoted by W. Ne/N, where N₂ is the number of samples in category and N is the total number of samples.

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: Performance on promoter design. The best performance is indicated bold, while the second- performance is underlined. This convention is followed in all subsequent tables.
Method
MSE 4
KS Test Statistic 4
Fluency +
BIT Diffusion
0.0395
D8PM-UNIFORM
0.0375
DDSM
0.0334
Dirichlet FM
0.0269
0.399
3.6538
D3
0.0219
0.052
3.5255
ATGC-Gen Gen-GPT
0.0289
0.048
3.5231
ATGC-Gen-Bert
0.0192
0.043
3.5904
5 Experiments
5.1 Promoter Generation
Dataset and task descriptions. This dataset contains 100,000 promoter sequences from GRCh38 (HG38) human reference genome. Fach sequence annotated with the transcription initiation profile, indicating the likelihood of transcription initiation activity at each base pair (fan 2014 We use the same DNA sequences in previous works (Avdeyev et al. 2023 Stark al. 2024): specifically splitting 1,024 base pair long DNA sequences around each annotated transcription start site position (Hon et al. 2017). The task is to interpret the transcription initiation profile into the corresponding DNA sequences of the same length.
Metrics. Following prior works (Avdeyev el al. 2023 Stark et al. 2024 we evaluate the generation performance using the mean squared error (MSE) between the predicted regulatory activity of the generated and original DNA sequences. The regulatory activity computed using the SEI framework (Chen et al. 2022), published deep learning model that predicts 21,907 regulatory features for a given DNA sequence. For our primary evaluation, we extract promoter-related activity by focusing on the H3K4me3 chromatin mark. In addition, report the Kolmogorov Smirnov (KS) statistic between the distributions of regulatory activity values from the generated and original sequences, as proposed in (Sarkar et al.) 2024), to assess distribution ional alignment. We further evaluate the flueney of the generaled sequences by HyenaDNA (Nguyen et al. )023), as described in Section to ensure that generated DNA is biologically plausible and structurally coherent.
Results and analysis. Table 2 shows the generation performances for the promoter sequences. The results show that ATGC-Gen-BERT achieves the best overall performance compared to previous basclines, demonstrating the strong modeling capacity of the masked recovery generation paradigm. In contrast, ATGC-Gen-GPT aggregates base-resolution activity features at the beginning of the sequence, which removes per-token alignment between the activity signals and the DNA sequence. This loss of fine-grained information leads to inferior performance in functional metrics. However, due to its autoregressive nature, ATGC-Gen-GPT better preserves the fluency and coherence of the generated DNA sequences.
5.2 Enhancer Generation
Datasct and task descriptions. This task involves two distinct datasots: one from fly brain (Janssons et al. 2022) and the other from human melanoma cells (Atak et al.) 2021]. We follow the same data split in the work by Sttrk et al. (0024), using 104k fly brain DNA sequences and 89k human melanoma DNA sequences, each consisting of 500 base pairs. The cell class labels are derived from the ATAC-seq experiments (Buenrostro et al. 2013), with 47 classes for the human melanoma dataset and 81 classes for the fly brain dataset. Our objective is to generate DNA sequences based on the given cell class labels.
Metrics. Following Stark et al. (2024), we evaluate the performance using the Fréchet Biological Distance (FBD) between the generated and original DNA sequences. To calculate the FBD score, we use pretrained classifier model (ttark et al.] 2024) to obtain the hidden representation and then compute the Wasserstein
8

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Performance on enhancer generation.
Fly Brain
Melanoma
Method
FBD 4
Diversity t
Fluency +
FBD +
Diversity t
Fluency +
Dirichlet FM
1.0404
0.8314
4.0512
1.9051
0.8395
3.7126
ATGC-Gen-GPT
0.5080
0.8309
4.0326
0.9228
0.8131
3.6852
Table 4: Performance ChIP-Seq DNA generation. "w/" indicates that DNA sequences are generated based on the biological properties, while 'w/o" indicates that the properties are not provided during generation.
Method
Binding Score 1 Diversity
Real Sequence
0.2319
0.0513
Random Sequence
0.0036
ATGC-Gen w/o Property
0.0747
0.1213
ATGC-Gen w/ Property
0.1176
0.1228
distance between Gaussians. Additionally, we assess performance using the diversity score and fluency score, similar to the metries proposed in Section These metrics provide a comprehensive evaluation of the generated DNA sequences.
Results and analysis. Table presents the results for enhancer DNA sequence generation. Our proposed ATGC-Gen-GPT achieves a significantly lower Fréchet Biological Distance (FBD) than baseline models, indicating its effectiveness in incorporating global property information and modeling cross- coos--oddlltyy sig- nals to generate biologically realistic sequences. In terms of generation quality, ATGC-Gen-GPP produces more fluent and coherent sequences than the flow matching- baaseinng-based model, although with reduced diversity. Additional results for ATGC-Gen-BERIT are provided in Appendix C.2 Compared to the GPT variant, ATGC-Gen-BERT demonstrates weaker performance capturing global property information, likely due to the limitations of the masked modeling objective in autoregressive-style generation.
5.3 ChIP-Seq Generation
Results and Analysis. Table 4 reports the results the ChIP-Seq generation task, described in Section 4 Since the task involves variable-length generation, we only evaluate ATGC-Gen-GPTI The model achieves a strong Binding Score when conditioned on biological propertics. It also achieves higher diversity compared to other ablations. The performance drop without property inputs highlights the importance conditioning on biological context. Nonetheless, the property-free model still performs better than random sequences, suggesting that ATGC-Gen captures meaningful sequence patterns even without explicit control signals.
6 Conclusion and Discussion
In this paper, we present ATGC-Gen, a controllable DNA sequence generator that integrates diverse biological properties via cross-modal encoding and leverages language models for conditional generation. Our method enables accurate, fluent, and biologically relevant sequence design across multiple tasks. To support this direction, we also construct a new dataset from ChIP-Seq experiments, providing a realistic benchmark for
binding generation.
Practical Usage and Applications. Controllable sequence design is closcly aligned with real-world neods in synthetic biology, such as promoter or enhancer engineering and transcription-factor binding site opti- mization. ATGC-Gen can assist researchers generating candidate sequences that satisfy multiple design constraints before downstream experimental sereening. Although this work does include wet-lab vali- dation, the proposed controllability interface aligns with how sequence constraints are applied in practical design settings.
9

---

Published in Transactions on Machine Learning Research (12/2025)
Limitations. Despite promising results, work is constrained by limited computational resources, pre- venting the training of larger models. Moreover, frequent evaluations during training, especially under autoregressive settings, introduce notable computational overhead.
Broader Impacts. Controllable DNA generation offers exciting opportunities for synthetic biology and genetic engineering. While such capabilities may accelerate the design of beneficial biological functions, they also raise concerns about misuse or the unintended synthesis of harmful sequences. Ensuring biological safety, responsible use, and ethical safeguards remains critical.
Future Directions. Future work may explore more advanced generation techniques such retrieval- augmented models, as well as the development of richer and biologically grounded benchmarks to drive progress in programmable genomics. Incorporating experimental feedback loops or active-learning stralegies may further enhance the practical utility of controllable DNA generation.
Acknowledgments
This work was supported in part by National Institutes Health under grant U01A007012.
10

---

Published in Transactions on Machine Learning Research (12/2025)
References
A promoter-level mammalian expression atlas. Nature, 507(7493):462 470, 2014.
David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 91):1477-69, 1985.
Zeynep Kalender Atak, Ibrahim Ihsan Taskiran, Jonas Demeulemeester, Christopher Flerin, David Mauduit, Liesbeth Minnoye, Gert Hulselmans, Valerie Christiaens, Ghanem-Eliat Ghanem, Jasper Wouters, et al. Interpretation of allele-specific chromatin accessibility using cell state-aware deep learning. Genome re- search, 31(6):1082 1096, 2021.
Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. In International Conference on Machine Learning, pp. 2276-1301. PMLR, 2023.
Helen M. Berman, John Westbrook, Zukang Feng, Gary Gilliland, T. N. Bhat, Helge Weissig, llya N. Shindyalov, and Philip E. Bourne. The Protein Data Bank. Nucleic Acids Research, 28(1):235-2444 01 2000. ISSN 0305-1048. doi: 10.1093/nar //////....... 1.235. CRL / 11t.1//..19///////.1 1. 235
Nadav Brandes, Dan Ofer, Yam Peleg, Naday Rappoport, and Michal Linial. Proteinbert: universal deep-learning model of protein sequence and function. Bioinformatics, 38(8):2102-2113 2022.
Jason D Buenrostro, Paul G Giresi, Lisa C Zaba, Howard Y Chang, and William Greenleaf. Transposition native chromatin for fast and sensitive epigenomic profiling of open chromatin, dna- bindbinding proteins and nueloosome position. Nature methods, 10(12):1213-1218, 2013.
Kathleen M Chen, Aaron K Wong, Olga G Troyanskaya, and Jian Zhou. A sequenco- gaquenco-based global map of regulatory activity for deciphering human genelics. Nature genelics, 54(7):940 999-999999 2022.
ENCODE Project Consortium et al. An integrated encyclopedia of dna elements in the human genome. Nature, (7444444) 2012.
Hugo Dalla- Torre, Liam Gonzalez, Javier Sendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkha- tim, et al. The nucleotide transformer: Building and evaluating robust foundation models for human genomics. bioRciv, pp. 2023-01 01, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for compulational linguistics: lumman language technologies, volume 1 (long and short popers), pp. 4171-41 1171-4186, 2019.
Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. arXiv preprint arxim:1707 02663, 2017.
Christopher Fifty, Jure Leskovec, and Sebastian Thrun. In-context learning for few-shot molecular property prediction. arXiv preprint ar arXiv:2310.0888 08863, 2023.
Vladimir Gligorijović, P Douglas Renfrew, Tomasz Kosciolck, Julia Kochler Loman, Danicl Berenberg, Tommi Valanen, Chris Chandler, Bryn Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure-based protein function prediction using graph convolutional networks. Nature communications, 12(1):3168, 2021.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXio preprint 2rXXi=2312.00752. 2023.
Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with language model. Science, 387(663)):850 858, 2025.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Chung- Chau Hon, Jordan A Ramilowski, Jayson Harshbarger, Nicolas Bertin, Owen JL Rackham, Julian Gough, Flena Denisenko, Sebastian Schmeier, Thomas M Poulsen, Jessica Severin, et al. An atlas of human long non-coding rnas with accurate 5 textquotesingle ends. Nature, 543(7644):199-00)) 204, 2017.
Jasper Janssens, Sara Aibar, Ibrahim Ihsan Taskiran, Joy N Ismail, Alicia Estacio Gomez, Gabriel Aughey, Katina Spanier, Florian V De Rop, Carmen Bravo Gonzalez-Blas, Marc Dionne, al. Decoding gene regulation in the fly brain. Nature, 601(7894):630 -646636366,6,-- 2022.
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder
representations from transformers model for dna-language in genome. Bioinformatics, 37(15):2112-2111 2021.
Nathan Killoran, Leo J Lee, Andrew Delong, David Duvenaud, and Brendan Frey. Generating and designing dna with deep generative models. arXiv preprint 1Xi.:712.0061 2017.
Xiang Li, John Thickstun, Ishaan Guirajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. Advances in Newral Information Processing Systems, 35:4328 4343, 2022.
Xiner Li, Masatoshi Uehara, Xingyu Su, Gabriele Scalia, Tommaso Biancalani, Aviv Regev, Sergey Levine, and Shuiwang Ji. Dynamic search for inference-time alignment in diffusion models. arXiv preprint arXiv:2503.08 02039, 2025.
Zehui Li, Yuhao Ni, William AV Beardall, Guoxuan Xia, Akashaditya Das, Guy-Bart Stan, and Yiren Zhao Discdiff: Latent diffusion model for dna sequence generation. arXiv preprint 3rXiv:2402.0607) 2024.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Frolutionary-scale prediction of atomic-level protein structure with language model. Science, 379(6637):1123 1130, 2023.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36:43177- 43201, 2023.
Eric Nguyen, Michael Poli, Matthew G Durrant, Armin W Thomas, Brian Kang, Jeremy Sullivan, Made- lena Y Ng, Ashley Lewis, Aman Patel, Aaron Lou, et al. Sequence modeling and design from molecular to genome scale with evo. bioRrie, pp. 2024- 02, 2024a.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow Callum Birch-Sykes Stefano Massaroli, Aman Patel, Clayton Rabidcau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling single nucleotide resolution. Advances in neural information processing systems, 36, 2024b.
Shen Nie, Fengyi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502. 29999999900999, 09992, 2025.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gencrative pre-training. 2018.
Ieva Rauluseviciute, Rafacl Riudavets-Puig Romain Blanc-Mathien, Jaime A Castro-Mondragon, Katalin Ferene, Vipin Kumar, Roza Berhanu Lemin,, Jérémy Lucas, Jeanne Chèneby, Danir Baranasic, et al. Jaspar 2024: 20th anniversary of the open-access database of transcription factor binding profiles. Nucleic acids research, 52(D1))D114 D182, 2024.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advonces in Neural Information Processing Systems, 02:13033-1301820 2024.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Anirban Sarkar, Ziqi Tang, Chris Zhao, and Peter Koo. Designing dna with tunable regulatory activity using discrete diffusion. bioRæin, 2024.
Yair Schiff, Chia- Thia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint or orXio:2403.0322) 2024.
Johannes Schimunek, Philipp Seidl, Lukas Friedrich, Daniel Kuhn, Friedrich Rippmann, Sepp Hochreiter, and Günter Klambauer. Context-enriched molecule representations improve few-shot drug discovery. arXiv preprint XrXi::200..0 09481, 2023.
Niklas Schmidinger, Lisa Schneckenreiter, Philipp Seidl, Johannes Schimunek, Pieter-Jan Hoedt, Johannes Brandstetter, Andreas Mayr, Sohvi Luukkonen, Sepp Hochreiter, and Günter Klambauer. Bio-xlstm: Generative modeling, representation and in-context learning of biological and chemical sequences. arXiv preprint 1..::2411.041000 2024.
Damiano Sgarbossa, Cyril Malbranke, and Anne-Florence Bitbol. Protmamba: homology-aware but aligument-free protein state space model. Bioinformatics, 41(6), 2025.
Hannes Stark, Bowen Jing, Chenyu Wang, Gabricle Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. arv preprinl arXiv:2402.0555 05841, 2024.
Xingyu Su, Haiyang Yu, Degui Zhi, and Shuiwang Ji. Learning to discover regulatory elements for gene expression prediction. arXiv preprint XrX2:22002. 13991, 2025.
Ibrahim Taskiran, Katina I Spanier, Hannah Dickmänken, Niklas Kempynck, Alexandra Pančíková, Eren Can Ekşi, Gert Hulselmans, Joy Ismail, Koen Theunis, Roel Vandepoel, et al. Cell-type-directed design of synthetic enhancers. Nature, (26((997):212-22) 2024.
Masatoshi Uehara, Xingyu Su, Yulai Zhao, Xiner Li, Aviv Regev, Shuiwang Ji, Sergey Levine, and Tommaso Biancalani. Reward-guided iterative refinement in diffusion models at test-time with applications to protein and dna design. Xix preprint arXiw:2502. 14944, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukass Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neurul information processing systems, 30, 2017.
Yingheng Wang, Zichen Wang, Gil Sadeh, Luca Zancato, Alessandro Achille, George Karypis, and Huzefa Rangwala. Le-plm: Long-context protein language modeling using bidirectional mamba. with shared projection layers. bioRriv, pp. 2024-10 10, 2024.
Bohao Xu, Yingzhou Lu, Chenhao Li, Ling Yue, Xiao Wang, Nan Hao, Tianfan Fu, and Jim Chen. Smiles- mamba: Chemical mamba foundation models for drug admet prediction. arXiv preprint arXi::2408.00600 2024.
Wangchunshu Zhou, Yuchen Elcanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan. Controlled text generation with natural language instructions. In International Conference on Machine Learning, pp. 42602-42613. PMLR, 2023a.
Zhengyang Zhou, Yunrui Li, Pengyu Hong, and Hao Xu. Multimodal fusion with relational learning for molecular property prediction. Communications Chemistry, 8(1):200, 2025.
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint Xix:222306. 15006, 2023b.
Jan Zrimec, Xiaozhi Fu, Azam Sheikh Muhammad, Christos Skrekas, Vykintas Jauniskis, Nora K Speicher, Christoph S Börlin, Vilhelm Verendel, Morteza Haghir Chehreghani, Devdatt Dubhashi, et al. Controlling gene expression with deep generative design of regulatory dna. Nature communications, 13(1555099 2022.
13

---

Published in Transactions on Machine Learning Research (12/2025)
A Data Processing for ChIP-Seq Experiment
Based on the data from the ChIP-Seq experiment, we perform data preprocessing. No license is needed for the data files and database table The raw data comes from the ENCODE project The preprocessing sleps are as follows:
Filtering by Binding Scores. Binding score measures how effectively DNA sequences bind to proteins. To focus the most relevant interactions, retain only the highest binding scores for each category. This step ensures that our analysis concentrates on the strongest and most meaningful DNA-protein interactions.
Alignment with SEI Framework. To ensure compatibility with the SEI framework used for downstream evaluation of DNA sequence funetionality, we align the cell types and protein names in our dataset with those recognized by SEI. Specifically, we retain the GM12878 cell type, which has the largest number of entries. Additionally, we filter the dataset to include only proteins whose names are supported by the SEI framework, enabling accurate and consistent evaluation.
Standardizing Input Sequence Lengths. The raw dataset contains pairs of DNA sequences and their associated binding transcription factor (TF) protein sequences, with lengths varying significantly, ranging from 50 to over 20,000 base pairs for DNA and up to over one thousand amino acids for proteins. Such variation poses challenges for modeling and computation. To ensure consistency and tractability, we filter the dataset retain only DNA sequenees shorter than 500 base pairs and protein sequences shorter than 1,000 amino acids. This standardization facilitates efficient training and stable convergence of the language models.
Embedding Protein Sequences. To represent the transcription factor (TF) proteins, we use the ESM-2- 3B protein language model (Lin et al. 2023 to extract contextual embeddings from the amino acid sequences. We retain the full-length output embeddings without applying any sequence-level pooling or aggregation, thereby preserving residue-leve resolution for downstream tasks.
After completing the preprocessing steps, we partition the dataset by chromosome: chromosomes 20 and 21 are used for validation, while chromosomes 22 and X are held out for testing. The remaining chromosomes are used for training. The detailed statistics of the cleaned and processed dataset are provided in Table
Table 5: ChIP-Seq dataset information after preprocessing.
# rows
# proteins
# split samples
55830
62
51800/2181/1849 /1499
B Implementation Details
We report the hyperparameters and training configurations used in our experiments. For ATGC-Gen- Bert, we adopt a BERT-style encoder with the bert-base configuration: 12 layers, 12 attention heads, and a hidden size of 768. For ATGC-Gen-GPT, we use a GPT-style decoder with 16 layers, 16 attention heads, and the same hidden size of 768. We use the AdamW optimizer with a learning rate of x 10-4 and a lincar warmup over the first 10% of training opochs. Model selection is basod on performance on the validation set. The batch size adjusted to fit within GPU memory constraints. All training and inference are conducted on NVIDIA A100-SXM-80GB GPUs.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Table 6: Performance on enhancer generation without property information.
Fly Brain
Melanoma Diversity f
Method
FBD +
Diversity f Fluency
FBD J
Fluency J
Dirichlet FM w/o Property ATGC-Gen-GPT w/o Property
15.2107
0.0697 0.0572
4.0847
5.3874
0.0696 0.0579
3.8251
14.6412
4.0229
8.3796
3.6329
Table 7: Performance between ATGC-Gen-Beel and ATGC-Gen-GPP GPT on enhancer generation.
Fly Brain
Melanoma
Method
FBD +
Diversity t
Fluency 4
FBD +
Diversity t
Fluency J
ATGC-Gen-GPPP
0.5080
0.8309
4.0326
0.9228
0.8131
3.6852
ATGC-Geen
27.63
0.8667
3.9609
45.27
0.8230
3.7560
C Additional Results
C.1 Enhancer Design without Property Information
To better understand the role conditioning signals, we follow prior work and evaluate ATGC-Gen- GPT in an uncontrolled seiting, where no properly information provided during generation (Stark et al. 2024).. As shown Table 6 ATGC-Gen-GITT results in a mixed performances compared with the Dirichlet FM baseline in this scenario. This result contrasts with the controllable generation sctting, where ATGC-Gen-GTI consistently outperforms Dirichlet FM by large margin (Table 3 The discrepancy highlights that ATGC- Gen is particularly effective when it can leverage external biological properties to guide sequence generation. In the absence of such conditioning, its advantage diminishes. Overall, these findings suggest that while Dirichlet FM remains competitive in unconditional generation, ATGC-Gen is better suited for property- conditioned lasks precisely the setting that is most relevant practical genomic design applications.
C.2 Enhancer Design with ATGC-Gen-BER
Table 7 prosents comparison between ATGC-Gen-BERT and ATG on the enhancer generation tasks. Despite both variants using the same sequence-level integration to encode global property informa- tion, ATGC-Gen-BERT performs substantially worse than ATGC-Gen-GPT in terms of Fréchet Biological Distance (FBD). This performance gap suggests that masked recovery generation, as used in ATGC-Gen- BERT, is less effective at capturing and utilizing global propertics compared to the antoregressive decoding in ATGC-Gen-GPT. In contrast, the autoregressive nature of GPT allows for sequential incorporation of such information, leading to better alignment with the desired biological conditions.
C.3 Effect of Unmasking Rate ATGC-Gen-Bert
can control the number of tokens unmasked at each step. For instance, may choose to unmask all masked tokens simultaneously unmask only one randomly selected token per step. Figure presents presentsss the results under varying unmasking rates, where the x-axis denotes the number of unmasking steps normalized by sequence length (c.g., 1.0 corresponds to unmasking one token per step for a 1024-length sequence). The figure shows that overall performance remains relatively stable across different unmasking schedules. Notably, using fewer steps (e.g., unmasking 10% of tokens per step, or 103 steps in total) yields comparable performance full-length generation (1024 steps), suggesting the potential for significant speedup. However, the extreme case of unmasking all tokens in single step results in a notable performance drop (MSF = 0.0334), indicating the importance of stepwise refinement. In Table 2 we report results using the most conservative setting, unmasking one token per step. While not the best-
During the generation process of XTGC-Gen-Berll
https://genone. uesc. www....../.ccceeeeee.eeeeee
https:// ucsc. edu/ bin/ http::/:hh........o..ee////eeeea
15

---

Published in Transactions on Machine Learning Research (12/2025)
0.194
0.0194
Mean Squared Error (MSE)
0.0193
0.192
O 0191
0.19)
0.0191
0.0190
0.0189
10
Number of Unmesking Steps Sequence Length
Figure 3: Effect of unmask ratio on ATGC-Gen- in promoter generation.
GABPA
7487
KLF4
7058/7497
Yy1
6242/7497
ETS1
6030/7497
Motif
SP1
5828/7497
NFYA
5437/7497
CREB1
5210/7497
4959/7497
ELK1
3242/7497
20
60 % of generated sequences with =1 hit
40
80
100
Figure Fraction of generated sequences containing least one occurrence of each selected transcription factor motif. Highly enriched motifs include GABPA, KLF4, and YYI.
performing configuration in Figure we adopt this setting to avoid selecting hyperparameters based on testing set performance, which could introduce bias.
C.4 Motif Enrichment in Generated Sequences
To further validate the biological plausibility of our generated regulatory sequences, we performed motif enrichment analysis using a curated panel of transcription factor (TF) binding motifs from the JASPAR database (Rauluseviciute et al. 2024 on the promoter design task described in Section 5.1 Specifically, we considered ten motifs corresponding to well-ostablished regulators of gene expression:
SP1 (MA0079.5)
KLF4 (MA0039.5)
NRFI (MA0506.3)
YY1 (MA0095.4)
NFYA (MA0060.4)
16

---

Published in Transactions on Machine Learning Research (12/2025)
Sequerces
50g2
EI51 FIS
CONT POCTTOOOOOO CER
GLEPEERA
4
200
400
60n
B00
1000
Pastion
Figure 5: Motif distribution across five randomly sampled generated sequences. Each box marks a delected motif occurrence at the corresponding genomic position.
ELK1 (MA0028.3)
ETS1 (MA0098.4 4)
GABPA (MA0062.4)
CREBI (MA0018.5)
E2F1 (MA0024.3)
We scanned all generated sequences for the presence of these motifs using position frequency matrices (PFMs) from JASPAR, followed by enrichment quantification.
Global motif enrichment. Figure 4 summmarize the fraction of generated sequences that contain at least one occurrecce of each TF motif. Highly enriched motifs include GABPA, KLF4, and YY1, with more than 80% of sequences exhibiting at least one instance. This demonstrates that the model captures essential cis-regulatory elements commonly associated with enhancer-like activity.
Motif positional distribution. To illustrate the positional organization of motifs, Figure shows motif annotations across five randomly selected generated sequences. Each motif occurrence is plotted at its genomic position, highlighting the combinatorial nature of motif clustering. The recurrence of canonical enhancer-associated motifs such as ETS1, SP1, and NFYA suggests that the generative model produces realistic cis-regulatory grammar beyond singlo-motif cnrichment.
17


Reviews:
[{"review_id": "KXrLFaatMn", "reviewer": "Reviewer_nR4f", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary\nThe authors propose two generative models for conditioned DNA sequence generation: an autoregressive model and a model that iteratively fills masked sequences. The models are evaluated on three datasets, with the third dataset newly introduced by the authors in this manuscript.\n\n### Strengths\n* **(S1 — Relevant problem setting).** Conditioned DNA sequence generation is an important and timely problem of interest to the community.\n* **(S2 — Competitive results).** The proposed models outperform strong baselines, including flow-based approaches such as Dirichlet FM.\n\n### Weaknesses\n* **(W1 — Limited discussion of existing work)**. Conditioned generation of biological sequences through context is not novel and has been studied previously (e.g. [1–3]). Also DNA language models exist (e.g. [2, 6]). The authors should incorporate existing models into their analysis and discuss existing literature in depth, highlighting similarities and differences. For example, in homology-aware models both the sequences and the conditioning information are protein-based, resulting in a uni-modal setup. By contrast, the proposed model includes multiple modalities. Combining different modalities in latent spaces is well established in the biological domain (e.g. bioactivity few-shot learning models combining structure and activity data [4–5]). The conditioning mechanism for DNA sequences in this manuscript has potential novelty, but this only becomes clear if discussed explicitly.\n* **(W2 — Lack of clarity on novelty).** Following from W1, the manuscript currently falls short of clearly articulating what is new about the proposed approach.\n* **(W3 — Incorrect and cluttered formulas).** Several equations are either inconsistent or misleading:\n    * Properties are denoted both as $C_i$ (p. 2) and $\\boldsymbol{s}_i$ (p. 4).\n    * Sequence embeddings $\\boldsymbol{h}_{e,i}$ are introduced without explanation — how are they derived from the raw sequence\n    * Formula (2) has inconsistent dimensions, as the weight matrix and embeddings appear swapped in the linear transformation.\n    * From Section 3.2 onwards, only $\\boldsymbol{H}_p$ is used as property context. Does this imply that only sequence-level context integration is employed? If so, why is a section on feature-level integration included?\n* **(W4 — Unclear dataset relevance).** The authors clearly describe the setup of the proposed dataset, but its relation to the storyline of the manuscript and to other DNA datasets remains unclear. If the main contribution is the model framework, why is a new dataset necessary? Why are existing DNA datasets insufficient?\n\n[1] Sgarbossa, Damiano, Cyril Malbranke, and Anne-Florence Bitbol. \"ProtMamba: a homology-aware but alignment-free protein state space model.\" Bioinformatics 41.6 (2025).\n\n[2] Schmidinger, Niklas, et al. \"Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences.\" arXiv preprint arXiv:2411.04165 (2024).\n\n[3] Wang, Yingheng, et al. \"LC-PLM: Long-context Protein Language Modeling Using Bidirectional Mamba with Shared Projection Layers.\" bioRxiv (2024): 2024-10.\n\n[4] Schimunek, Johannes, et al. \"Context-enriched molecule representations improve few-shot drug discovery.\" arXiv preprint arXiv:2305.09481 (2023).\n\n[5] Fifty, Christopher, Jure Leskovec, and Sebastian Thrun. \"In-context learning for few-shot molecular property prediction.\" arXiv preprint arXiv:2310.08863 (2023).\n\n[6] Schiff, Yair, et al. \"Caduceus: Bi-directional equivariant long-range dna sequence modeling.\" Proceedings of machine learning research 235 (2024): 43632.", "claims_and_evidence_yn": "No", "claims_explanation": "Because of W1 and W2 the relevance of this work remains unclear.", "audience_yn": "Yes", "audience_explanation": "The problem setting is relevant and of clear interest. The superior performance of transformer-based models over flow-based models is interesting.\n\nThe manuscript would be of interest to the TMLR audience, assuming that W1/W2 are adequately resolved.", "requested_changes": "* Critical — W1/W2. The authors should include missing literature and describe which ideas in their framework have already been explored in prior bio/chemistry publications. A discussion of similarities and differences is necessary to highlight the unique aspects of their approach. Without this, the manuscript risks being perceived as just another DNA language model.\n* Critical — W3. The mathematical foundation of the framework must be presented correctly. The authors should revise the manuscript to ensure that all formulas are dimensionally consistent and that symbols are properly introduced.\n* Critical — W4. The additional dataset is potentially valuable, but its relevance is unclear without proper context. The authors should explain why this dataset is needed to evaluate their models, and why existing DNA datasets would not suffice.\n* Critical — Clarity of approach. The description of feature-level integration is insufficient. The authors should clarify how this integration is performed and how it relates to the overall framework.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EBnmvDI1qm", "reviewer": "Reviewer_HsU2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes ATGC-Gen for controllable DNA sequence generation. The model is investigated with both decoder-only (GPT style) and encoder-only (BERT style) architectures. It also introduces a new dataset for DNA generation based on ChIP-Seq experiments. Experiments are performed to test ATGC-Gen with other diffusion models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is clear in general. It shows the models, formulations, results, dataset, and experiments clearly. \n\nThe paper shows that LMs are a natural fit due to their handling of symbolic sequences. The cross-model encoding for heterogeneous biological inputs is a clever design. \n\nThe empirical results show that the proposed method is better than the baseline methods.", "audience_yn": "Yes", "audience_explanation": "The community that focused on generative models, biological applications, and multimodal learning would be interested in the findings of the paper. \n\nHowever, it is better to show the proposed model's application on real scientific problems, such as biological structure reconstruction or discovery. Without that, the impact and contributions of the current paper is quite limited.", "requested_changes": "The paper is better to apply the model to real scientific applications rather than synthetic comparisons with other algorithms. \n\nIn the experiment session, no qualitative analysis, such as motif enrichment in generated sequences, or error cases. The baseline is strong but not exhaustive. It missed comparisons to other LMs like MambaDNA in generation mode.", "broader_impact_concerns": "Yes. There are ethical concerns related to the potential misuse of controllable DNA sequence generation in synthetic biology and genetic engineering. The codes that support reproducibility in the paper may cause ethical risk. \n\nThe paper has already included a \"Broader Impacts\" section (page 9) to acknowledge these issue.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mCMnTgcIXF", "reviewer": "Reviewer_hZrv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes ATGC-Gen, which uses transformer-based language models for controllable DNA sequence design, conditional on specific biological properties of interest, e.g. protein binding. The method encodes contextual information like cell types, protein sequences, and transcription activation signals, to train DNA sequence generator via autoregressive modeling (ATGC-Gen-GPT) or masked language modeling (ATGC-Gen-BERT).\n\nThe paper introduces a new dataset to capture protein-DNA binding patterns. In addition to this benchmark, the method is evaluated on existing promoter and enhancer generation tasks, using property-specific metrics (e.g. binding score) as well as 'fluency' (perplexity under a pretrained model) and diversity (of generated sequences). Different versions of ATGC-Gen outperform baselines, albeit narrowly in several cases.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper claims superiority of language models in DNA sequence generation over diffusion-based methods, since the former is amenable to discrete, variable-length generations. The experiments show improved performance of ATGC-Gen-GPT over other baselines and over ATGC-Gen-BERT (fixed-length generation) in many of settings and metrics, substantiating this claim.", "audience_yn": "Yes", "audience_explanation": "DNA sequence generation is a rich area of active research, and the comparisons between different approaches shown in this paper would be of interest to those researchers. The paper also proposes a new benchmark to evaluate DNA-protein binding, which would further help future work test their methods on this task.", "requested_changes": "I would appreciate some discussion on the following questions, which I believe are critical to establishing the impact of this work and to securing my recommendation.\n\n- How are the linear projections of the properties learned, before they are used as inputs to a transformer? \n- Since the results of the paper are mixed between ATGC-Gen-GPT and ATGC-Gen-BERT, what are the performance trade-offs between autoregressive training and masked-language-modeling? Can the authors provide any recommendations or actionable insights for practitioners?\n- There appears to be a performance trade-off between property-specific metrics like binding score vs diversity of sequence generation. Can the authors comment on this trade-off and whether their method can be adapted to favor one over the other?\n- Lal et al. [1] seems to be a closely related work that the paper is missing a citation for. Since it operates on the same problem setting, I would request the authors to either add a comparison to it in the experiments section, or at least discuss the work and situate ATGC-Gen  in the literature with respect to it.\n\n[1] Lal A, Garfield D, Biancalani T, Eraslan G. Designing realistic regulatory DNA with autoregressive language models. Genome Res. 2024 Oct 11;34(9):1411-1420. doi: 10.1101/gr.279142.124. PMID: 39322281; PMCID: PMC11529870.", "broader_impact_concerns": "Sufficiently discussed in the paper.", "additional_comments": "None.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)