Regulatory DNA Sequence Design with Reinforcement Learning
Authors: Zhao Yang
Bing Su
Chuan Cao
Ji-Rong Wen
Venue: ICLR 2025 Poster (2025)
Keywords: sequence optimize
generative models
ai4science
dna
rl
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: $\textit{Cis}$-regulatory elements (CREs), such as promoters and enhancers, are relatively short DNA sequences that directly regulate gene expression. The fitness of CREs, measured by their ability to modulate gene expression, highly depends on the nucleotide sequences, especially specific motifs known as transcription factor binding sites (TFBSs). Designing high-fitness CREs is crucial for therapeutic and bioengineering applications. Current CRE design methods are limited by two major drawbacks: (1) they typically rely on iterative optimization strategies that modify existing sequences and are prone to local optima, and (2) they lack the guidance of biological prior knowledge in sequence optimization. In this paper, we address these limitations by proposing a generative approach that leverages reinforcement learning (RL) to fine-tune a pre-trained autoregressive (AR) model. Our method incorporates data-driven biological priors by deriving computational inference-based rewards that simulate the addition of activator TFBSs and removal of repressor TFBSs, which are then integrated into the RL process. We evaluate our method on promoter design tasks in two yeast media conditions and enhancer design tasks for three human cell types, demonstrating its ability to generate high-fitness CREs while maintaining sequence diversity. The code is available at https://github.com/yangzhao1230/TACO.

Published as a conference paper at ICLR 2025
REGULATORY DNA SEQUENCE DESIGN WITH REINFORCEMENT LEARNING
Zhao Yang Bing Sult Chuan Cao
Ji-Rong Wen'
Gaalling School of Artificial Intelligence, Renmin University of China
Microsoft Research A14Science
ABSTRACT
Cis-regulatory elements (CREs), such promoters and enhancers, are relatively short DNA sequences that directly regulate gene expression. The fitness of CREs, measured by their ability modulate gene expression, highly depends the nu- cleotide sequences, especially specific motifss known as transcription factor bind- ing sites (TFBSs). Designing high-fitness CREs is crucial for therapeutic and bioengineering applications. Current CRE design methods are limited by two major drawbacks: (1) they typically rely on iterative optimization strategies that modify existing sequences and are prone to local optima, and (2) they lack the guidance of biological prior knowledge in sequence optimization. In this pa- per, we address these limitations by proposing a generative approach that lever- ages reinforcement learning (RL) to fine-tune a pre-trained autoregressive (AR) model. Our method incorporates data-driven biological priors by deriving com- putational inference-hasee rewards that simulate the addition of activator TFBSs and removal of repressor TFBSs, which are then integrated into the RL process. We evaluate our method on promoter design tasks in two yeast media conditions and enhancer design tasks for three human cell types, demonstrating its ability to generate high-fitness CREs while maintaining sequence diversity. The code is available https:///ittbb.ommmyyymg/o///2/2
INTRODUCTION
Cis-regulatory elements (CREs), such promoters and enhancers, are short functional DNA quences that regulate gene expression in a cell-type-specific manner (Fu et al. 2025). Promoters determine when and where a gene is activated, while enhancers boost gene expression levels. Over the past decade, millions of putative CREs (Gao & Qian 2020 have been identified, but these natu- rally evolved sequences only represent small fraction the possible genetic landscape and are not necessarily optimal for specific expression outcomes. It is crucial to design synthetic CREs with de- sired fitness (measured by their ability to enhance gene expression) as they have broad applications in areas such as gene therapy (Boye et al.] 2013), synthetic biology (Shao et al. 2024), precision medicine (Collins& Varmus 2015), and agricultural biotechnology [
Previous attempts to explore alternative CREs have relied heavily on directed evolution, which in- volves iterative cycles of mutation and selection in wet-lab settings (Wittkopp Kalay 2012: Heinz et al. 2015). This approach is sub-optimal due to the vastness of the DNA sequence space and the significant time and cost required for experimental validation. For example, 200 base pair (bp) DNA sequence can have up to 2.58 x 10 possible combinations (Gosai et al. 2024), far exceed- ing the number of atoms in the observable universe. Thus, efficient computational algorithms are needed narrow down the design space and prioritize candidates for wet-lab testing.
Massively parallel reporter assays (MPRAs) dde Boer et al.) 2020 have enabled the screening of large libraries of DNA sequences and their fitness in specific cell types. Recent studies have begun using fitness prediction models as reward models to guide CRE optimization, allowing the explo-
Work was done during internship at Microsoft Research Al4Science; 1ttp://332@mmmm....m Correspondence to: Bing Su (subingats gmail.com), Chuan Cao (chuancao. @266@ gmail.com)

---

Published as a conference paper at ICLR 2025
(A)
(B)
GATA2
HNF1B
REST
.TTTTCI
C
Activate
Repress
C[ 1000/ 255 14 402
102
367
Blood Cell
Liver Cell
Neural Cell
Figure (A) TFBSs are commonly represented as frequency matrices, indicating the frequency of each nucleotide appearing at specific positions within the binding site. (B) GATA2 and HNFIB specifically activate gene expression in blood cells and liver cells, respectively, while REST specifi- cally represses gene expression in neural cells.
ration of sequences that outperform natural ones Vaiihhav et al., 2022: de Almeida et al. 2024). These methods typically rely on iterative optimization strategies that modily existing or random se- quences. In each iteration, they modify previously selected sequences to generate new candidates. Although the search space is vast, these methods can only explore a limited local neighborhood through simple modifications, making it difficult for them to escape local optima. As result, these approaches often produce CREs with limited diversity. Moreover, these methods generally use generic sequence optimization techniques without incorporating biological prior knowledge.
Inspired by the success of using Reinforcement Learning (RL) to fine-tune autoregressive (AR) generative language models (Ouyang et al. 2022; Liu et al. 2024 Mo et al. 2024), we propose generative approach that leverages RL to fine-tune AR models for designing cell-type-specific CREs. Unlike previous methods that modify existing sequences, our approach enables the genera- tion of novel sequences from scratch by capturing the underlying distribution of CREs. We employ HyenaDNA (Nguyen et al. 2024b Lai et al., (004). state-of-the-art (SOTA) AR DNA generative model, and fine-tune it on CRES learn their natural sequence patterns, ensuring the generation of realistic and diverse sequences. During RL fine-tuning, we treat the current AR model as the policy network and use the fitness predicted by reward model as the reward signal to update our policy. This allows us to adjust the model parameters to generate CRE sequences that achieve high fitness while maintaining sequence diversity.
Additionally, we seamlessly incorporate domain knowledge of CREs into our RL process. The reg- ulatory syntax of CREs is largely dictated by the transcription factors (TFs) that bind to them (Gosai et al. 2024| de Almeida et al. 2024: Lal et al. 2024 Zhang et al. 2023 TFs are proteins that di- rectly influence gene expression by binding to specific sequence motifs within CREs, known as TF binding sites (TFBSs), and modulating transcriptional activity. For instance, F shows the motif pattern recognized by the GATA2 TF. Furthermore, the effects of TFs can vary widely depend- ing on the cell type. As shown in Figure (B), GATA2 and HNF/B B are TF's that specifically activate gene expression in blood cells and liver cells (Lal et al., 2024). respectively, while REST acts as repressor of gene expression in neural cells (Zullo 2019), illustrating the cell-type-specific nature of TF activity.
The effect of TF be broken down into its intrinsic role as an activator or repres- sor (de Almeida et al.) 2024) (referred to as its vocabulary) and its interactions with other TFs (Georgakopoulos-Soon Soares et al., (003). We find that using the frequency of TFBS occur- rences within a sequence as features can yield reasonably good fitness prediction performance when trained with a decision tree model, Light- GBM (Ke et al. 2017). As shown in Table the SOTA DNA model, Enformer, achieves Pearson correlation of 0.83 on the test set for predicting fitness in the HepG2 cell line using
Model
seast
haman
complex
defined
hepg2
k562
skash
Enformer
(Sequence Festure)
0.87
0.91
0.83
0.85
0.85
LightGRM (TBBS Frequency Feature)
Table I: Pearson correlation coefficient of differ- ent fitness prediction models on the test set. De- tails can be found in Section 3.3
2

---

Published as a conference paper at ICLR 2025
quence data as imput. In contrast, using only TFBS frequency features (without sequence informa- tion) results in a Pearson correlation of 0.65. This shows that TF frequency alone can still capture much of the predictive power, even without detailed sequence data. Notably, we believe that these frequency features implicitly reflect both the TT vocabulary and their interactions. Furthermore, we the trained LightGBM (Ke et al.) 2017) model infer whether each TFBS feature promotes or represses fitness using SHAP values Lundberg]2011
allowing us to explicitly integrate TFBS do- main knowledge into our RL process. Hence, we name our proposed method TACO: TFBS-Awar Cis- Fis-Regulatory Element Optimization, which combines RL fine-tuning of autoregressive models with domain knowledge of TFBSs to enhance CRE optimization.
Our main contributions are as follows:
We introduce the RL fine-tuning paradigm for pre-trained AR DNA models in CRE design, enabling the generated sequences to maintain high diversity while also exploring those with superior functional performance.
We incorporate TFBS information by inferring their regulatory roles and integrating their impact directly into the generation process, allowing for synergistic exploration guided by both data-driven and knowledge-driven approaches.
We evaluate our approach under different optimization settings on real-world datasets, in- cluding yeast promoter designs from two media types and human enhancer designs from three cell lines, demonstrating the effectiveness of TACO.
2 RELATED WORK
Conditional DNA Generative Models. DDSM (Avdeyev et al. 2023) pioneered diffusion models
as a generative approach for DNA design, using classifier-free guidance (Ho & Salimans 2022)
to generate DNA sequences based on desired promoter expression levels. Building
this work,
several studies have since employed diffusion models for CRE design (Li et al., 2024b DaSilva
ett al. 2024. Sarkar al., 2024). In addition diffusion models, regLM (Lal al. [224 utilized
prefix-tunin on the AR DNA language model HyenaDNA (Nguyen et al. 2024b). incorporating
tokens that encode expression strength fine-tune the model specifically for CRE design. However,
these generative methods are primarily designed to fit existing data distributions, which limits their
ability to design novel sequences with potentially higher fitness levels that have yet to be explored by humans.
DNA Sequence Optimization. DyNA PPO (Angermueller al. 2019 was an early exploration
of applying modern RL to biological sequence design. By improving the sampling efficiency of PPO (Schulman et al. 2017) and leveraging an AR policy, provided a general framework for bio-
logical sequence design. DyNA PPO and its subsequent works (Jain al. 2022 Zeng et al. 2024) primarily focused on advancing coneral-purpose sequence design algorithms, with an emphasis on optimizing short TFBS motifs (6-8 bp) in the context of DNA sequence design. With the availability of larger CRE fitness datasets. Vaishnav et al. (2022) applied genetic algorithms to design CREs. Recent works, such as Gosai et al. al. (2024), explored greedy approaches like Add (Sinai al.) 2020), simulated annealing (Van Laarhoven et al., 1987). and gradient-based SeqProp (Linder & Scelig 202I). Similarly, TTskiran et al. (2024) combined greedy strategies with directed evolution. However, these methods often start from random sequences or observed high-fitness sequences, leading to local optima and limited diversity. Recently, Reddy et al. (2024) proposed directly opti- mizing CREs using gradient ascent (GAs) on differentiable reward model trained on offline CRE datasets.
Motif-based Machine Learning in Scientific Data. Motifs are small but critical elements in scien- tific data, such as functional groups molecules or TFBS in DNA sequences. Explicitly modeling these motifs can provide significant benefits, as demonstrated in molecular optimization (Jin et al.) 2020 Chen et al. 2021), generation (Geng et al. 2023), property prediction (Zhang et al. 2202)). and DNA language models (An et al., 2022). In the context of DNA CREs, TFBSs are widely consid- ered the most important motifs. Our approach is heavily inspired by de Almeida et al. (2024). who observed that during direct evolution guided by reward model, there is tendency to first remove repressor TFBSs and then add enhancer TFBSs to optimize the sequences. We have incorporated this observed process into our RL framework.
3

---

Published as a conference paper at ICLR 2025
3 METHOD
3.1 PROBLEM FORMULATION
sequence of nucleotides with length I,, where I; E {A, C, G,T is the nucleotide at the i-th position. We assume the availability of a large-scale dataset of CRE sequences with fitness measurements D ((x)), where N is the number of sequences in the dataset and f(X) represents the fitness for X. On the complete dataset D, we train a reward model oracle to predict a CRE's fitmess, which is used for the final evaluation. Additionally, we partition subset of low-fitness sequences Diow C DD to train both an AR model and a reward model surrogate that reflects the distribution learned from the offline dataset. Different experimental settings may utilize different types of reward models to guide the RL optimization process if applicable. Our ultimate goal to generate sequences predicted to have high fitness by the oracle.
We define DNA sequence X (.1,-.. .LL
(XN, ([x))),
To align with RL terminology, we formulate sequence generation as a Markov Decision Process (MDP). The state 8; corresponds to the partial sequence generated up to time step i, while the action a, E {A, C, T} represents the nucleotide selected at position Our generative model serves as the policy To, which learns output probability distribution over possible actions (nucleotides) given the current state (partial sequence). The generation process terminates when the sequence reaches length L.
3.2 RL-BASED FINE-TUNING FOR AUTOREGRESSING DNA MODELS
Pre-training CRE-specific AR Model. We adapt MlyAnaaDNA (Nguyen et al. 2024b) for our AR model by continual training on Drow following Lal al. (202)). While HyenaDNA achieves strong performance on DNA tasks with linear complexity (Appendix C). it was originally trained on the whole human genome rather than short regulatory CRE sequences. To bridge this length and func- tional gap, we perform continual pre-training better capture CRE-specific regulatory patterns (Experimental evidence in Appendix C.T). We refer to this process as pre-training since it employs unsupervised learning on the sequence data.
The pre-trained AR model serves as our initial policy Ty in the RL framework, predicting the proba- bility of each nucleotide given previous ones. We formulate this as a sequential process where each action 0; E {AA,C (A,C,GG G, T selects a nucleotide at position i. The policy trained by minimizing:
L
min E Eno Dioe
- log To(a; a1,..., ai.".....-......... 0i-1)
where Taaa; a , -1=1) represents the probability assigned by the policy to selecting nucleotide a, at position given the sequence of previous actions (nucleotides). Pre-training on Dio helps the policy learn to generate sequences that already resemble the true CRE distribution (Jin et al. 2020 Chen et al. 2021), providing a good initialization for subsequent RL fine-tuning.
RL-Based Fine-tuning for AR DNA Models. Next, we formulate the RL fine-tuning process MDP, as illustrated in Figure 2 this formulation, the state s; corresponds to the partial sequence generated up to time step i, while the action a, represents the nucleotide selected by the policy mo- The reward r(s;-1, a;) is defined as a combination two types of rewards: TFBS reward Freas and fitness reward Tritess shown in equation
Friiness
if I, if a, results in TFBS E T, otherwise.
(()-1,
PrrBs(t) (t),
Here, Fiimees is applied when i is the final time step of the episode (i = L.), and represents the fitness value of the generated sequence as evaluated by the reward model. On the other hand, 7TFBS is a reward applied whenever a TFBS / E T =[t, 12, la...., selecting a, Details on how TFBSs are identified can be found in Appendix D The specific values
tn .t....,,,, is identified in the sequence after
of PIFES (t) are introduced in Section 3.3 Negative rewards are assigned for generating repressive TFBSs, while positive rewards are given for generating activating TFBSs, as shown in Figure The overall objective is to maximize the expected cumulative reward:
4

---

Published as a conference paper at ICLR 2025
Actions
Negative Reward
0
a3 G
a
55
Q7
A
Policy
Generation www.reeeeeettiooooo
Fimess
BO5
[TA
A
TATGAG
TATGAGT...A
50
$1
S4
S5
36
$7
S80
States
Figure 2: AR generation of DNA sequence. The action a; represents the nucleotide to be ap- pended to the sequence, and the state Si-1 is the concatenation of all actions taken up to time - 1. negative reward given if an action generates a repressive TFBS, while a positive reward given for an activating TFBS. The final sequence is evaluated using reward model (oracle) to obtain its fitness. BOS represents the beginning of the sequence, and ATCG denotes nucleotide bases.
max ((0) ENT Σ E((s)11,a))
where J(0) represents the expected cumulative reward, is the length of the sequence, and (( ) the reward at each step. This objective enables the policy to generate DNA sequences with desired regulatory properties by leveraging both reward model guidance and domain-spectie knowledge of TFBS vocabulary.
Auxiliary RL Techniques. To optimize the policy Te. we employ the REINFORCE algo- rithm (Williams) 1992) following previous studies in molecule optimization (Ghugare et al. 2024). Additionally, we leverage a hill climbing replay buffer (Blaschke et al. (0200. which stores and samples high-fitness sequences during training to further guide exploration. We also apply entropy regularization (Ghugare et al. 2024), which inversely weights the log probabilities of selected ac- tions. This approach effectively penalizes actions with excessively high probabilities, thereby dis- couraging overconfident actions and promoting exploration of less likely ones. This combination of techniques allows the model to effectively balance exploration and exploitation, resulting in im- proved performance complex DNA optimization tasks. Ablation experiments supporting these designs can be found Appendix G.2
3.3 INFERENCE OF TFBS REGULATORY ROLES
As illustrated in Figure 3 our ap- proach to inferring TFBS regulatory roles consists of two steps. First, train a decision tree-based fitness pre- diction model using TFBS frequency features as input. Second, we lever- age the model interpretability tech- (017)
CACGATTCGACAA..GAAAGGGGGG
Frequancy Sequence
+0.8
nique SHAP values (Lundberg (2017)
Black-Box Model
determine the regulatory impact of each TFBS feature.
Explainer
-3,9
+7.1
To infer the regulatory impact of each TFBS, we first define the TFBS frequency feature of se- quence as vector h(X) where (x, denotes the frequency of the i- th TEBS in sequence r.. This feature
Fitness Score: 4.0
Fitness Score: 4.0
Figure 3: A black-box LightGBM model takes TFBS occur- rences as input, and SHAP values infer their contributions to CRE fitness prediction.
vector represents the occurrence pattern of TFBSs within the sequence, making it suitable for tabular data modeling. Details on extracting TFBS features by scanning the sequence can be found in Ap- pendix D Given the tabular nature of this data, employ LightGBM (Ke et al. 2017), tree-based
5

---

where S is a subset of features not containing JSSU {i}) is the model prediction when feature included, and f(S) is the prediction when feature excluded. This equation ensures that SHAP values fairly distribute the impact of each feature according to its contribution. To infer the reward (t) for each TFBS & T = {t1, .3...., }. we compute the mean SHAP value over the entire dataset. If the mean SHAP value does not significantly differ from zero (p-value >> 0.05, Published as a conference paper at ICLR 2025
model known for its interpretability and performance tabular datasets, to fit the fitness values of sequences. Li chosen because decision tree models, in general, offer better interpretabil- ity by breaking down the contribution of each feature in clear, hierarchical manner. Details of the LightGBM model can be found in Appendix F. After training, we evaluate the model's performance using the Pearson correlation coefficient between the true and predicted fitness values, shown in Table [ This evaluation metric helps us quantity how well the LightGBM model captures the relationship between TFBS frequencies and fitness values.
Based on the trained LightGBM model, we use SHAP values (Lundberg 2017) to interpret the im- pact of each TFBS on the predicted fitmess. SHAP values provide theoretically grounded approach to attribute the prediction of model to its input features by calculating the contribution of each feature (in our case, each TFBS) to the prediction. The SHAP value for the i-th TFBS sequence X, denoted as (X), is computed as:
1)! I - IS| 1 (f(S
(X) =
Σ AAG
(4)
n!!
determined by hypothesis testing), we set the reward of t to zero:
a (t),), if p-value < 0.05, otherwise,
FreBs (1) = 1000
(5)
where a is a tunable hyperparameter, and (a is the mean SHAP value TFBS across the dataset. This approach ensures that only statistically significant TFBSs contribute to the reward, and a controls the magnitude of the reward. Analysis of cell-type-specific TFBSs based on SHAP values provided in Appendix D.I
3.4 SUMMARY OF OUR METHOD
In summary, our method integrates key components. First, fine-tune AR generative model, pre-trained on CRE sequences, using RL. to optimize sequence generation (Figure 2 Second, we use data-driven approach to infer the role of TFBSs in cell-type-specific context and incorporate these insights into the RL process (Figure 3h. The complete workflow is detailed in Appendix G Algorithm
EXPERIMENT
4.1 EXPERIMENT SETUP
Datasets. We conducted experiments both yeast promoter and human enhancer datasets. The yeast promoter dataset includes two types of growth media: complex (de Boer et al., 2020) and defined (Vaishnav et al. (022). The human enhancer dataset consists of three cell lines: HepG2, K562, and SK-N-SH (Gosai et al., 2024). All paired CRE sequences and their corresponding fitness measurements were obtained from MPRAs (Sharon et al., (012)). Our data preprocessing was based on Lal et al. (2024) (Appendix A).. The DNA sequence length in the yeast promoter dataset is 80, while it is 200 for the human enhancer dataset. Each dataset represents a cell-tyee-peciffi scenario due to distinct TF effect vocabularies and regulatory landscapes.
Reward Model and Pre-training Dataset. In Section 4.2 and Section we show experiments under different settings. In Section 4.2 assume access to ideal oracle that guides the RL process, where both guidance and evaluation utilize oracle scoring. The policy wa. however, is pre- trained exclusively on D In contrast, Section
4.3 presents offline Model-Based Optimization

---

Published as a conference paper at ICLR 2025
(MBO) setting, which assumes no access to the ideal oracle during the optimization process. Instead, we rely solely on a surrogate model trained on Diow for scoring, while the policy is pre- teeitraind on the complete dataset D. Only after generating the final batch of candidate sequences do we submit them to the oracle for evaluation. Architecture details of the reward models are provided in Appendix
Baselines. We compare TACO against several established optimization approaches, including Bayesian optimization as implemented in the FLEXS benchmark (Sinai et (2,0),00) and evolu- tionary algorithms such as AdaLead (Sinai et al. 2020) and PEX (Anand & Achim (2022), , well as covariance matrix adaptation evolution strategy (CMAES) (Auger & Hansen (0012) using one- hot encoding, Additionally, we adapt the SOTA protein optimization method LatProtRL (Lee al., 2024) for CRE optimization. Given the lack of a powerful backbone model like ESM (Jain et al. 2022 in the DNA domain (Evidence provided in Appendix C.2), we removed the ESM-based latent vector encoding from LatProtRI. and refer to the resulting model as DNARI.. DNARL. can be viewed as sequence mutation- based PPO algorithm (Schulman et al. 2017) enhanced with a replay buffer mechanism. We do not compare with Gradient Ascent (GAs)-based approaches (Reddy et al. 2024), as our method and other baselines do not rely on differentiable surrogates. For a detailed discussion GAs, please refer to Appendix
Evaluation Metrics. We use three evaluation metrics: Top, Medium, and Diversity. Top is de- fined the mean fitness value of the highest-performing 16 sequences from the optimized set X = (X1,....Xx: ), where K = 256 represents the total number of sequences proposed in each round. Both Medium and Diversity metrics calculated using the highest-performing 128 sequences selected from the full set of 256 sequences. Medium refers to the median fitness value among these 128 sequences, while Diversity quantifies the median pairwise distance between all pairs within these 128 sequences. These metrics align with those used in Lee et al. (2024), except for Novelty, which we omit because DNA sequences lack well-defined validity criteria, resulting in disproportionately high novelty scores with limited informative value. For details, see Appendix [F] We report the mean and standard deviation of all evaluation metrics across five runs with different random seeds. Standard deviations appear in parentheses in these tables.
Implementation Details. The architecture of our AR model is based on HyenaDNNA-IM following the approach outlined in Lal et al. (2024). In Section 4.2 we pre-trained all policies on the subset Diow- In Section 4.3 we directly used regLM as the initial policy, but without adding prefix tokens to simulate unsupervised training Lal et al. 2024 All experiments were conducted
a single NVIDIA A100 GPU. During optimization, we set the learning rate to 5e-4 for the yeast task and le-4 for the human task. The hyperparameter o, which controls the strength of the TFBS reward in equation 5 was set to 0.01. We min-max normalized all reported fitness values and the rewards used for updating the policy, while the reward models were trained the original fitness values. For each round, we generated K = 256 sequences, each with fixed length, consistent with the datasets. A total of L' 100 optimization rounds were conducted, and the metries were computed using the final nound's22 X
4.2 FITNESS OPTIMIZATION WITH AN ACTIVE LEARNING SETTING
In this setting, we assume that the oracle trained on the complete dataset D is accessible during the RL process. Following et al. (2024); Ghugare et al. (2024). we partitioned each dataset into three subsets-easy, middle, and hard based on fitness values, denoted as D Detailed partitioning strategies are provided in Appendix AA] For each difficulty level, we pre-trained the AR model on Drow to simulate optimization starting from low-fitness sequences.
Yeast Promoters. As shown in Ta- ble [2] optimizing yeast promoters is rel- atively easy, with most methods generat- ing sequences that significantly exceed the dataset's maximum observed fitness val- ues. For such sequences, the results are reported as 1. Therefore, we only present the results for the hard subset, while the complete results are available in Appendix
Methud AdaLead
Yous Promoter (Complex)
Youst Promoter (Dtined)
Top
Mediam
Diversity
Top
Medium
9:8
933 (2.59)
7.6 (0.89)
6.4 (0.55)
BD
5.6 (5.57)
(1.0))))
CMAES DNARL
0.79 (0.00)
30.0 (2.5)
0.44 (0.03)
30.4 (2.3)
7.7 (0.48)
10.2 (1.4)
TACO
52.8 (2.77)
40.6 (3.65)
Table 2: Results on yeast promoter datasets (hard).
large- 1m- 1
se-seqlen-hf
7

---

Published as a conference paper at ICLR 2025
HepG2-easy
HepG--mmddde
HepG2-hard
Medium
Diversity
Top
Medlum
Diversity
Top
Medium
Diversity
PEX
0,33 (0.02)
0.89
20.2 (6.57)
(0.04)
0.86 (0.04)
(7.12)
0.85
0.82 (0.02)
(2.65)
AdaLeod
(0.00)
0.75 (0.00)
5:2 (0:45)
(0.00)
(0.00)
(4.00)
(0.02)
(0.02)
(1.77
0.66 (0.06)
0.50 (0.00)
(1.6(891))
(0.05)
0.58 (0.05)
(7.8))
(0.04)
(0.08)
(5.07)
CMAES
0.61 (0.00)
0.42 (0.04)
(4.04)
0.57 (0.02)
0.43 (0.03)
(3.24)
0.59 (0.00)
0.43 (0.02)
77.2 (5.17)
DNARI.
0.79 (0.07)
0.71 (0.00)
(2.2(0.08)
0.63 (0. 14)
0.84 (0.00)
(0.00)
0.76 (0.04)
0.72 (0.01)
(3.43)
TACO
0.78 (0.01)
0.75 (0.01)
151.8 (2.39)
0,76 (3.01)
11775 (0.01)
(7.13)
0.76 (0.01)
0.74 (0.01)
131.8 (4.27)
K562-racy
K562-middle
K562-hard
Method
Top
Medium
Diversity
Top
Medium
Diversity
Top
Medium
Diversity
PEX
0,95 (0.01)
0.93 (0.01)
21.8 (9.68)
(3.01)
0.92 (0.01)
(1.62)
(0.01)
0.92 (0.02)
(1.34)
AdaLead
0.85 (0.01)
0.84 (0.01)
(1.00.00)
0.85 (0.01)
0.84 (0.01)
(1.87)
0.85 (0.01)
0.84 (0.01)
(.8((.54))
B0
0.70 (0.13)
0:65 (0.12)
(5.32)
0,76 (0.05)
0.70 (0.05)
(5.55)
0.74 (1.003
0.70 (0.04)
37.0 (6:52)
CMAES
0,70 (0.05)
0.42 (0.02)
78.8 (4.09)
0.79 (0.00)
0.50 (0.03)
76.0
0.73 (0.05)
0.47 (0.05)
76.8 (4.55)
DNARL
0.89 (0.04)
0.87 (0.01)
(3.72)
0,50 (0.02)
0188 (0.01)
26.3 (1.28)
0.89 (0.01)
0.87 (0.02)
17.5 (3.33)
TACO
0.2
0.91
124.6 (3.51)
(0.01)
0.50 (0,52)
(2.50
0:93
(0.01)
1256 (2.88)
SK-N-SH-easy
SK-N-SH-awdddle
SK-N-SH-hard
Method
Top
Medium
Diveritty
Top
Medium
Diversity
Top
Medium
Diversity
PEX
(0.01)
0.86 (0.00)
22:2 (5.93)
0,92 (0.02)
0.88 (0.01)
23.8 (7.85)
0.90 (0.02)
0.85 (0.03)
(2.74)
Adal.cod
0.84 (0.08)
0.82 (0.08)
(1.52
0.81 (0.06)
0.80 (0.06)
(3.05))
0.79 (0.05)
0.78 (0.05)
14.4 (4.45)
BO
0.68 (0.07)
0.62 (0.07)
39.8 (7.89)
0.71 (0.08)
0.64 (0.10)
40.4 (4.83)
0.71 (0.06)
0.63 (0.04)
39.9 (6.00)
CMAFS
0.73 (0.04)
0.45 (0.02)
(3.09)
0.74 (0.01)
0.45 (0.03)
(3.00))81)
0.74 (0.07)
0.44 (0.03)
(3.54)
DNARL
0.83 (0.21)
0.80
35.42
(0.01)
(0.01)
(1.55)
(0.01)
(3.213
TACO
(0.01)
(0.02)
(4.28)))77)
(0.01)
0.85 (0.01)
(2.10(
(0.00)
(0.01)
(3.14)
Table 3: Performance comparison human enhancer datasets.
Table 14
Among the baselines. only CMAES fails to fully optimize sequences to the maximum fitness value, although it demonstrates strong performance in terms of diversity. Our method not only achieves the maximum fitness but also exhibits the highest diversity compared to all other approaches.
Human Enhancers. Optimizing human enhancers presents a more challenging task. As shown in Appendix A Table [ the 90th percentile min-max normalized fitness values for HepG2, K562, and SK-N-SH in the dataset D are 0.4547, 0.4541, and 0.4453, respectively-Iess than half of the maximum observed. In Table 3 TACO demonstrates superior performance compared to the baselines. For the HepG2, PEX achieves the highest fitness score, but its diversity is typically below 20. In contrast, TACO attains SOTA fitness for K562 and SK-N-SH cell lines while maintaining significantly higher diversity across all datasets (over 1/3 higher than CMAES, which has the highest diversity among baselines).
Evaluation by Optimization Round. As shown in Figure f we present the evaluation results after each round of optimization. We observe that AdaLead, greedy-based algorithm, quickly finds relatively high-fitness sequences at the initial stages. However, its diversity drops rapidly, causing the fitness plateau and get stuck in local optima. In contrast, PEX demonstrates steady increase in fitness, but it consistently maintains a low diversity throughout. Only TACO not only achieves a stable increase in fitness but also maintains high diversity due to its generative model paradigm.
4.3 OFFLINE MODEL-BASER OPTIMIZATION
We present results of offline MBO (Reddy et al. 2024; Uehara et al.) 2024), where the dataset D partitioned into a subset Domine The AR model was pre-trained on D simulate real-world scenarios where sequences are abundant but their fitness labels are unknown (Uehara et al.] 2024). Meanwhile, we trained a fitness prediction model (surrogate) on the smaller Doftine to guide the RL process. The maximum fitness threshold in Doffine is defined as the 95th percentile of D. Unlike Section 4.2 where the true oracle is known, here we lack such knowledge. Details of why we need offline MBO and data partitioning strategies are provided in Appendix H Since the oracle is not visible during the optimization process, we introduce an additional evaluation metric: the average pairwise cosine similarity of the embeddings of the proposed sequences as generated by the oracle
8

---

Published as a conference paper at ICLR 2025
Top Fitness
Madian Fitness
Diversity
150
A500 HS HS-N--I
b.8
100
0.0
50
0.4
0.4
20
80
80
100
40
60
100
20
10
80
150
pou HS N
0.8
100
0.0
0.4
1000
20
450
35
100
0.0
0.00 H5
0.4
0.4
20
80
100
20
40
60
100
20
60
80
100
TACO
BO
PEX
AdaLead
Figure 4: Evaluation metrics by optimization round for TACO, BO, PEX and Adalead. Shaded regions indicate the standard deviation of 5 runs. The x-axis indicates the number of rounds.
model. This metric, referred to as Emb Similarity, quantifies the diversity of the final proposed sequences the latent featre space.
Table presents the results the K562 dataset. Under the offline MBO, the performance of all methods degrades compared to the oracle-guided setting. The overall trends across methods are consistent with those observed in Section 4.2 TACO achieves results in Top and Median that comparable to PEX while significantly outperforming other optimization methods in terms of Di- versity. The complete offline MBO results for all datasets are presented Appendix K The results of lowering the fitness threshold for the offline dataset are presented in Appendix J] We also include two conditional generative models, regLM (Lal et al. 2024 and DDSM (Avdeyev al., 2023) These methods maintain high diversity in the generated sequences; however, the fitness of the gen- erated sequences is generally inferior to that achieved by most optimization methods. See detailed discussion and implementation details of generative models in Appendix
Model
Top f
Medium
Diversity
Emb Similarity 4
PEX
0.76 (0.02)
0.73 (0.02)
15.8 (4.97)
0.97 (0.01)
AdaLead
0.66 (0.08)
0.58 (0.06)
63.2 (70.01)
0.88 (0.12)
BO
0.71 (0.07)
0.64 (0.08)
43.6 (6.91)
0.87 (0.04)
CMAES
0.66 (0.02)
0.44 (0.03)
79.2 (3.83)
0.35 (0.03)
reglm
0.69 (0.02)
0.47 (0.01)
149.60 (0.49)
0.38 (0.02)
DDSM
0.43 (0.00)
0.40 (0.00)
93.40 (0.49)
0.80 (0.00)
TACO
0.75 (0.09)
0.72 (0.10)
102.6 (20.14)
0.97 (0.04)
Table 4: Offline MBO results for human enhancers (K562).
4.4 ABLATION STUDY
Fine-tuning a pre-trained AR model with RL and incorporating the TFBS reward are our key con- tributions. We conducted ablation experiments in the setting described in Section 4.3 Results shown in 5
The effect of pre-training. Pre-training on CRE sequences proves to be highly beneficial. While the "w/o pretraining" setup (which uses a randomly initialized policy) occasionally discovers sequences with high fitness, it underperforms the Medium metric by 0.03, 0.12, and 0.03 compared to the

---

Published as a conference paper at ICLR 2025
second- best result across datasets. This demonstrates that pre-treeeeete vre-training allows the policy to begin in relatively reasonable exploration space, enabling it to identify a large number of suitable sequences more efficiently.
The effect of TFBS reward. Incorporating the TFBS reward enhances the Mcdium metric of TACO. The method outperforms the "w/o TFBS reward" baseline by margins of 0.02, 0.01, and 0.02, re- spectively. These prior-informed rewards guide the policy to explore a more rational sequence space efficiently. Moreover, the biologically guided TFBS reward is surrogate-agnostic, with the poten- tial to achieve a similar effect to the regularization applied to surrogates in Reddy et al. (2024), by avoiding excessive optimization towards regions where the surrogate model gives unusually high predictions. The differences in the Top and Diversity achieved by various models are relatively mi- nor, with no consistent conclusion. As the a increases from the default value of 0.01 to 0.1, our method shows improved performance in both Top and Medium metrics for K562 and SK-N-SH datasets. However, this improvement comes at the cost of a rapid drop in diversity. Interestingly, all metrics for the HepG2 dataset worsen as grows. We hypothesize that this discrepancy arises from the quality of TFBS reward, precomputed using the LightGBM model, varying across datasets. We recommend carefully tuning in real-world scenarios.
Dataset
Setting
Top f
Medium t
Diversity f
Emb Similarity
TACO 0.01)
0.69 (0.03)
0.60 (0.05)
141.2 (1.92)
0.82 (0.05)
w/o me-training
0.68 (0.00)
0.55 (0.02)
139.4 (2.30)
0.69 (0.02)
HepG2
w/o TFBS reward
0.66 (0.05)
0.58 (0.07)
140.8 (1.64)
0.81 (0.05)
0.1
0.65 (0.06)
0.58 (0.06)
138.6 (3.21)
0.86 (0.04)
TACO (a 0.01)
0.75 (0.09)
0.72 (0.10)
102.6 (20.14)
0.97 (0.04)
pre-training
0.66 (0.15)
0.59 (0.16)
103.6 (25.77)
0.83 (0.14)
K562
w/o TFBS reward
0.76 (0.07)
0.71 (0.08)
106.2 (20.90)
0,94 (0.05)
= 0.1
0.78 (0.01)
0.77 (0.01)
82.8 (4.02)
0.99 (0.00)
TACO (a 0.01)
0.68 (0.08)
0.62 (0.08)
121.4 (7.86)
0,90 (0.03)
wlo pre-training
0.69 (0.02)
0.57 (0.06)
131.8 (11.17)
0.74 (0.11)
SK-N-SH
w/o TFBS reward
0.67 (0.06)
0.60 (0.06)
111.6 (12.86)
0.89 (0.04)
= 0.1
0.71 (0.01)
0.65 (0.02)
121.2 (5.45)
0.90 (0.05)
Table 5: Ablation study on the effect of pre-training and TFBS reward.
5 LIMITATIONS
There are still several areas for improvement in our approach: (1) The TFBS candidates we use are derived from a fixed database, which bounds the upper limit of the TFBS reward. Exploring data-driven motif mining (Dudnyk et al. 2024) methods may help. (2) Currently, we infer the role of TFs based solely on TFBS frequency. In reality, interactions between TFs and their orientation can significantly impact their regulatory roles (Georgakoppollos-Sna Soorrss et al. 2023). Explicitly in- corporating these factors to model more complex TF activities could lead to further improvements. (3) Evaluating DNA sequence validity requires further attention. Unlike for molecules or proteins, defining valid DNA sequences is more challenging, necessitating additional metrics beyond trained reward models alone (Appendix F).
6 CONCLUSION
Designing CREs highly impactful task with increasing data availability. We propose TACO, generative approach that leverages RL. to fine-tune pre-trained autoregressive model. TACO incorporates biological priors through additional rewards for activator TFBS addition and repressor TFBS removal during the RL process. Our method generates high-fitness CREs while maintaining sequence diversity. To our knowledge, we are among the first to explicitly integrate TFBS motif information into machine learning- based CRE design. We believe this approach of incorporating such fundamental biological knowledge into generative models represents a promising direction for future research in the field.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported in part by the National Natural Science Foundation China No. 62376277 and No. 61976206, and Doubao large model fund.
REFERENCES
Weizhi An, Yuzhi Guo, Yatao Bian, Hehuan Ma, Jinyu Yang. Chunyuan Li, and Junzhou Huang. Modna: motif-oriented pre-training for dna language model. In Proceedings of the 13th ACM international conference bioinformatics, computational biology and health informatics, pp. 1-5, 2022.
Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant de- noising diffusion probabilistic models. arXiv preprint arXiv:2205.15011 2022.
Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In International conference on learning representations, 2019.
Anne Auger and Nikolaus Hansen. Tutorial ema-es: evolution strategies and covariance matrix adaptation. In Proceedings of the 14th annual conference companion Genetic and evolutionary computation, pp. 827-848, 2012.
Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. In International Conference Machine Learning, pp- 1276-1301. PMLR, 2023.
Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18 (10)11196-1033 2021.
Timothy L. Bailey, James Johnson, Charles E Grant, and William S Noble. The meme suite. Nucleic acids research, 43(W 1):W39- W49, 2015.
Gonzalo Benegas, Carlos Albors, Alan J Aw, Chengzhong Ye, and Yun S Song. Gpn-msa: an alignment-based dna language model for genome-wide variant effect prediction. bioRxiv, 2023.
Thomas Blaschke, Josep Arús-Pous Hongming Chen, Christian Margreitter, Christian Tyrchan, Ola Engkvist, Kostas Papadopoulos. and Atanas Patronov. Reinvent 2.0: an ai tool for de novo drug 5922, 2020.
design. Journal of chemical information and modeling,
Shannon Boye, Sanford L Boye, Alfred S Lewin, and William W Hauswirth. A comprehensive review of retinal gene therapy. Molecular therapy, 21(3):00--1 2013.
Binghong Chen, Tianzhe Wang, Chengtao Li, Hanjun Dai, and Le Song. Molecule optimization by explainable evolution. In International conference on learning representation (ICLR), 2021.
Francis Collins and Harold Varmus. A new initiative on precision medicine. New Joulaaa of medicine, 372(9):933-755, 2015.
Lucas Ferreira DaSilva, Simon Senan, Zain Munir Patel, Aniketh Janardhan Reddy, Sameer Gabbita, Zach Nussbaum, César Miguel Valdez Córdova, Aaron Wenteler, Noah Weber, Tin M Tunjic, et Dna-diffusion: Leveraging generative models for controlling chromatin accessibility and gene expression via synthetic regulatory elements. bioRxiv, 2024.
Bernardo P de Almeida, Christoph Schaub, Michaela Pagani, Stefano Secchia, Eileen EM Furlong, and Alexander Stark. Targeted design of synthetic enhancers for selected tissues in the drosophila embryo. Nature, 626(7997):207-211 2024.
Carl G de Boer, Eeshit Dhaval Vaishnav, Ronen Sadeh, Esteban Luis Abeyta, Nir Friedman, and Aviv Regev. Deciphering eukaryotic gene-regulatory logic with 100 million random promoters. Nature biotechnology, 3861),56-65, 2020.

---

Published as a conference paper at ICLR 2025
Kseniia Dudnyk, Donghong Cai, Chenlai Shi, Jian Xu, and Jian Zhou. Sequence basis of transcrip- tion initiation in the human genome. Science, 384(6694):cau00116. 2024.
Oriol Fornes, Jaime A Mastro-Mondragoo, Aziz Khan, Robin Van der Lee, Xi Zhang, Phillip A Richmond, Bhavi P Modi, Solenne Correard, Marius Gheorghe, Damir Baranašić, et al. Jaspar 2020: update of the open-access database of transcription factor binding profiles. Nucleic acids research, 48(D)):D87-DD 2020.
Xi Fu, Shentong Mo, Alejandro Buendia, Anouchka Laurent, Anqi Shao, Maria del Mar Alvarez- Torres, Tianji Yu, Jimin Tan, Jiayu Su, Romella Sagatelian, et al. A foundation model of tran- scription across human cell types. Nature, pp. 1-9, 2025.
Caixia Gao. The future of crispr technologies in agriculture. Nature Reviews Molecular Cell Biology, 19(5)2775-27, 2018.
Tianshun Gao and Jiang Qian. Enhanceratlas 2.0: an updated resource with enhancer annotation in 586 tissue/cell types across nine species. Nucleic acids research, 48(DI):D58-D66 2020.
Zijie Geng, Shufang Xie, Yingee Xia, Lijun Wu, Tao Qin, Jie Wang, Yongdong Zhang, Feng Wu, and Tie-Yan Liu. De novo molecular generation via connection-aware motif mining. In The Eleventh International Conference on Learning Representations, 2023.
Ilias Georgakopoulos-Soaree Chengyu Deng, Vikram Agarwal, Candace SY Chan, Jingjing Zhao, Fumitaka Inoue, and Nadav Ahituv. Transcription factor binding site orientation and order are major drivers of gene regulatory activity. Nature communications, 14(1):2333, 2023.
Raj Ghugare, Santiago Miret, Adriana Hugessen, Mariano Phielipp, and Glen Berseth. Searching for high-value molecules using reinforcement learning and transformers. In The Twelfth International Conference on Learning Representations, 2024.
Sager J Gosai, Rodrigo Castro, Natalia Fuentes, John C Butts, Kousuke Mouri, Michael Ala- soadura, Susan Kales, Thanh Thanh Nguyen, Ramil Noche, Arya S Rao, Machine- guided design of cell-type-targeting cis-regulatory elements. Nature, pp. 1-10, 2024.
Sven Heinz, Casey E Romanoski, Christopher Benner, and Christopher Glass. The selection and function of cell type-specific enhancers. Nature reviews Molecular cell biology, 16(3):144-154, 2015.
Jonathan Ho and Tim Salimans. arXiv:2207.125988 2022.
Classifier-free
diffusion
guidance.
arXiv preprint
Connie Huang, Richard W Shuai, Parth Baokar, Ryan Chung, Ruchir Rastogi. Pooja Kathail, and Nilah M Ioannidis. Personal transcriptome variation is poorly explained by current genomic deep learning models. Nature Genetics, 55(12):2056-2059 2023.
Moksh Jain, Emmanuel Bengio, Alex Hernandez-Gariia Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning. pp. 9786-98011 PMLR, 2022.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In International conference machine learning, pp. 4859. PMLR, 2020.
Alexander Karollus, Thomas Mauermeier, and Julien Gagneur. Current sequence- based models capture gene expression determinants in promoters but mostly ignore distal enhancers. Genome biology, 24(1):56, 2023.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: highly efficient gradient boosting decision tree. Advances neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2025
Avantika Lal, David Garfield, Tommaso Biancalani, and Gokcen Eraslan. reglm: Designing realistic regulatory dna with autoregressive language models. In International Conference on Research in Computational Molecular Biology, pp. 332-335, Springer, 2024.
Minji Lee. Luiz Felipe Vecchietti, Hyunkyu Jung, Hyun Joo Ro, Meeyoung Cha, and Ho Min Kim. Robust optimization in protein fitness landscapes using reinforcement learning latent space. In Forty-first International Conference on Machine Learning. 2024.
Siyuan Li. Zedong Wang. Zicheng Liu, Di Wu, Cheng Tan, Jiangbin Zheng, Yufci Huang, and Stan Z. Li. VQDNA: Unleashing the power of vector quantization for multi-species genomic sequence modeling. In Forty-firt International Conference on Machine Learning, 2024a.
Zchui Li, Yuhao Ni, William AV Beardall, Guoxuan Xia, Akashaditya Das, Guy-Bart Stan, and Yiren Zhao, Discdiff: Latent diffusion model for dna sequence generation. arXiv preprint
TrXiv:2402.06079. 2024b.
Johannes Linder and Georg Seelig. Fast activation maximization for molecular sequence design. BMC bioinformatics, 22:1-20, 2021.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2024.
Scott Lundberg. A unified approach to interpreting model predictions.
arXiv preprint
urXiv:1705.000 2017.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction the effects mutations on protein function. Advances in neural information processing systems, 34:29287-29303, 2021.
Wenyi Mo, Tianyu Zhang, Yalong Bai, Bing Su, Ji-Rong Wen, and Qing Yang. Dynamic prompt op- timizing for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26627-26636, 2024.
Eric Nguyen, Michael Poli, Matthew G Durrant, Brian Kang, Dhruva Katrekar, David B Li, Liam Bartie, Armin W Thomas, Samuel H King, Garyk Brixi, al. Sequence modeling and design from molecular to genome scale with evo. Science, 38666723))aad69336 2024a.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch- Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024b.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Aniketh Janardhan Reddy, Xinyang Geng, Michael H Herschl, Sathvik Kolli, Aviral Kumar, Patrick D Hsu, Sergey Levine, and Nilah M loannidis. Designing cell-typ--specific promoter sequences using conservative model-based optimization. bioRxiv, pp. 2024-06, 2024.
Anirban Sarkar, Ziqi Tang, Chris Zhao, and Peter Koo. Designing dna with tunable regulatory activity using discrete diffusion. bioRxiv, pp. 2024-05, 2024.
Yair Schiff, Chia Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov, Ca- duceus: Bi-directional equivariant long-range dna sequence modeling. In Forty-first International Conference on Machine Learning.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
13

---

Published as a conference paper at ICLR 2025
Jiawei Shao, Xinyuan Qiu, Lihang Zhang, Shichao Li, Shuai Xue, Yaqing Si, Yilin Li, Jian Jiang, Yuhang Wu, Qiqi Xiong, et al. Multi-layered computational gene networks by engineered tristate logics. Cell, 2024.
Eilon Sharon, Yael Kalma, Ayala Sharp, Tali Raveh-Sadka Michal Levo, Danny Zeevi, Leeat Keren, Zohar Yakhini, Adina Weinberger, and Eran Segal. Inferring gene regulatory logic from high- throughput measurements of thousands of systematically designed promoters. Nature biotechnol- ogy, 3066) 2012.
Sam Sinai, Richard Wang. Alexander Whatley, Stewart Slocum, Elina Locane, and Eric Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint arXiv:2010.02144 2020.
Ziqi Tang and Peter K Koo.
Evaluating the representational power of pre-trained dna language models for regulatory genomics. bioRxiv, pp. 2024-02, 2024.
Ibrahim Taskiran, Katina I Spanier, Hannah Dickmänken, Niklas Kempynck, Alexandra Pančíková, Eren Can Ekşi, Gert Hulselmans, Joy N Ismail, Koen Theunis, Roel Vandepoel, et al.
Cell-type-directed design of synthetic enhancers. Nature, 626(7997)2112-220, 2024.
Masatoshi Uehara, Yulai Zhao, Ehsan Hajiramezanali, Gabriele Scalia, Gökcen Eraslan, Avantika Lal, Sergey Levine, and Tommaso Biancalani. Bridging model-based optimization and generative modeling via conservative fine-tuning of diffusion models. arXiv preprint arXiv:2405.19673 2024.
Eeshit Dhaval Vaishnav, Carl G de Boer, Jennifer Molinet, Moran Yassour, Lin Fan, Xian Adiconis, Dawn Thompson, Joshua Z. Levin, Francisco Cubillos, and Aviv Regev. The evolution, evolvability and engincering of gene regulatory dna. Nature, 603((701): 2022.
Peter JM Van Laarhoven, Emile HL Aarts, Peter JM van Laarhoven, and Emile HL Aarts. Simulated annealing. Springer, 1987.
Ronald J Williams.
Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229 2569, 1992.
Cort J Willmott and Kenji Matsuura. Advantages of the mean absolute error (mae) over the root mean square error (rmse) in assessing average model performance. Climate research, 30(():79- 82, 2005.
Patricia J Wittkopp and Gizem Kalay. Cis-regulatory elements: molecular mechanisms and evolu- tionary processes underlying divergence. Nature Reviews Genetics, 13(1):59 69, 2012.
Tianhao Yu, Haiyang Cui, Jianan Canal Li, Yunan Luo, Guangde Jiang, and Huimin Zhao. Enzyme function prediction using contrastive learning, Science, 37966639) 2023.
Xi Zeng, Xiaotian Hao, Hongyao Tang, Zhentao Tang, Shaoqing Jiao, Dazhi Lu, and Jiajie Peng- Designing biological sequences without prior knowledge using evolutionary reinforcement learn- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 383-391, 2024.
Pengcheng Zhang, Haochen Wang, Hanwen Xu, Lei Wei, Liyang Liu, Zhirui Hu, and Xiaowo Wang- Deep flanking sequence engineering for efficient promoter design using deepseed. Nature com- munications, 14(1):6309, 2023.
Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self- supervised learning for molecular property prediction. Advances in Neural Information Process- ing Systems, 34:15870-15888 2021.
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana V Davuluri, and Han Liu. DNABERT-2. Efficient foundation model and benchmark for multi-species genomes. In The Twelfth Interna- tional Conference on Learning Representations, 2024.
Joseph M Zullo, Derek Drake, Liviu Aron, Patrick O' O'Hern, Sameer C Dhamne, Noah Davidsohn, Chai-An Mao, William Klein, Alexander Rotenberg, David A Bennett, et al. Regulation of lifespan by neural excitation and rest. Naurr, 574(7778):359-364 2019.
14

---

Published as a conference paper at ICLR 2025
A DETAILS OF DATASETS
Existing CRE fitness datasets were generated using Massively Parallel Reporter Assays (MPRAs), which enable high-throughput measurements of regulatory sequences. The yeast promoter dataset includes results from Iwo distinct media conditions: complex (de Boer et al. 2020 and de- fined (Vaishnav et al. 2022). In contrast, the human enhancer dataset consists of data from three different human cell lines (Gosai et al. 2024): HepG2 (a liver cell line), K562 (an erythrocyte cell line), and SK-N-SH (a neuroblastoma cell line). The fitness differences in human enhancers are substantial, which makes the optimization task more challenging. As shown in Table 6 the 90th percentile min-max normalized fitness values for HepG2, K562, and SK-N-SH in dataset D are 0.4547, 0.4541, and 0.4453, respectively.
Cell Line
75th Percentile
90th Percentile
HepG2
0.3994
0.4547
K562
0.3975
0.4541
SK-N-SH
0.3986
0.4453
Table 6: Enhancer fitness.
In Section 4.2
we adopted the dataset splits proposed by regLM (Lal et 2024). using their defined training set our full dataset, denoted as DD To simulate a progression from low-fitness to high- fitness sequences, we further partitioned into a subset Dew for pre-training the policy. Specifically, we define three difficulty levels-hard, medium, and easy-based on fitness percentiles of 20-40, 40-60, and 60-80, respectively, for both media conditions in the yeast dataset. Since yeast is a single-cell organism,
ensured that fitness levels remained consistent across both media. For the human enhancer datasets, we define the hard fitness range as values below 0.2, the medium range as values between 0.2 and 0.75, and the easy range values between 0.75 and 2.5. These thresholds were chosen to maintain fitness values below 0.2 in other cell lines, thereby simulating cell-type- specific regulatory scenario. The selected sequences within these fitness ranges form Diow. which is used for pre- pre-trainnn traininiing the policy.
B ENFORMER SERVES AS REWARD MODELS (ORACLE AND SURROGATE)
Enformer Avsec et al. 2021) is a hybrid architecture that integrates CNNs and Transformers, achieving SOTA performance across range of DNA regulatory prediction tasks. In study,
all CRE fitness prediction models are based on the Enformer architecture (Lal et al. 2024 Uchara
al. 2024). The primary distinction lies in the output: while the original Enformer model predicts 5,313 human chromatin profiles, we adapted it to predict single scalar value (Lal et al. 2024) representing CRE fitness.
The reward models for the human enhancer datasets retain the same number of parameters as the original Enformer. In contrast, for the yeast promoter datasets, we reduced the model size due to the simpler nature of yeast promoter sequences, following Lal et al. (2024). Specific architectural configurations are detailed in Table 7 For consistency, we directly utilized the pre-trained weights
provided by regLM (Lal et al., 2024) as our oracle. However, we independently trained our own surrogate model on Domlne.
Model
Dimension
Depth
Number of Downsamples
Human Enhancer
1536
Yeast Promoter
384
Table 7: Reward model hyperparameters.
15

---

Published as a conference paper at ICLR 2025
C DISCUSSION ON DNA FOUNDATION MODELS
Over the past year, there has been significant growth in the development of DNA foundation mod- els, with many new models emerging- However, most of these models, such as Caduceus (Schiff) et al.), DNABert2 (Zhou et al. 2024). and VQDNA (Li et al.] (024a), are based on BERT-style pretraining and lack the capability to generate DNA sequences. Among them, HyenaDNA (Nguyen et al. 2024b) is the only GPT-style DNA language model. Unlike traditional Transformer-basee architectures, HyenaDNA leverages a state space model (SSM), which provides linear computa- tional complexity, making it suitable for handling long DNA sequences with complex dependencies. Subsequent work based on HyenaDNA, such as Evo (guuyen et al. 2024a), has demonstrated the powerful DNA sequence generation capabilities of this architecture. Additionally, regL. (Lal al., 2024) has explored conditional DNA generation by employing a prefix-tuning strategy. where a cus- tomized token is used as the prefix of the DNA sequence guide the subsequent generation process. This approach has enabled reglm to effectively model context-dependent DNA sequence generation.
C.I EFFECT OF PRE-TRAINING ON SHORT CRES
Although HyenaDNA can be directly employed as an initial policy, its pre-training was conducted on IM-Iength sequences across the entire human genome. Therefore, described in Section 3.2 we further fine-tuned the initial oracle on CRE datasets. As shown in Table fine-tuning HyenaDNA on short functional CRE sequences leads modest performance improvements. We attribute these enhancements to the fine-tuning process, which exposes the model to a greater number of short sequences, thereby better aligning with the sequence lengths required for subsequent CRE de- sign tasks. Furthermore, this adaptation enhances the model's ability to capture functional regions specific to CREs.
Model
Top
Medium
Pre-trained HyenaDNA
0.749
0.723
Fine-tuned HyenaDNA
0.751
0.729
Table 8: Performance (hepg2 hard) comparison of pre-trained and fine-tuned HyenaDNA short CRE sequences.
C.2 LIMITATIONS OF CURRENT DNA FOUNDATION MDOELS
While there have been advancements in DNA foundation models, evidence suggests that they do yet match the capabilities of models like ESM (Vaishnav et al.) 22022). Specifically: (1) ESM embed- dings are known for their high versatility and are widely utilized in various downstream tasks, e.g., enzyme function prediction Yu et al.) (023). In contrast, as noted Tang & Koo (2024), DNA foun- dation model embeddings often perform no better than one-hot encodings. (2) ESM's language model head can achieve AUROC scores above 0.9 in pathogenic mutation prediction by directly cal- culating the log-likelihood ratio of reference and alternative alleles (Meier et al. 2021). However, DNA foundation models currently perform significantly worse, with AUROC scores below 0.6 as reported in [Bnnegas et al. (2023). (3) In addition to sequence- based DNA foundation models, some supervised DNA models have also been shown to exhibit limitations in distinguishing mutations across individuals Huang et al. (2023) and recognizing long-range DNA interactions Karollus et al.
2023
D TFBS SCAN AND FREQUENCY FEATURE PREPROCESSING
The Jaspar database (Fornes et al. 2020 provides detailed annotations of TFBSs. Each TFBS 6; corresponds to transcription f'ccorr that binds to it, regulating gene expression. Instead of repre- senting t, as a fixed sequence, it is described by a position frequency matrix M, E R1ax4 where L, is the length of the TFBS, and the four columns correspond to the nucleotides {A, C, G, T). The matrix encodes the likelihood of each nucleotide appearing at each position in the TFBS, making possible to capture variations in TF binding.
16

---

Published as a conference paper at ICLR 2025
We utilize FIMO (Find Individual Motif Occurrences) (Bailey et al. 2015) to scan each sequence for potential TFBSs. Given a sequence x and a matrix M,, FIMO evaluates each subsequence 2, in by calculating probabilistic score:
score M.) II P(na M[(R),
(6)
where P(ns Mk(k)) represents the probability of nucleotide ne occurring at position k in the matrix M, FIMO identifies the subsequences with the highest scores as potential occurrences of the TFBS.
frequency feature vector h(z) h2()) hn (X)] where (x) denotes the frequency of the i-th TFBS in se- quence x. This frequency feature vector is then used as input for the downstream prediction model. The use of frequency-band bbaee features, as opposed to binary indicators, captures the varying levels of TFBS occurrences in the sequence, allowing for a more nuanced understanding of the regulatory role of each TFBS. Given this tabular representation, we employ LightGBM (Ke et al.) (017), tree-based model known for its interpretability and effectiveness on tabular datasets, to predict the fitness values of sequences.
For each sequence X, FIMO outputs
D.1 CELL-TYPE SPECIFICITY OF TFBS FUNCTIONAL ROLES
After analyzing the contribution of each TFBS using SHAP values (Section 3.3 we further investi- gated whether TFBSs exhibit cell-type specific regulatory functions. Figure 5 and Figure 6 present Venn diagrams illustrating the functional classification of TFBSs across different conditions in yeast and human datasets, respectively. For yeast promoters (Figure(5) we observed that TFBSs maintain
Poztia
Ovartap
Wagetive TTES Overls
Menzal TP85 Owoke
Templre Settdd
seople
Defrec
Figure 5: Venn diagram categorizing TFBSs by their functional roles (positive, neutral, or negative) in yeast promoters across two media conditions (Complex and Defined). The substantial overlap across all categories indicates that TFBSs maintain consistent regulatory functions regardless of media composition, confirming the absence of condition-specifii TFBS activity within the same cell type.
consistent functional roles between Complex and Defined media conditions. This is evidenced by the substantial overlap in the Venn diagrams for positive, neutral, and negative regulatory categories. Since both conditions represent the same cell type with only differing media composition, this result confirms that TFBS functionality remains largely invariant to environmental changes within a single cell type in yeast. In stark contrast, the human enhancer analysis (Figure () reveals pronounced cell-type specificity in TFBS functionality. Across HepG2, K562, and SK-N-SH cell lines, we iden- tified numerous TFBSs that exhibit different regulatory roles depending on the cellular context. For instance, certain TFBSs function as positive regulators in one cell line while acting as negative reg- ulators or remaining neutral in others. The presence of cell line-specific TFBSs in each functional category (positive, neutral, and negative) provides compelling evidence for the context-dependent activity of transcription factors in human cells. These comparative findings highlight a fundamental difference in regulatory mechanisms: while yeast exhibits consistent TFBS functionality across dif- ferent environmental conditions within the same cell type, human enhancers demonstrate significant cell-type specificity in how transcription factors influence gene expression. This cell-type dependent
17

---

....... h. (X)], where (xx denotes the frequency of the i-th Published as a conference paper at ICLR 2025
POSTiN THIS Overko
Seutaal THIS Overup
Sagge THIS Getrip
36421
2ANS1
50N-1
Figure 6: Venn diagrams categorizing TFBSs by their functional roles (positive, neutral, or nega- tive) across three human cell lines (HepG2, K562, and SK-N-SH). The significant number of cell line-specific TFBSs in each category demonstrates that transcription factors can adopt distinct reg- ulatory functions depending on cellular context, revealing the cell-type specificity of TFBS activity in human enhancers.
regulatory flexibility in humans likely contributes the greater complexity and diversity of gene pression patterns observed across different human tissues. Our results emphasize the importance of considering the cell-type context when designing synthetic enhancers for human applications, as TFBS functionality cannot be assumed to remain constant across different cellular environments.
E DETAILS OF LIGHTGBM
We utilized LightGBM (ee et al. 2017) to train models that directly predict CRE fitness based on TFBS frequency features, enabling to infer the cell type-specific roles of individual TFBSs. To infer the regulatory impact of each TFBS, we first define the TFBS frequency feature of a sequence X vector h(X)= [hi (X), hx()). .....
TFBS in sequence X. The LightGBM model is trained to map the TFBS frequency features to the
corresponding fitness values of sequences, using the objective function:
Σ ((Fx)).(((()))) 7 XED
min
where S(X) is the true fitness value of sequence X, (h(()::) is the fitness value predicted by the LightGBM model parameterized by y using the TFBS frequency feature vector h(X). The term d ((b())))) represents distance metric measuring the discrepancy between the true and predicted fitness values.
For each dataset, independently trained LightGBM regression model. The specific hyperpa- rameters used in our model are listed in Table g in Section 4.2
Parameter
Value
Objective
Regression MAE GBDT
Metric
Boosting Type
Number of Leaves
63
Learning Rate
0.05
Feature Fraction
0.7
Seed
Random State
Table 9: Hyperparameters used for training the LightGBM regression model.
18

---

Published as a conference paper at ICLR 2025
yeast
human
Metric
complex
defined
hepg2
k562
sknsh
MAE
0.63
0.65
0.65
0.65
0.66
RMSE
0.63
0.64
0.56
0.57
0.58
Table 10: Ablation study comparing different metrics on CRE fitness prediction for yeast and human datasets for training LightGBM models
We experimented with various loss functions corresponding to the distance metrie in Equa- tion equation 7 specifically testing Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) dennedd below:
dxan(/((),,/(((()):))) =
Σ ((xx)
(8)
iDoow
X
dRMSE( (xx)s
Σ XED
(1(x) ((x)))(((x):)
(9)
Daw
We also varied other hyperparameters including learning rates {0.01, 0.05} and number of leaves (31, 63). Our preliminary experiments indicate that learning rate and the number of leaves have minimal impact on the results, while the choice of loss function significantly affects performance. The results for these two loss functions are shown in Table T0 We found that MAE consistently outperformed RMSE. This is likely because TFBS occurrences are highly sparse, and MAE tends to perform better with sparse features (Willmott Matsuura, 2005). Therefore, we selected MAE as the final loss function for training LightGBM models.
F CHALLENGES IN EVALUATING GENERATED DNA SEQUENCES
Unlike molecules and proteins (Uchara al.) 2024 which inherently possess well-defined physical and chemical properties, DNA sequences lack such structural constraints. For example, molecular structures are subject to physical properties like bond angles and energy states, while protein se- quences are evaluated based on their 3D folding stability and interactions, making it straightforward to filter out physically implausible designs. Therefore, in molecule and protein design, oracle- predicted fitness is often supplemented with physical property constraints to ensure the plausibility of generated candidates. This helps exclude a significant number of physically infeasible structures, enhancing the relevance of the optimization process. However, DNA sequences pose unique chal- lenge in this regard. Unlike molecules or proteins, DNA's plausibility cannot be easily assessed through physical properties, its functional attributes are primarily determined by its interaction with transcription factors and other regulatory proteins in a context-specific manner. Furthermore, current MPRA datasets are typically generated from random sequences, meaning there is inherent concept of plausibility in the data itself.
Our observations further highlight this challenge. In our experiments, we found that the novelty val- ues of generated DNA sequences were disproportionately high compared to the initial low-fitness sequences, making the novelty metric less informative. This behavior suggests that DNA sequences tend to diverge significantly from their starting points during optimization, regardless of their bio- logical relevance or plausibility. Due to these limitations, we exclude the Novelty metric and instead focus evaluating the generated sequences using Fitness and Diversity metrics, which better cap- ture the optimization objectives for CRE design.
However, believe that developing appropriate metrics for evaluating DNA sequence plausibility remains highly significant for CRE design. Beyond essential experimental validation (Gosai et al., 2024 Vaishnav al. (2022). future work could incorporate assessment methods proposed in Zeng ct al. 2024p Avdeyee et al. (2023) to more comprehensively evaluate the biological plausibility of generated sequences.
19

---

Published as a conference paper at ICLR 2025
G MORE DETAILS OF TACO
G.1 ALGORITHM OVERVIEW
The overview of our algorithm TACO (the setting is in Section 4.2 shown in Algorithm
Algorithm TACO: RL-Based Fine-tuning for Autoregressive DNA Models
Require: Low-fitness dataset Diow. TFBS vocabulary T, Oracle qu Pretrained AR model TT, Num- ber of Optimization Rounds E
Preprocessing:
2: Train LightGBM model on TFBS frequency features h(X) from dataset D
3: Compute SHAP values (x) for each TFBS tj
4: Update TFBS rewards TTFBS (t) based equation
5: for round e = to E do
6: Sample batch of sequences {Xxx} (K 256) from policy тe
7: for each sequence X do
8:
for time step
to I. do
9:
Generate nucleotide using Ty (as((a Observe state 8; (a),..., (4) if a, results in TFBS E then
11:
Assign reward r(s--1, a;) (tr
13:
else
Assign reward a()
15:
end if
16:
end for
17:
Obtain fitness reward Thimess from oracle Tv (Xx) Compute total reward R Ete r (8i-11 as) + Tittnsss
18:
19:
end for
20:
Update policy Ty using REINFORCE:
0 + (o(y(()))))))
21:
end for
G.2 THE EFFECT AUXILIARY RL DESIGNS
As illustrated in Figure 7 we systematically evaluate two critical components of our auxiliary de- signs presented in Section 3.2 the hill-climb replay buffer and entropy regularization. First, we investigate the impact of the hill-climb replay buffer, which preserves past experiences with high fitness values. Our results demonstrate that incorporating this replay buffer significantly enhances the maximum fitness values achieved during exploration, aligning with findings from previous search (Lee et al., 2024; Ghugare et al.] 2024p. Subsequently, we examine the effectiveness of entropy regularization, which is specifically designed to promote exploration by increasing policy randomness and preventing premature convergence to suboptimal solutions. Our experimental re- sults reveal that this approach successfully leads to improved action diversity, underscoring its value in facilitating a more comprehensive exploration of the solution space.
H OFFLINE MODEL-BASHR OPTIMIZATION
In Section
present results under active learning setting (eee et al. (024), which assumes easy access to a perfect oracle for evaluating generated CRE sequences in the optimization process. However, this setting can lead optimization processes that overfit to imperfect oracle (trained only with observed data).
Here, consider alternative offline model-based optimization(MBBO) setting (Reddy et al. 2024 [ehara et al. 2024), which assumes that accessing the true oracle costly. but some labeled offline data is available. In this setting, surrogate model is trained on the offline dataset guide the
20

---

Published as a conference paper at ICLR 2025
Abletion on Resloy Suffer
Ablarion Entropy Regularization
Aeg.tatration
1
0g5
10.00
Figure Ablation study on supporting RL designs.
optimization process, and the final sequences are evaluated by the oracle. This approach helps mitigate overfitting to a mannmadee oracle" trained on limited data.
Figure 8 illlustrates an example (a single run on the yeast complex dataset). The left panel shows the curve of the Top fitness predicted by the surrogate as the iterations progress. Since the optimization is guided by the surrogate, the curve continues to increase. Initially, the right panel (representing Top fitness as predicted by the oracle) also rises steadily. However, around iteration 80, there is sharp increase in the surrogate's 's predicted fitness, while the oracle's predicted fitness exhibits a brief spike before declining. This behavior suggests that at 80 iterations, the optimization pro- cess discovers seemingly high-fitness point. However, the surrogate, believing this direction to be correct, continues optimizing, leading to an overestimation of the fitness. The oracle's ' actual score, however, does not continue increase significantly. This example demonstrates that in real optimization processes, the surrogate can be misled by spurious data points, further emphasizing the importance of the offline MBO setting.
Surrggate)
Top (Prestcted
13
212
23
Figure 8: Left: The curve of Top fitness predicted by the surrogate during iterations. Right: The corresponding Top fitness predicted by the oracle. The discrepancy highlights the potential for the surrogate to overestimate fitness due to spurious data points, emphasizing the need for offline MBO settings.
Specifically, we still use the oracle for the final evaluation of sequences, but we sub-sample portion of the data to create a predefined offline dataset. Following [eeaaa et al. (2024), the sub-sampling strategy involves randomly splitting the dataset in half and selecting sequences with fitness values below the 95th percentile to simulate a real-world scenario where observed data may have lower ceiling. This dataset is referred optimization process proceeds similarly to Section 4.2 except that each iteration is guided by the surrogate, with the oracle used only for final quality evaluation of the generated sequences.
Doman- A surrogate model is trained on Domine and the
21

---

Published as a conference paper at ICLR 2025
1 DISCUSSION AND IMPLEMENTATION DETAILS OF CONDITIONAL GENERATIVE MODELS
Although the objectives of generative models and optimization methods differ, both aim to propose samples that deviate from the observed real-world data. To this end, include discussion and comparison with SOTA generative models.
Let the data distribution be denoted as P(z), where each data point is paired with label y (e.g., the fitness of a CRE). The full dataset observed in the real world is represented as D = {(2;, Yi) Wi))E= In biological sequence data, x typically follows a reasonable underlying distribution P(z), which can be approximated using generative model (z)e without requiring knowledge of y. However, directly sampling from (x) often yields sequences with low fitness, as the distribution of y values (e.g., high-fitness regions) is typically narrow and sparsely represented in the data. Thus, an unconditional generative model is generally ineffective for designing biological sequences.
To address this limitation, conditional generative modeling can be employed. By training model to approximate P(x y) using the offline labeled dataset Daftine. we can condition on high observed fitness values y to theoretically generate high-fitness sequences. Formally, given a dataset where y is partitioned into discrete bins or ranges (e.g. high-fitness values), the conditional generative model is trained to maximize the likelihood. Subsequently, sequences are generated by sampling a conditioned on values corresponding to high fitness. However, in practice, this approach often underperforms because the distribution of high y values is extremely narrow, and the model struggles to accurately capture this region.
We compare our method against recent generative models, including the autoregressive generative model reglm (Lal et al., 2024) and the discrete diffusion model DDSM (Avdeyev et al. 2023). For evaluations, we adopted conditional generation strategies for both models. Specifically: regLM: The official pretrained weights were used. Sequences were generated by conditioning on the prefix label corresponding to the highest fitness score in each dataset. DDSM: This model was trained on our offline dataset, where labels above the 95th percentile were set to y = 1, and the remaining labels were set to y = 0 following [Uehara al. (2024). The conditional diffusion model was then trained using this binary labeling scheme, and sequences were generated by conditioning on y 1 for evaluation.
As shown in Table 18 both regLM and DDSM exhibit high diversity in their generated sequences but fail to match the fitness values achieved by optimization based methods. This limitation arises because generative models are designed to fit the observed data distribution P(x y), and such, their generated sequences are inherently constrained by the data's fitness distribution. It is also worth noting that reglm utilized official pretrained weights, which may have been exposed to data with higher fitness scores than our offline dataset. Even with this advantage, it fails to outperform optimization-basse methods. In contrast, our method builds upon pretrained distribution P're and further proposes new sequences by iteratively optimizing / Ppre (z) through feedback from an oracle or surrogate. The ultimate goal to reshape the distribution so that high-fitness sequences become more accessible during sampling.
J ANALYSIS OF GRADIENT ASCENT'S PERFORMANCE IN OFFLINE MBO SETTINGS.
As stated in Reddy et al. (2024), in offline MBO settings, directly applying Gradient Ascent (GAs) to surrogate is expected to generate adversarial examples with poor performance. However, in our experiments, we did not observe this phenomenon. Instead, performing GAs directly led surpris- ingly good results, which is an unexpected finding. To the best of our knowledge, prior CRE design work has not extensively explored GAs, with the exception of Reddy et al. 2022 However, this study did not include ablation study on regularization terms. Therefore, in the context of machine learning-basse based DNA CRE design, it remains an open question whether directly applying GAs to a differentiable surrogate could generate adversarial examples with poor performance. We acknowl- edge that, in an ideal scenario, surrogates trained on local data may indeed generate adversarial examples, and we believe this issue warrants further attention in future research.
22

---

Published as a conference paper at ICLR 2025
To further address this concern, validated GAs performance across different fitness quantiles (95 shown in Figure 9a| 80 shown in Figure 99] 60 shown in Figure 9c) using K562 cells (our default offline MBO setting the 95th percentile). Our GAs implementation directly operates on the one- hot encoded probability simplex following Reddy et al. (2024), which allows for smooth updates during optimization. Therefore, we report both the results the me-hhot-nnoded simplex (Prob) and the hard-decoded sequences optimized (Sequence) in each iteration. We reported the scores predicted by both the surrogate and oracle for these two representations. Our findings indicate: (1) For the 95th percentile, as shown in Figure 99 the fitness in the sequence space initially rises sharply but then drops. For the 60th percentile, as shown in Figure [9c] a similar pattern is observed in the oracle's predictions within the Prob space. This reveals gap between the surrogate and the oracle, as the surrogate's predictions consistently increase. This aligns with our expectations of the offline MBO setting, i.e., the surrogate cannot fully reflect the oracle. (2) However, the oracle's predictions do show significant improvement at the start, indicating that directly applying GAs to the surrogate can still benefit the oracle's ' results. This suggests that, under the current CRE data partitioning strategy, even a surrogate trained on low-fitness subsets can reasonably capture the trends of the oracle's predictions (although the surrogate itself, having never encountered high- fitness data, predicts much lower upper bounds).
(a) Results of GAs using the 95th (b) Results of GAs using the 80th (c) Results of GAs using the 60th percentile subset.
percentile subset.
percentile subset.
Figure 9: Comparison of GA results using different percentile subsets (95th, 80th, and 60th).
Based on the current evidence, we believe that the observed good performance of GAs may be an inherent property (possibly related to the inherent data distribution of CREs and our current data partitioning strategy). Even a surrogate trained on the 60th percentile subset can achieve decent performance. This is an interesting question for future research in CRE design.
However, we emphasize that our primary focus on designing optimization algorithms rather than relying on differentiable surrogate. Our current offline MBO setting has already made the task more challenging, achieving the intended goal of designing an offline MBO setting. Nevertheless, do not yet fully understand why GAs does not lead to significantly poor results. Figuring this out is left for future work, but we believe it is crucial for werechine-larning--ee
CRE design.
Besidies, we have also added the results of different methods (including GAs) guided by a surrogate trained on the 60th percentile shown in Table [1 Table 12 and Table 13 It can be observed that, despite the significant gap between the surrogate and the oracle under the 60th percentile training, GAS still achieves relatively good performance. Notably, under the 60th percentile setting, PEX, which performes well at the 95th percentile, shows moderate results, while CMAES, which previ- ously performes the worst, achieves excellent performance. Our TACO, in this setting, continues to maintain SOTA results.
Model
Top t
Medium t
Diversity t
Emb Similarity 1
PEX
0.54 (0.02)
0.48 (0.02)
16.4 (5.13)
0.80 (0.03)
AdaLead
0.50 (0.05)
0.42 (0.02)
146.6 (2.61)
0.36 (0.03)
BO
0.46 (0.04)
0.41 (0.02)
41.2 (6.91)
0.71 (0.07)
CMAES
0.55 (0.04)
0.41 (0.02)
78.4 (3.97)
0.41 (0.05)
GAs
0.59 (0.01)
0.51 (0.01)
136.20 (0.40)
0.80 (0.01)
TACO
0.56 (0.08)
0.50 (0.08)
134.6 (14.03)
0.77 (0,08)
Table 11: Results of HepG2 (60 Percentile).
23

---

Published as a conference paper at ICLR 2025
Model
Top T
Medium f
Diversity 1
Emb Similarity +
PEX
0.50 (0.06)
0.45 (0.04)
13.6
0.87 (0.02)
AdaLead
0.62 (0.09)
0.49 (0.10)
138.6 (20.7)
0.55 (0.11)
BO
0.55 (0.03)
0.44 (0.03)
43.6 (7.37)
0.66 (0.11)
CMAES
0.66 (0.07)
0.47 (0.05)
79.0 (4.36)
0.49 (0.06)
GAs
0.52 (0.02)
0.43 (0.00)
126.60 (1.20)
0.84 (0.01)
TACO
0.63 (0.04)
0.48 (0.02)
132.4 (25.7)
0.62 (0.21)
Table 12: Results of K562 (60 Percentile).
Model
Top f
Medium f
Diversity t
Emb Similarity 1
PEX
0.50 (0.11)
0.47 (0.11)
19.6 (3.21)
0.74 (0.04)
AdaLead
0.55 (0.06)
0.44 (0.03)
141.6 (13.24)
0.54 (0.09)
BO
0.55 (0.09)
0.46 (0.05)
48.8 (11.65)
0.72 (0.09)
CMAES
0.61 (0.09)
0.44 (0.03)
75.2 (1.92)
0.50 (0.05)
GAs
0.48 (0.01)
0.42 (0.00)
140.00 (0.63)
0.59 (0.01)
TACO
0.69 (0.04)
0.57 (0.06)
135.6 (5.5)
0.78 (0.06)
Table 13: Results of S-KN S-N-SSH (60 Percentile).
K
MORE EXPERIMENTAL RESUTLS
Since many conclusions are consistent across different datasets and settings, include significant portion of the experimental results here in the Apeendix. The complete experimental results for
yeast under the oracle-guided optimization setting (Section 4.2) presented in Figure [4 The
results for offline MBO (Section 4.3) are detailed in Table Table 16 Table
Table 18 and
Table 19
Promoter (Complex)
middle
hard
Method
Top
Medium 1
Diversity
Top
Medium
Diversity
Top 1
Medium
Diversity
PEX
(1.14)
8.4
AdaLead
1
23.4 (1.3)
9.0 (1.58)
7.6 (0.89)
22.6 (1.34)
(5.577)
CMAES DNARL
0.78 (0.13)
30.2 (2.68)
0.85 (0.02)
(1.52)
0.79 (0.09)
30.0 (2.5)
8.6 (2.14)
10.2 (1.14)
7.7 (0.48)
TACO
$2.2 (1.92)
48.8 (5.36)
52.8 (2.77)
Yeast Promoter (Defined)
middle
hurd
Top
Medium f
Diversity 1
Top
Medium f
Diversity
Top 1
Medium
Diversity
9.2 (0.84)
AdaLead
8.0 (2.35)
(1.0)
6,4 (0.55)
BO
23.0 (1.58)
22.8 (2.28)
23.0 (3.07).8)
CMAES
0.26 (0.36)
30.0 (2.92)
0.48 (0.17)
29.8 (1.3)
0.44 (0.33)
30.4 (2.3)
DNARL.
11.6 (3.04)
18.5 (8.00
10.2 (1.14)
(2.77)
(4.64)
(3.68)
Table 14: Performance comparison yeast promoter datasets (Guided by the Oracle).
24

---

Published as a conference paper at ICLR 2025
Model
Top f
Medium t
Diversity t
Emb Similarity 4
PEX
1.16 (0.09)
1.12 (0.08)
11.4 (57.60)
0,98 (0.01)
AdaLead
1.06 (0.02)
1.00 (0.02)
57.6 (0.55)
0.95 (0.00)
BO
1.09 (0.02)
1.03 (0.03)
24.4 (4.77)
0.97 (0.01)
CMAES
1.06 (0.07)
0.70 (0.12)
29.20 (0.45)
0.75 (0.05)
regLM
1.02 (0.00)
0.94 (0.00)
59.00 (0.00)
0.91 (0.01)
ddsm
0.94 (0.02)
0.79 (0.01)
58.20 (0.40)
0.81 (0.01)
TACO
1.06 (0.01) 0.98 (0.01) 57.4 (1.34)
0.93 (0.01)
Table 15: Offline MBO (95 Percentile) results (yeast promoter, complex).
Model
Top
Medium f
Diversity t
Emb Similarity $
PEX
1.19 (0.15)
1,10 (0.16)
10.40 (2.61)
0.98 (0.01)
AdaLead
1.02 (0.04)
0.98 (0.04)
8.20 (1.79)
0.98 (0.01)
BO
1.06 (0.03)
1.02 (0.02)
26.00 (2.24)
0.97 (0.01)
CMAES
0.79 (0.10)
0.39 (0.12)
30.80 (2.05)
0.59 (0.05)
LegLM
0.98 (0.01)
0.89 (0.01)
58.80 (0.40)
0.90 (0.00)
DDSM
0.92 (0.02)
0.81 (0.00) 56.20 (0.40)
0.86 (0.01)
TACO
1.10 (0.05) 1.03 (0.04) 46.00 (1.87)
0.97 (0.01)
Table 16: Offline MBO (95 Percentile) results (yeast promoter, defined).
Model
Top t
Medium t
Diversity t
Emb Similarity 4
PEX
0.75 (0.01)
0.73 (0.01)
13.6 (4.51)
0.98 (0.01)
AdaLead
0.59 (0.01)
0.52 (0.04)
34.2 (59.15)
0,84 (0.16)
BO
0.65 (0.09)
0.61 (0.10)
40.2 (6.14)
0.83 (0.13)
CMAES
0.57 (0.03)
0.41 (0.03)
77.2 (2.28)
0.45 (0.04)
regLM
0.65 (0.01)
0.48 (0.02)
150.00 (0.00)
0.28 (0.02)
DDSM
0.41 (0.00)
0.41 (0.00)
15.40 (0.49)
0.99 (0.00)
TACO
0.69 (0.03) 0.60 (0.05)
141.2 (1.92)
0.82 (0.05)
Table 17: Offline MBO (95 Percentile) results (human enhancer, HepG2).
Model
Top 1
Medium t
Diversity f
Emb Similarity 1
PEX
0.76 (0.02)
0.73 (0.02)
15.8 (4.97)
0.97 (0.01)
AdaLead
0.66 (0.08)
0.58 (0.06)
63.2 (70.01)
0.88 (0.12)
BO
0.71 (0.07)
0.64 (0.08)
43.6 (6.91)
0.87 (0.04)
CMAES
0.66 (0.02)
0.44 (0.03)
79.2 (3.83)
0.35 (0.03)
regLM
0.69 (0.02)
0.47 (0.01)
149.60 (0.49)
0.38 (0.02)
DDSM
0.43 (0.00)
0.40 (0.00)
93.40 (0.49)
0.80 (0.00)
TACO
0.75 (0.09)
0.72 (0.10)
102.6 (20.14)
0.97 (0.04)
Table 18: Offline MBO (95 Percentile) results (human enhancer, K562).
25

---

Published as a conference paper at ICLR 2025
Model
Top t
Medium t
Diversity t
Emb Similarity 1
PEX
0.69 (0.01)
0.68 (0,00)
17.8 (3.90)
0.98 (0.01)
AdaLead
0.59 (0.08)
0.56 (0.08)
8.6 (2.30)
0.96 (0.03)
BO
0.61 (0.09)
0.52 (0.08)
42.4 (4.77)
0.80 (0.08)
CMAES
0.58 (0.05)
0.42 (0.03)
78.6 (1.14)
0.40 (0.06)
regLM
0.61 (0.00)
0.47 (0.01)
149.60 (0.49)
0.38 (0.03)
DDSM
0.54 (0.00)
0.49 (0.00)
102.20 (1.17)
0.91 (0.01)
TACO
0.68 (0.08) 0.62 (0.08)
121.4 (7.86)
0.90 (0.03)
Table 19: Offline MBO (95 Percentile) results (human enhancer, S-KN-SH).
26


Reviews:
[{"review_id": "qcJ5tx05eU", "reviewer": "Reviewer_2GBK", "summary": "In this paper, the authors introduce TACO, a method for designing DNA cis-regulatory elements (CREs). TACO leverages reinforcement learning (RL) fine-tuning of a pretrained DNA autoregressive generative model to maximize CRE activity while maintaining diversity.  The approach involves training an \"oracle\" model on activity data and extracting transcription factor binding motifs (TF motifs) by interpreting the oracle's extracted features using SHAP values.\n\nTACO formulates the Markov Decision Process (MDP) problem with an empty initial state, where actions involve appending nucleotides to the sequence at each step. Episodes terminate after T steps. The oracle's prediction serves as the final reward, while intermediate non-zero rewards are assigned based on the extracted TF motifs to guide the search process.  Specifically, negative rewards are given for repressive motifs, and positive rewards are given for enhancing motifs.\n\nThe authors compare TACO to standard optimization methods using the same oracle on datasets for promoter activity in yeast and enhancer activity in human cell lines.  Their results demonstrate that while TACO does not necessarily yield higher predicted activity for generated CREs, it achieves greater diversity compared to standard optimization methods.", "questions": "Questions/Comments:\n\n- Algorithm 1 appears too early in the paper, as it references elements and equations introduced later. Moving it further down would improve the flow.\n\n- Equations 1, 3, and 4 could be moved to the appendix, as they are well-known in the field.\n\n- Could the authors provide more details on how the maximum number of steps T is determined?\n\n- Could the authors clarify the following statement: \"We set the maximum number of optimization iterations to 100, with up to 256 oracle calls allowed per iteration.\" Is one optimization iteration equivalent to one episode? If so, does \"256 oracle calls\" imply that T=256?\nWhy did the authors retrain HyenaDNA from scratch on D_star? Why not start from a pretrained HyenaDNA model?\n\n- The authors claim a lack of strong encoder backbones like ESM in the DNA field. This seems inaccurate, given recent publications like Nucleotide Transformer, DNABert, HyenaDNA, Caduceus, Evo, Borzoi, Enformer, and many others. If the authors disagree with the claims from these papers, they should provide a detailed argument.\n\n- Using \"medium\" for both a dataset and a metric could be confusing.\n\n- The authors frequently cite Almeida et al. but do not use their fly enhancer data in the evaluation. What motivated this decision?\n\n- At the end of the \"conditional DNA generative models\" section, the authors state: \"However, these generative methods are designed to fit existing data distributions, limiting their ability to design sequences that have yet to be explored by humans.\" However, TACO seems to suffer from the same limitation, as its exploration space is bounded by the trained oracle, which is itself limited by the available data distribution. I would appreciate the authors' perspective on this.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem of generating diverse and high-fitness CREs is a crucial area of biological research, and I commend the authors for their contribution to this field. \n\n- I particularly appreciate their exploration of RL fine-tuning, a novel approach in this domain with promising potential, given its success in other areas. \n\n- The authors' meticulous design of the oracle for each benchmark is commendable. \n\n- The automated extraction of TF motifs using SHAP values from the trained LightGBM oracle, coupled with its use in reward shaping for guided search, is an interesting approach. \n\n- The paper is well-written and structured, ensuring clarity and ease of understanding for the reader. The technical details are well-presented, and the comprehensive literature review effectively covers important and recent works.", "weaknesses": "The primary weaknesses of this paper lie in the experimental setup and the choice of baselines.\n\n- Across all datasets, the authors report that all methods generate sequences with strong activity. This suggests that the tasks may be \"too easy,\" potentially undermining their suitability for benchmarking generative models like TACO, which aim to generate high-activity sequences.\n\n- While the included baselines are relevant, they rely on older optimization techniques that have been consistently outperformed by modern generative methods, such as autoregressive models and diffusion models, particularly in terms of generating diverse data. Consequently, the reported results, while valid, are not particularly insightful. A more compelling analysis would compare different generative techniques for CRE design. The authors acknowledge in their literature review the emergence of diffusion models (D3 and DNADiffusion) and autoregressive models (regLM) specifically for CRE design. Benchmarking TACO against these methods is essential.\n\n- The introduced diversity metric primarily ensures that generated sequences avoid collapsing to a single mode. While this highlights the known limitations of standard optimization techniques like genetic algorithms, it does not comprehensively evaluate the overall quality of sequences generated by a generative model like TACO. Given that predicted activity appears to be non-discriminative, incorporating additional quality and diversity metrics from a data distribution perspective would be beneficial. I recommend exploring the validation pipeline introduced in D3, which is highly relevant in this context.\n\n- The introduction of reward shaping based on automatically extracted TF motifs is presented as a key contribution; however, the results in Figure 5 do not strongly support its efficacy. Specifically, TACO achieves comparable diversity with alpha=0 (no reward shaping) while maintaining high sequence activity. Including TACO with alpha=0 in Tables 2 and 3 might demonstrate its performance, as it would likely achieve the highest activity while maintaining high diversity, the primary differentiating factor in these studies.\n\nAlthough the investigated domain and technique are interesting and relevant, and the paper is well-written, the authors do not fully demonstrate the added value of their method. The benchmark also lacks critical baselines. I would be willing to reconsider my assessment if (1) the authors compare their method against at least one of D3, DNADiffusion, and regLM; (2) they introduce additional metrics to evaluate generated sequence diversity; and (3) they provide more convincing evidence for the effectiveness of their reward shaping technique.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "puke0XUVMD", "reviewer": "Reviewer_tdUH", "summary": "The paper presents a novel method for designing cis-regulatory elements (CREs) using a reinforcement learning (RL) framework that enables the generation of high-fitness, cell-type-specific, and diverse sequences. The primary objective is to produce CRE sequences that are capable of enhancing gene expression. Current methods, often reliant on greedy algorithms or directed evolution approaches, lack biological insight for guiding the exploration of sequence space. These traditional methods tend to get trapped in local minima, resulting in CREs that are limited in diversity and difficult to interpret biologically.\n\nMain Contributions\n\nDevelopment of an RL Framework: The authors introduce a reinforcement learning framework that fine-tunes a pre-trained autoregressive generative model, HyenaDNA, for designing high-fitness, cell-type-specific CREs. This approach also aims to maintain sequence diversity, addressing limitations in traditional methods.\n\nIntegration of TFBS Modulation: The RL process actively incorporates transcription factor binding sites (TFBS) to encourage biologically meaningful changes, such as removing repressor motifs and adding activator motifs. This guidance helps enhance the regulatory potential of the generated sequences.\n\nEvaluation Across Multiple Contexts: The proposed method is tested across different tasks:\nAn enhancer design task for three distinct human cell types.\nA promoter design task across two yeast species in varied media conditions, demonstrating the framework’s adaptability to different biological contexts.", "questions": "1.What motivated the choice of the HyenaDNA model beyond it being the only published autoregressive DNA language model? Is its receptive field size of 1 million excessive for training on CREs, given that the DNA sequence lengths are only 80 and 200?\n\n2. How was the yeast Enformer model trained? Specifically, what sequences were used, and what was the target variable for regression?\nWhy do the fitness percentiles selected for D range from 20-80%? Why not use the full range, such as 10-90%, to potentially capture a wider diversity?\n\n3. Did you consider using cosine similarity as a distance metric when training the LightGBM models?\n\n4. Have you thought about adding a feature in the LightGBM model to consider interactions between two TFBS, requiring that they be present together or at a specific distance? This could capture complex TFBS interactions as discussed in Georgakopoulos-Soares et al., \"Transcription factor binding site orientation and order are major drivers of gene regulatory activity\" (Nature Communications, 2023). (This could be added to the discussion.)\n\n5.Could you clarify whether a specific threshold or adaptive strategy is used to define \"low likelihood\" sequences in the regularization approach, and how this affects the exploration-exploitation balance in sequence generation?\n\n6.Why did you choose six sequences for the \"Top\" evaluation metric, and why generate a total of 128 sequences?\n\n7. Could it be more informative to report fitness values close to, but not capped at, 1 to show variability among high-fitness sequences? Additionally, could you provide the standard deviation of fitness scores across generated sequences for each method in Table 2? If the standard deviation is 1, could u sample more sequences? Would you also consider reporting on 'Low' or 'Bottom' fitness sequences to better understand model limitations and areas for improvement?\n\n8. Given that diversity appears beneficial for finding novel targets, would continuing beyond 100 iterations lead to further exploration and potentially better results? Additionally, could you elaborate on how initial conditions influence the observed diversity and fitness metrics, and whether different starting points could impact the algorithm’s optimization effectiveness?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper makes several contributions to the field of DNA sequence design:\n\nBiologically Guided Sequence Design: By integrating TFBS modulation into the RL framework, the approach maintains biological relevance, encouraging the addition of activators and removal of repressors, which enhances the model's potential for generating functional CREs.\n\nInnovation in CRE Design: The reinforcement learning paradigm offers a new way to explore the sequence space more effectively, overcoming the limitations of greedy and evolution-based methods that often produce low-diversity sequences.\n\nComprehensive Evaluation: The authors rigorously evaluate the framework’s performance on both enhancer and promoter design tasks, providing evidence for its flexibility and applicability to various regulatory elements and cell types.\n\nOverall, the paper is written well and of good quality.", "weaknesses": "The paper would benefit from a more extensive Discussion section to contextualize the results further. \n\nMinor Points\nInclude references to Appendix B/C in the introduction or caption of Table 1.\nIn the preliminary experiment described in the introduction, add a reference to Appendix E or specify the number of TFBS scanned for each model.\nIn Figure 2, clarify the meaning of the BOS token and C or T action symbols.\nFigure 3 is missing \"A\" and \"B\" labels in the visuals.\nIn Appendix F, correct the sentence, \"Our results indicate that only the metric has a significant impact on the final performance. The ablation results are summarized in Table 7.\"\nExplicitly describe the regularization technique (entropy regularization) discussed in Section 4.3 and reference it in the RL Implementation Details section.\nSection 4.3 might be better suited as an appendix to improve readability, and increase space for a larger discussion/conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z6ZzBBTn1F", "reviewer": "Reviewer_BjdH", "summary": "The authors propose a reinforcement learning (RL) method for designing cis-regulatory elements. The method uses a pre-trained autoregressive model of DNA sequences as its policy and fine-tunes this model throughout the training process. The method incorporates domain knowledge by adding a reward that encourages generated sequences to contain known transcription factor binding site motifs. The authors test their method against a number of relevant baselines on a set of yeast promoter and human enhancer design tasks, showing improved performance over the baselines.", "questions": "* The intermediate TFBS reward only gives a reward to sequences that contain known TFBS motifs. This seems like it limits the model to recombining known motifs, rather than learning to design new motifs. Since TFBS motif design is a task that is of interest to the community (as is mentioned in Related Work), the form of the reward may limit the practical applicability of the method. Can the author's clarify whether their method is able to design novel TFBS motifs? If not, can the author's clarify whether this is an important limitation or not?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is clearly written. The problem and method are well-motivated and clearly explained.", "weaknesses": "* The evaluation tasks in this paper do not represent a realistic scenario for biological sequence design and do not follow established practices from the literature. Biological sequence design with machine learning is best characterized as an offline model-based optimization (MBO) problem, where there exists a fixed dataset associating sequences to fitness, but intermediate fitness measurements can not be collected during the process of optimization. This is because biological experiments are expensive but highly parallelizable, so it it usually only economical to collect a large number of measurements at once, rather than a small number of intermediate measurements. A possible approach to solving this offline MBO problem is train a model to predict fitness from sequence, and then use this \"surrogate\" model to guide an optimization procedure (which could be RL or another process). Note that the surrogate model is sometimes misleadingly referred to as an oracle. The most important concern with these methods is that the surrogate model is not a good model of the ground truth fitness function in regions that are out-of-distribution (OOD) of the fixed training set. Thus, the major methodological advancements in offline MBO have been aimed at controlling the optimization such that it does not produce sequences that are OOD of the training set. This problem is discussed extensively in, e.g., \"Conditioning by adaptive sampling\"  (2019) by Brookes et al., \n and \"Conservative Objective Models for Effective Offline Model-Based Optimization\" (2021) by Trabucco et al. In the paper under review, the authors also train a surrogate/oracle model to guide their optimization but make two choices that minimize the practical relevance of their evaluation tasks:\n    1. They train the oracle on the complete dataset, even though the intention of the task is to design high fitness sequences using only low fitness data. Therefore, when they query the oracle to calculate rewards, they are treating the oracle as a ground truth fitness function that they are collecting intermediate rewards for. As discussed above, this is unrealistic for biological sequence design.\n    2. They use predictions from the oracle as the final evaluation criteria. This again treats the oracle as the ground truth fitness function. \nThe combination of choices (1) and (2) mean that the evaluation tasks simply measure the ability of the method to optimize the oracle. As discussed in the citations above, optimizing the oracle without controlling for OOD concerns will usually result in designing unrealistic sequences that are OOD of the initial training set. Evaluation of methods for biological sequence require careful consideration of these factors, which has led to the introduction of evaluation frameworks such as FLEXS (referenced in the paper); these factors must be taken into account when designing tasks to evaluate the TACO method introduced in this paper.\n* A recent method has been introduced that uses Conservation Model Objectives (COMs) to design CREs ([\"Designing Cell-Type-Specific Promoter Sequences Using Conservative Model-Based Optimization\" (2024) by Reddy, et. al.](https://www.biorxiv.org/content/10.1101/2024.06.23.600232v1)). This method considers the factors discussed in the previous bullet, and contains strong validation with in-vitro experiments. It is not cited in the paper under review and not compared against. In order to be a strong contribution, the paper under review must cite this COMs paper and compare against the method. \n* The novelty of the method seems overstated as written. In particular, RL has been applied to biological sequence design in Angermueller et. al, 2019, which is not made clear in the Related Work section. Given this, the novelty of the authors' method is in the use of an autoregressive model as the policy, and the addition of the TFBS reward. This should be made more clear by making clear the contributions of Angermueller et. al, 2019 and how the authors' method improves over this.\n* As mentioned above, the authors introduce two novel concepts to the use of RL in sequence design. In order to understand the impact of these concepts, they should be fully ablated on all of the evaluation tasks. In particular, it would be informative to know how a different policy model performs with the TFBS reward and how the autoregressive policy performs without the TFBS rewards on all of the design tasks. This will clarify the key contributions of the paper.\n* Figures 3A and 5A are below acceptable quality for a publication at ICLR. Both appear to be screenshots and contain either unreadable or poorly labeled axes/subplots. \n* The References section contains incorrect and inconsistent citation formatting. In particular, many titles are lower case when they should be upper case and inconsistent information is included in the citation (e.g. sometimes URLs are provided and other times they are not). Also, many citations refer to an arXiv pre-print, rather than the published version of a paper; this must be checked and fixed.\n\nMinor errors:\n* Second sentence of abstract is grammatically incorrect (should \"CRE\" be plural?)\n* Equation 1: p_theta should be pi_theta\n* Line 368: (Hansen) is an incomplete reference", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)