Ctrl-DNA: Controllable Cell-Type-Specific Regulatory DNA Design via Constrained RL
Authors: Xingyu Chen
Shihao Ma
Runsheng Lin
Jiecong Lin
BO WANG
Venue: NeurIPS 2025 spotlight ()
Keywords: Regulatory DNA sequence design
Reinforcement Learning
Constrained optimization
Genomic language model
Primary area: machine_learning_for_sciences

Abstract: Designing regulatory DNA sequences that achieve precise cell-type-specific gene expression is crucial for advancements in synthetic biology, gene therapy and precision medicine. Although transformer-based language models (LMs) can effectively capture patterns in regulatory DNA, their generative approaches often struggle to produce novel sequences with reliable cell-specific activity. Here, we introduce Ctrl-DNA, a novel constrained reinforcement learning (RL) framework tailored for designing regulatory DNA sequences with controllable cell-type specificity. By formulating regulatory sequence design as a biologically informed constrained optimization problem, we apply RL to autoregressive genomic LMs, enabling the models to iteratively refine sequences that maximize regulatory activity in targeted cell types while constraining off-target effects. Our evaluation on human promoters and enhancers demonstrates that Ctrl-DNA consistently outperforms existing generative and RL-based approaches, generating high-fitness regulatory sequences and achieving state-of-the-art cell-type specificity. Moreover, Ctrl-DNA-generated sequences capture key cell-type-specific transcription factor binding sites (TFBS), short DNA motifs recognized by regulatory proteins that control gene expression, demonstrating the biological plausibility of the generated sequences.

Ctrl-DNA: Controllable Cell-Type-Specific Regulatory DNA Design via Constrained RL
Xingyu Chen 1:23-
Shihao Ma 12..
Runsheng Lin'
Jiecong Lin
Bo Wang123t
University of Toronto
Vector Institute for Artificial Intelligence University Health Network *Changping Laboratory
Abstract
Designing regulatory DNA sequences that achieve precise cell-type-specific gene expression is crucial for advancements in synthetic biology, gene therapy and precision medicine. Although transformer-based language models (LMs) ef- fectively capture patterns in regulatory DNA, their generative approaches often struggle produce novel sequences with reliable cell-type-spectne activity. Here, we introduce Ctrl-DNA, a novel constrained reinforcement learning (RL) frame- work tailored for designing regulatory DNA sequences with controllable cell-type specificity. By formulating regulatory sequence design biologically informed constrained optimization problem, we apply RI. autoregressive genomic LMs, enabling the models to iteratively refine sequences that maximize regulatory ac- tivity in targeted cell types while constraining off-target effects. Our evaluation human promoters and enhancers demonstrates that Ctrl-DNA consistently out- performs existing generative and RL-based approaches, generating high-fitness regulatory sequences and achieving state-of- sate-of-theeart cell-type specificity. Moreover, Ctrl-DNA Ctri-DNA-generatee sequences capture key cell-type-speciiic transcription factor binding sites (TFBS), short DNA motifs recognized by regulatory proteins that control gene expression, demonstrating the biological plausibility the generated sequences.
Code available at: ggthub.com/bowwag-bbbCCii
1 Introduction
Cis-Regus Re-RRegulatryy elements (CRE), such as promoters and enhancers, are critical DNA sequences that control gene expression. The ability to engineer DNA sequences with precise regulatory activities has widespread implications in biotechnology, including gene therapy, synthetic biology, and precision medicine 2 particularly desirable challenging goal is designing CREs that drive high gene expression in target cell type while maintaining controlled or limited fitness in off-target cell types. CRE's regulatory function largely determined by its transcription factor binding sites (TFBSs), which are short DNA motifs recognized by transcription factors (TFs) that mediate gene regulation in cells. The presence or absence of specific TFBSs directly influences the fitness of sequence across different cellular contexts. Although millions of regulatory sequences have evolved
"Equal contribution.
Corresponding author: bo veangorettrrineiiutte.cc CCE fitness is defined as the ability to enhance gene expressions.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

naturally [3,, these sequences are not optimized for targeted biomedical applications. For example, despite the human body comprising over 400 distinct cell types, very few cell-type-peecific promoters have been identified [4]. This scarcity highlights the need for novel, engineered CREs with precise cell-type specificity. However, the design space for regulatory sequences is immense: a 100-base sequence yields approximately 4100 possibilities, making purely experimental approaches expensive and impractical.
Massively parallel reporter assays (MPRAs) have significantyy advanced our ability to evaluate large libraries of DNA sequences for their cell-type-spece fitness [5] 6 Building upon these assays, recent deep learning approaches leverage predictive models as reward functions to guide the optimization of CREs [ 77. However, these methods typically rely on iterative optimization strategies based on mutating existing or randomly initialized sequences, which often limits the sequence diversity and can trap optimization in local optima. Furthermore, enforcing complex constraints in multiple cell types is non-trivial in these frameworks.
Recent studies have adapted autoregressive language models (LMs) for regulatory DNA sequence design, successfully capturing functional sequence patterns and enabling the generation of sequences with desired properties, such as enhanced gene expression levels [8]. However, these models primarily imitate the distribution of known sequences, limiting their ability to explore novel regions of the sequence landscape. Reinforcement learning (RL) has emerged as an approach to finetune genomies LMs for optimizing DNA sequence design. However, existing RL-based approaches for cell-type- specific CRE design typically focus on maximizing fitness in target cell type without accounting for fitness in other cell types (9). To date, integrating explicit constraints within RI. frameworks to suppress off-target regulatory activity using genomic LMs remains unexplored. Furthermore, conventional RL optimization strategies often depend on accurate value models and dense reward signals, introducing increased difficulty and inefficiency when navigating the vast DNA sequence space with complex biological constraints and sparse fitness reward signals.
To address these limitations, we introduce Cirl-DNA, reinforcement learning framework for con- trollable cell-tye-sseccffe regulatory DNA design via constrained RL. To the best of our knowledge, this work represents one of the first efforts to design regulatory DNA sequences with precise and controllable cell-type specificity. Building on recent advances in RL RI--based fine-tuning framework based on pre-trained autoregressive genomic LMs. Our method avoids value model training by incorporating batch-normalized rewards, enabling stable and efficient optimization across multiple cell types. Ctrl- DNA supports explicit cell-type-specific constraints, enabling the generation of sequences with high expression in target cell types while constraining off-target activities. We evaluate Cirl-DNA on human promoter and enhancer design tasks across six cell types. Our results show that Cirl-DNA consistently outperforms existing generative and RL-based methods, achieving higher activity in target cell types while improving constraint satisfaction in non-target cell types. We also show that Ctrl- Etrl-DNA-gee -ggeeratd sequences maintain substantial sequence diversity and effectively capture biologically meaningful, cell- -plecyype-specffic regulatory motifs.
[TO II [2], we develop an
policy gradients directly from
Our main contributions are as follows:
We develop a novel constraint- RL framework for CRE design, utilizing Lagrange multipliers explicitly and effectively to control cell-type specificity. To our knowledge, this represents one of the first efforts to incorporate constraint-based optimization into regulatory sequence generation.
By directly computing policy gradients from batch-normalized biological rewards and constraints, our method eliminates the need for computationally expensive value models, enabling efficient learning under complex CRE design constraints.
Through extensive empirical evaluations on human promoter and enhancer design tasks, we demonstrate that Ctrl-DNA consistently outperforms existing generative and RL-based meth- ods, achieving higher targeted regulatory activity with state-of-the-arr cell-type specificity.
2
Related Works
DNA Sequence Design Optimization: Optimization strategies complement generative models by explicitly steering sequences toward desired functions. Classical evolutionary algorithms, such as

---

genetic algorithms, iteratively refine sequences using fitness predictors, but they are often computa- tionally expensive and may converge to suboptimal solutions [33. To improve efficiency, heuristic techniques such as greedy search have been developed, incrementally editing sequences toward higher predicted performance []. Gradient-based approaches leverage differentiable surrogate mod- els (e.g., neural predictors like Enformer) to perform gradient ascent directly in sequence space 4 [4]. Although computationally efficient, these methods often initialize from random or high- fitness observed sequences, reducing the diversity of generated sequences. Reinforcement learning (RL) offers a powerful framework that combines generative modeling with goal-directed optimization. DyNA-PPO [15] demonstrated the effectiveness of deep RL for DNA design, outperforming random mutation- based methods. GelowNets further advanced this direction by learning stochastic policies that align with reward distributions, enabling diverse exploration of sequence space [16]. More recently, TACO 9 used RI. to fine-tune pretrained DNA language models with biologically informed rewards. However, these approaches primarily focus on optimizing fitness in single target cell type, without mechanisms to suppress or constrain activity in undesired cell types.
Constrained Reinforcement Learning: Constrained Reinforcement Learning (CRL), or often formulated as constrained Markov decision processes (CMDPs), addresses the critical challenge of optimizing policies under explicit constraints. Early foundational studies include actor-critic methods [17] and constrained policy optimization with function approximation [[8] Subsequent studies have explored integrating constraints into RL, often using Lagrangian methods that introduce non non-stationarity into rewards [19 20 (1]. Regularized policy optimization augments standard objectives with Kullback-Leibler (KL) or trust-region constraints [22]. and is widely used in both
single-task [23] and multi-task settings 24 CRI has been applied in language generation to suppress undesired outputs through techniques like Lagrangian reward shaping [1] 12 25). balancing primary objectives with safety constraints. However, such approaches remain largely unexplored in regulatory DNA design, where sparse rewards and multiple cell-fype-specific constraints introduce significant challenges for standard constrained RL frameworks.
Generative Models for Biological Sequence Design: Deep generative models have advanced the design of functional DNA sequences. ceffusion-based approaches have emerged as promising tools, beginning with the Dirichlet Diffusion Score Model (DDSM), which creates promoters based on expression levels [26]. Building on this foundation, several subsequent studics have further developed diffusion models for designing regulatory DNA sequences (27 28 29). Researchers have also leveraged Generative Adversarial Networks (GANs) for regulatory sequence design, with 7 creating cell-type-specifie enhancers in Drosophila and humans, and ExpressionGANN 30 generates yeast promoters that outperform natural sequences in expression efficiency. Autoregressive genomic language models have recently been applied to model DNA sequences, learning statistical patterns from large-scale genomic datasets. For example, RegLM[8] fine-tuned the HyenaDNA model [31] using prefix tokens that encode expression levels, allowing the generation of enhancers with controlled activity. However, despite producing biologically plausible sequences, generative models typically replicate distributions observed in training data, constraining their ability to explore novel, out- of-distribution solutions. This inherent limitation underscores the need for integrating generative approaches with optimization frameworks such as reinforcement learning.
3 Methods
3.1 Problem Formulation
We formulate DNA sequence design as a constrained Markov decision process (CMDP). A DNA sequence is defined as X = (æ1, ,2, ) E VL, where V = (A, C, G. T} is the nucleotide vocab- ulary and is the sequence length. The CMDP is defined as M = (S, A. p. Rg { (6, 1m 1). where S is the state space, A = V is the action space. p(s++1 8t, oz) is a transition function that appends nucleotide a to the current sequence prefix st. The sequence is evaluated by m black-brr box reward functions { R; VL. + R where we denote reward Ru as the CRE fitness in target cell, and reward R2 for 2 1 as CRE fitness in off-target cell types. The values 6, E R. is the constraint threshold for tff-target cell i. At each time step 4, the agent observes state 8, (x),- (....... 11-1) E S, selects an action a, AA according to policy Tp(ae st), and transitions to the next state 8141- Rewards (X)) are only calculated at the terminal step L.
3

---

Pre Pre-trained DNA LM (e.g., HyenaDNA)
Ctrl-DNA
Cell- Soppe-Seecffic Generation
CREs
Measured activity
GATAAT G A ? G A ?? ?? G ?
0.9
0.2
0.3
0.3
0.2
HepG2
K562
SKN-SH
Figure 1: Overview of the Ctrl-DNA framework for controllable regulatory sequence generation. Ctrl-DNA builds on pre-trained autoregressive DNA language model and applies constrained reinforcement learning to guide sequence generation toward high fitness in target cell type (e.g.. HepG2) while suppressing off-target fitness (c.g.. K562, SK-N-SH), enabling the generation of CREs with strong cell-type specificity.
Our objective is to learn a policy Ty that maximizes the expected CRE fitness in the target cell type while ensuring off-target fitness remains within the specified constraints. Formally, we aim to solve:
max Ex [X)]
s.t. Ex [())) & 6,, V (m....,m).
(1)
For clarity, we define (.) = Exwag [X) as the expected reward for cell type i, where (0) referred to task rewards and (0) for i referred to as constraints (also called tff-target rewards throughout this paper).
3.2 Constrained Batch-wise Relative Policy Optimization
We now describe approach for solving the constrained reinforcement learning problem for CRE sequence generation introduced Section 3.T Most deep-learning-bert bounin constrained RL methods rely training can significantly increase training complexity. Moreover, reward signals that are sparse and only available at the end of generated sequence may further complicate the training of value models that need accurate values at each step [33
more value models to estimate expected returns and costs [1] 32 12], which
To address this, we adapt work in K3
for our constrained DNA sequence design task, avoiding value network training for each cell type while enforcing constraints on off-target cell CRE fitness. We adopt a primal-dual approach based on Lagrangian relaxation, which introduces adaptive multipliers to enforce constraints while optimizing the main objective.
10
Lagrangian Relaxation and Constrained RL. The Lagrangian relaxation of the constrained objective in Eq. defines primal-dual optimization problem:
Lug (8,
min Jo(0) - Σ (J)((()) - 5.) X20
where 6, is a user-specified threshold and A 2 0 is dual variable for constraint i.
In practice, solve primal-dual policy optimization by taking iterative gradient ascent-descent steps of the policy parameter 0 and Lagrange multiplier As:
m78
0c+1 = =A+ + 170 V V (0, A) = 6b Vo Jo(O) Σ A1
(3)
A1,++1 Av. 7A. V,, Ling (0, A) A 72. X. (1. J((O))]
where 7/0 and 7),, are learning rates. 1: denotes the optimization step. This min-max formulation seeks saddle point that maximizes reward while satisfying constraints I1]
Our Methods. As commonly done in reinforcement learning
34
35
VoJ(O) is calculated by
policy gradient methods where (0) = E v Vo log To(a;
V, represents surrogate
signal such as rewards, state-action values or advantage estimates compute advantages using learned value functions, avoid value network training by drawing
36
While standard approaches

---

inspirations from the Group Relative Policy Optimization (GRPO) framework [TO, 33 GRPO estimates advantages by comparing outputs generated from the same prompt. In contrast, we propose batch-level variant for CRE sequence optimization, where advantages are computed by grouping sequences within each training batch.
Formally, for each objective i E {O....,m) m). the normalized advantage for sequence X, is defined as A (x(,)))R, where (x) the reward assigned by the i-th reward function, and R;, a R2)) denote the batch mean and standard deviation of R3 over the current batch of sequences. To guide
=(2t,
policy updates under constraints, we the Lagrange multipliers form convex combination of advantages from different cell types. We define the Lagrangian advantage as:
A0 = m
A
(5)
where m is the number of constraints, and X! is the Lagrange multiplier applied for constraint i. This encourages the policy favor sequences with high target rewards while discouraging those that violate constraints.
To estimate (0) during policy updates, we adopt a clipped surrogate objective with KL regular-
ization 10 35
B
T min (P clip,
Cpolicy (0) = B
KL(TT Teet)
(6)
where To and wold denote the current and previous policy networks. Trer is the reference model, which usually is the initial policy model. Here, p. x=(a]le)) is the importance sampling ratio, and
clip, clip..p:") (p) clip(p," 1 c) restricts large policy updates. The coefficient / controls the strength of the KL divergence penalty, and c sets the clipping threshold.
To adaptively enforce constraints, we update the Lagrange multiplier X; based on batch-level constraint satisfaction. For each constraint i, we define the multiplier loss as (Am)) =
$ 1E,,(RR - 6.) As where ò; is the constraint for cell type i. This formulation increases the penalty off-target cell types whose predicted fitness exceeds the constraint thresholds, while reducing the weight of those that already satisfy the constraints.
With this setup, the primal-dual updates (Equation 3 4
become:
0x+1 02 + ne V VeCpalicy(O).. (0),
A1,+1 Ar, TA: X., (ammltipplieAAs))...
Detailed pseudocode for the full algorithm and gradient functions are provided in Appendix A.
Empirical Designs. To improve training stability and model performance, we introduce several empirical modifications. First, we maintain a replay buffer of previously generated sequences and mix them with samples from the current policy. This helps reduce variance in batch-level reward statistics and leads to smoother advantage estimation. Second, we clip each Lagrange multiplier A; to the range [0, 1]. which prevents overly aggressive constraint enforcement and stabilizes the dual updates. Lastly, to prevent the main objective from being overwhelmed when constraint weights are large, we clip its coefficient in Eq. 5 as min(1, Σ X²), ensuring sufficient signal for optimizing the target reward.
3.3 Regularizing Generated Sequences via TEBS Frequency Correlation
Although Ctrl-DNA effectively optimizes sequence generation under specified constraints, the result- ing sequences may still deviate from biologically realistic distributions due to reward hacking [7]. To further regularize the generated distribution toward biologically plausible patterns, introduce an additional reward term based on the correlation between transcription factor binding site (TFBS) frequencies in generated and real sequences.
Specifically, we first compute TFBS frequencies from reference set of real DNA sequences. For each TFBS, calculate its occurrence frequency across these real sequences, forming reference
5

---

frequency vector Geat- Similarly, for each sequence generated by the policy To, we compute corresponding motif frequency vector Agen- We then quantify the similarity between generated and real sequence distributions using the Pearson correlation coefficient: (Tras(X) = Corr( Treal) for each generated sequence X.
We treat HifeS as an additional constraint reward function to maintain realistic TFBS patterns in generated sequences. However, to prevent the policy from overfitting to correlation alone and generating sequences that merely replicate the real distribution, we apply a clipped upper bound on the corresponding dual multiplier ATBB That is, from Equation we clip AT BS to the range [0, Amar] where Xmaz < 1, where Amax is predefined hyperparameter. This clipping mechanism ensures a balanced optimization process that maintains realistic TFBS frequencies without overly constraining policy exploration or the main optimization objective.
TFBS information is widely used in CRE design 8 29] However, existing methods typically incorporate TF motifs either as post-hoe evaluation metrics or through explicit tiling strategies during sequence design. A closely related approach is TACO (9) which trains LightGBM model predict sequence fitness from motif frequencies and derives motif-level rewards from SHAP values. In contrast, our method bypasses the need for additional predictive models by directly aligning the motif frequency distribution of generated sequences with that of real sequences. This removes the potential biases introduced by model training and provides a more reliable regularization signal from real biological distributions.
4 Experiments
4.1 Experimental Setup
Datasets. We evaluate our method on human promoter and enhancer datasets 138 39
The enhancer dataset includes 200 bp sequences from three cell lines: HepG2, K562, and SK-N-SH. The promoter dataset consists of 250 bp sequences from Jurkat, K562,, and THP1. Each dataset contains sequence- fitness pairs across the three respective cell types, with fitness measured via massively parallel reporter assays (MPRAs) [40]. We adopt the preprocessing pipeline described in [8] to preprocess all datasets. Please refer Appendix A.4 for details.
Motif Processing and Real Sequence Statistics. We obtain human-specific position probability matrices (PPMs) and pairwise motif correlation data from the JASPAR 2024 database [41]. Following the preprocessing procedure described in [8) we retain a curated set of 464 human transcription factor motifs. For each task, identify real sequences by selecting those in the top 50th percentile for fitness in the target cell type and the bottom 50th percentile in off-target cell types. We then apply FIMO 42 to scan for motif occurrences and compute motif frequency distributions, which serve as the reference motif distribution for optimization and evaluation.
43 Reward Models. We train reward model for each cell type using the Enformer architecture following protocols from prior works on cis-regulatory element (CRE) design X
9
4 29
38
For the human enhancer task, directly adopted the pretrained model weights from
For
human promoters, we trained a separate reward model using the data split from [4] (70% train, 20%
validation, 10% test). Training was conducted using the AdamW optimizer with learning rate of le-4 and MSE loss, over 20 epochs. The checkpoint with the best validation performance was used for evaluation.
All reward models are based on an Enformer architecture 444, combining convolutional and Trans- former layers, which has shown strong performance in DNA regulatory prediction tasks. Both the enhancer and promoter models share the same architecture: 11 layers with a hidden dimension of 1536.
Models and Baselines. For sequence generation, we fine-tune HyenaDNA 31 an autoregressive ge- nomic LM pre-trained on the human genome, our policy model. We compare our proposed method, Cirl-DNA, against diverse set of baselines, including evolutionary algorithms (Adalead [45), BO 46), CMAES [7] PEX 48 generative models (RegLM [8), and reinforcement learning PPO-Lagrangian
approaches (TACO PPO
[12 ) Full baseline details are provided in
Appendix A.5

---

Human Enhancer Datasets
b
Human Promoter Datasets
0.8
0.8
Hept2 Finoee (-1]
JARKAT Fimees 1
Hap02 Finoss
Fitress
0.6
0.5
0.5
0.4
0.4
0.8
0.6
0.4
0.8
0.6
0.4
0.2
0.7
0.5
0.6
0.5
0.4
0.3
6.2
KOG2 Fimees (+-)
SK--SH Fitress [-]]
THP1 F tieess (--)
K562 Fitness (+))
0.9
0.6
0.6
KS02 Fineoo
0.8
K562 Fitress
0.5
0.5
0.4
0.4
K062
0.5
0.2
0.2
0.8
0.6
0.4
0.6
0.4
SKGN-SH Filnss (+-
Hapd2 Fineess (--)
THP1 Fitnees
JURKAT Finess (-)
6.9
O
0.8
SHSNS Fitrees (-)
0.8
SKISH Finooa
Fitnees
8.7
TNP1 Firncs
0.7
THP1 Fitnoss
0.7
0.6
0.6
0.6
0.5
0.5
96
0.5
0.4
0.4
0.8
0.8
944
822
0.6
0.5
0.4
0.3
HapG2 Fithess (-)
KB82 Fiiness (+-)
KS82 Fittness (+)
JURKAT Fimess (+)
Add
CMAFS
Regi PPO
TAGO
CHI-ONA (8-1.38.4)
Can-OMA (0-0.8)
PER
PrD-agg
CHH-ONA (5+0.5)
Figure 2: Pairwise fitness comparison of generated CREs highlights Ctrl-DNA's 's cell-type specificity. Each subplot compares mean + s.d. fitness in two human cell lines (y target, off off-targett) points in the top-right denote sequences with high on-target and low off-target fitness. Baseline methods shown in pastel colors, while Ctrl-DNA variants (6 0.3/0.4, 0.5, 0.6) are connected in red dotted lines, illustrating the trade-off as constraint strength increases and Ctrl-DNA's 's dominance in the top-right corner for both enhancer (a) and promoter (b) datasets.
Evaluation Metrics. To assess the performance of each method, we report median rewards ( Median) over the generated sequences in the final round. Additionally, we report Reward Difference (AR) to quantify the average difference between the target cell's fitness and the fitness across off-target cell types, indicating the cell-type fitness specificity. Motif Correlation is defined as the Pearson correlation between TFBS frequencies in generated sequences and real sequences. A higher correlation indicates greater aligmment with biologically plausible motif distributions. Lastly, Diversity is caleulated as the Shannon entropy of the generated sequences in the final round, reflecting the model's 's ability to explore diverse solutions. We report the mean and standard deviation of each metric over 5 runs initialized with different random seeds.
4.2 Ctrl-DNA optimizes enhancer and promoter sequences under cell-type-specifii constraints
We evaluate Ctrl-DNA on two regulatory sequence design tasks using the Human Enhancer and Human Promoter datasets. Results are presented in Figure [ In these plots, points positioned further to the right indicate lower fitness in off-target cell types (i.e., better constraint satisfaction), while points higher on the vertical axis indicate higher fitness in the target cell type. Methods that appear in the upper-right corner achieve the best trade-off between maximizing target cell fitness and minimizing off-target expression.
7

---

Table 1: Performance comparison across methods for each target cell type on the Human Enhancer and Human Promoter datasets. For each target, we report AR (1) and motif correlation (1). Constraint thresholds are set to 0.5 for all six cell types. Note that K562 refers to the K562 cell type in the Human Promoter dataset. Motif Corr! computed using 90th-per reference sequences.
Cell Type
Mutric
Adal.cad
B0
CMAFS
PEX
RegL
PPO
TACO
PPO-Tag
Chrl-DN
H2p
Moai
AR
0:00 (0.04)
0.45 (0,03)
-0.0 (0.08)
0.21 (0.10)
0.15 (0.08)
0.01 (0.01)
0.31 (0.04)
(0.3)
0.22 (0.07)
0.16 (0.0)
(0.06
0.41 (0.14)
-0.05 (0.01)
0.30 (0.03)
.0.02.0.35)
0.41 (0.02)
0.43
0.49 (8,01)
K962
Modif Coer
AR
0.49
0.36 (0.05)
0.08 (0.18)
0.08 (002)
0:03 (0.02)
0.06 (0.04)
0.17 (0.03)
0.21 (0.06)
(.000
0.19 (0.02)
0.35 (.11
0.14 (0.04)
0.28
(.11(000))
0.64 (0.05)
0,0 (0.05)
0,51 (8.02)
0.54 (8.01)
SK N S11
AR
Cour
0.35 (0.13)
0.00 (0.07)
-001 (0.0))
0.05 (0.05)
0.03 (0.04)
0:00 (0.01)
(000)00))
0.17 (0.02)
0.18 (0.01)
0.14 (0.0))
0.23 (0.01)
(0.0))
0.08 (002)
0.11 (0.12)
0.42 (0.05)
0000
0.37 (0.11)
0.25 (8.04)
JURKAT
AR
0.41
0.09 (0.06)
0.19
0.04 (0.03)
0.30
0.00 (0.01)
0.15 (0,03)
(0.05)
0.09 (0.01)
(0.02)
0.04 (0.12)
(0.11)
0.05
(.35())11)
0.11 (0.01)
(0.33)
0.25 (0.01)
(0.02)
K362"
Monif
AR
-0.12 (0.01)
0.60 (0.15)
-0:15
0.13
.0.17 (0.02)
0.24 (0.08)
4.04 (0002)
0.61 (0.02)
-0.08 (0.08)
0.42 (0.04)
.0.09 (0.85)
0.39 (0.11)
-1.10 (0.02)
0.50
J0.22 (0.04)
0.42 (0.22)
0.75 (6.06)
0.12 (8,12)
TIPP
Modd Coer
(0.00)
0.25 (0.09)
0:20
0.39 (0.06)
0.84 (8.01)
0.29 (0.01)
(0.01)
(0.03)
(0.10)
(0.03)
0:42 (0.10)
(0.01)
0.25 (0.04)
Musif Moii
0.16 (0.13)
0.06 (0.15)
0.06 (0.04)
0.04 (0.02)
0.29 (0.01)
0.37 (0.04)
0.33 (0.01)
-0.02 (0.07)
0.60 (8.12)
For enhancers, we evaluate performance under three constraint thresholds (6 0.3, 0.5, 0.6). Across all thresholds, Cirl-DNA consistently achieves the highest target-cell fitness while satisfying the off-target constraints. IPO-Lagrangian (PPO-Lag) struggles to balance optimization and constraint satisfaction, likely due to the difficulty of training value networks under sparse, sequence- reward signals. Notably, while methods such as TACO and CMAES achieve relatively high expression in the target cell type, they fail to suppress taf-target fitness, leading to poor cell-type specificity.
The promoter design task a more challenging task because all three target cell types are mesoderm- derived hematopoietic cells, which share substantial transcriptional similarity (4] We test under three constraint thresholds (8- = 0.4, 0.5, 0.6). Cirl-DNA outperforms all baselines in maximizing target cell-type fitness and satisfying constraints at ő = 0.5 and 0.6. However, no method, including Cirl- DNA, successfully reduces THP1 fitness below the stricter threshold of 5 = 0.4. We hypothesize that this is due to the data distribution: the 25th percentile of THP1 fitness is already 0.49 (Appendix(AA)) indicating that most sequences exhibit high expression in this cell type. Despite this challenge, when THPI is treated as off- off-target cell, Ctrl-DNA still achieves the lowest THPI fitness among all methods.
Across both enhancer and promoter tasks, Ctrl-DNA consistently achieves the best trade-off between optimizing target cell types and enforeing cell-type-specific constraints, substantially outperforming existing RL and generative baselines. Interestingly, we observe clear trade-off when enforcing stricter constraints: as the constraint threshold decreases, the fitness in the target cell type slightly declines. This trend likely arises because stricter constraint enforcement potentially narrows the feasi- ble sequence space, making it more challenging to simultaneously optimize target cell-type activity and minimize off- off-target expression. For completeness, we compared our constrained formulation to a scalarized differential seterrentialxppresssiiio (DE) objective and found that DE imposes a fixed trade-off that underperforms our adaptive Lagrangian approach; see Appendix A.8 for details. Despite this inherent difficulty, Ctrl-DNA's 's constraint-awar optimization framework remains highly effective, demonstrating robustness in maintaining superior target-cell fitness even under rigorous constraint conditions.
4.3 Ctrl-DNA captures biologically relevant motifs with higher specificity
Besides the fitness of generated CRE sequences in each cell type, we also evaluate the sequences with three other metrics: reward difference (AR), motif correlation and diversity. In Table I we can observe that Ctrl-DNA achieves highest reward differences across all cell types in both human promoter and enhancers, indicating it is better at optimizing DNA sequences for sell-type-specife fitness. For motif correlation, Ctrl-DNA also achieves stronger performance across most cell types, except for THP1 in promoter design. As noted in Section 4.2, THP1 fitness values are skewed, with the majority of sequences in the dataset exhibiting fitness around 0.5. Since motif correlation is evaluated against sequences near the 50th percentile (see Section the resulting motif frequency distribution may not accurately reflect the high-activity sequences we aim to design.
8

---

Liver
Erythropoietic
2.5
Enhancer HepG2
2.0
Promoter K562
10-
+
Motif Fraction
2.-
08-
2.0
1.50
Diversity
1.5
08-
1.00
1.0
24-
04-
0.5
02
0.55
50
60
96551
GAU
d4tnt 1861
PEr Pig beco
cscckis (oecsis On cauchit (ivol))
Anala
frop
1200
csoNs cotte (1 Car
peos Lag
Figure 3: (a) Fraction of Sor--DN--geneaaad enhancers containing selected cell-type-specific transcription factor (TF) motifs. (b) Diversity scores of generated sequences for HepG2 enhancers (left) and K562 promoters (right) across different methods.
To further investigate this discrepancy, we extract motif's from promoter sequences in the 90th percentile of THPI fitness, applying significance threshold of q < 0.05 to avoid false positives. We then re-evaluate the motif correlation between generated sequences and this more selective reference set. These results, denoted as Motif Corrf in Table I show that Ctrl-DNA outperforms all baselines under this stricter setting. suggesting that they tend to align with non-informative or broadly distributed motifs. Despite being regularized using motifs from less selective reference set, Cirl-DNA successfully prioritizes the most discriminative motifs during optimization. Finally, the correlation improvements persist when evaluated against alternative reference sets explicitly designed to reduce circularity (Appendix A0 with consistently high correlations across cell lines.
contrast, most baseline methods exhibit reduced motif correlation,
To further demonstrate that Ctrl-DNA selects more cell diyeediscriminntive motifs during sequence generation, we evaluated the frequency of known cell-type- sell-type-specific TFBS in the generated sequences. motifs. In particular, we examined generated sequences for liver-specific and wepthhopoitticsppeeeeee As shown in Figure 3p. Cirl-DNA-geneed generated sequences for HepG2 (a liver-derived cell line) show the highest frequency of liver-specffic motifs such as HNF4A and HNF4G. Similarly, sequences generated for K562 (an erythropoietic lineage cell line) contain the highest frequency of erythropoietic-ppeeie motifs such as GATAI and GATA2. These findings indicate that Ctrl-DNA not only optimizes for target-cell fitness, but also learns regulatory patterns that reflect underlying cell-type specificity.
Finally, we assess the diversity of generated sequences in Figure P and Appendix [A.6] Ctrl-DNA achieves comparable or higher diversity than most baselines, indicating its ability to generate diverse sequences without sacrificing regulatory control.
4.4 Ablation Study
Constraint Formulations. To investigate alternative constraint enforcement strategies, we explored several other constrained methods from current works. First, we adapt the loss from Interior-poin Policy Optimization (IPO) 49,, referring this variant as Ctrl-DNA-IPO. Second, we implement a log-barrier penalty on constraint rewards following [50]. which we denote as Ctrl-DNA-Log. All experiments are conducted using a constraint threshold of 0.5. See Appendix(A.. [or detailed setup. As shown in Table 2 Cirl-DNA-Log suppresses off-target rewards effectively but fails to maintain high fitness the target cell type. In contrast, Cirl-DNA-IPO improves target reward but does not enforce constraints adequately. These results highlight that our proposed formulation strikes better balance between optimizing target cell-type fitness and minimizing off off-target expression.
we introduced a correlation-based regularization using TFBS motif frequencies to promote biologically plausible sequences. By changing the upper bound on the TFBS multiplier (Amax) we can limit the weight we put on this regularization. From Table 2 we observe that increasing Amax from 0.0 to 0.1 improves motif correlation without substantially degrading other metrics. In certain cell types, such as JURKAT, a higher value of Amax also leads to improved optimization performance (See Appendix Table 77) This supports the utility of TFBS regularization in guiding sequence generation. However, since our comparisons use motif frequencies computed from a loosely matched reference set, we recommend tuning Amax carefully in practical applications depending the reliability of the available ground truth.
TFBS Regularization. In Section 3.3

---

Table Ablation study on constraint formulation and TFBS regularization. We compare variants of Ctrl-DNA using alternative constraint handling methods (Cirl-DNA-IPO, Ctrl-DNA-Log) and varying TFBS regularization strengths (Amax). Results are reported on the Human Enhancer dataset (target cell: HepG2), with constraint threshold d = 0.5 in both experiments. See Appendix A.7 for complete results across all datasets.
Method
Target: HopG2
HepG2 f
SK-N-SH
Motif Comelation
Diversity
Ctrl Bol-DNA-LLE Log
0.24 (0.02)
0.24 (0.03)
0.21 (0.94)
0.02 (0.06)
0.16 (0.13)
1.62 (0.08)
Chi-DNN--IPO
0.74 (0.01)
0.86 (0.02)
0.83 (0.02)
-0.10 (0.02)
0.39 (0.14)
1.58 (0.13)
Ctrl-DNA (Ama = 0.00)
0.78 (0.02)
0.40 (0.06)
(.303)
0.42 (0.02)
0.33 (0.12) 12)
1.82 (0.02)
(mma 0.01)
(0.01)
(0.04)
(0.02)
(0.02)
0.16 (0.10)
Ctrl-DNA (Amax 0.1)
0.77 (0.01)
0.36 (0.04)
0.31 (0.04)
0,44 (0.03)
0.43 (0.07)
1.82 (0.04)
5 Discussion
Designing cell-type-specific cis-regulatory sequences presents a challenging optimization problem that involves balancing competing objectives. Our proposed method, Ctrl-DNA, achieves strong per- formance across both enhancer and promoter datasets, outperforming evolutionary and RL baselines in maximizing target cell-type fitness while satisfying off-target constraints. In addition, Cirl-DNA supports explicit control over constraint thresholds, enabling flexible and controllable CRE sequence design. By computing Lagrangian advantages directly from batch normalized rewards without train- ing value models, Ctrl-DNA offers a lightweight and effective solution to constrained CRE DNA sequence generation.
Nevertheless, the ability to enforce constraints is inherently limited by the data distribution. For instance, in the THP1 promoter dataset, large proportion of sequences exhibit high baseline activity, making it difficult to enforce stricter constraints such as 5 = 0.4. This challenge affects both the accuracy of the learned reward model and the capacity of Ctrl-DNA suppress expression in such settings. These observations highlight the importance of considering datasct-specific characteristics when setting constraint thresholds evaluating constrained RL methods.
Although Ctrl-DNA already demonstrates robust performance, there are several directions for im- provement. First, tuning Lagrange multipliers remains empirical. Future work could explore adaptive control methods such as proportional iapootioonal-nntgraall derivative controllers [5T]. Second, additional bi- ological constraints could be incorporated to further improve the plausibility and functionality of generated sequences. Finally, our current framework limited to reinforcement learning fine-tuning on autoregressive models. As next step, we plan extend Ctrl-DNA other structures such as diffusson--bsed DNA models.
10

---

References
[1] Sager J Gosai, Rodrigo I Castro, Natalia Fuentes, John C Butts, Kousuke Mouri, Michael Alasoadura, Susan Kales. Thanh Thanh L Nguyen. Ramil R Noche, Arya Rao, et al. Machine-guided design of cell-type-targeting cis-regulatory clements. Nature, pages 1-10, 2024.
[2] Shannon E Boye, Sanford L Boye, Alfred $ Lewin, and William W Hauswirth. A comprehensive review of retinal gene therapy. Molecular therapy, 21(3):509- 519, 2013.
[3] Tianshun Gao and Jiang Qian. Enhanceratlas 2.0: an updated resource with enhancer annotation in 586 tissue/cell types across nine species. Nucleic acids research, 48(D1):D58-DD) D64, 2020.
[4] Aniketh Janardhan Reddy, Xinyang Geng, Michael Herschl, Sathvik Kolli, Aviral Kumar, Patrick Hsu, Sergey Levine, and Nilah Ioannidis. Designing cell-type-specific promoter sequences using conservative basel-based optimization. Advances in Neural Information Processing Systems, 37:93033 -305999 2024.
[5] Ryan Tewhey, Dylan Kotliar, Daniel Park, Brandon Liu, Sarah Winnicki. Steven K Reilly. Kristian Andersen, Tarjei S Mikkelsen, Eric S Lander, Stephen Schaffner, et al. Direct identification of hundreds of expression expression-modulatinn variants using a multiplexed reporter assay. Cell, (55(6))551-11 2016.
[6] Carl G de Boer, Eeshit Dhaval Vaishnav, Ronen Sadeh, Estehan Luis Aheyta, Nir Friedman, and Aviy Regev. Deciphering eukaryotic gene-regulatory logic with 100 million random promoters. Nature hiotechnology. 38(1):56-65, 2020.
[7] Ibrahim I Taskiran, Katina Spanier, Hannah Dickmänken, Niklas Kempynck, Alexandra Pančíková. Eren Can Ekşi, Gert Hulselmans, Joy N Ismail, Koen Theunis, Roel Vandepoel, al. Cell-type-dreetee design of synthetic enhancers. Nature, 626(7997):212- 2212222202222 2024.
[8] Avantika Lal, David Garfield, Tommaso Biancalani, and Gokcen Eraslan. Designing realistic regulatory dna with autoregressive language models. Genome Research, 34(9):1411-142) 2024.
[9] Zhao Yang, Bing Su, Chuan Cao, and Ji-Rong Wen. Regulatory dna sequence design with reinforcement learning. In The Thirteenth International Conference Learning Representations.
[10] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement 2025.
learning. arXiv preprint arXiv:2501.
[11] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 21.31. PMLR, 2017.
[12] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement lcarning. arXiv preprint arXiv: 1910.01708, 7(1):2, 2019.
[13] Ecshit Dhaval Vaishnav, Carl G de Boer, Jennifer Molinet, Moran Yassour, Lin Fan, Xian Adiconis, Dawn A Thompson, Joshua Z Levin, Francisco A Cubillos, and Aviv Regev. The evolution, evolvability and engineering of gene regulatory dna. Nature, 603(7901):455 2022.
[14] Johannes Linder and Georg Seelig. Fast activation maximization for molecular sequence design. BMC bioinformatics. 22:1 22:1-20, 2021.
[15] Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In International conference on learning representations, 2019.
[16] Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garee Garcia, Jarrid Rector- Brotor, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning, pages 9786-9801. PMLR, 2022.
[17] Vivek S Borkar. An actor-critic algorithm for constrained markov decision processes. Systems & control letters, 2403)200--13, 2005.
[18] Shalabh Bhatnagar and Lakshmanan. An online actor-critic algorithm with function approximation for constrained markov decision processes. Journal of Optimization Theory and Applications, 153:688-708, 2012.
[19] Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint 108.v:1805.11074, 2018.

---

[20] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133-9143, PMLR, 2020.
[21] Yuhao Ding and Javad Lavaei. Provably efficient primal-dual reinforcement learning for emdps with non-stationary objectives and constraints. In Proceedings of the AAAI Conference on Artificial Intelligence volume 37, pages 7396-7404, 2023.
[22] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages
PMLR, 2015.
[23] Abhas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv: /806, 06920, 2018.
[24] Ted Moskovitz, Michael Arbel, Jack Holder, and Aldo Pacchiano. Towards an understanding of default policies in multitask policy optimization. In International Conference on Artificial Intelligence and Statistics, pages 10661-10683 PMLR, 2022.
[25] Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D Dragan, and Stephen McAlcer. Confronting reward model overoptimization with constrained rihf. arXiv preprint urXiv:2310.04373. 2023.
[26] Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. In International Conference on Machine Learning, pages 1276-1301. PMI.R, 2023.
[27] Simon Senan, Aniketh Janardhan Reddy, Zach Nussbaum, Aaron Wenteler, Matei Bejan, Michael I Love, Wouter Meuleman, and Luca Pinello. Dna-diffusion: Leveraging generative models for controlling chromatin accessibility and gene expression via synthetic regulatory elements. In ICLR 2024 Workshop on Machine Learning for Genomics Explorations, 2024.
[28] Zchui Li, Yuhao Ni, Tim August B Huygelen, Akashaditya Das, Guoxuan Xia, Guy-Bart Stan, and Yiren Zhao. Latent diffusion model for dna sequence generation. arXiv preprint arXiv:2310.06/500 2023.
[29] Anirban Sarkar, Ziqi Tang. Chris Zhao, and Peter K Koo. Designing dna with tunable regulatory activity using discrete diffusion. bioRxiv, pages 2024- 05, 2024.
[30] Jan Zrimee, Xiaozhi Fu, Azam Sheikh Muhammad, Christos Skrekas, Vykintas Jauniskis, Nora Speicher, Christoph S Börlin, Vilhelm Verendel, Morteza Haghir Chehreghani, Devdatt Dubhashi, ct al. Controlling gene expression with deep generative design of regulatory dna. Nature communications, 13(1):5099, 2022.
[31] Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Worow, Callum Birch-Sykes. Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, al. Hyenadna: Long-range genomic sequence madeling single nucleotide resolution. Advances in neural information processing systems, 36:43177-43201, 2023.
[32] Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 13:15338-15349, 2020.
[33] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 24Xiv:2402.03338 2024.
[34] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[35] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint urXiv:1707.0634 2017.
[36] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. figh-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506 02438, 2015.
[37] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[38] Sager J Gosai, Rodrigo Castro, Natalia Fuentes, John C Butts, Susan Kales, Ramil R Noche, Kousuke Mouri, Pardis C Sabeti, Steven K Reilly, and Ryan Tewhey. Machine Machine-guided design of synthetic cell type-specific cis-regulatory elements. bioRxiv, 2023.
12

---

[39] Aniketh Janardhan Reddy, Michael H Herschl, Xinyang Geng, Sathvik Kolli, Amy X Lu, Aviral Kumar. Patrick D Hsu, Sergey Levine, and Nilah M loannidis. Strategies for effectively modelling promoter-driven gene expression using transfer learning, bioRxiv, pages 2023-02, 2024.
[40] Eilon Sharon, Yacl Kalma, Ayala Sharp. Tali Raveh-Sadka. Michal Levo, Danny Zeevi, Leeat Keren, Zohar Yakhini, Adina Weinberger. and Eran Segal. Inferring gene regulatory logic from high-throughput measurements of thousands of systematically designed promoters. Nature biotechnology, 3016):521-530, 2012.
[41] Jaime A Castro Mondragon, Rafael Riudavets-Puus leva Rauluseviciute, Roza Berhanu Lemma, Laura Turchi, Romain nate-e, Jeremy Lucas, Paul Boddie, Aziz Khan, Nicolás Manosalva Pérez, et al. Jaspar 2022: the 9th release of the open-access database of transcription factor binding profiles. Nucleic acids research, D0SDI):DISS-D173, 2022.
[42] Timothy L Bailey, James Johnson, Charles E Grant, and William S Noble. The meme suite. Nucleic acids research, 43(W1)::WW) 3339 W49, 2015.
[43] Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grahska- Barwinska, Kyle R Taylor, Yannis Assacl, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10): 196- 2021.
[44] Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grahska- Barinkk,, Kyle Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10): 120- 2021.
[45] Sam Sinai, Richard Wang, Alexander Whatley, Stewart Slocum, Elina Locane, and Eric D Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint arXiv:2010.02141, 2020.
[46] Jasper Snoek. Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing xystems, 25, 2012.
[47] Nikolaus Hansen. The ema evolution strategy: comparing review. Towards new evolutionary computa- tion: Advances in the estimation of distribution algorithms, pages 75-102, 2006.
[48] Zhizhou Ren, Jiahan Li, Fan Ding. Yuan Zhou, Jianzhu Ma, and Jian Peng. Proximal exploration for model- guided protein sequence design. In International Conference on Machine Learning, pages 18520-18536. PMLR, 2022.
[49] Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence. volume 34, pages 4940-4947, 2020.
[50] Masatoshi Uchara, Xingyu Su, Yulai Zhao, Xiner Li, Aviv Regev, Shuiwang Ji, Sergey Levine, and Tommaso Biancalani, deward-guided iterative refinement in diffussin models at test-time with applications to protein and dna design. arXiv preprint arXiv:2502. 14944, 2025.
[51] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133-9143, PMLR, 2020.
[52] Friong Liu, Yu-Chang Wu, Xiaobin Huang, Chengrui Gao, Ren-Jian Wang, Ke Xue, and Chao Qian. Pareto set learning for multi-objective reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 18789-18797, 2025.
[53] Chenyu Wang. Masatoshi Uehara, Yichun He, Amy Wang, Tommaso Biancalani, Avantika Lal, Tommi Jaakkola, Sergey Levine, Hanchen Wang, and Aviv Regev. Fine-tuning discrete diffusion models via reward optimization with applications dna and protein design. arXiv preprint arXiv:2410.13643. 2024.
[54] David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. International conference on machine learning, pages 773-782, PMLR, 2019.
13

---

A Technical Appendices and Supplementary Material
A.1 Training Details for Ctrl-DNA
We provide a high-level overview of our constrained reinforcement learning procedure for optimizing constraint-aware regulatory sequence generation. We also present the hyperparameter settings used in our experiments in Table All models were trained using the Adam optimizer with a learning rate specified in Table with 100 training epochs.
Experiments were conducted on a single NVIDIA A100 GPU with 40 GB of memory. Each experiment typically required between 1 to 2 hours of wall-clock time. The experiment results reported are mean performances across five seeds.
Algorithm 1 Ctrl-DNA
Require:
Initialized policy no, Reference policy Tet Lagrange multipliers (X, reward functions R4 )- reference TFBS frequency Treal, constraint thresholds {6,). learning rates ro, 7x+ hyperparameters c, replay buffer B, batch size B, replay batch size B,
1: for each training iteration do
2: Update Rold = क
3: Sample B sequences {X;}ʃ= from policy Told
4: Compute rewards RR (Xx)) for = ,,...,
(j) Compute TFBS frequency agen and correlation RIFBS (X2) Corr(graal (gral 9gen
(i)
5:
6:
Treat ATrns as additional reward Rm and append to the reward set Sample B, sequences from replay buffer and merge with current batch
7:
Compute normalized advantage: A(3) (.(X))-ff for each reward R; ((R))
8:
9:
Compute clipped main reward coefficient:
10:
ag = min (a, m
-E A;)
11:
Construct mixed advantage:
Áo)_ 0ig 1
E MA
12:
Add current batch {(x, (r(())) to replay buffer B for each policy update step do
14:
15:
Update policy parameters 0 using Pplicy (0) (Eq. 6p
16:
for each constraint -1, I,..., m + do
Update A using:
Lruutiplier (Ax) B+B, ΣØ (R,(X,) 5,) X
end for
20:
end for
21:
end for
Table Experiment Hyperparameters.
Hyperparameter
HepG2
K562
SK-N-SH
JURKAT
K562
THP1
Batch Size
256
256
256
256
256
256
Replay Buffer Batch Size
24
24
24
24
24
24
Policy Learning Rate (17)
le-4
le-4
le-4
le-4
le-4
le-4
Multiplier Learning Rate (7x)
3e-4
3e-4
3e-4
3e-4
3e-3
3e-3
KL Value Coefficient (3)
0.2
0.2
0.2
0.2
0.2
0.2
TFBS Multiplier Upper Bound (Amax)
0.1
0.1
0.1
0.1
0.1
0.1
A.2 Policy Optimization Objective
As discussed in Section
3.2 the objective for updating policy parameters is defined as:
Cpolicy (0)
min ((,))(0)) clip, ( ((0)) 8 KL( TTe))
(7)
B
14

---

where p:
==(()(())) is the importance sampling ratio, and clip. = clip(p - c, +
applies clipping for stability. Following 33 we assume one policy update per iteration, allowing
Told To for simplification.
The gradient of this objective with respect becomes:
1
Vo log To(a )
V puliicy (0)
B
i=1
Teerlas (i) (0)
Trerlay (5)
8q -1
3.
V
log
(8)
To (a()
To (1)
To
s(11)
B T
[a
-1) 1
AG) -
Vo log Tola (
B
(8z)) To (a(a
(
A.3 Lagrange Multiplier Update
We apply Lagrangian relaxation to enforce soft constraints off-target cell types. The gradient of multiplier objective with respect to each Lagrange multiplier A; is given by:
X, mmultiplir (Ax) =
(R((X)) - 6.),
j=1
where (X,, is the constraint-specific reward (e.g., off-target activity) for sample X,, and o; is the user-defined constraint threshold.
A.4 Dataset
In this section, describe the datasets used in experiments. The human enhancer dataset contains cis-regulatory clement (CRE) activity measured by MPRA across three cell lines: HepG2 (liver cell line), K562 (erythrocyte cell line), and SK-N-SH (neuroblastoma cell line). Each sequence this dataset is 200 base pairs long.
The human promoter dataset contains promoter activity (CRE fitness) measured from three leukemia- derived cell lines: JURKAT, K562, and THP1, All three are mesoderm-deriveer hematopoietic cell lines and share high biological similarity. Each sequence in this dataset is 250 base pairs in length. Compared to enhancer datasets that span multiple germ layers and tissue types, optimization and constraint satisfaction in the promoter dataset is more challenging due to the biological similarity between the cell lines
We provide percentile statistics of normalized activity scores in Tables 4 and 5 Notably, in the THP1 cell line, even the 25th percentile activity reaches 0.49, suggesting a right-skewed distribution. This distributional bias may partially explain the increased difficulty in constraining THP1 activity, as discussed in Section
Table 4: Percentile statistics of normalized activity scores across cell types in Human Enhancer datasets
Cell Line
25th Percentile
50th Percentile
75th Percentile
90th Percentile
HepG2
0.34
0.36
0.40
0.45
K562
0.34
0.36
0.40
0.45
SK-N-SH
0.35
0.37
0,40
0.45
15

---

Table 5: Percentile statistics of normalized activity scores across cell types in Human Promoter datasets.
Cell Line
25th Percentile
50th Percentile
75th Percentile
90th Percentile
JURKAT 0.35
0.38
0,44
0.54
K562
0.23
0.26
0.32
0.40
THP1
0.49
0.51
0.53
0.59
A.5 Baselines
In this section, we provide detailed descriptions of all baseline methods compared in the main paper:
hill- hill-climbing algorithm with mutation rate =1 (applied as μ/L per position), recombination rate =022, and greedy threshold k= 0.05 for selecting parents with fitness above (1-k) of the current best. Recombination is disabled (p= 0), and each candidate evaluated with v= 20 model queries.
AdaLead 45
is implemented
Bayesian Optimization (BO) 45 A black-box optimization method that models the fitness function with a Gaussian process surrogate and selects new candidates by maximizing acquisition function.
CMA-ES [7] A population- based evolutionary algorithm that adapts a multivariate Gaus- sian distribution over iterations. We apply CMA-ES on one-hot encoded sequence repre- sentations with population size of 16 and initial search variance of 0.2. The number of iterations is scaled to the model query budget.
An evolutionary approach that prioritizes generating high-fitness variants with minimal mutations relative to the wild-type sequence. It generates candidates by applying random mutations high-fitness frontier sequences, stratified by Hamming distance from the wild-type. The frontier neighbor size is set to 5, and predictions are processed in batches of 64.
PEX 48
RegLM 8 An autoregressive language model trained to generate cis-regulatory el- ements (CREs) conditioned on cell-type-specific fitness profiles. We fine-tune the (6.55M parameters) for 16 epochs, using the W optimizer (learning rate 0.0001, batch size 1024) and cross-entropy loss, with inverse-frequency label sampling to address class imbalance.
configuration
reinforcement learning method based on REINFORCE (34] which incorpo- rates transcription factor motif rewards to guide generation toward high-fitness sequences. Its implementation follows the original paper's 's hyperparameter settings. The TFBS reward coefficient is set to 0.01. We use a batch size of 256 and run optimization for 100 iterations. The learning rate is fixed at le-4 across all datasets.
TACO [9:
widely used policy optimization algorithm that updates policies using clipped surrogate objectives to ensure stable training. We use Generalized Advantage Estimation (GAE) to compute the advantage function. Optimization is performed using the Adam optimizer with learning rate of le-4. The value network shares the same pretrained HyenaDNA backbone as the policy network, with linear head on top of the final hidden state to predict value estimates. We use a batch size of 256 and train for 100 iterations.
PPO (35):
PPO-Lagrangian 12 A constrained variant of PPO that incorporates Lagrangian relax- ation balance main rewards and constraint satisfaction. For each reward component (i.e.. each cell type), separate value network is trained to estimate the expected return for that component. These value networks share the same architecture as in PPO. However, due to GPU memory constraints, we apply LoRA to fine-tune only the MIP and mixer layers in the HyenaDNA model. The optimizer, learning rate (le-4), batch size (256), and total training iterations (100) are kept consistent with the PPO setup.
16

---

A.6 Sequence Diversity
In this section, we report the full results of sequence diversity for Ctrl-DNA and baseline methods as shown in Figure
HepG2
K562
SK-N-SH
2.5
2.0
20
2.0
Enhancer
1.5
1,5
1.5
1.0
1.0
0.5
0.5
0.5
0.0
AdaL.cs
80
CMAES
PEX
Regg
PPO
De-ONA (D-D
TACO
PRO Lag
Cer DONAA (Dx0)
can ONA (dr0)
Adal ear
80
CMAES
PEX
Regt td
PPO
CMLONA (D-0.
TACO
PRD Lag
Cat RILONA (pm0) 5
COLDNA (090
AdilLen
80
CMAE8
PEX
Regt
PPo
TACO
CeloNuA doed
Can come bo
JURKAT
K562
THP1
2.00
2:00
2.5
+
1.25
1.25
1.5
1.00
1.00
1.0
0.76
0.75
0.5
0.25
0.00
0.00
-0.5
AdaLead
B0
CHAES
PEX
RegLMM
PRO
TACO
Car-DNA (6x0) Al
PPD-Lag
Cer-AA (5-d
CAr-NN (6-0
AdaLesd
BD
CHAES
PEX
Regi M
PRO
TACO
CRroNA (en0 41
Ppo-Lag
CHr-NA (6-0.g)
caI-DNA (6-0
Adoleaad
BO
CMAES
PEX
Al
PPO
TACO
Cat-ONA (6-0 4)
PPC-Leg
Car-DNA (6-0
CaI-NA (doa
Figure 4: Sequence diversity scores for generated sequences on the human enhancer and promoter
datasets. Higher values indicate greater variability among generated sequences.
A.7 Extended Ablation Results
This section provides additional results and implementation details for the ablation experiments introduced in Section 4.4] We evaluate ablations on both the Human Enhancer and Human Promoter datasets.
Ctrl-DNA-Logg Following the reward-guided approach proposed in [50, implement log-barrier transformation the constraint reward. Specifically, we define a log-augmented reward as
m
Riog (X) = (x( +
log (max (6, - (x(),)))),
(9)
where Rg is the target reward, R; is the constraint reward for i 2 1, c is the threshold, and is a small constant for numerical stability. We then compute normalized advantages using this transformed reward:
(j) Rlog (X₂) Riog Log
A
(10)
a (Ping)
We replace the mixed advantages in the original loss function (Equation with 1 All other settings (c.g., surrogate loss, clipping, KL regularization) remain the same as Ctrl-DNA.
17

---

Table Ablation study for Ctrl-DNA across three target cell types in Human Enhancer datasets.
Method
Target: HcpG2
HopG21
K562
SK-N-SH 4
AR
Correlation
Diversity
Cul-DNA-Log
0.24 (0.02)
0.24 (0.03)
0.21 (0.04)
0.02 (0.06)
0.16 (0.13)
1.62 (0.08)
Chi-DNA-IPPO
0.86 (0.02)
(0.02)
(0.14)
1.58 (0.13)
Ctrl INA Amaz 0.00)
0.78 (0.02)
0.40 (0.06)
0.33 (0.03)
0.42 (0.02)
0.33 (0.12)
1.82 (0.02)
Ctrl-DNA Amaz 0.01)
0.77 (0.01)
0.34 (0.04)
0.30 (0.02)
(0.02)
0.16 (0.10)
1.84 (0.01)
Chrl DNA Amar
0.77 (0.01)
0.36 (0.04)
0.31 (0.04)
0.44 (0.03)
0.43 (0.07)
1.82 (0.04)
Method
Target: K562
1
HepG2
SK-N-SH
Motif Correlation
Diversity
Cul-DNA-Log
0.28 (0.01)
0.16
0.19 (0.02)
0.11 (0.02)
0..47)
1.68 (0.16)
Cul-DNA-IPO
0.84 0.93 (0.01)
0.65 (0.09)
0.73
0.39 (0.05)
1.08
Chrl-NN (Amaa 0.00)
0.43 (0.01)
0.35 (0.01)
0,54 (0.01)
(0.02)
1.72 (0.02)
Cir.DNAA (Amas 0.01)
(0.01)
(0.05)
(0.01)
(0.06)
(0.03)
(0.05)
Cui-DNA (omaz =.1)
0.93 (0.01)
0.43 (0.01)
0.35 (0.01)
0.54 (0.01)
0.51 (0.02)
1.82 (0.04)
Method
Target: SK-N-SH
SK-N-SI1 f
HepG2
K562
Motif Correlation
Diversity
DNA-Log Chi--NN-IPPO
0.46 (0.07)
0.05 (0.01)
0.05 (0.01)
0.41 (0.07)
0.12 (0.02)
1.68 (0.08)
0.91 (0.04)
0.87 (0.01)
0.70 (0.01)
0.13 (0.03)
0.13 (0.04)
1.66 (0.07)
Cul-DNA (Amaz = 0.00)
0.83 (0.04)
0.57 (0.11)
0.38 (0.07)
0.35 (0.10)
0.35 (0.06)
1.78 (0.03)
Cui-DNA (Amas =0.01)
(0.05)
(0.03)
(0.03)
1.84
Cur-DNA Oomas 0.1)
0.86 (0.02)
0.54 (0.13)
0.44 (0.01)
0.37 (0.11)
0.51 (0.02)
1.82 (0.04)
Table Ablation study for Cirl-DNA across three target cell types in Human Promoter datasets.
Method
Target: JURKAT
1
Marif Correlation
Divensity
Cui-DNA-Log
0.46 (0.02)
0.17 (0.02)
0.45 (0.02)
0.15 (0.01)
0.11 (0.36)
1.61 (0.07)
Cul-DNA-IPO
(0.14)
0.31 (0.29)
1.22 (0.38)
Ctr- DNA (Ama = 0.003)
0.59 (0.11)
0.19 (0.03)
0.49 (0.02)
0.25 (0.09)
0.28 (0.36)
1.56 (0.09)
(Aroce 0.01)
(0.12)
(0.02)
(0.02)
(0.10)
0.18 (0.31)
1.56 (0.07)
Cui-DNA Amnaz 0.1)
0.69 (0.09)
0.38 (0.02)
0.49 (0.01)
0.25 (0.01)
0.69 (0.01)
1.69 (0.03)
Method
Target: KS62
K562 1
JURKAT
TIIP1
Motil Comelation
Diversity
Chi-DNA- Log
0.26 (0.04)
0.29 (0.03)
0.44 (0.03)
-0.11 (0.06)
0.60 (0.01)
1.49 (0.1
Ctrl-DNA-IPO
0.59 (0.02)
0.59 (0.06)
0.69 (0.09)
-0.11
0.57 (0.05)
1.65 (0.14)
Cul-DNA (emax 0.00)
0.61 (0.04)
0.49 (0.03)
0.49 (0.01)
0.11 (0.03)
0.61 (0.16)
1.63 (0.06)
Cirl-DNA (Amas 0.01)
(0.03)
0.49 (0.01)
(0.05)
Ctrl-DNA Avmax 0.1)
0.58 (0.03)
0.43 (0.06)
0.50 (0.01)
0.12 (0.02)
0.75 (0.06)
1.64 (0.04)
Method
Target: TIPP
THPI
1
AR
Motif Correlation
Diversity
Cul-DNA-LLA Log
0.51 (0.01)
0.10
0.16 (0.01)
0.38 (0.02)
0.42 (0.05)
1.50 (0.03)
Cul-DNA-IPP
0.88
0.59 (0.06)
0.42 (0.11)
0.38 (0.01)
0.57 (0.05)
1.77 (0.06)
Ctrl-DNA (Amaz 0.00)
0.92 (0.01)
0.51 (0.02)
0.33 (0.05)
0.50 (0.03)
0.23 (0.07)
1.22 (0.01)
Cul-DNA (maz 0.01)
0.50 (0.01)
0.32
(0.02)
0.23 (0.02)
(0.01)
ChrlDNN (Amaa 0.1)
(0.00)
0,51 (0.01)
0.31 (0.03)
0,51 (0.01)
0.25 (0.04)
1.81 (0.01)
18

---

Ctrl-DNA-IPO. Based on Interior-poinn -oinnt Policy Optimization (IPO) 49), we incorporate the log- barrier directly into the optimization objective. The surrogate loss becomes:
B T
ΣΣ min { p C (0) B
AD clip, (p: (o0))) -B KL Told) Σ
Φ (150
(11)
i=1
where the log-harries penalty is defined as
oJ log 6; J:m))
(12)
with > 0 controlling the sharpness of the approximation to the indicator function. A larger l yields a tighter barrier. We set 50 in our experiments. Note that, unlike the main Ctrl-DNA method, this variant does not compute mixed advantages in Equation Instead, we compute advantages using only the target reward:
Ro(X,) Ro o(Ho
A
(13)
A.8 Differential Expression Optimization
To assess whether scalarized objective suffices for cell-type-specifie design, we replace the original constrained reward in all optimizers with the differential expression (DE) objective [4]. Concretely, for each target cell type and set of off-target cell types O, we optimize single reward of the form
DE(x) Ro(x) Σ (R,(2)).
where Ro (x) are the cell activity of sequence in the target cell type. We evaluate TACO, AdaLead, PEX, and Ctrl-DNA (ours) on the human enhancer tasks by substituting their training rewards with DE(z); we also include Ctrl-DNA-DE variant that uses DE its reward during policy updates. The rest of the training and evaluation protocol is unchanged from the main experiments.
We additionally include Simulated Annealing (SA) for probabilistic exploration and Fast SeqProp (FSP) for gradient-basee based refinement 38 These two methods optimize MinGap, which a specific scalarization similar to DE. The DE-based evaluations therefore directly comparable to their standard objective.
As reported in Table 8 DE-based baseline variants consistently perform worse than Ctrl-DNA, even in terms of DE values. This is because using a DE reward imposes a fixed trade-off between target and off tfrretr expression, which often results in suboptimal solutions when the objectives conflict. This fixed weighting lacks the ficxibility to adjust the balance between objectives throughout training [52). In contrast, our Lagrangian approach allows for more effective coordination of competing goals, resulting in better performance.
Figure 2 further shows that Ctrl-DNA supports variable off-target thresholds (e.g., 0.3, 0.5, 0.6). allowing users to tailor optimization to specific biological requirements. Once the constraint is satisfied, the model focuses maximizing on-target activity within the feasible region. In contrast, DE optimization can lead the model to suppress off-target activity, even at the cost reducing on-target activation. For example, when optimizing for SK-N-SH, Ctrl-DNA-DE tends to suppress off-target expression at the expense of decreasing SK-N-SH cell activity.
A.9 Reward Model Generalization
We optimized and evaluated sequences using the same pretrained reward model from regLM 4 trained on promoter data with chromosomes' 7, 13, 21, and 22 held out. To test robustness to reward model shifi (and potential reward hacking), we introduced two evaluation settings: (1) optimization uses the re reward model, while evaluation uses separate "full" reward model trained on the entire dataset (including all chromosomes); (2) optimization uses a reward model trained on random 80% subset of the full data, while evaluation again uses the same full reward model as in Setting 1.
19

---

Table 8: DE optimization study for Ctrl-DNA across three target cell types in Human Enhancer datasets.
Method
Target: HepG2
Hepp22 1
K562 4.
SK-N-SI1 5
AR
Ctrl-DNA
0.77 (0.01)
0.36 (0.04)
0.31 (0.04)
0.44 (0.03)
Ctrl-DNA-DE
0.67 (0.02)
0.47 (0.02)
0.37 (0.03)
0.25 (0.02)
TACO-DE
0.58 (0.05)
0.46 (0,04)
0.39 (0.01)
0.16 (0.03)
AdaLead-DE
0.45 (0.03)
0.46 (0.05)
0.43 (0.13)
-0.01 (0.02)
Pex DE
0.67 (0.06)
0.57 (0.03)
0.61 (0.04)
0.06 (0.03)
FSP
0.62 (0.03)
0.36 (0.02)
0.36 (0.04)
0.26 (0.03)
SA
0.61 (0.01)
0.33 (0.01)
0.33 (0.01)
0.28 (0.03)
Method
Target: K562
K562 1
HepG2 L
IK-N-SH| 1
A Rt
Cri-DNA
0.93 (0.01)
0.43 (0.01)
(0.00)
0.54 (0.01)
Cul-DNA-DE
0.87 (0.04)
0.43 (0.01)
0.35 (0.02)
0.48 (0.03)
TACO-DE
0.80
0.48 (0.01)
0.49
(0.01)
Adal DE
0.54 (0.16)
0.49 (0.12)
0.49 (0.15)
0.05 (0.03)
Pex-DE
0.81 (0.04)
0.60 (0.05)
0.61 (0.08)
0.18 (0.07)
TSP
0.67 (0.07)
0.35 (0.02)
0.32 (0.04)
0.33 (0.04)
(0.07)
(0.01)
0.33 (0.01)
0.33 (0.04)
Method
Target: SK-N-SH
SK-N-SH 1
HepU2 J
K562 1
AR
Cul-DNA
0.86 (0.02)
0.54 (0.13)
0.44 (0.01)
0.37 (0.11)
Cul-DNA-DE
0.68 (0.02)
0.26 (0.02)
0.28 (0.04)
0.41 (0.02)
TACO DE
0.54 (0.02)
0.42 (0.04)
0.42 (0.09)
(0.01)
AdaLead-DE
0.45 (0.03)
0.43 (0.03)
0.45 (0.05)
0.01 (0.02)
DE
0.77 (0.03)
(0.09)
0.67 (0.09)
0.09 (0.02)
FSP
0.54 (0.03)
0.39 (0.02)
0.36 (0.01)
0.16 (0.02)
0.54 (0.04)
0.37 (0.05)
0.35 (0.04)
0.18 (0.05)
In all cases we use the human enhancer dataset targeting HepG2-specific activity, and the evaluation protocol matches the main experiments.
As shown in Table g both settings have trends and values consistent with the main results on expression predictions, indicating that the observed gains are not artifacts of overfitting to the training reward. In particular, evaluating under a distinct, stronger reward model (Settings 1-2) preserves the relative performance of Ctrl-DNA, suggesting robustness to evaluator shift and minimal reward hacking.
Table 9: Reward model generalization evaluation on the human enhancer task (HepG2 target).
Method
HepG2 1
K562 $
SK-N-SH
AR t
Ctrl-DNA
0.77 (0.01)
0.36 (0.04)
0.31 (0.04)
0.44 (0.03)
Setting I
0.76 (0.02)
0.35 (0.03)
0.35 (0.02)
0.40 (0.04)
Setting 2
0.82 (0.04)
0.37 (0.08)
0.36 (0.02)
0.45 (0.01)
A.10 Extended TFBS Reward Analysis
potential concern is the use of frequeeye correlation both as regularizer during training and as an evaluation metric. To address concerns of circularity, we conducted an additional analysis using distinct reference sets for evaluation, created by splitting sequences at the 90th and 50th percentiles based on their on and off target activities. The motif frequency correlation results are evaluated by using the new reference sets (Table TO.. Although this filtering significantly reduced the dataset size (leaving only 18 sequences for JURKAT, 4 for K562, and 7 for THP1 out of the original 12,335), Ctrl- DNA continued to demonstrate strong performance, substantially outperforming baseline methods across all tested cell lines with correlations of 0.37 for JURKAT, 0.52 for K562, and 0.60 for THP1.
20

---

Table 10: Motif-frequency correlation under 90/50 percentile selection.
Cell Type
Cir-DDA
Adal.cad
BO
CMA-FS
PFX
PPO-1ag
PPO
TACO
JURKAT
0.37 (0.02)
0.23 10)
(0.17)
0.19 (0.06)
0.34 (0.02)
0.27 (0.15)
0.32 (0.12)
0.39 (0.09)
K562
0.52 (0.11)
0.54 (0.23)
0.13 (0.25)
0,14 (0.05)
0.31 (0.02)
(0.18)
0.31 (0.13)
0.21 (0.14)
THPI
0.60 (0.02)
0.16 (0.13)
0.06 (0.08)
0.06 (0.04)
0.04 (0.02)
-0.02 (0.07)
0.37 (0.04)
0.33 (0,01)
Beyond circularity, we emphasize that the TFBS reward is global: we first compute motif- freiu-ffre vector from a reference set real sequences, then use the Pearson correlation between this reference vector and the motif frequencies of generated sequences. This encourages distributional alignment of motif usage rather than direct motif matching. in contrast to methods such as TACO. Despite this coarse-grained signal, Ctrl-DNA still captures cell-type-specifee motifs (e.g., HNF4A in HepG2, GATA in K562), indicating that specificity emerges from optimizing the global reward conjunction with explicit off-target constraints (see Fig. 3a).
Regarding threshold choice, we follow prior work 8 and adopt percentile-basee selection (on- target 2 50th; off-target I 50th) to reduce sensitivity to cross-cell-type scaling and normalization. While stricter 90th- -0th-percentile cutoff for on-target activity can enrich for highly specific sequences, yields too few samples for reliable TFBS enrichment estimates (e.g.. 18/4/7 for JURKAT/K362/THH in promoters; none across all three cell types in the human enhancer set among ~660k candidates). To further guard against overfitting upper-bounded weights for each TFBS regularization term (Section 3.3) which regularizes the influence of any single motif channel.
spurious motif correlations, Ctrl-DNA employs learnable,
A.11 Effect of Pretraining Optimization
We use pretrained HyenaDNA backbones without task-specific fine-tuning prior to reinforcement learning. To quantify the impact of pretraining, ablate the model initialization the HepG2 enhancer task by comparing a pretrained backbone to a randomly initialized one. Training and evaluation protocols match the main setup. We do not employ fine-tuned backbones in the main results because prior work [6] and our early trials showed minimal benefits over pretraining alone.
Table 11: Pretraining ablation on the HepG2 enhancer task.
Method
HepG2 f
K562 4
SK-N-SH +
AR
Pretrained
0.77 (0.01)
0.36 (0.04)
0.31 (0.04)
0.44 (0.03)
Random Init.
0.71 (0.01)
0.49 (0.03)
0.42 (0.02)
0.25 (0.01)
Pretraining materially improves target activation while reducing off-target expression, yielding substantially higher R (Table IT Notably, TACO also uses pretrained HyenaDNA backbone (Fig. 2) yet underperforms Ctrl-DNA, indicating that pretraining helps but does not explain the performance gap; the optimization framework remains the dominant factor.
A.12 Additional Experiments
We extend our human enhancer experiments with two additional baselines. Specifically, add DRAKES, a diffusion-basees generator for regulatory sequences (53), and CbAS, a model-based design method that conditions search distribution on meeting property event via importance- weighted maximum likelihood (54). Results are reported in Table Across all three target cell types (HepG2, K562, and SK-N-SH), Ctrl-DNA consistently achieves the highest -prget-specifi activity and the largest R. indicating superior ability to enhance the target cell type while suppressing off ofr-tt expression. For example, when targeting HepG2, Ctrl-DNA improves AR to 0.44 0.03, compared to near-zero or negative shifts for CbAS and DRAKES, demonstrating effective specificity control. Similar gains are observed for K562 (A 0.54 # 0.01) and SK-N-SH (AR = 0.37 + 0.11). These results highlight the advantage of Ctrl-DNA in handling multiple constraints compared to diffusion and model-based methods optimized for single objectives.
21

---

Table 12: Additional experiments for Ctrl-DNA across three target cell types in Human Enhancer datasets.
Method
Target: Hept2
HepG2
K562 4
SK-N-SH1
AR
Cirl-DNA
0.77 (0.01)
0.36 (0.04)
0.31 (0.04)
0.44 (0.03)
(0.03)
0.79 (0.04)
0.73 (0.03)
(0.02)
DRAKES
0.68 (0.004)
0.78 (0.007)
0.71 (0.007)
-0.07 (0.005)
Method
Target: K562
Hep L
SK-N-SH 4
Cut-DNA
0.93 (0.01)
0.43 (0.01)
0.35 (0.01)
0.54 (0.01)
ChAS
0.90 (0.01)
0.66 (0.02)
0.69 (0.01)
0.23 (0. 18)
DRAKES
0.83 (0.003)
0.69 (0.002)
0.80 (0.001)
0.08 (0.002)
Method
Target: SK-N-SH
SII
HepG2 L
K562 |
AR
CrI-DNA
(0.02)
(0.13)
(0.01)
10.1
ChAS
0.81 (0.05)
0.68 (0.08)
0.81 (0.06)
0.06 (0.01)
DRAKES
0.79 (0.001)
0.80 (0.003)
0.69 (0.002)
0.05 (0.002)
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Question:
Answer: [Yes]
Justification: All primary claims made in the abstract and introduction are substantiated by our experimental findings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Yes, we discuss the limitations of our method the final paragraph of the Discussion section. Specifically, we acknowledge the challenges in generalizing to broader genomic contexts and limitations posed by dataset distribution. We also discuss limitations related to training stability in sparse-reward settings and potential extensions to incorporate more complex biological constraints.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
22

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only lested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This work primarily proposes empirical method for applying constrained reinforcement learning to cis-regulatory DNA sequence design. does not introduce new theoretical results or formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:AAl code and training details, including hyperparameters, training settings, environment specifications, and pseudocode, will be provided in the supplementary material to ensure full reproducibility of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
23

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide all training code and instructions to reproduce the main experi677 mental results in supplementary materials. The repository includes detailed documentation environment setup, data preprocessing. training, and evaluation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /nips.cc/publi/ggu.dee
wwwpannisssoooooooooico for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
24

---

6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide comprehensive implementation and training details in Section B and Section A, including data preprocessing, model architecture, optimizer settings, and hyperparameters. These details are sufficient to reproduce the results reported in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For all experiments, used multiple random seeds and reported both the mean and standard deviation (or confidence interval) averaged across them.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix A provides details on the computational resources used, including GPU type, memory, and training time.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
25

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:
Answer: [Yes]
Justification: This research adheres to the NeurIPS Code Ethics. It uses publicly available datasets under appropriate licenses, does not involve human subjects sensitive data, and poses minimal risk of harm. All contributions are properly credited, and results are reported transparently to support reproducibility.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The paper includes a discussion of broader societal impacts. We highlight potential positive outcomes, such as enabling cell-type-specifee gene regulation design for therapeutic applications. We also address risks, including the potential misuse of generative models for harmful synthetic biology applications, and recommend that use be restricted to academic and clinically approved research settings.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
26

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The models and datasets used in this work focus on regulatory DNA sequence design for scientific research. They do not pose foreseeable risks of misuse, and no high-risk assets such as language models, generative image models, or scraped personal data are involved.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: All external datasets and software tools used in this work are publicly available and appropriately cited in the manuscript. We ensured that their licenses (e.g.. MIT, CC-BY) permit academic use and complied with terms use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo//aaae.e cam/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release all code and pretrained models used in this work supplementary materials. The materials include detailed instructions for reproducing our experiments, including data pre-processing, training, and evaluation.
Guidelines:
The answer NA means that the paper does not release new assets.
27

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve human subjects or any form crowdsoureing; all data are in-silico and model-based.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve human subjects any form of crowdsoureing: all data are in-silico and model-based.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are not used in any core methodological component of this work
Guidelines:
28

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, non non-standard components.
Please refer to our LLM policy (https://neurip:: /ttp://pp.ccc//offfeeeee//////// for what should or should not be described.
29


Reviews:
[{"review_id": "fcK5vugeMd", "reviewer": "Reviewer_TfWB", "summary": "Designing cell-type-specific cis-regulatory elements (CREs) such as promoters and enhancers are vital to the success of gene therapies. This work proposes a constrained RL-based method called regCon to design such CREs. regCon fine-tunes an autoregressive genomic language model using constrained RL to maximize reward model-predicted expression in a target cell type while maintaining predicted expression in non-targeted cells below a specified threshold. regCon uses Lagrange multipliers to control cell-type-specificity and policy gradients for optimizing the policy. Instead of adopting the conventional approach of using learned value functions, regCon instead modifies the Group Relative Policy Optimization (GRPO) framework to use batch-level information to compute advantage estimates – this incentivizes the model to produce sequences that have rewards similar to the best sequences in the batch. Finally, to ensure that the generated sequences retain transcription factor (TF) binding motifs observed in real cell-type-specific sequences (and thus reduce reward hacking), the authors add a constraint reward function that incentivizes the model to retain these motifs. The authors thoroughly benchmark regCon using two datasets that measure CRE activity and compare its performance to many baselines. regCon outperforms all baselines convincingly, produces cell-type-specific TF-binding motifs, and further ablation analyses validate the usefulness of various components.", "questions": "I have mentioned my questions and suggestions in the previous section.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "This is a strong submission in my opinion with well-motivated techniques, thorough benchmarking, and clear writing. I provide more details below.\n\nQuality: \n- The paper is technically sound and all parts of the regCon framework are clearly motivated and build on relevant prior work\n- Claims made in the paper are convincingly supported by the results - it is clear from the analyses that regCon is a potent sequence optimizer for designing cell-type-specific CREs since it clearly outperforms baselines.\n- Have the authors tried using a relatively simple reward function such as the average difference in expression between the target and non-target cells (like in Reddy et al. [1]) instead of using the constraint-based reward function? Adding this result would make the case for using the constraint-based reward function more convincing.\n- Although the authors benchmark many different baselines, they do not benchmark prior biological sequence design methods such as those proposed by Reddy et al. [1], Linder et al. [2], Brookes et al. [3], Gosai et al [4], etc. Adding some of these as additional baselines might make the case for regCon even more convincing.\n\nClarity: In general, this paper is very well-written and easy to understand. My only suggestion is to have more details about how exactly the datasets are processed. The authors currently reference another paper and say they use the same pipeline but it would be useful to explicitly write out the processing steps in the appendix for the sake of reproducibility.\n\nSignificance:\n- In my opinion, this could be a high impact submission given that it builds on prior work to provide a sequence optimization framework that could be of broad use to computational biologists. It also aims to solve a difficult yet important problem.\nThis work would also be of interest to RL researchers given its innovations\n- Apart from cell-type-specific CRE design, this framework could be adapted for solving other constraint-driven optimization problems that are common in biology such as protein or RNA design.\n\nOriginality: I am quite familiar with biological sequence design techniques but I am not as familiar with the latest RL research. However, to the best of my knowledge, this paper presents many original ideas such as using the constrained RL framework for cell-type-specific CRE design, using batch-level GRPO, and incorporating TF-binding motif retention into the optimization process. As mentioned previously, the authors provide clear motivations for these ideas and how they build on prior work.\n\nReferences:\n1. Reddy AJ, Geng X, Herschl M, Kolli S, Kumar A, Hsu P, Levine S, Ioannidis N. Designing cell-type-specific promoter sequences using conservative model-based optimization. Advances in Neural Information Processing Systems. 2024 Dec 16;37:93033-59. \n2. Linder J, Bogard N, Rosenberg AB, Seelig G. A generative neural network for maximizing fitness and diversity of synthetic DNA and protein sequences. Cell systems. 2020 Jul 22;11(1):49-62.\n3. Brookes D, Park H, Listgarten J. Conditioning by adaptive sampling for robust design. InInternational conference on machine learning 2019 May 24 (pp. 773-782). PMLR.\n4. Gosai SJ, Castro RI, Fuentes N, Butts JC, Mouri K, Alasoadura M, Kales S, Nguyen TT, Noche RR, Rao AS, Joy MT. Machine-guided design of cell-type-targeting cis-regulatory elements. Nature. 2024 Oct 23:1-0.", "was_revised": true, "final_justification": "This submission was already very strong and the authors further improved it during the discussion period by providing more baselines. I have no other issues with the paper, and retain my original recommendation to accept it.", "rebuttal": "We appreciate the reviewer’s recognition of our method’s novelty, the concreteness of our experiments, and the framework’s potential for broader biological design tasks. We also thank the reviewer for the constructive feedback. Please find our reponse below regarding the added baselines experiemnts and clarification on dataset processing. \n\n\n###  **Comparison with simple DE-based reward functions and other baselines**\n**Summary**: \n\nWe added the requested baselines and differential expression (DE) as a scalar reward. Across enhancer and promoter settings, regCon yields higher on target expression while meeting off target constraints. Scalar DE optimization underperforms and often increases DE by suppressing off target at the cost of on target signal.\n\n**Detail response:**\n\nWe have added experiments involving the baselines mentioned: Reddy et al. [1], Linder et al. [2], Brookes et al. [3], and Gosai et al. [4], as well as additional experiments using DE as a scalar reward function (denoted with a \"-DE\" suffix in Table 1-3).\n\nSpecifically, Gosai et al. [4] proposed three generation strategies: AdaLead, Fast SeqProp (FSP), and Simulated Annealing (SA). Since AdaLead was already included in our main experiments, we additionally implemented and benchmarked FSP and SA. Due to time and resource constraints, we evaluated:\n\n- Brookes et al. [3] and Gosai et al. [4] (FSP and SA) on the human enhancer dataset, alongside the DE-based variants of our baselines (Table 1-3).\n- Reddy et al. [1] and Linder et al. [2] on the human promoter dataset (Table 4-6).\n\nAs shown across all tables, DE-based optimization consistently underperforms compared to regCon. Scalar DE optimization applies a fixed linear trade-off (Target - Off), which tend to converge to suboptimal optimization space[5]. In contrast, Lagrangian methods dynamically adjust penalty coefficients through learned multipliers, enabling the model to escape poor local optima and achieve more effective optimization.\n\nFurthermore, maximizing DE as a scalar reward can lead the model to aggressively suppress off-target expression, even at the expense of reducing on-target activation. This behavior is evident in the performance of regCon-DE in Table 3, where the method achieves higher DE but at the expense of significantly lower on-target expression compared to regCon. In contrast, regCon enforces explicit off-target constraints and directs optimization toward increasing on-target activity within the feasible region, leading to higher on-target cell activity and more biologically meaningful solutions.\n\n\n\n**Table 1.**\n| **Method**           | **HepG2 ↑**     | **K562 ↓**     | **SK-N-SH ↓** | **ΔR**        |\n|----------------------|-----------------|----------------|----------------|----------------|\n| regCon               | 0.77 (0.01)     | 0.36 (0.04)    | 0.31 (0.04)    | 0.44 (0.03)    |\n| regCon-DE            | 0.67 (0.02)     | 0.47 (0.02)    | 0.37 (0.03)    | 0.25 (0.02)    |\n| TACO-DE              | 0.58 (0.05)     | 0.46 (0.04)    | 0.39 (0.01)    | 0.16 (0.03)    |\n| AdaLead-DE           | 0.45 (0.03)     | 0.46 (0.05)    | 0.43 (0.13)    | -0.01 (0.02)    |\n| Pex-DE               | 0.67 (0.06)     | 0.57 (0.03)    | 0.61 (0.04)    | 0.06 (0.03)    |\n| Brookes et al. [3]   | 0.74 (0.03)     | 0.79 (0.04)    | 0.73 (0.03)    | -0.02 (0.02)  |\n| FSP [4]                  | 0.62 (0.03)     | 0.36 (0.02)    | 0.36 (0.04)    | 0.26 (0.03)    |\n| SA  [4]                 | 0.61 (0.01)     | 0.33 (0.01)    | 0.33 (0.01)    | 0.28 (0.03)    |\n\n**Table 2.**\n| **Method**           | **K562 ↑**     | **HepG2 ↓**     | **SK-N-SH ↓** | **ΔR**        |\n|----------------------|-----------------|----------------|----------------|----------------|\n| regCon               | 0.93 (0.01)     | 0.43 (0.01)    | 0.35 (0.01)    | 0.54 (0.01)    |\n| regCon-DE            | 0.87 (0.04)     | 0.43 (0.01)    | 0.35 (0.02)    | 0.48 (0.03)    |\n| TACO-DE              | 0.80 (0.01)     | 0.48 (0.01)    | 0.49 (0.02)    | 0.50 (0.01) |\n| AdaLead-DE            | 0.54 (0.16)     | 0.49 (0.12)    | 0.49 (0.15)    | 0.05 (0.03) |\n| Pex-DE               | 0.81 (0.04)     | 0.60 (0.06)    | 0.61 (0.08)    | 0.18 (0.07) |\n| Brookes et al. [3]   | 0.90 (0.01)     | 0.66 (0.02)    | 0.69 (0.01)    | 0.23 (0.18) |\n| FSP [4]                  | 0.67 (0.07)     | 0.35 (0.02)    | 0.32 (0.04)    | 0.33 (0.04) |\n| SA [4]            | 0.67 (0.07)     | 0.34 (0.01)    | 0.33 (0.01)    | 0.33 (0.04) |\n\n**Table 3.**\n| **Method**           | **SK-N-SH ↑**     | **HepG2 ↓**     | **K562 ↓** | **ΔR**        |\n|----------------------|-----------------|----------------|----------------|----------------|\n| regCon               | 0.86 (0.02)     | 0.54 (0.13)    | 0.44 (0.01)    | 0.37 (0.11)    |\n| regCon-DE            | 0.68 (0.02)     | 0.26 (0.02)    | 0.28 (0.04)    | 0.41 (0.02)    |\n| TACO-DE              | 0.54 (0.02)     | 0.42 (0.04)    | 0.42 (0.09)    | 0.12 (0.01) |\n| AdaLead-DE            | 0.45 (0.03)     | 0.43 (0.03)    | 0.45 (0.05)    | 0.01 (0.02) |\n| Pex-DE               | 0.77 (0.03)     | 0.64 (0.09)    | 0.67 (0.09)    | 0.09 (0.02) |\n| Brookes et al. [3]   | 0.81 (0.05)     | 0.68 (0.03)    | 0.81 (0.06)    | 0.06 (0.01) |\n| FSP [4]          | 0.54 (0.03)     | 0.39 (0.02)    | 0.36 (0.01)    | 0.16 (0.02) |\n| SA [4]            | 0.54 (0.04)     | 0.37 (0.05)    | 0.35 (0.04)    | 0.18 (0.05) |\n\n**Table 4.**\n| **Method**           | **JURKAT ↑**     | **K562 ↓**     | **THP1 ↓** | **ΔR**        |\n|----------------------|-----------------|----------------|----------------|----------------|\n| regCon               | 0.69 (0.09)     | 0.38 (0.02)    | 0.49 (0.01)    | 0.25 (0.01)    |\n| Reddy et al. [1]     | 0.47 (0.02)      | 0.31 (0.01)   | 0.51 (0.01)    | 0.06 (0.01)    |\n| Linder et al. [2]    | 0.40 (0.05)     | 0.25 (0.03)    | 0.51 (0.01)    | 0.03 (0.05)    |\n\n\n**Table 5.**\n| **Method**           | **K562 ↑**     | **JURKAT ↓**     | **THP1 ↓** | **ΔR**        |\n|----------------------|-----------------|----------------|----------------|----------------|\n| regCon               | 0.58 (0.03)     | 0.43 (0.06)    | 0.50 (0.01)    | 0.12 (0.02)    |\n| Reddy et al. [1]     | 0.41 (0.01)      | 0.48 (0.02)   | 0.75 (0.01)    | -0.12 (0.01)   |\n| Linder et al. [2]    | 0.24 (0.03)     | 0.39 (0.03)    | 0.51 (0.01)   | -0.21 (0.05)    |\n\n**Table 6.**\n| **Method**           | **THP1 ↑**     | **JURKAT ↓**     | **K562 ↓** | **ΔR**        |\n|----------------------|-----------------|----------------|----------------|----------------|\n| regCon               |0.92 (0.01)     | 0.51 (0.01)    | 0.31 (0.03)    | 0.51 (0.01)    |\n| Reddy et al. [1]     | 0.55 (0.02)    | 0.39 (0.01)    | 0.29 (0.01)    | 0.20 (0.01)   |\n| Linder et al. [2]    | 0.51 (0.01)     |0.41 (0.07)    | 0.22 (0.03)   | 0.20 (0.03)    |\n\n### **Clarity: add dataset processing details**\nThank you for the suggestion. We will add an Appendix section that details data processing steps to enhance reproducibility. \n\nThank you again for your helpful feedback. We’re happy to address any further suggestions or questions.\n### **References:**\n[1] Reddy AJ, Geng X, Herschl M, Kolli S, Kumar A, Hsu P, Levine S, Ioannidis N. Designing cell-type-specific promoter sequences using conservative model-based optimization. Advances in Neural Information Processing Systems. 2024 Dec 16;37:93033-59.\n\n[2] Linder J, Bogard N, Rosenberg AB, Seelig G. A generative neural network for maximizing fitness and diversity of synthetic DNA and protein sequences. Cell systems. 2020 Jul 22;11(1):49-62.\n\n[3] Brookes D, Park H, Listgarten J. Conditioning by adaptive sampling for robust design. InInternational conference on machine learning 2019 May 24 (pp. 773-782). PMLR.\n\n[4] Gosai SJ, Castro RI, Fuentes N, Butts JC, Mouri K, Alasoadura M, Kales S, Nguyen TT, Noche RR, Rao AS, Joy MT. Machine-guided design of cell-type-targeting cis-regulatory elements. Nature. 2024 Oct 23:1-0.\n\n[5] Liu, E., Wu, Y. C., Huang, X., Gao, C., Wang, R. J., Xue, K., & Qian, C. (2025, April). Pareto set learning for multi-objective reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 18, pp. 18789-18797)."}, {"review_id": "356vcrKKi3", "reviewer": "Reviewer_49ng", "summary": "This paper introduces regCon, a constrained generative model for DNA sequence design based on constrained reinforcement learning (RL). It aims to increase the cell-type specificity in DNA sequence generation by maximizing fitness on target cell type and minimizing it on off-target ones. It leverages the Lagrangian techniques to enforce the constraints. They benchmark it on MPRA datasets of enhancers and promoters of various cell types show that the generated sequences have high cell type specificity compared to other models suc as regLM. Overall, regCon is an innovative methods for the task of DNA sequence design.", "questions": "- How do you choose the best hyperparameters? If you want to design CREs for a new cell type, how would you choose the hyperparameters?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Adding constraints to the RL framework for DNA generation is innovative and significant.\n\n- The paper is written clearly and explains all the details step by step.\n\n- It is the first os such models that explicitly formulates the off-target fitness minimization.\n\nWeaknesses:\n\n- The models seem to have a lot of hyperparameters.", "was_revised": true, "final_justification": "After reading the rebuttal, my concerns are resolved, and I tend to keep my score.", "rebuttal": "We thank the reviewer for the positive feedback and the insightful question.\n\n**How to choose the best hyperparameters**\n\nWe performed a grid search over key hyperparameters such as the KL coefficient, learning rate, and multipliers. We select hyperparameters by running experiments across different hyperparameters and monitoring reward-based metrics (e.g., target activity, constraint satisfaction) on the generated sequences.\n\n**How to choose the hyperparameters for a new cell type**\n\nWhen designing for a new cell type, we use the corresponding reward model to guide optimization. In practice, we initialize with hyperparameters that worked well on similar cell types and find that our method is relatively robust across a wide range of settings. As shown in Table 3 (line 477 in the manuscript and also down below), many cell types share the same hyperparameters, suggesting generalizability across targets.\n\nWe acknowledge that our method involves a number of hyperparameters. This is in part due to its foundation in Lagrangian-based reinforcement learning [1]. However, many of these hyperparameters can be reused across cell types. For the reviewer’s convenience, we have included the hyperparameter table from the appendix in the rebuttal below. Users can start from the configurations we provide and refine them through a lightweight grid search based on training dynamics.\n\n**Experiment Hyperparameters**\n\n| Hyperparameter                          | HepG2 | K562 | SK-N-SH | JURKAT | K562 | THP1 |\n|----------------------------------------|:-----:|:----:|:-------:|:------:|:----:|:----:|\n| Batch Size                             |  256  | 256  |   256   |  256   | 256  | 256  |\n| Replay Buffer Batch Size               |   24  |  24  |    24   |   24   |  24  |  24  |\n| Policy Learning Rate ($η_θ$)   | 1e-4  | 1e-4 |  1e-4   |  1e-4  | 1e-4 | 1e-4 |\n| Multiplier Learning Rate ($η_λ$)| 3e-4 | 3e-4 |  3e-4   |  3e-4  | 3e-3 | 3e-3 |\n| KL Value Coefficient (β)              |  0.2  | 0.2  |   0.2   |  0.2   | 0.2  | 0.2  |\n| TFBS Multiplier Upper Bound ($λ_{max}$) | 0.1 | 0.1 |  0.1   |  0.1   | 0.1  | 0.1  |\n\nThank you again for your constructive review and positive comments. Please let us know if you have any further questions!\n\n**Reference**\n\n[1] Ray, A., Achiam, J., & Amodei, D. (2019). Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7(1), 2."}, {"review_id": "wQWMBeVYnK", "reviewer": "Reviewer_saSR", "summary": "This work proposes `regCon`, a novel RL method for designing cell-type-specific CREs. `regCon` formulates this task as a constrained optimization problem, aiming to maximize gene expression activity in a target cell type while satisfying user-defined thresholds for off-target cells. This framework fine-tunes a pre-trained genomic language model using a primal-dual approach based on Lagrangian relaxation. Following the recently proposed GRPO framework, it computes advantages directly from batch-normalized rewards to guide policy updates. Furthermore, to enhance biological plausibility, `regCon` incorporates a regularization term based on the correlation of TFBS frequencies between generated and real sequences. Evaluations on human promoter and enhancer datasets show that `regCon` consistently outperforms existing baselines. It successfully generates regulatory sequences with high activity in the target cell type while effectively suppressing off-target activity, achieving controllable cell-type specificity.", "questions": "- Line 474 states that training was conducted for 100 epochs. Does 'epoch' in this context mean 'iteration'? The term 'epoch' is less common in standard reinforcement learning terminology.\n- Why was the problem formulated as a constrained optimization problem? For instance, Reddy et al. [1] define the difference of expression (DE) as a single objective. The authors should clarify the motivation for their chosen formulation.\n- Are most of the implemented baselines single-objective optimizers (i.e., without explicit constraints)? It seems that if the objective were defined as DE, other methods could also be easily adapted to a multi-objective setting.\n- Is the core difference between PPO-Lagrangian and `regCon` the use of a GRPO-like algorithm for advantage estimation? Why is the performance gap between them so large, and what is the primary source of `regCon`'s improvement over PPO-Lagrangian?\n- How were the penalty coefficients ($\\lambda$) initialized at the beginning of training?\n\n[1] Reddy, Aniketh Janardhan, et al. \"Designing cell-type-specific promoter sequences using conservative model-based optimization.\" Advances in Neural Information Processing Systems 37 (2024): 93033-93059.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper is well-written and easy to follow.\n- The problem of designing cell-type-specific CREs is of significant practical importance. The authors' choice to address this using a multi-objective optimization framework is a valuable contribution.\n- The proposed method shows a significant performance improvement over the baselines.\n\nWeaknesses\n- The specific training process for the reward model and details of data partitioning should be more clearly specified in the paper. This includes, but is not limited to, the data splitting principles and the reward model's training strategy.\n- Is the same reward model used for both guiding the optimization and for the final evaluation? This setup could potentially lead to reward hacking.\n- The paper lacks an ablation study on the contribution of the pre-trained model. Results from a randomly initialized HyenaDNA and from the base, un-fine-tuned HyenaDNA would be valuable additions.\n- The description of the baseline implementations is insufficient. While Appendix C provides a high-level overview of each method, it omits critical implementation details.", "was_revised": true, "final_justification": "The authors have addressed my two main concerns in their rebuttal: 1) the rationale for reward model selection, and 2) the respective contributions of the constrained RL formulation and GRPO. Therefore, I am raising my rating from 4 to 5.", "rebuttal": "We thank the reviewer for their thoughtful feedback and voting to accept our paper. We address each point individually below. “W/Q” numbers the weakness or question, followed by our response.\n\n### **[w1]**\nWe follow the training protocols described in [1] for our reward models. For the human enhancer task, we directly adopted the pretrained model weights from [1]. For human promoters, we trained a separate reward model using the data split from [2] (70% train, 20% validation, 10% test). Training was conducted using the AdamW optimizer with a learning rate of 1e-4 and MSE loss, over 20 epochs. The checkpoint with the best validation performance was used for evaluation.\n\nAll reward models are based on an Enformer architecture [3], combining convolutional and Transformer layers, which has shown strong performance in DNA regulatory prediction tasks. Both the enhancer and promoter models share the same architecture: 11 layers with a hidden dimension of 1536. We will add these details in the revised appendix.\n### **[w2]**\nOur method optimized and evaluated sequences using the same pretrained reward model from regLM [4], trained on promoter data excluding chromosomes 7, 13, 21, and 22. To assess potential reward hacking, we introduced two additional settings:\n\nSetting 1: Optimization used the regLM reward model; evaluation used a separate full reward model trained on the full dataset (including all chromosomes).\n\nSetting 2: Optimization used a reward model trained on a random 80% subset of the full data; evaluation used the same full reward model from Setting 1.\n\n| **Method**           | **HepG2 ↑**     | **K562 ↓**     | **SK-N-SH ↓** | **ΔR**        |\n|----------------------|-----------------|----------------|----------------|----------------|\n| regCon               | 0.77 (0.01)     | 0.36 (0.04)    | 0.31 (0.04)    | 0.44 (0.03)    |\n| Setting 1            | 0.76 (0.02)     | 0.35 (0.03)    | 0.35 (0.02)    | 0.40 (0.04)    |\n| Setting 2            | 0.82 (0.04)     | 0.37 (0.08)    | 0.36 (0.02)    | 0.45 (0.01)    |\n\n\nIn both settings, we conducted experiments on the human enhancer dataset, targeting HepG2 cell-specific activity. We observed consistent performance trends across all metrics (target expression, off-target suppression, motif enrichment, and diversity), comparable to our main results.\n\nThese findings suggest that *regCon* generalizes well, and that the observed performance gains are not an artifact of overfitting to the reward model used during optimization.\n\n\n### **[W3]**\n\nWe clarify that our experiments use pretrained HyenaDNA models without task-specific fine-tuning prior to reinforcement learning. To assess the role of pretraining, we compared pretrained and randomly initialized models on HepG2 enhancer optimization. We did not use fine-tuned models in our main experiments, as prior work [6] and our own early results both showed minimal performance differences between pretrained and fine-tuned models.\n\n| Method               | HepG2 ↑         | K562 ↓         | SK-N-SH ↓      | ΔR            |\n|----------------------|-----------------|----------------|----------------|----------------|\n| Pretrained           | 0.77 (0.01)     | 0.36 (0.04)    | 0.31 (0.04)    | 0.44 (0.03)    |\n| Random Init.         | 0.71 (0.01)     | 0.49 (0.03)    | 0.42 (0.02)    | 0.25 (0.01)    |\n\nWe found that the randomly initialized model underperformed the pretrained model. Notably, as shown in Figure 2 in our manuscript, TACO also leverages a pretrained HyenaDNA model but still underperforms our method, indicating that while pretraining offers improvements, our framework provides more effective optimization.\n\n### **[W4]**\nDue to the rebuttal space constraints, we are unable to include full baseline implementation details here, but we will provide them in the revised appendix.\n### **[Q1]**: \nThank you for pointing this out. In our implementation, we use the term \"epoch\" to refer to one full pass over the collected batch of trajectories. We agree that \"iteration\" would be a more appropriate term and will revise it in the manuscript.\n### **[Q2 & 3]**: \nTo investigate scalarized objectives, we ran additional experiments comparing regCon and the top three baselines (TACO, AdaLead, and PEX), each adapted to optimize differential expression (DE). We also tested a variant of regCon (regCon-DE) trained with DE as the reward.\n\nOur method offers two main advantages over DE objectives:\n\n- Explicit constraint control:\nAs shown in Figure 2 in our paper, regCon supports variable off-target thresholds (e.g., 0.3, 0.5, 0.6), allowing users to tailor optimization to specific biological requirements. Once the constraint is satisfied, the model focuses on maximizing on-target activity within the feasible region. In contrast, DE optimization can lead the model to suppress off-target activity, even at the cost of reducing on-target activation. For example, when optimizing for SK-N-SH (Table 2), regCon-DE tends to suppress off-target expression at the expense of decreasing SK-N-SH cell activity.\n\n- Improved DE optimization:\nMost of the baselines are single-objective optimizers. Although they can be adapted to use DE as a reward, our results (Tables 1 and 2) show that these DE-based variants consistently perform worse than regCon, even in terms of DE values. This is because using a DE reward imposes a fixed trade-off between target and off-target expression, which often results in suboptimal solutions when the objectives conflict. This fixed weighting lacks the flexibility to adjust the balance between objectives throughout training [7]. In contrast, our Lagrangian approach allows for a more effective coordination of competing goals, resulting in better performance.\n\nTable 1\n| **Method**           | **HepG2 ↑**     | **K562 ↓**     | **SK-N-SH ↓** | **ΔR**        |\n|----------------------|-----------------|----------------|----------------|----------------|\n| regCon               | 0.77 (0.01)     | 0.36 (0.04)    | 0.31 (0.04)    | 0.44 (0.03)    |\n| regCon-DE            | 0.67 (0.02)     | 0.47 (0.02)    | 0.37 (0.03)    | 0.25 (0.02)    |\n| TACO            | 0.58 (0.05)     | 0.46 (0.04)    | 0.39 (0.01)    | 0.16 (0.03) |\n| AdaLead            | 0.58 (0.05)     | 0.46 (0.04)    | 0.39 (0.01)    | 0.16 (0.03) |\n| Pex            | 0.58 (0.05)     | 0.46 (0.04)    | 0.39 (0.01)    | 0.16 (0.03) |\n\nTable 2\n| **Method**           | **SK-N-SH ↑**     | **HepG2 ↓**     | **K562 ↓** | **ΔR**        |\n|----------------------|-----------------|----------------|----------------|----------------|\n| regCon               | 0.86 (0.02)     | 0.54 (0.13)    | 0.44 (0.01)    | 0.37 (0.11)    |\n| regCon-DE            | 0.68 (0.02)     | 0.28 (0.04)    | 0.26 (0.02)    | 0.41 (0.02)    |\n| TACO            | 0.54 (0.05)     | 0.42 (0.09)    | 0.42 (0.02)    | 0.12 (0.01) |\n| AdaLead            | 0.58 (0.05)     | 0.46 (0.04)    | 0.39 (0.01)    | 0.16 (0.03) |\n| Pex            | 0.58 (0.05)     | 0.46 (0.04)    | 0.39 (0.01)    | 0.16 (0.03) |\n\n### **[Q4]**\nYes, a core difference lies in the advantage estimation method. regCon uses a GRPO-inspired approach based on batch-normalized rewards, eliminating the need for a value network. In contrast, PPO-Lagrangian relies on Generalized Advantage Estimation (GAE), which depends on training a value network.\n\nAs noted in Section 4.2 (Lines 246), training a value network from sparse, sequence-level rewards is inherently unstable, since it must infer accurate token-level estimates. regCon avoids this issue by directly using normalized rewards, leading to more stable policy updates and potentially explaining the observed performance gap.\n### **[Q5]**: \nFor each target cell type, we initialize three penalty coefficients (λ): one for each off-target constraint (λ = 0.5 for JURKAT, K562, SK-N-SH, HepG2; 0.7/0.3 for THP1, reflecting the [0,1] range of λ) and one for the TFBS constraint (initialized at 0.0 with an upper bound of 0.1). These values were selected by grid search to balance expressions in different cell types and motif enrichment. Although Lagrangian methods can be sensitive to initializations[8], we observed stable performance across all cell types with the same initializations. As noted in the line 332, we plan to incorporate PID controllers [6] to improve λ adaptation in future work.\n### **Paper Formatting Concerns**\nWe will revise Figure 2 to use more distinguishable colors and marker shapes for improved clarity.\n\nThank you again for your valuable suggestions, which have helped improve the clarity and contribution of our work. Please let us know if you have any more questions.\n\n[1] Lal, Anusha, et al. \"Designing realistic regulatory DNA with autoregressive language models.\" Genome Research, vol. 34, no. 9, 2024, pp. 1411–1420.\n\n[2] Reddy, A. J., et al. \"Strategies for effectively modelling promoter-driven gene expression using transfer learning.\" bioRxiv, 2024, doi:10.1101/2023.02.01.526753.\n\n[3] Avsec, Žiga, et al. \"Effective gene expression prediction from sequence by integrating long-range interactions.\" Nature Methods, vol. 18, no. 10, 2021, pp. 1196–1203.\n\n[4] Yang, Zhen, et al. \"Regulatory DNA sequence design with reinforcement learning.\" The Thirteenth International Conference on Learning Representations, 2025.\n\n[5] Schulman, John, et al. \"High-dimensional continuous control using generalized advantage estimation.\" arXiv preprint arXiv:1506.02438, 2015.\n\n[6] Stooke, Adam, Joshua Achiam, and Pieter Abbeel. \"Responsive safety in reinforcement learning by pid lagrangian methods.\" International Conference on Machine Learning. PMLR, 2020.\n\n[7] Liu, Enbo, et al. \"Pareto set learning for multi-objective reinforcement learning.\" Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 18, 2025, pp. 18789–18797.\n\n[8] Ji, Jiaming, et al. \"Safety Gymnasium: A unified safe reinforcement learning benchmark.\" Advances in Neural Information Processing Systems, vol. 36, 2023, pp. 18964–18993."}, {"review_id": "OC4Q9ELxgK", "reviewer": "Reviewer_t2FQ", "summary": "The paper presents regCon, a framework that combines an autoregressive DNA language model with constrained RL, resulting in the generation of cell type-specific enhancers or promoters. The method limits the activity of the generated sequences in off-target cell types while maximizing activity in the on-target cell type. The generated sequences are predicted to be specific, diverse, and contain biologically reasonable transcription factor binding motifs.", "questions": "1. A comparison to one or more diffusion modeling approaches would be useful, as several methods have been proposed recently to optimize diffusion models for cell type-specific DNA generation. These would be a conceptually similar benchmark for regCon.\n\n2. Motif frequency correlation is used as a regularization term and also as an evaluation metric in table 1. Is this term the same, and if so, is this circular?\n\n3. How is the TFBS frequency used for regularization computed? Is it based on the 50th percentile or 90th percentile of sequences? Would it not be better to set this threshold to an absolute value based on the target level of cell type-specificity, or to always use the 90th percentile? The choice of 50th percentile (Line 217) was confusing as the aim of the paper is to generate sequences with extremely high specificity.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: The paper stands out in the quality of evaluation. Particularly, the authors compare their method not only to similar methods based on autoregressive LMs, but to a wide range of different approaches. They also evaluate the model on cell types that are very similar to each other, which has not typically been done in previous studies.\n\nClarity: Greater clarity on the computation and use of TFBS correlation would be appreciated.\n\nSignificance: This method addresses a significant problem in the field, that of generating extremely cell type specific regulatory elements. This problem is critical for many industrial and therapeutic applications and such regulatory elements are extremely rare in nature.\n\nOriginality: The method is quite original in the field. Previous work on using autoregressive language models for DNA sequence generation was limited in utility as it did not optimize the generated sequences to be extremely cell type-specific, which is a highly desirable property for gene therapy and other applications. This method effectively addresses that limitation, potentially making autoregressive language models more valuable to design sequences outside the naturally occurring range of functionality.", "was_revised": true, "final_justification": "I am satisfied with the revisions (addition of a diffusion model baseline and clarification of the method). I have increased my score slightly and have also increased my confidence.", "rebuttal": "We appreciate the reviewer’s recognition of our method’s originality, comprehensive evaluation, and potential for designing highly cell-type-specific regulatory elements. We also thank the reviewer for the constructive feedback. Please find our reponse below regarding the added diffusion baseline experiemnts and clarification on the TFBS frequency reward. \n\n### **[Q1]**\n\nThank you for this valuable suggestion. To address your comment, we performed an additional benchmark comparison with DRAKES[1], a recently proposed diffusion based method optimized using RL that has demonstrated success in human enhancer generation for the HepG2 cell line. Specifically, we used their publicly available checkpoint fine tuned on HepG2 and followed their experimental setup to further fine tune the model separately for K562 and SK-N-SH cell lines. It is important to note that DRAKES optimizes only a single objective, which is maximizing the expression in one target cell type. As shown in the newly added benchmark results, regCon consistently outperforms DRAKES by achieving higher expression levels in the target cell lines (0.77 versus 0.68 for HepG2, 0.93 versus 0.83 for K562, and 0.86 versus 0.79 for SK-N-SH) and significantly lower off target expressions. These results highlight the advantage of regCon in handling multiple constraints compared to diffusion based methods optimized for single objectives.\n\n| **Method**           | **HepG2 ↑**     | **K562 ↓**     | **SK-N-SH ↓** | **ΔR**        |\n|----------------------|-----------------|----------------|----------------|----------------|\n| regCon               | 0.77 (0.01)     | 0.36 (0.04)    | 0.31 (0.04)    | 0.44 (0.03)    |\n| DRAKES               | 0.68 (0.004)     | 0.78 (0.007)    | 0.71 (0.007)    | -0.07 (0.005)|\n\n| **Method**           | **K562 ↑**     | **HepG2↓**     | **SK-N-SH ↓** | **ΔR**        |\n|----------------------|-----------------|----------------|----------------|----------------|\n| regCon               | 0.93 (0.004)     |  0.43 (0.01)   | 0.35 (0.007)    |    0.54 (0.01) |\n| DRAKES               | 0.83 (0.003)     | 0.69 (0.002)    | 0.80 (0.001)    | 0.08 (0.002)|\n\n| **Method**           | **SK-N-SH ↑**     | **K562 ↓**     | **HepG2↓** | **ΔR**        |\n|----------------------|-----------------|----------------|----------------|----------------|\n| regCon               | 0.86 (0.02)     | 0.44 (0.01)    | 0.54 (0.13)    |    0.37 (0.11)|\n| DRAKES               | 0.79 (0.001)     | 0.80 (0.003)    | 0.69 (0.002)    | 0.05 (0.002)|\n\n### **[Q2]**\n\n\nThank you for raising this important question. We acknowledge the potential limitation arising from the use of motif frequency correlation as both a regularization term and an evaluation metric. To address concerns of circularity, we conducted an additional analysis using distinct reference sets for evaluation, created by splitting sequences at the 90th and 50th percentiles based on their on and off target activities. Although this filtering significantly reduced the dataset size (leaving only 18 sequences for JURKAT, 4 for K562, and 7 for THP1 out of the original 12,335), regCon continued to demonstrate strong performance, substantially outperforming baseline methods across all tested cell lines with correlations of 0.37 for JURKAT, 0.52 for K562, and 0.60 for THP1. \n\n| Cell Type | *regCon*       | AdaLead     | BO          | CMAES       | PEX         | PPO-Lag        | PPO         | TACO        |\n|-----------|------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| JURKAT    | ***0.37 (0.02)***| 0.23 (0.10) | 0.10 (0.17) | 0.19 (0.06) | 0.34 (0.02) | 0.27 (0.15) | 0.32 (0.12) | ***0.39 (0.09)*** |\n| K562      | ***0.52 (0.11)***| ***0.54 (0.23)*** | 0.13 (0.25) | 0.14 (0.05) | 0.31 (0.02) | 0.18 (0.18) | 0.31 (0.13) | 0.21 (0.14) |\n| THP1      | ***0.60 (0.02)***| 0.16 (0.13) | 0.06 (0.08) | 0.06 (0.04) | 0.04 (0.02) | -0.02(0.07) | 0.37 （0.04）| 0.33（0.01）|\n\nWe also want to emphasize that using motif frequency correlation as a reward promotes generalizability, as shown in Figure 3a of our manuscript. Specifically, the TFBS reward is computed by first deriving a motif frequency vector from a reference set of real DNA sequences, then calculating the Pearson correlation between this reference vector and the motif frequencies in generated sequences. This encourages the model to capture global frequency patterns rather than explicitly match individual motifs, in contrast to prior methods like TACO. Despite this global formulation, regCon still successfully generates cell type–specific motifs such as HNF4A for HepG2 and GATA1 for K562, demonstrating that it acquires this ability purely through optimization on a coarse-grained reward. This highlights the robustness and generalizability of regCon’s optimization strategy.\n\n\n### **[Q3]**\nThank you for the insightful comment. In our experiments, we selected the sequences with activity above the 50th percentile in the target cell type, and off-target sequences as those with activity below the 50th percentile in non-target cell types. We then computed each motif’s occurrence frequency from these selected sequences. The motifs were selected following the protocol in [2].\n\nThis percentile-based thresholding, as adopted from prior work [2,3], helps mitigate differences in expression scaling across cell types and avoids reliance on absolute activity values, which may vary due to experimental noise or normalization schemes.\n\nWhile we agree that using a more stringent threshold, such as the 90th percentile for on-target activity, could better capture highly specific sequences, doing so significantly reduces the number of selected sequences. For instance, in the human promoter dataset, using a 90th/50th percentile split for on/off-target activity results in only 18 sequences for JURKAT, 4 for K562, and 7 for THP1, out of a total of 12,335. In the human enhancer dataset, no sequences satisfy such criteria across all three cell types among around 660,000 candidates. These sample sizes are insufficient to support reliable transcription factor binding site (TFBS) motif enrichment analysis. Therefore, we opted to use a relaxed threshold to ensure sufficient statistical power and robustness in estimating TFBS frequencies.\n\nImportantly, to mitigate potential overfitting to spurious motif correlations, we use a learnable weight for each TFBS regularization term and enforce an upper bound on these weights, as described in Line 193.\n\nThat said, we agree with the reviewer that using a 90th percentile cutoff would be preferable in scenarios where sufficient data is available, and we consider this a promising direction for future work.\n\nThank you again for your valuable suggestions, which have helped improve the clarity and contribution of our work. Please let us know if you have any more questions.\n**Reference:**\n\n[1] Wang, C., Uehara, M., He, Y., Wang, A., Biancalani, T., Lal, A., ... & Regev, A. (2024). Fine-tuning discrete diffusion models via reward optimization with applications to dna and protein design. arXiv preprint arXiv:2410.13643.\n\n[2] Lal, A., Garfield, D., Biancalani, T., & Eraslan, G. (2024). Designing realistic regulatory DNA with autoregressive language models. Genome Research, 34(9), 1411-1420.\n\n[3] Yang, Z., Su, B., Cao, C., & Wen, J. R. Regulatory DNA Sequence Design with Reinforcement Learning. In The Thirteenth International Conference on Learning Representations."}]

Author rebuttal:

Meta-review: This paper presents regCon, which models the cell-type-specific regulatory DNA design problem as a constrained optimization problem and utilizes the GRPO algorithm as the RL framework for sequence optimization. Through evaluations on human promoter and enhancer benchmarks, regCon consistently outperforms existing baselines.

Initially, reviewers maintained a consistently positive attitude toward this work, but several major concerns remained: 1) insufficient details in experimental descriptions, e.g., dataset partitioning and baseline selection; 2) potential reward hacking issues; 3) how to separately evaluate the benefits brought by the constrained optimization problem formulation and batch-level GRPO, respectively; 4) missing diffusion model baselines. After the rebuttal, reviewers acknowledged that these issues have been addressed. The AC thus believes this paper is now ready for acceptance. The camera-ready version should include the discussions and results provided in the rebuttal.
Decision: Accept (spotlight)