Designing Cell-Type-Specific Promoter Sequences Using Conservative Model-Based Optimization
Authors: Aniketh Janardhan Reddy
Xinyang Geng
Michael H Herschl
Sathvik Kolli
Aviral Kumar
Patrick D Hsu
Sergey Levine
Nilah M Ioannidis
Venue: NeurIPS 2024 poster (2024)
Keywords: ML applications
computational genomics
computational biology
model-based optimization
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We propose a workflow to design cell-type-specific promoters while accounting for various practical considerations and demonstrate its efficacy in a difficult setting.

Abstract: Gene therapies have the potential to treat disease by delivering therapeutic genetic cargo to disease-associated cells. One limitation to their widespread use is the lack of short regulatory sequences, or promoters, that differentially induce the expression of delivered genetic cargo in target cells, minimizing side effects in other cell types. Such cell-type-specific promoters are difficult to discover using existing methods, requiring either manual curation or access to large datasets of promoter-driven expression from both targeted and untargeted cells. Model-based optimization (MBO) has emerged as an effective method to design biological sequences in an automated manner, and has recently been used in promoter design methods. However, these methods have only been tested using large training datasets that are expensive to collect, and focus on designing promoters for markedly different cell types, overlooking the complexities associated with designing promoters for closely related cell types that share similar regulatory features. Therefore, we introduce a comprehensive framework for utilizing MBO to design promoters in a data-efficient manner, with an emphasis on discovering promoters for similar cell types. We use conservative objective models (COMs) for MBO and highlight practical considerations such as best practices for improving sequence diversity, getting estimates of model uncertainty, and choosing the optimal set of sequences for experimental validation. Using three leukemia cell lines (Jurkat, K562, and THP1), we show that our approach discovers many novel cell-type-specific promoters after experimentally validating the designed sequences. For K562 cells, in particular, we discover a promoter that has 75.85\% higher cell-type-specificity than the best promoter from the initial dataset used to train our models. Our code and data will be available at https://github.com/young-geng/promoter_design.

Designing Cell-Type-Specific Promoter Sequences Using Conservative Model-Based Optimization
Aniketh Janardhan Reddy" Xinyang Geng" Michael H. Herschl*.t, Sathvik Kolli, Aviral Kumar, Patrick D. Hsu Sergey Levinet, Nilah M. Ioannidis University of California, Berkeley
Abstract
Gene therapies have the potential to treat disease by delivering therapeutic genetic cargo to disease-associated cells. One limitation to their widespread use is the lack of short regulatory sequences, or promoters, that differentially induce the expression of delivered genetic cargo in target cells, minimizing side effects in other cell types. Such sell-type-specific promoters are difficult to discover using existing methods, requiring either manual curation access large datasets of promoter-driven expression from both targeted and untargeted cells. Model-based optimization (MBO) has emerged as an effective method design biological sequences in an automated manner, and has recently been used in promoter design methods. However, these methods have only been tested using large training datasets that are expensive to collect, and focus on designing promoters for markedly different cell types, overlooking the complexities associated with designing promoters for closely related cell types that share similar regulatory features. Therefore, we introduce a comprehensive framework for utilizing MBO to design promoters in a data- efficient manner, with an emphasis discovering promoters for similar cell types. We use conservative objective models (COMs) for MBO and highlight practical considerations such best practices for improving sequence diversity, getting estimates of model uncertainty, and choosing the optimal set of sequences for experimental validation. Using three leukemia cell lines (Jurkat, K562, and THP1), we show that our approach discovers many novel sell-type-specifie promoters after experimentally validating the designed sequences. For K562 cells, in particular, we discover a promoter that has 75.85% higher cell-type-pecciffiity than the best promoter from the initial dataset used train models. Our code and data will be available at https://gitub..oom/youm www.:/unngggggggggooteeeeree
Introduction
Gene therapies treat diseases through the delivery and expression of therapeutic genetic cargo in cells and tissues. Expression is controlled by a promoter sequence, a short regulatory DNA sequence (typically up to few hundred base pairs (bp) long) placed upstream the coding region. An ideal promoter for gene therapy would differentially induce expression in targeted cells while repressing expression in all other cells to increase effectiveness and reduce side-effects; i.e. it should be wetttyee-spccfffe and induce differential expression (DE). However, although there are over 400 types of cells in the human body [I, very few sell-type-specine promoters are known.
Traditional methods to design cell-type-speche tiling known rigullaglyy elements (CREs) or transcription factor (TF) binding motifs ((2 among others). These methods are difficult to automate and are not guaranteed to work, especially in less
promoters rely heavily on manual curation or involve
Equal contributions
Corresponding authors: aniketh @ berkeley.edu young.geng @ berkeley.edu, michael herselleeehhhhhh beechl@eeeeerreeeeeu nilah@ berkeley.edu
pdhsu berkeley.edu, svlevine @ cees. wwr.sebbrreee..c..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

studied cell types. Directed evolution can be used to automate promoter design but is expensive to perform, typically requiring many experimental validation steps to continually improve designs, and does not make the best use of available data. Offline model-based optimization (MBO) has recently been used to design promoters and other CREs in a data-driven manner (Linder et al. 77]. Wang et al. Jores et al. (9. LaFleur et al. [00], Gosai et al. [1] among others). The offline MBO approach involves building machine learning (ML) models of promoter-driven expression (PE) using experimental measurements and then optimizing the promoter sequence using model predictions as surrogates for experimental measurements. Offline MBO has the potential to accelerate promoter discovery by being automated and relatively data-efficient; however, the field is lacking generalizable MBO-based framework for designing cell-type-speciffe promoters in a data-efficient manner, while systematically accounting for practical considerations including:
1. Data-efficiency: Measuring PE is expensive and time-consuming, necessitating the effective use of available data during design. Most previous work on offline MBO for promoter design has used large PE datasets for model training (e.g. from massively parallel reporter assays, MPRAs) that are only available for few well-studied cell types, and their modelling strategies have not been evaluated when designing cell-type-specif promoters for less-studied cell types using small PE datasets.
2. Minimizing adversarial designs: Using models for optimization, rather than just prediction, presents key additional challenges, since naively optimizing designs based on predicted DE can lead to adversarial designs that fool a model into outputting desirable values while having undesirable values in reality. Previous promoter design workflows using optimization techniques such as in silico directed evolution, gradient ascent, or generative models do not account for such designs.
3. Selecting design candidates while accounting for sequence diversity and uncertainty in estimates of sequence's goodness: Offline MBO can design many sequences that are predicted to be cell-type-specific by the PE model. However, due to limited budgets and time for wetlab experiments, typically only a subset of the candidates can be validated. Thus, we need a systematic approach to choose promising candidates from larger set of designs in a principled manner. During selection, it is important to consider both the diversity of the chosen candidates and the uncertainty in model predictions. Choosing diverse candidate designs is important (1) to improve the chances of successfully discovering a sell-iype-specific promoter upon experimental validation, since we lack complete information about how a sequence controls PE, and (2) to explore sequence space more effectively during several rounds of design, leading to progressively better models and designs. is also important to choose candidates that have low uncertainty in model predictions while also balancing their potential goodness. This sequence selection process is crucial for a principled MBO workflow and has been overlooked by previous work.
Here, we present a systematic framework and case study for applying offline MBO to the real world problem of promoter design, backed by empirical evidence from wetlab experiments evaluating DE in three closely related blood-based cell types. To improve data-efficiency through transfer learning, we follow previous work [12 and pretrain sequence-based deep learning (DL.) models of PE using large existing MPRA datasets that measure PE in different contexts, and fine-tune them using small PE dataset collected from the target cell types. To address adversarial designs, extend the conservative objective models (COMs) framework [13] to our setting: while fine-tuning models, we use a loss term that reduces the predicted DE of adversarial designs. These fine-tuned models are then used to design cell-type-ppecific promoters by performing gradient ascent on the inputs. Upon generating multiple candidate sequences, we propose a selection scheme to choose diverse yet effective subset of final candidates that uses uncertainty estimates from an independently trained ensemble. We use our workflow to design cell-type-specific promoters for three leukemia cell lines (Jurkat, K562, and THP1) and present experimental evidence of its effectiveness. Existing studies have designed promoters for markedly different cell types derived from different tissues and germ layers [1]. a simpler problem since such cell types typically have very different regulatory environments with different transcription factors (TFs) expressed. We show that our MBO approach works in more difficult setting where we aim to design promoters that are only expressed in one of three hematopoietic leukemia cell lines. Our designs improve upon the cell-type-speciffe sequences from the fine-tuning dataset, improving 72.12% of sequences in Jurkat cells and 80.48 % in K562 cells. Additionally, in K562 cells, we identify a promoter with 75.85 % higher DE than the best promoter from the fine-tuning dataset. These results suggest that applying our approach over multiple rounds of modelling, design, and experimental validation will enable the discovery of promoters with even higher cell-type-specifiiity. by shifting the training distribution towards higher DE sequences at each iteration.
of
2

---

2 Related work
Promoter design methods: Promoters have traditionally been designed using heuristics and manual curation. For example, Selvakumaran et al. [3] and Yun et al. 4 found cell-type-specific promoters for ovarian and breast cancer cells, respectively, by identifying genes that are differentially expressed in these cells and showing that their promoters exhibit celltyype-seciffcitty Nissim et al. [S] designed call-type-seccfffe promoters for ovarian and breast cancer cells by identifying differentially expressed TFss and tiling their binding motifs. However, these studies do not provide a generalizable method to design sell-type-specifie promoters for any given cell type. In an attempt to address this issue, recent work has sought to develop more automated and data-driven design methods. Kotopka and convolutional neural network (CNN) models alongside simple optimization techniques such in silico directed evolution and gradient ascent to design promoters for high expression in yeast and various plant cells, respectively. Wang et al. [ generated E. coli promoters by training generative adversarial network (GAN) natural E, coli promoters and then using a PE predictor to further filter generated sequences. Gosai et al. [I] trained a sequence- based CNN model of PE using large MPRA dataset from three very different cell lines. They used this model with three design algorithms to produce cell-tye-sspecife promoters that they show be effective and diverse. Here, we build on previous work in three crucial ways. First, we exploit the benefits of pretraining on large existing PE datasets to build accurate models for target cell types in more data efficient manner compared to previous methods. Second, leverage COMs [3]. a powerful offline MBO method, to design promoters using these models more effectively, minimizing adversarial designs. Finally, we propose a sequence selection strategy that explicitly accounts for sequence diversity and model uncertainty to choose a subset of sequences for experimental validation from larger set of design candidates.
Smolke [4] and Jores et al. [9] used sequence-
Offline MBO for designing biological sequences: Apart from COMs, other offline MBO methods have been proposed for designing biological sequences. While COMs only requires discriminative models, most of the other methods rely on generative models. For example, Brookes et al. [15] built variational autoencoders (VAEs) that generate desirable designs by iteratively improving the VAE and its designs, guided by a design model. Jain et al. [16] used GFlowNets for sequence design, another class of generative models that are trained to output designs that span all modes of some design model's ' prediction distribution. Linder et al. [7'ss deep exploration networks (DENs) are generative models that are trained to output diverse yet desirable sequences and again use design model predictions to guide the training process. They experimentally validated their method and showed that alternative polyadenylation (APA) sites designed using DENs were better than those designed using gradient ascent. Here we use DENs a baseline in our analyses, since it was validated using wetlab experiments.
3 Background and motivation
The cell-type-specific promoter design problem: A typical promoter design experiment ([17 14|
[0] among others) starts by identifying target cell types and defining objective such as increasing the absolute or differential expression levels of some gene. Then, an initial set of candidate promoters is prepared by leveraging existing data sources and heuristics, even using random sequences. To test the efficacy of these promoters, reporter assays that collect PE measurements for large batch of promoters simultaneously are used (batch sizes range from a few hundred to many thousands or even millions). Using these initial PE measurements, better promoters can be designed using various methods such as directed evolution, shuffling motifs enriched in desirable sequences, or offline MBO. The designed promoters then need to be experimentally validated using another reporter assay. If a sufficiently desirable promoter is not discovered, the design and validation steps can be repeated.
Here, we focus on designing short cell-type-specific promoters in a data-constrained setting. In this setting, we have a set of target cell types C, from which we have a small PE dataset a few thousand sequences and corresponding PE measurements from all cell types in C. Current pooled oligonucleotide synthesis methods impose length constraints on DNA sequences. This technical limitation restricts high-throughput synthesis and measurement capabilities to promoters of at most a few hundred base pairs (bp). For any target cell type te E C, we define the DE of a promoter x (i.e. its celltyye--peciffcitt) as:
DE (IC) 1)
Σ [PE PE (x)),
(1)
1) ocgito
3

---

where (E C, is the experimentally measured expression value induced by x in cell type c. Our goal is design sequences that maximize DE the average difference in PE between the target cell type te and the other cell types.
Offline MBO: An offline MBO algorithm aims to produce designs that maximize some objective function, using a provided static dataset D {(x; mi)) of designs x; and a corresponding mea- surement of the objective value y/- The algorithm analyzes this dataset and produces an optimized candidate design x", which is evaluated against the true objective function. This process often involves learning a proxy objective fo mapping designs to corresponding objective values, which we hereafter refer to the design model. Optimization is then performed to find an input which maximizes this learned design model: x = arg maxx fx(x), using methods such as in silico directed evolution, gradient ascent, by building generative models.
Here, we use offline MBO algorithms to design promoters that maximize the objective function mentioned in Eqn [ When several rounds of promoter design are performed, can use all available PE measurements from previous rounds as the "static" dataset for offline MBO. In our workflow, we propose additional selection step after running offline MBO that diversifies the final designs if needed, making the pipeline more suitable for multi-round optimization.
Our MBO workflow for designing promoters in data-constrained settings
This section presents our approach to design cell-type-specific promoters that accounts for key practical considerations. It is summarized in Figure 1 with a detailed flowchart in Figure S.2
4.1 Building models of promoter-driven expression in data-efficient manner
As previously mentioned, offline MBO algorithms need a design model that approximates the objective function. When designing cell-type-specific promoters for some target cell type te E C, our objective function is DE (x) that is in turn computed using all PE* (x), c C, values. Therefore, instead of directly modelling DE (x), we simultaneously model all PE PE*(x) values using multi-task learning (MTL). This allows us use one design model to design cell-type-ppecifi promoters for all E C. In this section, we provide guidelines for building these models based on the findings of Reddy et al. [1]. who studied model architectures and transfer learning strategies for modelling PE. Model architecture:
Recent work has shown that genomic data can be accurately modelled using sequence- eoquence-based models models that take in one-hot encoded DNA sequences as input to predict associated experimental assay measurements such as gene expression, histone modifications, and TF-binding (e.g. (0). ) Reddy et [12] find that a model consisting of convolutional layers followed by transformer layers (MTLucifer) outperforms other architectures when used to model PE without any transfer learning.
Transfer learning to boost accuracy: Effective offline MBO hinges accurate design models, and building accurate models often requires large datasets. However, collecting large datasets with experimental PE measurements in multiple cell types is expensive and time-consuming In such data-constrained settings, transfer learning can be very beneficial. Reddy et al. [2] showed that pretraining sequence-based models large related genomic datasets before fine-tuning smaller
Fine-tuning using conservative regular for
Dasigrad aequeeose with high predicted diferrenial expression
Finel selection batoncring optimaity and divercity
Pretraiiing
Pretrained model
Design model
Optimizer
For nest runn disign
Uniraiied
Experimental measurement of promolen-drienn ecpression
designad promolers
Prelsiiing data
modill
Experimental prometer-drivn expression values from chosen cel types
Figure 1: Our workflow for designing cell-type-specific promoters. Five main steps are highlighted in grey: (1) pretrain base model using existing large genomic datasets; (2) fine-tune the pretrained model using the experimentally measured PE data collected far, while also using conservative regularizer; (3) use gradient ascent- based optimizer to design sequences that have high predicted DE; (4) apply a final sequence selection algorithm that balances optimality and diversity to choose smaller subset of the designed sequences; (5) experimentally measure the PE of the selected designed sequences. The last four steps can be repeated when running multiple rounds of design iterations.

---

PE dataset from target cells leads to significantly better modelling of the smaller dataset compared to training exclusively on the smaller dataset. The three best upproaches that they identified are:
1. Linear probing of Enformer [20] predictions using Lasso [21]: Enformer is large sequence- based model (again, consisting of convolutional and transformer layers) trained on thousands of epigenomic datasets from humans and mice, allowing it to accurately learn sequence features that control gene expression. Since it expensive to train Enformer (3 days on 64 TPU v3 cores), we often need to rely on the pretrained models published by Avsec et al. [20]. limiting flexibility in terms of changing architectures or using alternate deep learning (DL) librarics. It can also be difficult to use Lasso with gradient- based optimization techniques when performing offline MBO. Therfore, this approach is only recommended when the published pretrained Enformer model can be easily incorporated into your code base without any architectural changes, and when the offline MBO algorithm you are using is not reliant on gradients (e.g. CbAS [15).
2. Fine-tuning Enformer by using its embeddings to predict PE in target cells: The performance of this approach is very similar to the previous approach. Moreover, using fine-tuning instead of Lasso allows us to easily employ recommended when the pretrained Enformer model is compatible with your code base and you do not wish make any changes to its architecture that would necessitate re-training.
madient-bbased offline MBO methods. Thus, this approach is
3. Pretraining MTLucifer on large existing MPRA datasets before fine-tuning it to predict PE in target cells: Certain MPRA datasets (e.g. (22 23 measure PE in selected cell lines from large number of sequences. These data are useful for transfer learning, since models can learn a broad set of features that are generally useful for predicting PE. This approach is very inexpensive (33 hours for pretraining on a single Nvidia A40 GPU), which makes it easier to experiment with architectures and hyperparameters. Hence, this approach recommended when you want to use custom architectures for modelling your data, or if you want flexibility in the DL libraries used to build your code base.
Using any of these approaches should yield an accurate model of PE in our target cells. However, since we focus on performing offline MBO using COMs, which require differentiable design models, only the last two approaches are compatible with our workflow.
4.2 Performing offline MBO while mitigating adversarial designs using COMs
Once we have accurate design models, we can couple them with offline MBO algorithms to design promoters, However, during the design process, we need to address adversarial designs that might arise due to the distribution shift problem. We describe the problem below and how it can be mitigated using COMs [33. Then, we specify how COMs can be used to design cell-type-specific promoters.
Distribution shift problem in offline MBO and conservative regularization: While we can train an accurate design model fe(x), it may still suffer from generalization failures common to supervised regression models, especially when designed sequences are far away from the training data's ' distribution i.e. when there is distribution shift. Such adversarial sequences can be easily designed by optimizers by heavily mutating training sequences, especially those that are already good. To alleviate this issue, we suggest learning and using conservative models of the objective function, or COMs [13], as design models. These models are trained using a conservative regularizer that penalizes high predictions on the set of potentially undesirable promoters μ(x) that are not in the training set but appear promising under the current design model, thus preventing the optimizer from designing adversarial promoters. Concretely, conservative models use the following loss:
min 0
Ex-D [(fo(x) -) +a (Ex~ai [x)] Ex~D (x)))
supervised loss
Im conservative regularizer
where a is the conservatism coefficient and controls the degree of "conservatism".
Training design models for designing cell-type-speeifie promoters using conservative regulariza- tion: The previous subsection mentions two approaches for building differentiable design models. In either approach, we have a pretrained model (either a pretrained Enformer or MTLucifer model) that is fine-tuned using a small dataset from the target cells. To incorporate conservative regularization into the fine-tuning process, we modify the loss. Let us denote the model prediction for PE cell
I As an alternative to fine-tuning a pretrained network, which involves updating all weights in the network, one can extract embeddings or outputs from the pretrained network and fit simple linear model to make predictions for downstream task without updating the full pretrained network. This is called linear probing.
5

---

type as PE and the predicted DE in c as (x), then the overall fine- fine-tuning objective is:
min Σ Ci [PPE - (E + a (Ex-g, [DE[()) Ex (x)))
(3)
CEC
In contrast to the canonical formulation of COMs in Equ 2 we use DE in the regularization term, a value derived from the model's expression predictions, instead of using expression itself, as our goal is to maximize DE and we want avoid adversarial sequences with erroneously high predicted DE using the conservative regularizer. Furthermore, since we use a single design model for all target cells, the regularization term needs to be computed for all target cells. In contrast, the canonical COMs formulation works for only one objective function. Similar to Trabucco et al. [3], to get distribution (a(x) consisting of sequences with potentially overestimated DE in in every training step, we can use the Adam optimizer [24] to perform T steps of gradient ascent on DEG (x) starting
from sequences in the training batch (in our experiments: learning rate 0.5, 8 = 0.9, B32 = 0.999, and T = 100). Since the DNA sequences are discrete, we perform gradient ascent in the probability simplex in the one-hot encoded space, parameterized by a softmax function. At the end Tf steps of gradient ascent, we perform a hard clipping that the resulting sequence is valid one-hot encoding. The full fine-tuning objective is optimized until convergence and the model checkpoint with the lowest validation set loss is used the design model. Importantly, it is difficult choose an priori. Therefore, we train multiple design models with different a values and use all of them to design candidate promoters (in our experiments, E {0, 0.0003, 0.001, 0.003, 0.01, 0.03)).
Designing promoters using gradient ascent: After training a design model, starting from each promoter in the fine-tuning dataset (henceforth referred as the starting sequence), we apply the same gradient ascent-based optimization process as that used to build px(()) to generate a design. This generation is performed using every design model (trained using different ar values). yielding large pool of candidate designed sequences.
4.3 Balancing diversity, optimality, and uncertainty for final sequence selection
From the design process described in the previous subsection, we get a large pool of candidate cell- type-specific promoters. Unlike typical offline MBO problems where an algorithm must produce only single design for evaluation, as discussed in Section 3 designed promoters typically evaluated in batches (say, of size K). Moreover, multiple iterations of promoter design can be performed. In this section we describe an algorithm to choose K promoters for experimental validation from the large set of candidates for cell type c, accounting for these facets of the promoter design problem.
Given that we have multiple design models, we cannot simply take the top K designs with highest predicted DE, since the predictions come from different models, making them difficult to compare with each other. Taking the top designs might also yield sequences that are very similar to each other. Finally, we also want to account for model uncertainty during selection and ideally choose high performing sequences that have low uncertainty.
Using an ensemble to get pessimistic estimates of DE that account for model uncertainty: To uniformly evaluate all candidate designed sequences, and potentially get more accurate predictions. we build an ensemble model consisting of models with slightly different architectures, using ciiiin-alidation-tee -tst split of the fine-tuning data that different from the one used to build the design models. The constituent models are still pretrained prior to fine-tuning, but the conservative regularizer is not used during fine-tuning since these models are not directly used for design. To reduce the computational overhead for building the ensemble, one can use the same pretrained backbone in all constituent models but use different kinds of fine-tuning layers in each model. In our experiments, we use 36 models in the ensemble (Appendix E. To account for model uncertainty,
we use the ensemble to compute DE, (x), a pessimistic estimate of the DE in cell type c, for all candidates. The pessimistic estimate is defined as the lower confidence bound of the constituent models' predictions and is computed as the mean minus the standard deviation of the predictions. Using this pessimistic estimate in the final selection process allows us to choose sequences that most models are confulent about, further reducing the risk of choosing adversarial designs.
Final selection algorithm: Our ultimate goal is to maximize the expected efficacy of the designs in the set of final sequences selected for experimental validation. Intuitively, this can be achieved by: (i) ensuring that our designs have high predicted DE, and (ii) ensuring that the designs are as diverse as possible. Diversity is important as designs predicted to be optimal might not actually be optimal when they experimentally validated. Having diverse set of sequences increases the likelihood

---

of some design being optimal, since we cover broad region of the sequence space. We define a sequence set S's diversity as:
D(S) = XES,
Σ
H(xxx') + x(xxxx)
where H. is the normalized Hamming distance between two sequences (ensuring overall diversity) and IC is the normalized Euclidean distance between the 6-mer frequency vectors of two sequences (ensuring diversity in potential TF-binding motif's composition). The distances are normalized by dividing them by their maximum possible values. To instantiate this intuition into a concrete strategy, } such that S is the optimal solution we aim to select set of K sequences S X2, .x
to the following optimization problem:
S == arg max
Σ DE, (x)+ XES
where (x) denotes a pessimistic estimate of the DE computed using ensemble, and B is the diversity coefficient that controls the fitness vs. diversity trade-off. While in theory we can obtain S by optimizing over all sequences, doing so is computationally intractable. So, the optimization in Eqn is performed using a greedy algorithm over a large subset of candidates from the previous step. The subset consists of candidates that have positive ensemble average predicted DE, and also have high predicted expression in the target cell type since this is also crucial for gene therapy applications. The algorithm sequentially chooses K sequences in each step, given set of sequences S' already
chosen to be in the final set, it computes DE, (x) + BD(S' U (x)) for every candidate sequence not in S", and chooses the sequence that maximizes this value for inclusion in the final set.
Making adjustments to the design process based on performance and diversity metrics We discuss common problems that may arise during design and provide heuristics to address them. Designed sequences are generally not predicted to have higher DE than the starting sequences from the fine-tuning dataset: This issue might arise when the conservative regularizer is too power- ful, making it difficult for the optimizer to design good sequences. Here, reducing the conservatism coefficient should help. However, there might also be problems with the dataset that hinder the optimizer. For example, if the fine-tuning dataset has very few sequences that drive some level of DE in the target cell type, the design models might be poor at modelling such sequences. To determine if this is the case, one can look at the distribution of DE values in the training set and if there are too few sequences that have positive DE in the target cell type (e.g. less than 25% of the dataset), the design problem might be too difficult to solve using the available data.
4.4
Designed sequences are not diverse: The diversity of set of designs can be measured using multiple metrics. We focus on base pair entropy and average pairwise edit distance (can be any edit distance e.g. Hamming distance) as they are easy to interpret. If we are designing sequences of length L. base pair entropy can be computed at every position in the sequence by determining the frequency of the 4 DNA bases across all designed sequences. For diverse sequence set, this metric should be close to 2 at every position (near uniform usage of the 4 bases, entropy computed using binary logarithms). The average pairwise edit distance should be comparable to (e.g. L/2). If these metrics are low, the diversity coefficient 33 can be increased to boost diversity. Using more diverse set of starting sequences during optimization with gradient ascent can also improve the diversity of the designed sequences since we can converge upon different optima of the objective landscape.
Implementation and experimental evaluation of our approach
Next, we use the workflow proposed in the previous section to design 250 bp long cell-type- -specccccc promoters for three leukemia cell lines: Jurkat, K562, and THP1. As mentioned in the introduction, designing such promoters is difficult since these cell lines are all mesoderm-derived hematopoietic cells compared to cell lines derived from different tissues germ layers. Moreover, Jurkat and THP1 cells are relatively understudied and lack large sources of PE measurements. We also perform experimental validation of the designed sequences to show the effectiveness of our approach.
2 Designs with ensemble average predicted PE in target cell type e greater than the 90th percentile of predicted PE in for the fine-tur tuning dataset's sequences.
7

---

5.1 Setup
Model training (design and ensemble models): To train design models, we use an architecture similar MTLucifer [22] (shown in Figure (S.T). We pretrain the models large existing MPRA datasets before fine-tuning them using a small PE dataset consisting of 17, 17,104 measurements collected by Reddy et al. [2] from Jurkat, K562, and THP1 cells (dataset is described in more detail in Appendix We use this approach over fine-tuning Enformer, since our codebase uses Jax [25] for efficiently performing conservative regularization, and pretrained Enformer models are not available for Jax. Following Section 4 we train 5 design models, each with a different a value. We also build an ensemble for final sequence selection using a different split of the same dataset. Appendix [E] describes the training process in more detail.
Sequence design: We design 4,000 sequences for each cell type using our workflow. This number was chosen based on experimental constraints, and to keep the total number of sequences close to the size of the original training dataset. Each design model is used to design 117 candidate sequences (one design for every starting sequence from the fine-tuning dataset) using the process described in Section 4.2 and the final 4,000 sequences are chosen from these candidates using the algorithm from
Section 4.3 As shown later in this section, the designed sequences are naturally diverse, and we set the diversity coefficient/ 3 to zero since we did not need to improve diversity.
Baselines: We also design sequences using two baselines, in order to determine the usefulness of various components of our workflow:
1. Motif tiling:
This heuristic method to design cell type-specific promoters is similar in spirit to that proposed by Nissim et al. [5], and aims to improve upon known high DE sequences. We use it to determine the overall usefulness of our approach for designing promoters compared to traditional methods. Briefly, the motif tiling approach inserts motifs associated with high DE into sequences from the fine-tuning dataset that already have high DE, with the goal of increasing their DE even further. We run it for the three target cell types to design 520, 630, and 515 sequences for Jurkat, K562, and THP1 cells, respectively. Appendix [.T presents the details of this method.
2. DENs: In Section 4.2 we use gradient ascent to optimize starting sequences and produce designs that increase the design model-predicted DE. However, as detailed in the Section 2 many previous studies use generative models for this purpose. To determine which optimization algorithm is better, we replace gradient ascent with DENS in our workflow and produce 2000 sequences per cell type (with 3 = 10). We specifically use DENs, as their designs were experimentally validated by Linder et al. 7 and they are explicitly trained to produce diverse sequences. Appendix [2 describes DENs in more detail.
Experimental validation of sequences: We use similar protocol as Reddy al. tally determine the DE of designed sequences (Appendix D) Along with the designed sequences, we also measure the DE of the top sequences from the fine-tuning dataset (top 100 sequences for each cell type, so 300 sequences total), and of ~00 sequences from the fine-tuning dataset whose expression roughly uniformly spanned the full range of PE values. These sequences with measurements in both the fine-tuning set experiment and the validation experiment are used to determine whether designed sequences improve upon their starting sequences, using a procedure described later this section. In total, we measure the DE of 20,741 sequences. After filtering out sequences with less than 100 reads in any experiment, we are left with 14,315 high-quality measurements for use in all downstream analyses.
experimen-
5.2 Main Results Evaluating the diversity of designed sequences: We analyze the diversity of the designs in terms of their mean base pair entropy and the mean Hamming distance between any two designs (Table TT. Overall, our approach produces highly diverse sequences for all three cell types. We note that for THP1, DENs produce significantly less diverse designs. This is possibly due to the difficulty of the design problem- - the fine-tuning dataset has few sequences with high DE in THPI, leading to the design model having fewer modes. This might make it difffiuult for a generative model trained from scratch to discover the design model's 's modes. Since gradient ascent starts from the sequences in the fine-tuning dataset, it can more easily discover the various modes. Finally, to make sure that our designs are distinct from the fine-tuning sequences, we also analyze the mean Hamming distance between a design and the closest fine-tuning sequence (Table (ST1). We see that our designs are substantially different from the fine-tuning sequences, with designs being 125-140 bp away from the closest fine-tuning sequence on average. Thus, given a diverse fine-tuning dataset, our forkflow produces designs that are not only distinct from one another but are also different from the fine-tuning
8

---

Number of sequences
Mean base pair entropy
Mean Hamming distance
Method
Jurkat
K562
THP1
Jurkat
K562
THP1
Jurkat
KS62
THP1
Our approach
4000
4000
4000
1.87
1.96
1.96
176,18
183.73
184.17
DENs used instead of grad. Motif tiling
2000
2000
2000
1.89
1.87
1.00
178.17
177.39
108. 108.22
630
515
1.79
1.90
1.92
168.71
179.37
180.95
Table I: Quantifying the diversity of designs. Mean base pair entropy (using binary logarithms, maximum value 2) and mean pairwise Hamming distance (maximum value 250) shown for the designed sequences. Mean base pair entropy is calculated by determining the entropy of each position across all designs and then averaging these values.
sequences, even with the application of conservative regularization, which might otherwise inhibit diversity in certain scenarios.
Does our approach design sequences that are better than the corresponding starting sequences? Next, we analyze the overall effectiveness of our workflow and compare it to motif tiling. Since we aim to design promoters that have higher cell-yype-seecifccity than promoters in the fine-tuning dataset, we need to quantify the improvement in DE from the starting sequences to the designed sequences. Direct comparison of DE values is challenging due to differences in experimental conditions and potential confounding factors like batch effects, and re-measuring the DE of all starting sequences inefficient. To address this issue, range for any given cell type to calibrate DE values between experiments. Since these common sequences have DE values from both experiments, can compute percentile scores for both the starting and corresponding designed sequences among this common set. The design process is considered effective if the designed sequence achieves higher percentile score than the starting sequence. This calibration is valid since the ranks of the sequences in the common set are concordant between experiments, as evidenced by the high Spearman correlation between DE values measured by Reddy et al. [12] and our experiments: 0.953 for Jurkat, 0.921 for K562, and 0.950 for THP1.
common set of sequences that span the expression
Sequences designed using workflow
Jurkot (n=2550)
K562 (n-3709)
THP1 (n-3564)
100
1839/2550 (72.12%)
25
Designed sequence K362 DE Kile
100
2985 3709 (90. 49%)
60
Kale
1000
1333/3564 (37.40%)
25
20
20
Designed sequerce
15
40
Designed sequence
15
10
10
20
20
20
71. 2550 (27.88%)
(19.52%)
75
100
75
100
Starting jurkat
Starting K362
Starting sequence THP1 DE
Sequences designed using motif tiling
Jurket (n=417)
K562 (n=593)
THP1 (n=480)
14
100
(14.39%)
16
/9/ (9.27%)
Designed sequence TIP Xie
75
(15. 62.622%)
80
12
Dasignad sequence K562
80
10
80
100
Desigred sequence
60
60
60
80
60
40
40
40
40
20
20
20
20
357
538 (90.738)
405
(84.384)
73 Starting sequeecce KS62 DE %ile
25
50
75
100
Starting sequence Jurkat DE
Starting sequence THP1 DE
Figure 2: Comparing the DE of designed sequences with the DE of corresponding starting sequences. The top row of plots compare the percentile scores of starting and designed sequences from our approach, and the bottom row of plots compare the percentile scores for starting and designed sequences from motif tiling. Each column represents sequences designed for one of the three cell types. The y line is shown in red, with the number of sequences above it highlighted in the top left corner and the number below it highlighted in the bottom right corner

---

Jurkat
K562
THP1
M.W.W. 7.2490-75
M.W.W. p 8.701e:06
M.W.W. 1.148B-01
Junat DE
K502
THP
-1
Gradi ent ascent DENS
Gredient ascent Optimizer design
DENS
Gradient ascent Optimizer for design
DENS
Optimizer design
Figure 3: Box plots comparing the DE of sequences designed using gradient ascent as the optimizer in our workflow vs. those from DENs. p-values are computed using Mann-Whitney, Wilcoxon (MWW) test that tests whether sequences from gradient ascent have higher DE than sequences from DENs.
Figure Souree the percentile scores of the starting and designed sequences from our approach and from motif tiling. Sequences designed for Jurkat and K562 using our approach improve upon the majority of corresponding starting sequences, highlighting the overall effectiveness of our approach in this difficult setting. In fact, we design promoter for K562 that has 75.85% % higher DE (=5.17) than the best high DE sequence from the fine-tuning dataset (=2.94). On the other hand, we observe that using motif tiling to improve on high DE sequences is ineffective, almost always leading to worse sequences. We note that our workflow tends choose designs that come from relatively mediocre starting sequences. This is possibly because most sequences in the fine-tuning dataset have DE similar to these sequences, likely making the design and ensemble models accurate and confident near such sequences. Since our workflow significantly improves upon such sequences, we should be able to discover progressively better sequences over multiple rounds of model training, design, and experimental evaluation.
However, we fail to improve upon most starting sequences for THPI. This is likely because of the difficulty of the design problem. As mentioned previously, there are few high DE sequences in the fine-tuning dataset for THPI. Thus, it might be difficult for design models to be accurate the space of high DE sequences, leading poor designs. In such cases, collecting more experimental data for training might be the only effective way to improve designs.
Gradient ascent is better than DENs for designing sequences: Finally, compare gradient ascent to DENs (Figure 3 to determine the better optimizer for use our workflow. We observe that gradient ascent is generally
effective than DENs. Given that gradient ascent is also easier to implement and tune, we recommend it over generative models such as DENs.
6 Discussion and limitations
Designing sell-type-specine promoters is erucial step in the effective application of gene delivery technologies. We introduce a novel workflow for promoter design based on COMs [13], tailored for real-world applications due its data efficiency and ability to produce diverse, non-adversarial designs. We apply this approach to the challenging task of designing cell-type-specific promoters for three leukemia cell lines. We then experimentally validate designs, demonstrating effectiveness for two out of three cell lines.
We also present limitations that practitioners should consider. The success of our workflow depends on the accuracy of the design models, and effectiveness across the three cell lines was related to design model accuracy for each cell line (Table S.2). This resulted failure for THP1 cells, where we had lower quality fine-tuning line-tuning data with few high DE sequences. Pretraining data may also affect performance, and many existing MPRA datasets used for pretraining come from K562 cells, likely explaining our superior performance in K562 compared to Jurkat. Finally, conservative regularization may inhibit diversity in certain cases as using conservatism leads to designs that are close to the training data. Although we do not observe this in our experiments (Tables S.I and []. using fine-tuning dataset with limited sequence diversity might lead to very similar designs and addressing this issue is good avenue for future work. Our MBO workflow highlights these and other practical considerations. Our approach is valuable for practitioners targeting wide range of cell types, and will inspire further research in this area.
10

---

Acknowledgments
This work was partially supported by an H2H8 Graduate Research Grant to AJR, a Schmidt Futures grant to SL, and grants from the U.S. National Institutes of Health (R00HG009677) and the UC Noyce Initiative for Computational Transformation to NML. NMI is a Chan Zuckerberg Biohub San Francisco Investigator.
This research used the Savio computational cluster resource provided by the Berkeley Research Com- puting program at the University of California, Berkeley (supported by the UC Berkeley Chancellor, Vice Chancellor for Research, and Chief Information Officer).
References
[1] Tabula Sapiens Consortium, The tabula sapiens: multiple-organ, single-cell transcriptomic atlas of humans. Science, 37666594))cabb4486,, 2022.
[2] Carol H Miao, Kazuo Ohashi, Gijsbert A Patijn, Leonard Meuse, Xin Ye, Arthur R Thompson, and Mark A Kay. Inclusion of the hepatic locus control region, an intron, and untranslated region increases and stabilizes hepatic factor ix gene expression in vivo but not in vitro. Molecular Therapy, 166):222-5333 532, 2000.
[3] Muthu Selvakumaran, Rudi Bao, Anne PG Crijns. Denise C Connolly, Jillian K Weinstein, and Thomas C Hamilton. Ovarian epithelial cell lineage-specific gene expression using the promoter of a retrovirus-like element. Cancer research, 61(4):1291-129) 2001.
[4] Hye Jin Yun, Young-Hwa Cho, Youngsun Moon, Young Woo Park, Hye-Kyoung Yoon, Yeun- Ju Kim, Sung-Ha Cho, Young-IIl Lee, Bong-Su Kang, Wun-Jae Kim, et al. Transcriptional targeting of gene expression in breast cancer by the promoters of protein regulator of cytokinesis 1 and ribonuclease reductase 2. Experimental & Molecular Medicine, 40(3):345-3533 2008.
[5] Lior Nissim, Ming-Ru Ru Wu, Erez Pery, Adina Binder-Nis Nissim, Hiroshi I Suzuki, Doron Stupp, Claudia Wehrspaun, Yuval Tabach, Phillip A Sharp, and Timothy K Lu. Synthetic rna-based immunomodulatory gene circuits for cancer immunotherapy. Cell, (1(5)::138-11555) 2017.
[6] Ming-Ru Wu, Lior Nissim, Doron Stupp. Frez Pery, Adina Binder-Nissim, Karen Weisinger, Casper Enghuus, Sebastian R Palacios, Melissa Humphrey, Zhizhuo Zhang, al. A high- throughput screening and computation platform for identifying synthetic promoters with en- hanced cell-state specificity (specs). Nature communications, 10(1):1-10, 2019.
[7] Johannes Linder, Nicholas Bogard, Alexander B Rosenberg, and Georg Seelig. A generative neural network for maximizing fitness and diversity of synthetic dna and protein sequences. Cell systems, 11(1):49-62. 2020.
[8] Ye Wang, Haochen Wang. Lei Wei, Shuailin Li, Liyang Liu, and Xiaowo Wang. Synthetic promoter design in escherichia coli based on a deep generative network. Nucleic Acids Research, 48(12))6403-6412, 2020.
[9] Tobias Jores, Jackson Tonnies, Travis Wrightsman, Edward S Buckler, Josh T Cuperus, Stanley Fields, and Christine Queitsch. Synthetic promoter designs enabled by a comprehensive analysis of plant core promoters. Nature Plants, 7(6):42 -088,22555,,0 2021.
[10] Travis L LaFleur, Ayaan Hossain, and Howard M Salis. Automated model- design of synthetic promoters to control transcriptional profiles in bacteria. Nature communications, 13 (1):5159, 2022.
[11] Sager J Gosai, Rodrigo Castro, Natalia Fuentes, John C Butts, Kousuke Mouri, Michael Alasoadura, Susan Kales, Thanh Thanh I. Nguyen, Ramil R Noche, Arya S Rao, et al. Machine- guided design of cell-type-targeting cis- cis-regulatory elements. Nature, pages 1-10, 2024.
[12] Aniketh Janardhan Reddy, Michael H Herschl, Xinyang Geng, Sathvik Kolli, Amy X Lu, Aviral Kumar, Patrick Hsu, Sergey Levine, and Nilah M loannidis. Strategies for effectively modelling promoter-driven gene expression using transfer learning. bioRxiv, 2024. doi: 10. 1101/2023.0..24.52999
URL. htttps://wwat biorxiv. hit.arvvw..r//cattte///r//////22
19/2023. 02. 24. 529941

---

[13] Brandon Trabucco, Aviral Kumar. Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. International Conference on Machine Learning, pages 10358-10368. PMLR, 2021.
[14] Benjamin J Kotopka and Christina D Smolke. Model-driven generation of artificial yeast promoters. Nature communications, 1 11(1)22113, 2020.
[15] David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 773-782. PMLR, 09-15 Jun 2019. URL/https://procediin
mlr. pptpse//9/bboooosss html
[16]
Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Bonaventure P. Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequence design with GFlowNets. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9786-9801. PMLR, 17-23 Jul 2022. URL mlr press/ v162 jain22a....... html
[17] Michael R Schlabach, Jimmy K Hu, Mamie Li, and Stephen J Elledge. Synthetic design of strong promoters. Proceedings of the national academy of sciences, 107(6):2538-2543 2010.
[18] Jian Zhou and Olga G Troyanskaya. Predicting effects of noncoding variants with deep learning- based sequence model. Nature methods, 1110099911-344 2015.
[19] Vikram Agarwal and Jay Shendure. Predicting mına abundance directly from genomic sequence using deep convolutional neural networks. Cell reports, 31(7), 2020.
[20] Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska- Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10): 190--103, 2021.
[21] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology. 58(1):267-28) 1996.
[22] Jason Ernst, Alexandre Melnikov, Xiaolan Zhang. Li Wang, Peter Rogov, Tarjei S Mikkelsen, and Manolis Kellis. Genome-seale high-resolution mapping of activating and repressive nu- cleotides in regulatory regions. Nature biotechnology. 34(1 I): 180-1190, 2016.
[23] Joris Arenshergen, Ludo Pagie, Vincent D FitzPatrick, Marcel de Haas, Marijke P Baltissen, Federico Comoglio, Robin H van der Weide, Hans Teunissen, Urmo Vōsa, Lude Franke, et al. High-throughput identification of human snps affecting regulatory element activity. Nature genetics, (1))): 160-11669,1,,, 2019.
[24] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:///rxxiy orrg abs/ 1412.
[25] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Handerman-Milmn and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL
http://githhn htt.:tth..ooooeeee hwtp://ww...ooogooo////////////e google/jaaaaaaoooo
[26] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
arXiv:1606.084.11 2016.
[27] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199) 2014.
12

---

[28] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[29] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 568: 127063, 2024,
[30] Joris van Arensbergen, Vincent D FitzPatrick, Marcel de Haas, Ludo Pagie, Jasper Sluimer, Harmen J Bussemaker, and Bas van Steensel. Genome-wide mapping of autonomous promoter activity in human cells. Nature biotechnology, 35(2): (652)):455-553 2017.
[31] Rajiv Movva, Peyton Greenside, Georgi Marinov, Surag Nair, Avanti Shrikumar, and Anshul Kundaje. Deciphering regulatory dna sequences and noncoding genetic variants using neural network models of massively parallel reporter assays. PLoS One, 14(6):c0218073, 2019.
[32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Internatoonal Conference on Learning Representations, 2019. URL httpps://openreview http://rrrviieeee/oruuurl
Cd=Bkg6RiCqy7
[33] Charles Grant, Timothy L Bailey, and William Stafford Noble. Fimo: scanning for occurrences of given motif. Bioinformatics, 27(7):1017-1018, 2011.
[34] Jeff Vierstra, John Lazar, Richard Sandstrom, Jessica Halow, Kristen Lee, Daniel Bates, Morgan Diegel, Douglas Dunn, Fidencio Neri, Eric Haugen, et al. Global reference mapping of human transcription factor footprints. Nature, 183(7718)7729-73, 2020.
[35] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICC 2015: 18th International Conference, Munich, Germany, October 5.9, 2015, Proceedings, Part III 18, pages 234-24 241. Springer, 2015.
13

---

A Supplementary figures and tables
MPAA
SFarpe KENA
andedkings
512 unit per byer
Whe suring hesch
2uaee MLD with 542 urit
TeP
206 10 longth
512 lengt preserving
1C24 length
1054-
Fins embeddngg
embiddingg each poultin
kamnll
Figure S.I: Design model architecture: the convolutional layers use GELU activation [26], dropout with 0.1 probability [27]. and are followed by group norm layer (28) with each group size being 16. The MLP layers in the fine-tuning heads also use GELU activation. We apply the RoPE position embeddings [29] at each attention layer of the transformer.
Number of sequences
Average Hamming
distance to closest
fine-tuning sequence
Method
Jurkat
K562
THPI
Jurkat
K562
THP1
Our workflow
4000
4000
4000
126.46
132.17
140.27
Using DENs instead of grad. asc.
2000
2000
2000
149.63
155.44
152.89
Motif tillng
520
630
515
96.43
90.17
98.99
Table S.1: Average Hamming distance to the closest fine-tuning sequence for designs from every method Even though our workflow optimizes sequences from the
(maximum value 250, minimum value fine-tuning dataset to produce designs, the designs are generally quite distinct from the fine-tuning sequences. Since DENs are generative models that produce designs from random noise vectors, they are trained properly, we expect their designs to be different from the fine-tuning sequences. Finally, motif tiling inserts motifs into sequences from the fine-tuning dataset. Thus, expect its designs to be relatively more similar the fine-tur mee-tunng sequences than designs from other methods.
14

---

B Detailed flowchart describing the steps in our workflow
Chrosing approprien
Enformer model computiii with your
Use MTluudller mosel aftrr pratariing publisnne
Use pichureed
Experimento dass from sargel cell tyes
Enferner
consorvative eeudarizaioo
a (0.0.0053, (00) 0,003, 0.81,000)
Trkining consercation stjjettoo
Decrease the reg ilrtzatinn levels reduce uefncc the ocneevaatve regata might strong
Det of design mode is each tranned
opt
Troik ng mony diforrnt models with sigiity output usinng dota
we have sufficiont posilive DE in the targel cel?
Are we able to produce
sign
the starting sequeiices?
prodicted DE than
Ensemble model
More primerial data reight need be collected the design preblem to be feasãro
Large set of conditire soqueences
Ensartibe madil prediciiree casdickea sunparicaes
Fillr out sequ moss with negatice ensemble groed
ensomble
Compute normakkedd ack cotarce bataeon every par auquaricso
Compuse nurma iced Eudi doan distance Detaneen the Inequancy vadion every pair ascuAniiss
coothoord
aigartten
Findl designs
distance between designs sufiidenly high?
No: incraaze tb
lo ldrtion of Iine dosigns
Figure S.2: Flowchart that shows every step our workflow for a practitioner.
15

---

C Experimental data used for training design models
We use the PE measurements collected by Reddy et al. 12 for designing cell type-specific promoters. They provide 17,104 PE values in Jurkat, K562, and THPI cells derived from 250 bp long manually designed promoters. They set out to measure the PE of 20,000 promoters chosen using heuristics that try to maximize the number of differentially expressed promoters. Nearly 50% of the tested promoters are from differentially expressed endogenous genes, another ~ 40% are crafted by assembling known and de-novo motifs which are abundant the promoters of differentially expressed endogenous genes, and the remaining 10% of promoters originate from highly expressed endogenous genes' promoters. Each promoter is integrated upstream of a minimal CMV promoter and the enhanced green fluorescent protein (EGFP) reporter gene within lentiviral vector. The resulting expression in each cell line post-trane transduction is quantified by the levels of induced fluorescence. They get 17,104 PE values of adequate quality with two replicates and average the values from each replicate get the final PE values we use these values in our experiments.
This data was released with their codebase https//ggghabbooom/waaattt//nnnt AGPL-330 license.
under an
D Experimental protocol for measuring promoter-driven expression of designed sequences
We used the same protocol as Reddy et al. 122 with the following modifications for testing the newly designed sequences. First, the cycle number for PCR amplification of the Twist oligopool was reduced from 12 to 10 cycles to reduce amplification bias, and the lentiviral backbone used CMV promoter instead of an RSV promoter to drive the expression of the cargo from the lentiviral transfer plasmid to increase titer. To generate lentivirus, we used the LV-MAX kit (ThermoFisher A35684) following the protocol for IL flask. followed by the same concentration method. For lentiviral titration and full scale transduction, we used the same protocol, except we did not perform spinfection as this was deemed not necessary for adequate transduction efficiency. 24 hours after transduction, medium containing 8ug/ml. polyhrene and lentivirus was changed to fresh medium, and cells were al allowed to recover for another 48 hours before performing puromycin selection. This additional time allowed for better expression of the puromycin resistance gene before selection, increasing functional titer. Full scale transductions were performed at >500X library coverage and at an MOI <0.3. Cells were sorted using the same general methodology at a similar coverage; however, gDNA was extracted using the NucleoSpin Blood L (Machery-Nagel 740954) with the addition of RNAse A (NEB T3018L, 8L) following the manufacturer's instructions. No changes were made to NGS library preparation, but sequencing performed using Illumina NovaSeq X Series, pooled with other NGS libraries with non overlapping indices.
E Training of design and ensemble models and their prediction performance
In this section, we detail the process we use to train our design and ensemble models. We train 5 design models with varying values of the conservatism coefficient a (a {0, 0.0003,, 0.001, 0.003, 0.03). We also highlight their prediction performances.
E.1 Pretraining details
Here, we provide details about our pretraining process. All design models and all constituent models of the ensembles are pretrained before being fine-tuned. Following Reddy et al. [12], we pretrain our models using data from Sharpr-MPRA [22] and SuRE MPRA (30 23]. SuRE MPRA measures the expression induced by 150-500bp genomic fragments from 4 individuals from 4 different populations in the K562 and HepG2 cell lines. 2.4B and 1.2B fragments were found be expressed in K562 and HepG2 respectively. Most fragments have very low expression and training on all measurements time-consuming Thus, Reddy et al. [12] define a classification task using this data that subsets the data and bins each sequence into one of 5 expression bins. We also use this classification task for pretraining. Sharpr-MPRA is smaller MPRA that measures the expression from ~ 487K 145bp sequences centered at DNase I peaks in K562 and HepG2 cells and in two different settings. Reddy et al. 112 use preprocessed version of this data from Movva et al. (31] that builds a regression task with 12 outputs (2 replicates for expression measured in 2 settings in cell lines, and 4 outputs that
16

---

correspond to the average expression across replicates), We use the same formulation for pretraining our models.
During pretraining, our models' output layers produce the probability of sequence belonging to each of the expression bins for the SuRE MPRA-bas pretraining task and directly predicts the expression values for the Sharpr-MPR A- A-based pretraining task. We minimize the sum of the negative log-likelihood (NLL) loss for the SuRE MPRA task and the mean squared error (MSE) loss for the Sharpr-MPRA task (since both tasks have distinct sequences, a training sequence only contributes to one of the two loss terms, the other loss is set to zero for that sequence).
We use the dataset splits defined by Reddy et al. [12 and train with batch size of 448 using the AdamW optimizer [32) with lc-4 learning rate and 3e-3 weight decay. We train the models for 20000 steps in total, with cosine learning rate decay schedule that decays to O. We retain the best checkpoint that is selected by monitoring the validation loss throughput the training process.
E.2 Building an ensemble by combining models with slightly different architectures
The ensemble uses constituent models that slightly modify the design model architecture presented in Figure [S.]- each model uses a different type of MLP layer (shown in red in Figure [S.1] in the fine-tuning head. Within the of constituent models, we vary the depth (2, 4, or layers). number of hidden units (512, 1024, or 2048), and activation functions (tanh, GELU, ReLUU, or SiLU (26) of
the MI.P layers. This gives us constituent models for ensembling.
E.3 Fine-tuning details
After pretraining, all design models fine-tuned using certain split of the PE dataset collected by Reddy et al. [12 (Split in Table S.2). The ensemble's constituent models are fine-tuned using a different split (Splits 2 in Table S.2) In each split, following Reddy al. [12.. we use 70% of the assayed promoters for training, ~ T0% for validation, and ~ 20% for testing. Since there are distinct classes promoters the dataset with varying levels of GC content, our splits are stratified by both promoter class and GC content.
When fine-tuning the design models using the conservative regularizer, the full fine-tuning objective from Eqn 3 is optimized for up to 200 steps using the AdamW optimizer (32) (batch size = 512, learning rate 5e-5, weight decay 3e-3, 31 0.9, 32 0.999) with a cosine learning rate decay schedule that decays to 0 after 10 warm-up steps and model checkpoints with the lowest validation set loss are retained. Since the THP1 data is noisier compared to data from Jurkat and K562 cells (illustrated by Reddy et al. [12] reporting significantly lower concordance between replicate PE measurements), our fine-tuned models have generally lower prediction performance for this cell line. Furthermore, there are few promoters in the training data with high DE and PE in THP1. These problems make difficult to design sequences for THP1 using sequence optimization we often could not find many designs with high predicted DE and PE in THP1 using the vanilla DE definition presented Eqn I during fine-tuning and sequence optimization. Therefore, we slightly modified the DE definition for THP1 cells to boost the number of such desirable designs (especially those with high predicted PE in THPI since this a requirement for gene therapy applications). This definition is used both during fine-tuning (i.e. in Eqn 3) and sequence optimization:
DETHPI (x) = (ICI-1) -
[1.5 * PETHPI (x) - -PE
(5)
The constituent models of the ensemble are fine-tuned without the conservative regularizer using the AdamW optimizer with batch size of 512, 5e-5 learning rate, 3e-3 weight decay. 31 = 0.9. 32 0.999, and with a cosine learning rate decay schedule that decays to 0 after 10 warm-up steps. They are fine-tuned for upto 250 steps, and we retain the best checkpoints that are selected by monitoring the validation loss throughput the training process.
17

---

E.4 Prediction performance
The prediction performance of our models on their respective test sets is shown in Table S.2] These results indicate that all of our models are accurate predictors of PE in the target cell types, justifying their usage in designing promoters.
Jurkat
K562
THP1
Model
Dataset Split
Design model with a - 0
Split
0.699
0.666
0.705
0.698
0.644
0.542
Design model
0.0003
Split
0.700
0.706
0.698
0.512
Design model with 0.001
Split
0.700
0.666
0.706
0.698
0.645
0.542
Design model with 0.003
Split
0.701
0.667
0.706
0.698
0.642
0.543
Design model with o 0.01
Spli 1
0.200
0.668
0.705
0.098
0.639
0.545
Ensembble model used for final selection
Split
0.702
0.6699
0.69
0.6900
0.627
0.529
Test Set Replicate Concordance
Split
0.819
0.735
0.737
0.666
0.730
0.483
Table S.2: Prediction performance obtained using our models.
F
Details about baselines
In this section, we describe the baselines in more detail.
F1
Designing cell type-specifie promoters using motif tiling
In this section, we detail how design cell type-specific promoters using motif tiling. First, to identify motifs that might contribute to DE, we use FIMO 33
with default settings to detect instances
of clustered TF-binding motifs defined by Vierstra et al. 34 in the sequences assayed by Reddy
et al. [12]. and retain detected motif occurrences with q-value 0.01. Let's now consider designing cell type-specific promoter for Jurkat. For every motif, we first run Welch's t-lest to determine if sequences that contain it have higher expression in Jurkat than sequences that do not contain it, and retain motifs with v-values < 0.01. Then, for every motif, we run pairwise Welch's t-tests to determine if its presence leads to higher expression in Jurkat compared to K562 or THP1. Motifs that have positive effect sizes in both t-tests (i.e. leads higher expression in Jurkat compared to both K562 and THPI) with q-values < 0.01 are retained as those that could be contributing towards DE in Jurkat. Then, these motifs are used to design two sets of sequences one set of sequences is designed by inserting the same motif into a background sequence as many times as possible while separating the motifs by 10bp (we get sequences per motif using different background sequences), and the second set is designed by randomly sampling motifss (weighted by average effect size from the two pairwise tests) from the list retained motifs and inserting as many of them as possible into background sequence while separating the motifs by 10bp. While generating each sequence, the background sequence is a randomly chosen sequence from those assayed by Reddy et al. [12] that exhibits a DE of at least 2 and also has PE in the target cell that is greater than the 90th percentile of PE. Inserted motif sequences are sampled from the motif's 's position weight matrix (PWM). The same process is repeated for K562 and THP1.
We discover 4. 26, and motifs that may be causing DE in Jurkat, K562 and THP1 respectively. Thus, get 20, 130, and 15 sequences in Jurkat, K562, and THP1 respectively by tiling the same motif repeatedly. Then, for each cell line, we design 500 sequences by randomly sampling motifs. Therefore, we get total of 520, 630, and 515 sequences for Jurkat, K562, and THP1 respectively using this design method.
F22 Deep Exploration Networks (DENs)
DENs are generative models that are trained output diverse sequences that maximize a design model's 'ss predictions. The generator takes random noise as input and transforms it into a sequence PWM. We use UNet-style 35 generator that first transforms the noise vector into a sequence PWM-sized matrix (i.e. of size (250, 4)). Then, it applies 6 downsizing convolutional layers followed by 5 upsizing convolutional layers. final convolutional layer then pools information across the final set of filters' outputs to produce the sequence PWM. The PWM is used to sample sequences that are fed to the design model to get its predictions and fitness-baser loss that trains the DEN to output high- sequences is computed. Additionally, to explicitly increase the diversity of the generated
https:/rresourcessuuu wwtieess..stuoiieeeeeee http::uwwww.....eeee///////eeeee -lstrerffuuuuterrinieeb.ttttt
18

---

sequences. in every training step, random noise vectors are input to the DEN in pairs to get two sequence PWMs per pair. Then, diversity-basee loss is computed that incentivizes the sequences generated using the two different noise vectors to be distinct from each other, both in sequence and design model embedding space. An entropy-based loss also minimized to reduce the entropy the PWM output by the DEN at every position.
Thus, when training a DEN to generate cell type-specific promoters for a target cell [Jurkat, K562, THP1), the training objective we use is:
2 Σ
2
min
(3g
+
Σ
(g))
j=1, (us)
j=1,
fitness loss
1
+ B diivrritty max
- -0.3 + max oE 1000 N
jjh Uj.k
2
j=1
k=
sir:gs (M1)
g3 (ugg)
soquence- diversity
Ro(s)).Ro((
+ diveriiyy max
0.. +
Σ
2
(11 Ro
j=1
(5))
(j fj~ (wa)
ewweeddnggg-ii based diversity loss
+ B Bentropy max
1.8
loga 4
Σ - go((u) loga (go(u1) + 10 10- $)
N
entropy loss
where the set of trainable parameters of DEN Js which outputs N 250 base pairs long sequence PWMs by taking 1 or Ut2 200-dimensional random noise vectors sampled from the uniform distribution over [-1, 1J, as inputs. From the sequence PWMs output by gas we sample two one-hot encoded sequences per noise vector denoted by sj and qj. These sequences are then input to the trained design model that predicts PE induced in each of the three cell types. Then, the predicted DE in the target cell induced by sequence x is given by Eqn[[] The fitness loss maximizes this predicted DE. The other loss terms increase sequence diversity and reduce entropy in the sequence PWM. Here, $j is the one-hot encoded base pair position & in sj (similarly for gj,x), Ro(s,) is an embedding for 8j extracted from the design model fe. 9a (ur)) is the probability distribution over base pairs in the sequence PWM go at position k. Finally, the coefficients fuivensay and Bentopy are used to weight the diversity and entropy losses relative to the fitness loss and to one another. They can be varied to regulate the diversity work by Linder et al. [7] that proposed DENs for more details on the method, We tune the various hyperparameters reflected in the training objective by observing the overall quality of the generated sequences.
fitness trade-off. We refer readers to the original
We train total of 15 DENS per target cell type, each using a different design model compute the fitness loss, or making different diversity vs. fitness trade-off by using different favveriity values. We have 5 different design models, each trained using a different conservatism coefficient a (Table We also try 3 different 3divesiity values - 1. 5, and 10, yielding 15 DENs in total. Each DEN is used generate 20000 sequences and we use the final sequence selection algorithm from Section 4.3 choose the final set of 2000 sequences per target cell.
19

---

G
Computational resources used for training
Our workflow is mostly run on system equipped with 128 CPU cores and 8 Nvidia 100 GPUs with 80GB VRAM each. Pretraining takes around 3.5 hours and it takes around 2 hours to fine-tune design model and optimize sequences using gradient ascent. We performed a hyperparameter sweep consisting of pretraining runs to tune model architectures. Overall, we estimate that it took around 50 hours on this machine to produce designs using our workflow.
DENs trained system equipped with 32 CPU cores and 8 Nvidia A5000 GPUs with 24GB VRAM each. It takes about 3 hours to train each model on a single GPU. Overall, we estimate that took around 20 hours this machine produce designs using DENs.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We support all claims made in the abstract and introduction using workflow description and experimental evaluation sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, they are discussed in the last section titled "Discussion and limitations".
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
We do not present any theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide high-level details about the methods used to produce our experi- mental results in Section a|d provide more details in the appendix with the experimental protocol in Section D training details in Section E] and details about the baselines in Sec- Our code, data, and instructions to reproduce our main results will be available at httpgghtwwhooooooooooggg/g////ee _deiign
[io
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
22

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Our code, data, and instructions to reproduce main results will be available at httpp::/giiuub.com/m.y young- www.aooooooooggggggoooooottteeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppcccppubccc/uuuccccec
Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details about our workflow are provided in Section 4 the high-level setup for our experiments is described in Section 5 and the appendix has more details about the training and testing in Sections and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Wherever appropriate, we add error bars and details about the significance testing (only used to produce Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: These details are provided in appendix section G
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics attpps//neerr htt.r/pp....pbublll/tcccccccduaa
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and our work conforms to it in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We talk about the potential positive impacts of this work in the abstract and introduction. We do not think our work can have any negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not think our paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The only existing asset we use the data released by Reddy et al. 12 We mention that it was released with their codebase under an AGPL-3.0 license in appendix
section C
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code, data, and instructions to reproduce our results will be available at
https://github.com//ye homoulubooooooouo---gggggooooooo peprsnn We provide details on how we collected the data in appendix section
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not use crowdsourcing or human subjects to collect data.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not crowdsourcing human subjects to collect data.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "Y0r3iCTjZc", "reviewer": "Reviewer_zTWW", "summary": "The paper presents a comprehensive guide for designing cell-type-specific promoter sequences using a conservative model-based optimization (MBO) approach. The primary goal is to develop promoters that drive gene expression specifically in target cells while minimizing off-target effects in closely related cell types. The authors propose a detailed workflow that incorporates data efficiency, sequence diversity, and model uncertainty. This method is validated through empirical experiments on blood cancer cell lines, demonstrating its effectiveness compared to traditional and simpler optimization methods.", "questions": "1. How do we balance diversity and conservative regularization? They seem to be contradictory. \n2. Could the authors add decision-tree-like guideline figures at the end of the paper? It will help biologists better understand the workflow.", "limitations": "The authors have addressed the limitations.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The manuscript is well-written, and it is easy to follow. The problem of promotor design is important and it has driven increasing interest. The authors here present a comprehensive work and a practical guide for effectively designing the cell type specific promoter. The wet lab experiment further strengthens this paper.", "weaknesses": "1. I believe many generative-based models are specifically designed for DNA-sequence design, e.g., the genetic algorithm, simulated annealing, and other RL-based models. Why do the authors only focus on \"gradient ascent\"? It may introduce more adversarial effects compared with the other methods. \n2. Lack of sufficient figures to better visualize the proposed workflow and practical guide.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and for recognizing the significance of our work. In our responses below, we first justify the use of gradient ascent vs. other optimizers such as genetic algorithms and simulated annealing, due to its computational efficiency. Then, we clarify that our workflow can be used to produce diverse designs, even when conservative regularization is used, due to our use of multiple design models that are trained using different regularization strengths, as well as our final sequence selection algorithm that balances optimality and diversity to select the final set of designs for wet lab validation. Finally, to better illustrate our workflow in the final version of the paper, we will be adding a decision-tree-like figure as suggested by the reviewer - a draft of this figure is in the document attached to the general rebuttal.\n\n1. **Gradient ascent is used instead of other optimizers due to its computational efficiency, and the conservative objective mitigates adversarial effects:** While discrete optimizers such as genetic algorithms and simulated annealing are suited for making discrete mutations to a DNA sequence during optimization, it can be computationally expensive (requiring many samples and filtering) to use them in a large search space such as ours (250 bp sequence = $4^{250}$ possible sequences), since these algorithms do not use the function’s derivative to take a step towards an optimum. Similarly, we did not experiment with RL-based models for DNA-sequence design since it is difficult to integrate them with the conservative objective models framework, which requires a fast optimizer during training to mine for potentially adversarial sequences. On the other hand, gradient ascent is computationally efficient and allows us to quickly discover an optimum. Furthermore, in preliminary experiments, we experimented with adding discrete mutations during training and sequence optimization in addition to performing gradient ascent, but we did not observe a significant improvement in sequence quality vs. just using gradient ascent (as measured by design model-predicted differential expression). Overall, simplicity and efficiency considerations push us to use gradient ascent instead of other discrete optimizers.\n\n     As the reviewer points out, it is true that gradient ascent algorithms can produce adversarial effects. However, we utilize a conservative objective to mitigate them, as described in Section 4.2 in the paper. The other optimizers are also not immune to producing adversarial effects – if run for a sufficient number of steps, any of the aforementioned algorithms can produce adversarial sequences since the designed sequences can be arbitrarily different from the training set.\n\n2. **Collecting sequences from design models trained using different conservative regularization levels, and using our final sequence selection algorithm produces a diverse yet desirable set of final designs:** As the reviewer notes, strong levels of conservative regularization could hinder the diversity of designed sequences. In order to maintain a high level of diversity, we obtain a large set of candidate sequences using many different levels of regularization as mentioned in lines 240-242 (controlled by the $\\alpha$ parameter in Eqn 3). Then, we use the algorithm in Section 4.3 to choose a subset of sequences for wet lab validation that are diverse yet desirable. This combination of multiple regularization levels and the final sequence selection algorithm yields a diverse final sequence set as illustrated in Table 1, overcoming potential issues of an over-regularized model producing a narrow distribution of candidate sequences. That said, we agree that fundamentally, conservatism (to stay close to the data) and diversity (going beyond the data in several ways) are at odds with each other more broadly from an algorithmic perspective, and studying these topics is a good avenue for future work, even for general ML, outside of biological problems, as we will discuss in the final version of paper.\n\n3. We will add additional figures that illustrate our guidelines to the final version of the paper. A draft decision-tree-like figure is in the document attached to the general rebuttal. We will refine this figure for the final manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OMVBn45i8g", "reviewer": "Reviewer_XucD", "summary": "The paper outlines a workflow to designing promoter sequences that are specific to cell types, especially closely related cell types. The proposed approach consists of five steps:\n\n1.\tPretraining a model on existing massively parallel reporter assay (MPRA) datasets, which are large but restricted to a few well-studies cell types.\n2.\tFine-tuning the pretrained model on experimental data with the aid of a conservative regularizer.\n3.\tA gradient ascent-based approach to design sequences that have high predicted differential expression (DE).\n4.\tSelection of smaller subset that is both optimal and diverse.\n5.\tExperimental verification of designed promoter sequences.\nThe authors compare the performance of the proposed workflow to two existing approaches – motif tiling and deep exploration networks (DENs) on experimental data on both optimality and diversity metrics. Their results show instances where their workflow either outperforms existing methods in optimality or yields more diverse promoter sequences than DENs.", "questions": "Questions:\n\n1.\tWhat is the “linear probing” in line 179 referring to?\n2.\tDoesn’t the conservative regularizer limit the method’s ability to “discover” new promoter sequences? Is it possible that there are promoter sequences that are unlikely to directly evolve from the sequences in the fine-tuning dataset?\n3.\tWhy are 6-mer frequency vectors used to compute the Euclidean distance $\\mathcal{K}$?\n4.\tWhy should the base pair entropy of a diverse sequence set be 2? Based on the definition used in the paper, should this not be $ln(4)$?\n5.\tWhy are different metrics used to compare the DEs of the proposed workflow against motif tiling (Figure 2) and DENs (Figure 3)?\n\nSuggestions:\n\n1.\tOn line 142, it is not immediately clear if the “some objective function” being referred to is different from the true objective function. While this becomes clearer later in the paper, a rewording of this line may improve clarity.\n2.\tPoints 1 and 2 in Section 4.1 could be merged as the primary barrier to using Enformer is the cost of retraining it to suit the user’s setting.\n3.\tThe titles in Figure 2 are difficult to read. Moving them to the caption of each sub-figure may improve readability.\n4.\tIn Figure 3, there is a visible improvement in DE only for Jurkat cells. Are there any other reasons for why the proposed workflow is preferable to DENs for the other cell types (computational savings perhaps because of the use of a discriminative model in the workflow vs generative models in the DEN)? A restatement of these reasons will strengthen the case for the proposed workflow.\n5.\tThe authors could be more concise. For instance, the information in the section from line 98 to line 106. The information here is largely similar to that contained in the outline of the workflow at the end of the introduction (Section 1). Similarly, the discussion on the diversity-based selection (starting on line 274) serves no role in their final algorithm as $\\beta$ is set to zero. Being more concise may enable the authors to include more relevant details and improve the clarity of their writing.", "limitations": "The authors have made a good faith effort to acknowledge the limitations of their work in Section 6. I believe that their method is also limited in discovering novel promoter sequences due to the use of the conservative regularizer. Since their design method starts from sequences in the finetuning data to evolve the design sequences, there is likely a limit on how divergent they are from the seen data. This is somewhat also borne out by the poorer performance of their workflow on THP1 cells, which are scarcer in their training data.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is very clear with an intuitive flow. It begins with a description of the desiderata for any workflow for designing promoter sequences, followed by a summary of the related work and a series of preliminaries to acquaint the reader with the objectives of the problem being considered. I also appreciate the references to related works and motivations for the design decisions in Section 4. This delineates the differences in the proposed workflow compared to existing works. The authors are also explicit with the details of their model architectures, training and experiments. This is readily visible from the great level of detail in the relevant sections of the appendix, as well as a clear statement of their objective function and its components (see Eq. 3 and 4). \n\nBesides the writing, the inclusion of a wet lab experimental evaluation of the designed promoter sequences lends a lot of credibility to their work as it exhibits the practical utility of the proposed workflow. Suitable baselines have also been chosen for comparison of performance, and consequently make substantive claims to the utility of the proposed workflow.", "weaknesses": "The paper has no glaring weaknesses. However, there are some places where the authors have made statements that are confusing or unsubstantiated.\n\n1.\tThe claim of data efficiency over small PE datasets is not substantiated. It appears that the authors accomplish this via fine-tuning over the smaller dataset. However, they take their cue for this from existing offline MBO algorithms, so this likely cannot be claimed as a key benefit to their proposed workflow.\n2.\tThe outlined workflow uses an ensemble of fine-tuned models (without the conservative regularizer) to compute a pessimistic estimate of the DE of a given sequence and cell pair. Why should we expect such an ensemble to estimate a DE that is consistently lower than the true PE? Additionally, this ensemble comprises models with slightly different architectures, but no further specifics are given as to what these architectures are and how they are chosen?\n3.\tThe selection algorithm outlined in Section 4.3 appears to be greedily optimizing for an objective that balances optimality and diversity. However, the authors then state on line 339 that they set the diversity coefficient ($\\beta$) to 0 as the workflow naturally designed diverse promoter sequences. This renders most of the discussion between lines 274 and 294 moot to their workflow.\n4.\tIn the conservative regularizer, the authors penalize high predictions for both unseen and potentially undesirable sequences, summarized as $\\mu(\\mathbf{x})$. However, it is not clear how the formulation in Eq. 2 handles unseen sequences as the second term $\\mathbb{E}_{\\mathbf{x}\\sim D} \\left[ f\\_{\\theta} (\\mathbf{x}) \\right] $ is restricted to sequences seen in the dataset(s).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments and for recognizing the significance of our work. In the responses that follow, we clarify that our data-efficiency claim is due to the pretraining step which is missing in previous promoter design workflows – we will make this clear in the final version of our manuscript. Next, we justify why we expect the pessimistic estimate of DE to underestimate the true DE due to its definition as the lower confidence bound (LCB) of the ensemble’s constituent models’ predictions. We point to more details about the ensemble’s architecture and explain our architectural choices – our explanation will be added to the final manuscript. Then, we explain that we retain the diversity component of our final selection algorithm to allow a practitioner to boost diversity if necessary. Using an additional analysis and results from Table 1, we show that our designs represent diverse regions of the sequence space that are also distant from the space of training sequences, but we acknowledge that certain desirable promoters will be undiscoverable – we will discuss this more in the final manuscript. We also address the other weaknesses and questions mentioned by the reviewer. We hope that our responses make a stronger case for the acceptance of our paper.\n\n**Addressing weaknesses identified by the reviewer:**\n1. **Our workflow’s data-efficiency comes from the pretraining step that is absent in existing promoter design workflows:** The main reason we claim data-efficiency is indeed due to the pretraining step (lines 71-74 and Section 4.1). Our modeling strategies are derived from Reddy et al. (2024), who showed that pretraining on related genomic datasets can boost downstream performance when the pretrained model is fine-tuned using a small PE dataset. As we point out in the related work section, existing MBO-based promoter design workflows are reliant on large MPRA datasets and train their design models exclusively on these large datasets. By adopting the transfer learning strategies identified by Reddy et al. (2024) in our workflow, we can obtain more accurate design models using the same amount of data from target cells when compared to existing workflows that do not leverage transfer learning, thereby supporting our data-efficiency claim. We will make this point clearer in the final version of our manuscript.\n\n2. **Since the pessimistic estimate of DE is computed as the lower confidence bound (LCB) of the ensemble’s constituent models’ predictions (lines 269-270), we expect it to underestimate the true DE if the ensemble is accurate:** If we have reasonably accurate models of DE, the mean prediction from the ensemble should be an accurate estimate of the true DE. Since the LCB is computed as the mean prediction minus the standard deviation across predictions, this pessimistic estimate should underestimate the true DE, provided the ensemble is reasonably accurate.\n\n     **Ensemble model architectures:** Details about the ensemble’s constituent models are in Section D.2 of the appendix. Since it is expensive to pretrain many different architectures, we use the same pretrained backbone in all constituent models while using different output layer architectures during fine-tuning. In particular, we vary the depth and number of hidden units of the output layers. We also vary the activation functions, as using different activation functions in the constituent models’ output layers allows us to approximate the PE function using different piecewise functions, thereby improving the ensemble’s overall robustness.\n\n3. **Diversity component of final selection algorithm is retained to have the flexibility to boost diversity if necessary:** While we set the diversity coefficient to zero for sequences designed using gradient ascent, we set it to 10 when using DENs (line 351), as the top designed sequences produced by DENs were not as diverse in our preliminary experiments as those produced using gradient ascent. We retained this ability to modulate diversity in our workflow for the sake of flexibility – if a practitioner observes that the top designed sequences are very similar to one another, the diversity coefficient can be increased to avoid selecting highly similar sequences in the final sequence set.\n\n4. **The conservative regularizer only penalizes potentially undesirable (i.e. potentially adversarial) sequences among those not seen in the training set – it does not penalize all unseen sequences:** We will improve the clarity of this description in line 217. The second term in Eqn 2 prevents uniform underestimation of the function being modeled across all sequences by maximizing the expected value of the function over the training set (see Trabucco et al. (2021) that presents the COMs framework for more details).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "naG5TzgLCG", "reviewer": "Reviewer_ipDz", "summary": "Building on existing models, the authors used MBO to design promoters in a data-efficient manner, with a particular focus on discovering promoters for similar cell types. This approach was tested on three relatively similar blood cancer cell lines, demonstrating that the method successfully identified numerous new cell type-specific promoters after experimentally validating the designed sequences.", "questions": "1. The model lacks innovation and primarily relies on the application of existing models.\n2. The design process proposed by the author is roughly similar to the existing biological sequence design algorithm process, and the screening based on model uncertainty is also mentioned in GflowNets. GflowNets also has strategies to increase diversity and optimize properties. The author's contribution is more of a simple combination of current method processes. What is the author's innovation?\n3. If the author's contribution is to propose a workflow, then should they use more extensive sequence design models in the workflow, rather than just using COMs, to prove the effectiveness of the design algorithm?\n4. In terms of experimental evaluation, the authors should include more comparative models. Given the many existing biological sequence design algorithms, the authors need to demonstrate the advantages of their algorithm.", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A design process is proposed, building on existing models, which uses MBO to design promoters in a data-efficient manner. This approach has some impact on promoter optimization.", "weaknesses": "The authors did not focus much on improving the design of the model itself, instead relying on existing models. The innovation in the model is limited, and the experimental evaluation is not comprehensive, as it did not assess a wider range of biological sequence design algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and address their concerns below. First, we clarify that the novel contribution of our work is our cell-type-specific promoter design workflow. Although we use existing modeling strategies and MBO algorithms, we combine them in a novel way to tackle a difficult but important problem – an accepted axis of novelty at NeurIPS as per the guidelines. Unlike existing approaches, our elaborate workflow accounts for many practical considerations such as data-efficiency, avoiding adversarial sequences, and obtaining diverse sequences (PDF attached to general rebuttal illustrates it). And unlike most other offline MBO works that rely on computational evaluations, which are often unrepresentative of real-world experiments, we validate our workflow using expensive and time-consuming wet lab experiments. Next, we also explain the benefits of our workflow over GFlowNets, which are generally harder to train and tune, do not account for adversarial examples, and have not been validated using real-world experiments. Finally, we justify the limited number of benchmarks, as the cost of running wet lab experiments is extremely high and our budget did not allow us to benchmark more methods. We hope that the reviewer can reconsider their assessment based on our responses.\n1. **Our contributions:** We would like to clarify that the specific models used in our work are not the main contribution. Instead, the focus of this paper is to propose a novel workflow for discovering cell-type-specific promoters using offline MBO that accounts for various practical considerations. Our workflow has the following advantages over existing promoter design workflows (detailed in the Related Work section):\n      - Compared to traditional workflows that use heuristics and manual curation, our workflow is automated and generalizable and can be used to design promoters for even relatively understudied cell types.\n      - We improve on existing MBO-based workflows in three main ways. First, we use the transfer learning strategies identified by Reddy et al. (2024) to train models in a data-efficient manner - existing workflows do not employ transfer learning and rely on large training datasets that are expensive to collect. Second, we use Trabucco et al. (2021)'s COMs framework to mitigate adversarial designs - existing workflows use simple optimization techniques (e.g. vanilla gradient ascent) that are more prone to adversarial designs. Finally, as experiments are expensive and a limited number of sequences can be tested, we propose a final sequence selection step to choose a small subset of diverse yet desirable sequences for testing from a large pool of candidate designs – this step is also missing in existing workflows.\n\n      Although we use the modeling strategies identified by Reddy et al. (2024) and the COMs framework, we combine them to solve an important problem in a novel way – an accepted axis of novelty at NeurIPS as per the guidelines. We also add a novel sequence selection step to balance diversity, optimality, and uncertainty. This novel workflow has the potential to improve gene therapies and should be of interest to many ML researchers working on synthetic biology problems such as promoter design. Moreover, we validate the utility of our workflow by performing real-world wet lab experiments - most MBO methods are never validated using wet lab experiments. We show that our workflow is effective and outperforms both a traditional design method (motif tiling) and an offline-MBO-based method (deep exploration networks - DENs) that had previously been validated using wet lab experiments. These points support the novelty and significance of our work.\n2. **GFlowNets are harder to train, do not account for adversarial examples and have not been validated using real-world wet lab experiments:** Our workflow improves on existing promoter design workflows in many ways, especially by accounting for practical considerations such as data-efficiency, adversarial designs, diversity, and uncertainty. The reviewer mentions that GFlowNets also try to optimize a given property while retaining diversity. However, since GFlowNets are generative models, they require specialized architectures that are significantly harder to tune and train vs. training a discriminative model as in our workflow. They also do not account for adversarial designs. Most importantly, GFlowNets have not been validated using real-world wet lab experiments to the best of our knowledge (Jain et al., 2022). DENs are also generative models that are aimed at producing diverse desirable designs and have been validated using real-world wet lab experiments (Linder et al., 2020), which is why we chose DENs as a comparator to benchmark our approach. Figure 3 shows that our workflow outperforms DENs.\n3. **High experimental costs limit our ability to benchmark more methods:** Most existing MBO methods for biological sequence design do not validate their designs using wet lab experiments – instead, their evaluations are based on computational oracles whose predictions are not always reflective of real measurements. In our wet lab experiments, we show that our workflow is indeed capable of designing cell-type-specific promoters. While we benchmark against DENs and motif tiling, the reason for not benchmarking more design methods is the extremely high cost of performing these experiments. They are also time consuming, and took us ~6 months to complete. Since our budget limited the total number of sequences that could be tested, and since we require a significant number of designs from every method to thoroughly validate it, we chose to evaluate a representative set of methods. In particular, we chose motif tiling as representative of widely-used traditional methods, and we chose DENs as the only MBO algorithm that has been evaluated for biological sequence design using wet lab experiments. Our workflow outperforms both of these methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
In the attached document, we provide an additional table that shows the average Hamming distance to the closest training set sequence for designs from every method. This table illustrates that designs from our workflow are quite distinct from the training set sequences, differing by 125-140 bp on average. We also provide a draft figure that details the various steps of our workflow.

Meta-review: The tools and guides developed in this submission were unanimously viewed as a timely, relevant, and well executed contribution to promoter sequence design. The clarifications provided in the rebuttal further helped clarify the contribution and how it compares to existing work. As such, I agree with all three reviewers in that it should be published in NeurIPS and encourage the authors to include their main arguments from the rebuttal into the camera ready version of the paper.
Decision: Accept (poster)