Learning to Discover Regulatory Elements for Gene Expression Prediction
Authors: Xingyu Su
Haiyang Yu
Degui Zhi
Shuiwang Ji
Venue: ICLR 2025 Oral (2025)
Keywords: Gene Expression
Deep Learning
Sequence Modeling
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: We consider the problem of predicting gene expressions from DNA sequences. A key challenge of this task is to find the regulatory elements that control gene expressions. Here, we introduce Seq2Exp, a Sequence to Expression network explicitly designed to discover and extract regulatory elements that drive target gene expression, enhancing the accuracy of the gene expression prediction. Our approach captures the causal relationship between epigenomic signals, DNA sequences and their associated regulatory elements. Specifically, we propose to decompose the epigenomic signals and the DNA sequence conditioned on the causal active regulatory elements, and apply an information bottleneck with the Beta distribution to combine their effects while filtering out non-causal components. Our experiments demonstrate that Seq2Exp outperforms existing baselines in gene expression prediction tasks and discovers influential regions compared to commonly used statistical methods for peak detection such as MACS3. The source code is released as part of the AIRS library (https://github.com/divelab/AIRS/).

Published as a conference paper at ICLR 2025
LEARNING TO DISCOVER REGULATORY ELEMENTS FOR GENE EXPRESSION PREDICTION
Xingyu Su'", Haiyang Yu'", Degui Zhi & Shuiwang Ji
'Texas A&M University, "The University of Texas Health Science Center at Houston (xingyu.se su, haiyang, s jjijet amu. edu, {degui. zhi 2hih emc.edu
ABSTRACT
We consider the problem of predicting gene expressions from DNA sequences. A key challenge of this task is to find the regulatory elements that control gene expressions. Here, we introduce Seq2Exp, Sequence to Expression network explicitly designed to discover and extract regulatory elements that drive target gene expression, enhancing the accuracy of the gene expression prediction. Our approach captures the causal relationship between epigenomic signals, DNA se- quences and their associated regulatory elements. Specifically, we propose de- compose the epigenomic signals and the DNA sequence conditioned on the causal active regulatory elements, and apply an information bottleneck with the Beta dis- tribution to combine their effects while filtering out non-causal components. Our experiments demonstrate that Seq2Exp outperforms existing baselines in gene ex- pression prediction tasks and discovers influential regions compared to commonly used statistical methods for peak detection such as MACS3. The source code is released as part of the AIRS library httts://gthubbcomdiveabaa//l
INTRODUCTION
Gene expression serves fundamental process that dictates cellular function and variability, pro- viding insights into the mechanisms underlying development (Pratapa et al. 2020), disease (Cook-
son et al. 2009 Emilsson et al.) 2008), and responses external factors (Schubert et al. 2018).
Despite the critical importance of gene expression, predicting it from genomic sequences remains a challenging task due to the complexity and variability of regulatory elements involved. Recent
advances in deep learning techniques (Avsec al., 2021 Gu & Dao [023] Nguyen et al. [024
Badia-i Mompel et al., 2023) have shown remarkable capabilities and performance in modeling long sequential data like language and DNA sequence. By capturing intricate dependencies within ge- nomic data, these techniques provide deeper understanding of how regulatory elements contribute to transcription (Aristizabal et al. 2020).
To predict gene expression, DNA language models are usually applied to encode long DNA se- quences with subsequent predictor to estimate the gene expression values (Avsec al. 2021 Nguyen et al. 2024) Gu & Dao 2023 Schiff et al., 2024). However, those language models are typically designed to encode DNA sequences alone, overlooking the specific environments like dif- ferent cell types, which leads to suboptimal performance. Instead of predicting the gene expression only using DNA sequence, which invariant across cell types, a more biological relevant formu- lation is to predict gene expression levels using both DNA sequence and epigenomic signals. For example, GraphReg (Karbalayghareh et al. (022) uses epigenomic signals as input data to predict gene expression values. However, it does not integrate DNA sequences and epigenomic signals in unified manner to improve gene expression prediction. EPInformer (Lin et al.) 2024) uses statisti- cal methods to identify the epigenomic signal peaks, and focuses regulatory elements identified by those peaks. Although obtaining better results, EPInformer still neglects the complex relation- ship between DNA sequences, epigenomic signals and regulatory elements, which is essential for improving prediction accuracy.
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
The task of predicting gene expression levels given the DNA sequences and epigenomic signals presents several challenges. First, epigenomic signals can be measured by a variety of experimental techniques, including ChIP-seq. DNase-seq. Hi-C, each with their own biases and limitations (Con-) sortium et al., 2012 Bernstein et al. 2010 Moore et al. 2020). Additionally, the regulatory elements influencing target gene expression
often sparse and may involve long-range interactions, making them challenging to identify and integrate into predictive models. These complexities highlight the need for models that can effectively discover the actively interacted regulatory elements with the target gene on long DNA sequences.
In response to these challenges, we propose designed to improve gene expression prediction by selectively extracting relevant sub-sequences from both DNA sequences and epigenomic signals. Since DNA sequences and epigenomic sig- nals capture different aspects of biological information, their integration offers deeper insights. For example, Hi-C/HiChIP data reveals the physical interaction frequency between distal DNA regions, and DNase- seq reflects the functional activity of regulatory elements. Effectively incorporating these signals along with DNA sequences can be highly beneficial for addressing the above challenges for gene expression prediction task. Specifically, in this work, suggest the causal relationship be- tween genomic data and gene expression to guide the learning process as depicted in Figure [ Inspired by the causal relationship, one based on DNA sequences and the other on epigenomic signals. The proposed Seq2Exp first em- ploys a generator module learn token-level mask based on both DNA sequences and epigenomic signals, to extract DNA sub-sequences. Then, the predictor module is applied on these extracted sub- sequences to predict gene expression. With information bottleneck. Seq2Exp can effectively filter out non-causal parts by constraining the mask size, ensuring that only the most relevant regions are extracted. Overall, the incorporation of the DNA sequences and epigenomic signals systematically discovers regions that are likely to influence gene expression.
P Seq2Exp (Sequence to Expression), a novel framework
decompose the mask learning process into two components:
We summarize our contributions here:
We propose a framework articulating the causal relationship between epigenomic signals, DNA sequences, target gene expression and related regulatory elements.
Based on the causal relationships, our framework is proposed to combine the mask probability distribution from DNA sequences and epigenomic signals, and filtering out non-causal region via information bottleneck.
The proposed Seq2Exp achieves SOTA performances compared to previous gene expression prediction baselines, and demonstrates the extracted regulatory elements serve better sub- sequences compared to statistical peaks calling methods of epigenomic signals such as MACS3.
2 RELATED WORKS AND PRELIMINARY
2.1 TASK DESCRIPTION
Let Xsqq= = [x1, 2r.] denote the DNA sequence with length L, where each token x; E RA411 one-hot vector representing a nucleotide from the set (A, C, G, T}. For this DNA sequence, the corresponding epigenomic signals are denoted as Xsig = [s1.... SLL. where s; E Rdxl represents d different signals. By using both the DNA sequence and epigenomic signals, the task aims predict the target gene expression denoted as Y E R. To achieve this target, we propose our framework to extract the active regulatory elements by learning token-level binary mask M [m1,'.. .zz), where my E {0, 1} or a soft mask M where m, E [0, 1].
Specifically, in our implementation, each example contains one target gene. We first identify the transcription start site (TSS) of the target gene, then select input sequences X and X consist of 200, 000 base pairs, centered on the TSS. Then, the entire sequences provide sufficient contextual information for accurate prediction of the target gene expression value Y.
2.2 RELATED WORKS
DNA language model has been proposed recently to apply language machine learning models to long DNA sequences (Nguyen et al. 2024 Gu & Dao. 2023 Schiff et al. 2024) and solve vari-
2

---

Published as a conference paper at ICLR 2025
ous downstream tasks. Two notable methods in this area HyenaDNA (N guyen al. 2024 and Caduceus (Schiff et al. 2024). HyenaDNA utilizes the Hyena operator (Poli et al. (023) to pro- cess long DNA sequences. Caduceus introduces bidirectional Mamba (Gu & Dao 2023) for DNA
sequences, providing linear complexity for long sequence modeling while also considering the re- verse complement of the DNA sequences. These methods offer powerful approach for modeling long sequence data, such as DNA, and can be fine-tuned for tasks like gene expression prediction. However, they usually only considers DNA sequences input, and do not explicitly consider the additional epigenomic signals during the prediction. Since these signals often carry meaningful in- formation, such as physical interaction frequency and functional activity, incorporating them into the model could further enhance its performance the gene expression prediction task.
Gene expression prediction is one of the fundamental tasks in bioinformatics (Segal et al., 2002 Numerous studies (Agarwal & Shendure] 2020 Karbalayghareh et al. 2022, Avsec et al., 2021 Lin et al.) 2024) have attempted to predict gene expression values directly from DNA sequences. Enformer Avsec et al. )021), for instance, tries to only encode DNA sequences as input and em- ploys convolutional and transformer blocks to predict 5,313 human genomic tracks and 1,643 mouse tracks. In contrast, GraphReg (Karbalayghareh et al.) 2022), incorporates graph attention network to account for Hi-C/HiChIP interactions between DNA sub-sequences, improving gene expression predictions by considering physical interaction frequencies. However, both methods either rely on epigenomic signals or DNA sequences as input data, without integrating both data rypes. Recently, EPInformer (Lin et al. 2024) has advanced this approach by integrating both DNA sequences and epigenomic signals for gene expression prediction. EPInformer first identifies enhancer regions from the DNA sequences based on DNase-seq signals, treating epigenomic signals as enhancer features, and then use promoter-enhancer interactions for gene expression prediction. Despite this progress, EPInformer selects enhancer regions solely based on epigenomic signal peaks, overlooking the com- plex relationships between DNA sequences, epigenomic signals, and predicted gene expression val- ues. This highlights the need for machine learning methods capable of learning to extract relevant regions in more comprehensive manner.
2.3 BACKGROUND OF INFORMATION BOTTLENECK
To effectively extract active regulatory elements from DNA sequences, is important to understand the concept of the information bottleneck. The information bottleneck method is a widely used technique in machine learning on tasks for images (Alemi et al., 2016; Chen et al.) [2188. language
data (Belinkov et al., 2020 Lei et al. 2016 Paranjape et al. 2020 Bastings et al. 2019 Jain et al.) 2020) or graph data (Wu al., 2020 Miao al.] 2022). Its goal is maximize the mutual in- formation between compressed representations Z and the target variable Y, expressed as I [(Z; Y), while controlling the information extracted from the input X. Note that in the proposed method, Y represents the target gene expression. A straightforward approach would be to set Z = X, but this retains the full complexity of X, which makes the optimization process challenging, especially with the long and noisy nature of DNA sequences.
To address this, researchers impose a constraint on the information transferred from X to Z, ensuring that T(X: Z) < Ic. where is information constraint that allows capture only the most critical compressed representations. The information bottleneck objective becomes maximizing:
L = -
(1)
where 3 is a hyperparameter that balances the trade-off between compression and relevance, How- ever, directly optimizing this objective is challenging. To overcome this, Chen et al. (0188) proposes to maximize a lower bound approximation, which leads to minimizing the following expression:
L
E (p) log qs (m/l))] BKL (Z
where (Z(x)) is a parametric approximation of Z, q ZZ) is a variational approximation of the true distribution p(y: /2), and r(2) approximates the marginal distribution p(Z).
3

---

Published as a conference paper at ICLR 2025
Mwm
Gene
Xuy: Eolgenomic Signats
Re Indenbrieed from Measurement
Hels
Rug
Xuy Sequenoe
R.
Target
Figure I: Causal relationships between epigenomic signals, sequence, gene expression Y and related regulatory elements.
3 PROPOSED METHODS
In this section, we present our framework Seq2Exp. We first present our motivation for predicting gene expression with learnable extraction of effective regulatory elements. We illustrate the causal relationship among regulatory elements, epigenomic signals and DNA sequences as shown in Fig- ure I. Motivated by this structural causal model (SCM) (Pearl 2009 Pearl al.) 20000 Wu al.) 2022 our framework provides learnable approach to effectively extract effective regulatory ele- ments, considering both DNA sequences and epigenomic signals, through an information bottleneck mechanism.
3.1 CAUSAL RELATIONSHIP AMONG REGULATORY ELEMENTS, DNA SEQUENCE AND EPIGENOMIC SIGNALS
The interactions between target gene and regulatory elements are complex, particularly when mul- tiple potential regulatory elements are involved. Mcanwhile, long sequences and distal interactions require a large search region, further complicating the discovery of effective regulatory elements that influence target gene expression. In this study, we take use of epigenomic signals X sig from labo- ratory experiments as well as the DNA sequence seq for target gene expression Y, and formulate their relationships with the proposed three categories of regulatory elements.
Rg: Regulatory elements that have the potential to interact with target gene. However, they might not influence target gene expression if they are inactive a specific cell type or are distant from the target gene.
Rm Regulatory elements discovered from measurement. Typically, the region with strong mea- sured epigenomic signals, such as peaks in DNase-seq. are more likely influence the gene expression. However, there are usually multiple genes within sequence and the association of Rm, with target gene remains unknown.
Rag: Regulatory elements actively interacted with target gene. identified as the causal com- ponent for the final target gene expression Y.
The causal relationship between these variables is depicted in Figure] Note that each variable cor- responds to distribution and link represents causal connection. The flow of this SCM illustrates the perspective of data generation.
Xseg -- Rg. The DNA sequence consists of Rg and other non-causal parts.
Rag - Y. The causal part Rag directly influences the final gene expression. For example, an active enhancer interacting with gene can significantly impact its expression.
Rm- The key causal component Rag is shared by both Rg and Rm It can be de- tected through epigenomic signals in laboratory experiments and also participates in interactions with the target gene.
Rg -- Rag
4

---

Published as a conference paper at ICLR 2025
Rm X sig. sig usually contains strong observable signals, such peaks in DNase-seq. whereas regions without such signals often provide limited useful information.
3.2 TASK OBJECTIVE
Based on information bottleneck, Equation rather than selecting specific sub-sequences. To directly select regulatory elements, we define the latent representations as Z M XX where .f a binary variable controlling the selection of each DNA base or soft mask M indicating the importance of cach DNA base. We assume that each selection is independent given the input sequence X, i... XXM II, pmm. Following the method of Paranjape et al. (2020). the objective becomes:
2 describes how to learn compressed representations Z
N
L RJ
Bpo(matzal- log go (vslmm @ 2,)] + 3K (p) r(m)).
(3)
where the first term is the task-specific loss, such prediction, and the second term imposes a constraint on the learned mask m, aligning with the predefined distribution r(m) without conditioning any specific sequence In our case, this second term to enforce sparsity in the learned regulatory elements.
square enror in DNA gene expression
3.3 DECOMPOSITION OF SEQUENCES AND SIGNALS
By using information bottleneck shown in Equation 3 our primary focus is on estimating P XX), i.e, learning the mask from the input sequences. Given that the input X consists of both DNA sequences and epigenomic signals, we need to estimate p Xxx](( Xeig)).
Assumption (Conditional Independence of Sequences and Signals). We assume that, conditioned
on the selection of regulatory elements M, the DNA sequences and epigenomic signals are condi- tional independent each other, i.e.,
p(Xsig: Xeeg [M = P(Xxtg)M))((XeelM
Assumption [ based on the causal relationships outlined in Section 3.1 The selected sub- sequences of full given sequence, represented by M @ X, can be viewed the optimal regulatory elements (Rag) for a specific gene in a particular cell type. From a data generation perspective, both the regulatory elements detected through measurements (R,,) and those interacting with the gene (Ry) originate from the optimal regulatory elements (Rag). Therefore, given the optimal regulatory elements, the distributions eig .M) and p(X seggl) should be independent of each other.
Proposition 1. Based on Assumption 7 the estimation of pa (M can be decomposed into terms involving Xsng and ssi. Specifically, we have
Po Po(M|X) x pa, sog))oo) (M(Xsig).
(5)
where pe, (M)) and pe, (M (esg) represent the contributions from the DNA sequence and the epigenomic signals, respectively.
The detailed proof of this decomposition is provided in Appendix A.1 Proposition T allows us to factorize the estimation of PM[M]X into two independent components, corresponding to the DNA sequence Xeeg and the epigenomic signals Xaig. As a result, can independently estimate Pe, (M)X (eeeg) and pe, [M]Xsig) which simplifies the overall estimation process. This decompo- sition based on the assumption that, conditioned on the selection of regulatory elements m, the DNA sequences and epigenomic signals are independent, thus enabling more efficient and targeted modeling of each component.
3.4 MASK DISTRIBUTION
With the conditional independence property shown in Proposition I the estimation of the mask M can be decomposed into two components: one based on DNA sequences pe, X and the other epigenomic signals pe, (M Xaig). We assume that both components follow the Beta distribution, as described in Assumption The sampled values from the Beta distribution represent the probability of selecting specific base pairs from DNA sequence.
5

---

Published as a conference paper at ICLR 2025
Assumption 2 (Mask Distribution). We assume that the soft mask my follows the Beta distribution, i.e, m, ~ Beta(ax ax 3).
Unlike the binary hard mask M, the soft mask m, takes values between and 1, making it more suitable for the Beta distribution. The hard mask M can then be obtained by applying threshold to the soft mask. For the implementation, we apply both hard mask version and soft mask version.
There are several reasons for choosing the Beta distribution. First, the Beta distribution typically quantifies success rates (DeGroot & Schervish 2013] Gelman et al. 2013). The input parameters and 3 represent the weights for selecting and not selecting the base pair, respectively. Therefore, when a > B, the base pair is more likely to be selected, and vice versa. Second. as both a and 3 in- crease simultaneously, the selection process will exhibit lower variance, indicating more confidence in the selection. Third, the product of two Beta distributions, when properly normalized, results in another Beta distribution. This ensures that the distributions within the framework remain in the same family, simplifying subsequent mathematical calculations and providing consistent fitting objectives for the models.
Based on these properties of the Beta distribution, we assume that both Ps, (m, (Xeeg) and Pe, (Xig) follow Beta distributions, but with different parameters a and B.
Proposition 2. Given pod (m,)X uct of these distributions also follows a Beta distribution, with parameters:
Beta(an, B1) and pe,
~ Beta( 32), the prod-
Psd (ma (ima//Xee)tee Xee (ma.))
~ Beta( + + - 1, 3--
(6)
The proof of Proposition 2 is provided in Appendix A.2 When combining the probability distribu- tions learned from the DNA sequence and signals, Proposition ensures that the resulting distribu- tion remains within the same family, And the final mask m, is then obtained through the combined Beta distribution. Specifically, deep learning models are applied in our framework to learn these two distributions by predicting the parameters a and B.
3.5 SPARSE OBJECTIVE
In this part, we focus on the mask prior distribution r(m). From the objective in Equation 3 the KL divergence between (m, (x;) and r(m:) needs to be computed. To simplify this calculation, we assume the prior distribution of the soft mask r(m,) follows the Beta distribution well. Therefore, we have r(m,)) ~ Beta(a3, 3a), where ag and Bg are related to the sparsity mask.
The expectation of the Beta distribution is
Blmm] 1
033
03 33'
where μ approximately represents the proportion of regulatory elements within the DNA sequences. Therefore, by setting the hyperparameters a' and A3, the sparsity of the mask is taken into consid- eration, acting as a bottleneck to filter out non-causal parts.
4
MODEL DESIGNS
4.1 ARCHITECTURE
As shown in Figure our proposed model generate the mask distribution po Po(M (M X) from the DNA
sequences and epigenomic signals (Xseg: Xsig), and an predictor, go ( @ X), provides gene expression values from the masked sequences Z = M X. Those two modules will be trained together.
Generator. As outlined in Section 3.4 we aim to generate a mask M to identify the critical regions within the DNA sequences. To achieve this, we first learn a soft mask m,, which is a probabilistic representation of each base pair's 'ii' relevance, where m. [0, 1]. The soft mask is modeled using the Beta distribution, whose parameters a1 02 B1, and 32 3a-are
are estimated from the combination of DNA sequences and epigenomic signals, as detailed in Proposition! 2

---

Published as a conference paper at ICLR 2025
Input
Generator
X 60y
a,
P(m, X) - Bela
a
X,,
1
g. Y
XOM M
Predictor
Figure 2: Pipeline of proposed architectures. The input data contains the DNA sequence X and epigenomic signals X sig- A deep learning model fo is then applied to Xwy to learn the cor- responding parameters for the Beta distribution 0 while a2, .2 are obtained from X sig in non-parameterized manner. By combining these two beta distributions, p(m,|X) is obtained and used generate the mask for actively interacted regulatory elements. The selected elements are then fed into the predictor model 96 to provide the final target gene expression.
For the parameters derived from the DNA sequences, the neural network fo is used to predict and B1. Specifically, we have
a1 31 fo(X
(8)
where the network fo outputs the L-dimensional parameters an and Bi, with L being the length of the input DNA sequence. Each position in the sequence is associated with a pair of values and 31, which parameterize Beta distribution for that base pair.
For the parameters related to epigenomic signals, we use the intuition that higher signal values increase the likelihood of selecting the corresponding base pair. To capture this, we directly use the epigenomic signal values the parameter a2. which influences the selection weight for each base pair. The parameter $2, representing a selection threshold, is set as a fixed constant. Specifically, we define
02 X B2 C..-
By the above modeling procedure, simplify the modeling process, making the learning of ag and 32 non-parametric while maintaining the influence of signal strength without introducing additional learnalle parameters.
After estimating the parameters, based on Proposition 2 the soft mask m, is sampled from the combined Beta distribution, Pe, (m, (pm,,,mmml sig) Beta(a; 022 1, B1 B2 - 1), which represents the probability of selecting each base pair in the sequence. This probabilistic formulation allows us model the selection process effectively.
Finally, for the hard mask version, a threshold is applied to the soft mask to generate the hard mask, M [l,, 2 Cm), where Cm is the threshold (c.g., 0.5). The hard mask M provides binary decision for selecting or ignoring specific base pairs. Through this approach, we model the mask generation process by leveraging both DNA sequences and epigenomic signals, combining parametric and non-parametric methods for more efficient region selection.
After obtaining the mask M, we apply it the input sequences extract the relevant sub-sequences, represented as M @ X. The extracted sub-sequences are then fed into secondary neural network, denoted by Iø. to estimate the probability distribution of the target gene expression Y. The conditional distribution is expressed 4s (Y [M @ X), where represents the parameters of the network, and M @ X refers to the masked input sequences.
Predictor.
To incorporate epigenomic signals alongside the DNA sequences, the input X is formed by con- catenating the one-hot encoded DNA sequence embeddings with the epigenomic signal values. This combined input allows the model leverage both DNA sequence information and epigenomic sig- nals, enhancing the model's predictive capability during the estimation process.
4.2 OPTIMIZATION
To optimize the loss function introduced in Equation 3 it is essential that every step remains dif- ferentiable allow for gradient- optimization during training. After obtaining the parameters
7

---

Published as a conference paper at ICLR 2025
of the Beta distribution through the neural network pe. generate the soft mask m by sampling from this distribution. To maintain differentiability, we treat the Beta distribution as a special case of the Dirichlet distribution (Figurnov et al.] 2018; Bishop 2006). Using the reparameterization trick, we achieve differentiable sampling from the Dirichlet distribution by first sampling from the Gamma distribution and then normalizing the results (Figurnov et al., 2018). This method ensures that the sampling process remains differentiable with respect to the parameters a and B, allowing for efficient backpropagation and optimization.
During inference, instead of sampling from the Beta distribution, we directly use the expected value of the Beta distribution as the soft mask m, for each base pair. The expected value of Beta distribution with parameters and | is given by B[m,] = a+g which provides a deterministic and efficient way to generate the soft mask without introducing randomness during inference, thus stabilizing the model's s predictions.
For the soft mask version, we multiply the soft mask value. And for the hard mask version, when the soft mask m, is obtained, we need to convert into a hard binary mask to make final selections for each base pair. To retain differentiabilltyy in this process, we apply the straight-through estimator (STE) commonly used Gumbel-Softmax (Jang et al.] 2016). The STE allows us to make the forward pass non-differentiable by applying hard threshold (c.g., setting values greater than 0.5 to and others to 0), while during the backward pass, the gradient is propagated through the soft mask as if it were continuous. This approach ensures that the model can learn effectively while using discrete decisions during the forward pass, preserving differentiability in the overall optimization process.
EXPERIMENTS
5.1 SETTINGS
5.1.1 DATASETS AND INPUT FEATURES
In this study, we aim to predict gene expression by modeling CAGE values, as it serves as key indi- cators of gene expression levels. Since gene expression varies across different cell types, we focus on two well-studied cell types: K562 and GM12878, both commonly used in biological research. The CAGE data are sourced from the ENCODE project (Consortium et al. 2012), and we follow the methodology Lin et al. (2024) to predict gene expression values for 18,377 protein-coding genes.
For the input data, we utilize the HG38 human reference genome to provide the reference DNA sequences. Additionally, the model incorporates several types of epigenomic signals:
DNase-seq data is used to capture chromatin accessibility by identifying regions of the genome that are open and accessible to transcription factors and other regulatory proteins. The signals are extracted from big Wig files, providing genome-wide distributions of chromatin accessibility.
H3K27ac ChIP-seq data is used detect histone modifications, specifically the acetylation of lysine 27 on histone H3, which is often associated with active enhancers and promoters. This data is also extracted from big Wig files provide genome-wide information on histone modification patterns.
Hi-C data is processed to caleulate the contact frequencies between each base pair and the target transcription start site (TSS), following the ABC pipeline method as described by Fulco et al. (2019).
Furthermore, we incorporate additional features such as mRNA half-life and promoter activity from previous studies (Lin al.) 2024), which improve the model's prediction accuracy on gene expres- sion levels. The detailed information about these signals can be found in Appendix A.3
detailed description of data acquisition, preprocessing, and extraction, including downloading. filtering, and alignment, is provided in Appendix A.3
5.1.2 BASELINES
To benchmark our model's performance, we compare against several well-established baselines in gene expression prediction:
8

---

Published as a conference paper at ICLR 2025
Enformer (Avsec et al. 2021): : A widely used deep learning model for gene expression prediction, designed to capture rongerange interactions across DÑA sequences. Enformer employs the CNN and transformer architecture to model the DNA sequence to get the gene expression.
HyenaDNA (Nguyen et al. 2024): A cutring-edge method for modeling long DNA sequences, building the Hyena (Poli al., (0023) operator, which introduces novel way to handle long- range dependencies efficiently. HyenaDNA is designed to maintain high accuracy while signifi- cantly reducing computational complexity compared to traditional transformer-basee models.
Mamba (Gu & Dao 2023): A long-sequence modeling approach based on the state space model (SSM), offering linear computational complexity. Mamba is specifically tailored for efficiently handling long sequences, making it highly scalable while retaining strong predictive performance. Caduceus (Schiff et al., 2024): The state-of-the-art model for long genomic sequence modeling. built upon the Mamba architecture. Caduceus is optimized for learning rich representations of genomic sequences. In our study, we utilize Caduccus- A classification layer appended to evaluate its performance on specific task.
By-CCottact (ABC) model (Fulco et al. (219)) for gene expression prediction. EPInformer utilizes DNase-sed seq peak data to define potential regulatory regions and applies an attention mechanism to aggregate enhancer signals. By leveraging both epigenomic and spatial information, EPInformer effectively models the enhancer information for gene expression prediction.
EPInformer (Lin et al.) 2024): A recently developed model extends the
Note that the size of the field of view of Enformer, HyenaDNA, Mamba, Caduceus long- range DNA sequence, while the EPInformer is the extracted potential enhancer candidates based on DNase-seq measurement following ABC model [5]. Moreover, our proposed Seq2Exp also has the view of long-range DNA sequence, since the generator will take the long-range DNA sequence to extract the relevant sub-sequences for the prediction.
5.1.3 EVALUATION METRICS
We employ the following evaluation metrics to assess the performance of our model and baselines on the gene expression regression task: Mean Squared Error (MSE) measures the average squared dif- ference between the predicted and target gene expression values, capturing large deviations strongly. Mean Absolute Error (MAE) evaluates the absolute differences between predicted and actual val- ues, providing more direct measure of average prediction error. Pearson Correlation is used to assess the linear correlation between the predicted and actual gene expression values, reflecting how well the model captures the relative ordering of gene expression. While MSE and MAE focus the absolute errors predictions, Pearson Correlation assess the model's ability to capture relative ranking and overall trends in the data.
5.1.4 IMPLEMENTATION DETAILS
We evaluate model performance using cross-chromosome validation strategy. The model is trained on all chromosomes except those designated for validation and testing. Specifically, chromosomes 3 and 21 are used as the validation set, and chromosomes 22 and X are reserved for the test set. The inclusion of chromosome X is particularly challenging due to its unique biological characteristics compared to autosomes, thus providing more stringent test of the model's robustness.
Both generator Pe and predictor is are based on Caduceus architecture (Schiff et al. 2024), )), an advanced long-sequence model considering the bi-direction and RC-equivariance for DNA. Specif- ically, we train for 50,000 steps on a 4-layer Caduceus architecture from scratch with a hidden dimension of I 128, and more hyperparameters can be found in the Appendix A.4
All experiments were conducted on a system equipped with an NVIDIA A100 80GB PCIe GPU.
The input sequences consist of 200,000 base pairs, centered around the promoter regions of the target genes, providing sufficient contextual information for accurate gene expression prediction.
5.2 RESULTS OF GENE EXPRESSION PREDICTION
Table I presents the gene expression results based on CAGE values. Enformer, HyenaDNA, Mamba, and Caduceus are all DNA sequence-baid methods, relying solely on DNA sequences without incorporating epigenomic signals. Among these, Caduceus achieves the best performance. We

---

Published as a conference paper at ICLR 2025
Table I:
Performance on Gene Expression CAGE Prediction. The top performance over all the
methods are highlighted in bold. Underline indicates that the best performance over all the baselines.
K562
GM12878
MSE
MAE 4
Pearson
MSE
MAE
Pearson f
Enformer
0.2920
0.4056
0.7961
0.2889
0,4185
0.8327
HyenaDNA
0.2265
0.3497
0,8425
0.2217
0.3562
0.8729
Mamba
0.2241
0.3416
0.8412
0.2145
0.3446
0.8788
Caduceus
0.2197
0.3327
0.8475
0.2124
0.3436
0.8819
Caduceus w/ signals
0.1959
0.3187
0.8630
0.1942
0.3269
0.8928
EPInformer
0.2140
0.3291
0.8473
0.1975
0.3246
0.8907
Seq2Exp-hard Seq2Exp-soft
0.1863
0.3074
0,8682
0.1890
0.3199
0.8916
0.1856
0.3054
0.8723
0.1873
0.3137
0.8951
Table 2: Comparison with MACS3 Gene Expression CAGE Prediction.
K562
GM12878
MSE L
MAE L
Pearson 1
Mask Ratio
MSE L
MAE L
Pearson 1
Mask Ratio
Seq2Exp-hard 0.1863
0.3074
0.8682
6.88%
0.1890
0.3199
0.8916
6.32%
Seq2Exp-retrain
0.1979
0.3168
0.8623
10.00%
0.1887
0.3177
0.8941
10.00%
MACS3
0,2195
0.3155
0.8435
13.61%
0.2310
0.3651
0.8634
15.95%
further evaluate Caduceus by incorporating epigenomic signals, concatenated with the one-hot DNA sequence embeddings as input features. EPInformer, which explicitly extracts enhancer regions based DNase-seq measurements, outperforms other baselines. This highlights that selecting key regions based on epigenomic signals yields better results.
Finally, proposed model, Seq2Exp, achieves the best performance overall. By using the Ca- duceus sequence model as both the generator and predictor, and incorporating epigenomic signals additional features the predictor, Seq2Exp explicitly learns the positions of regulatory elements from both DNA sequences and epigenomic signals, resulting in superior performance. We propose two versions of Seq2Exp. Seq2Exp-hard is have a binary mask, and Seq2Exp-soft takes use of soft mask values to denote the importance, resulting in an even better performances regarding the CAGE prediction task.
5.3 COMPARISON WITH PEAK DETECTION METHOD
Table compares the performance of Seq2Exp with regions identified through peak calling by MACS3 (Zhang et al. 2008) on DNase-seq epigenomic signals. While DNase-seq is a crucial technique for identifying the positions of regulatory elements, statistical peak-calling methods, such as MACS3, can be considered a simple approach for measuring these elements. Our results show that Seq2Exp significantly outperforms MACS3 in terms of predictive performance. Seq2Exp-hard utilizes a hard binary mask. Seq2Exp-retrain builds on a soft mask, and explicitly select the top 10% of base pairs and retrain the predictor model using only the selected ones. Both models outperform MACS3, suggesting the ability of discovering regulatory elements.
6 CONCLUSION
In this work, we introduced Seq2Exp. a framework for gene expression prediction that learns criti- cal regulatory elements from both DNA sequences and epigenomic signals. By generating binary mask to identify relevant sub-sequences, Seq2Exp reduces input complexity and focuses on key regions for prediction. Our experiments demonstrate its effectiveness identifying important reg- ulatory elements and improving predictive performances, though current evaluations are limited to two cell types and several epigenomic signals.
For the future direction, expanding the framework to more cell types and integrating diverse epige- nomic data will be important for validating its generalizability. Beyond gene expression, applying this approach to other tasks related to regulatory element discovery and sequence analysis presents exciting research opportunities. Developing pretraining models focused on regulatory element ex- traction could also advance the field, enhancing generalization across genomic tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Research reported in this publication was supported in part by the National Institute on Aging of the National Institutes of Health under Award Number UOI AG07010 12 and ARPA-H under Award Num- ber LAYIAX000053 The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.
REFERENCES
Vikram Agarwal and Jay Shendure. Predicting mmna abundance directly from genomic sequence using deep convolutional neural networks. Cell reports, (1),,, 2020.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv: 2016.
Robin Andersson, Claudia Gebhard, Irene Miguel-Escalade Ilka Hoof, Jette Bornholdt, Mette Boyd, Yun Chen, Xiaobei Zhao, Christian Schmidl, Takahiro Suzuki, et An atlas of active enhancers across human cell types and tissues. Nature, 507(7493):455-461, 2014.
Maria J Aristizabal, Ina Anreiter, Thorhildur Halldorsdottr, Candice Odgers. Thomas W McDade, Anna Goldenberg, Sara Mostafavi, Michael S Kobor, Elisabeth B Binder, Marla B Sokolowski, et al. Biological embedding of experience: primer on epigenetics. Proceedings of the National Academy of Sciences, 237636) -22269,, 2020.
Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinska. Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18 (10):1196-1203 2021.
Pau Badia-i Mompel, Lorna Wessels, Sophia Müller-Dott, Rémi Trimbour, Ricardo O Ramirez Flo- res, Ricard Argelaguet, and Julio Rodr-Ro.rig Gene regulatory network inference the era of single-cell multi-omics. Nature Reviews Genetics, 24(11):739-754, 2023.
Joost Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with differentiable binary variables. In 57th Anmual Meeting of the Association for Computational Linguistics, pp. 2963-2977. ACL Anthology, 2019.
Yonatan Belinkov, James Henderson, et al. Variational information bottleneck for effective low- resource fine-tuning. In International Conference on Learning Representations, 2020.
Bradley E Bernstein, John A Stamattyannoooulls,, Joseph F Costello, Bing Ren, Aleksandar Milosavljevic, Alexander Meissner, Manolis Kellis, Marco A Marra, Arthur L Beaudet, Joseph R Ecker, et al. The nih roadmap epigenomies mapping consortium. Nature biotechnology, 28(10): 1048, 2010.
C.M. Bishop. Pattern Recognition and Machine Learning.
Information Science and Statis- tics. Springer, 2006. ISBN 9780387310732. URL https//bookssgoolle.cooo Pid=
KINoQgAACAAI
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning explain: An information-theoretii perspective model interpretation. In International conference ma- chine learning, pp. 883-892. PMLR, 2018.
ENCODE Project Consortium et al. An integrated encyclopedia of dna elements in the human genome. Nature, 489(7414):57,20) 2012.
William Cookson, Liming Liang. Gonçalo Abecasis. Miriam Moffatt, and Mark Lathrop. Mapping complex disease traits with global gene expression. Nature Reviews Genetics, 1843):184-194, 2009.
M.H. DeGroot and M.J. Schervish. Probability and Statistics. Pearson custom library. Pearson Edu- cation, 2013. ISBN 9781 292025009. URL hhtp:://wwwssppooocooooooo//l/la

---

Published as a conference paper at ICLR 2025
Job Dekker, Andrew S Belmont, Mitchell Guttman, Victor O Leshyk, John T Lis, Stavros Lomvar- das, Leonid A Mirny, Clodagh C O' ''shea, Peter Park, Bing Ren, et al. The 4d nucleome project. Nature, 2490 2017.
Valur Emilsson, Gudmar Thorleifsson, Bin Zhang, Amy S Leonardson, Florian Zink, Jun Zhu, Sonia Carlson, Agnar Helgason, G Bragi Walters, Steinunn Gunnarsdottir, et al. Genetics of gene expression and its effect on disease. Nature, 452(7186):423 428, 2008.
Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterization gradients. Ad- vances in neural information processing systems, 31, 2018.
Charles P Fulco, Joseph Nasser, Thouis Jones, Glen Munson, Drew T Bergman, Vidya Subrama- nian, Sharon R Grossman, Rockwell Anyoha, Benjamin R Doughty, Tejal A Patwardhan, et al. model of enhancer-promotee regulation from thousands of crispr perturba- tions. Nature genetics, 51(12):1664-166) 2019.
A. Gelman, J.B. Carlin, H.S. Stern, D.B. Dunson, A. Vehtari, and D.B. Rubin. Bayesian Data Analysis, Third Edition. Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, 2013. ISBN 9781499840955. URL httpss////bbooos.goggg//// gow.gee..w//boossss//////A/AA..0
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
Sarthak Jain, Sarah Wiegrefffe, Yuval Pinter, and Byron C Wallace. Learning to faithfully rationalize by construction. In Proceedings of the Association for Computational Linguistics (ACL). 2020.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, arXiv preprint arXiv: 01 144, 2016.
Alireza Karbalayghareh, Merve Sahin, and Christina S Leslie. Chromatin interaction- aware gene regulatory modeling with graph attention networks. Genome Research, 32(5):930-99) 0440009944,-0 2022.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 107-117, 2016.
Jiecong Lin, Ruibang Luo, and Luca Pinello. Epinformer: scalable deep learning framework for gene expression prediction by integrating promoter-enhancer sequences with multimodal epige- nomic data. bioRxiv, pp. 2024-08, 2024.
Siqi Miao, Mia Liu, and Pan Li. Interpretable and generalizable graph learning via stochastic atten- tion mechanism. In International Conference on Machine Learning, pp. 15524 15543. PMLR, 2022.
Jill E Moore, Michael J Purcaro, Henry E Pratt, Charles B Epstein, Noam Shoresh, Jessika Adrian, Trupti Kawli, Carrie A Davis, Alexander Dobin, et al. Expanded encyclopaedias dna elements in the human and mouse genomes. Nature, 583(7818)))) 19:69997710 2020.
Eric Nguyen, Michael Poli. Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024.
Bhargavi Paranjape, Mandar Joshi, John Thickstun, Hannaneh Hajishirzi, and Luke Zettlemoyer. An information bottleneck approach for controlling conciseness in rationale extraction. arXiv preprint arXiv:2005. 200022005.0005.,, 2020.
Judea Pearl. Causal inference in statistics: An overview. 2009.
Judea Pearl al. Models, reasoning and inference. Cambridge, UK: CombriddgeUiiversitypreess, 19(2):3, 2000.
Michael Poli, Stefano Massaroli, Eric Nguyen. Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pp. 28078.28078. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Aditya Pratapa, Amogh P Jalihal, Jeftrey N Law, Aditya Bharadwaj, and TM Murali. Benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data. Nature methods, 17(2):147-54,, 2020.
Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03224, 2024.
Michael Schubert, Bertram Klinger, Martina Klünemann, Anja Sieber, Florian Uhlitz, Sascha Sauer, Mathew Garnett, Nils Blüthgen, and Julio Saez-Rodriguez. Perturbation genes reveal signaling footprints in cancer gene expression. Nature communications, 9(1):20, 2018
Eran Segal, Yoseph Barash, Itamar Simon, Nir Friedman, and Daphne Koller. From promoter se- quence to expression: a probabillittic framework. In Proceedings the sixth annual international conference on Computational biology, pp. 263-272, 2002.
Tailin Wu, Hongyu Ren, Pan Li, and Jure Leskovec. Graph information bottleneck. Advances in
Neural Information Processing Systems, 33:20437-20448, 2020.
Ying-Xin Wu, Xiang Wang, Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. arXiv preprint urXiv:2201.12877 2022.
Yong Zhang, Tao Liu, Clifford Meyer, Jérôme Eeckhoute, David S Johnson, Bradley E Bernstein, Chad Nusbaum, Richard M Myers, Myles Brown, Wei Li, et al. Model-based analysis chip-seq (macs). Genome biology, 9:1-9, 2008
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DERIVATION OF SEQUENCE AND SIGNAL DECOMPOSITION
For the mask distribution (m(m) (m), we aim to decompose For simplicity, we omit the parameter 0 in the following derivation. By applying Bayes' theorem, we obtain
p(m|X) [amlXsee Xsig)
(Xxssg Xsig (sgg)m))(m)
(10)
p(X Xsig)
x
(Xig m)p(m),
where p(Xsegg seg; Im) p( p(Xseylmm)t ( X sig m) is based on Assumption | and p(Xseg: Xsig) represents the input data, which is independent of the learning process.
Applying Bayes' theorem again p(X (m) and ((Xsigg /m), we have
X) x p(Xseq m mn)pp(Xxss mm)ppm)
p(m| eeyg )p( )p( (Xeeg) p(m] (eig) sigg) p(m)
p(m)
p(m)
(11)
p(m] p(m Xaig)
p(m)
where safely omit segg and sig). For the marginal distribution p(m), make it to be a prior distribution with constant predefined parameters, allowing us to omit it as well. Thus, we derive
p(m|X) x p(m]Xxee (pm) (
(12)
which corresponds to Proposition
A.2 BETA DISTRIBUTION PRODUCT
The probability density function for a Beta distribution is given by
Beta(x; a, B) x-
(13)
Given that both p(m,/ seg) and p(m,/ follow Beta distribution, have
p(m,l X x a: -1 2) 81
(14)
p(m, (Xig)) x x0 202-1(11 - 2x)
Multiplying these distributions yields
p(m,) (xnal(seq)Pde (o Xoig) na 201400-220 21-3)
(15)
Beta(m,; 01 - 02 - 1, + 82 1).
Note that the parameters of Beta distribution must lie within the range (0, oc). thus we require a1 + a2 and 3 + B3 > 1 to ensure a valid distribution.
A.3 DATA PROCESSING
The gene expression is different for different cell types. In this work, we consider the well-studied cell type K562 and GM12878.
CAGE. Cap Analysis of Gene Expression (CAGE) is the primary target for prediction this work. Each gene is assigned CAGE value to quantify its expression level. CAGE is high-throughput sequencing technique primarily used to map transcription start sites (TSS) and quantify gene ex- pression. It provides comprehensive profile of promoter usage and alternative TSS across different genes, quantifying the number of RNA molecules initiating at each TSS, thereby reflecting gene transcriptional activity.
14

---

Published as a conference paper at ICLR 2025
Table 3: Hyperparameter values and their search space (final choices are highlighted in bold).
Hyperparameters
Values
# Layers of Generator Layers of Predictor Hidden dimensions
4
128
Or3, A3
[1,9], [10,90], [10, 190), [10, 10), [10, 1.11]
# training steps Batch size
50000, 85000
8
Learning rate
le 1c-3. 3, 5c-4, le 4, 5e
Scheduler strategy
CosineLR with Linear Warmup le-5
Initial warmup learning rate
Min learning rate
le-4
Warmup steps
5,000
Validation model selection criterion
validation MSE
In this study, we use CAGE data from the FANTOM5 project (Andersson et al., 2014] (K562: CNhs1 GM12878: CNhs12333), We follow the procedures outlined in Agarwal & Shendure (2020) and Lin et al. (2024) to derive the target values for each gene.
DNase-seq. DNase-seq (DNase hypersensitive site sequencing) is technique used to identify regions of open chromatin within the genome. It pinpoints areas that are less compacted by nucleo- somes. typically corresponding to promoters, enhancers, and transcription factor binding sites. The value derived from DNase-see seq represents the frequency of DNase cleavage at specific sites, with higher values indicating regions that are more accessible to regulatory elements.
We obtained the DNase-seq data from the ENCODE project (Consortium et al. 2012) (K562: ENCFF4140GC; GM12878: ENCFF960FMM), We directly downloaded the data in big for- mat, as it provides genome-wide distribution of DNase-seq values.
H3K27ac. H3K27ac refers the acetylation of lysine 27 on histone H3. trat-tranllaaiinnl modifi- cation associated with active enhancers and promoters. High levels of H3K27ac in a genomic region indicate that it is likely active enhancer promoter, playing significant role in gene expression regulation.
We also obtained H3K27ac data from the ENCODE project (Consortium al. 2012) (K562: ENCFF465GBD; GM12878: ENCFF798KYP), again in big format, which provides the value distribution across the genome.
Hi-C. Hi-C measures the three-dimensional (3D) organization of genomes by capturing physical interactions between different regions of DNA. This technique helps researchers understand how DNA is folded and structured within the nucleus. Hi-C data provides information about genome contacts, but due to technical limitations, it often has low resolution (typically 5,000 base pairs), meaning we can only observe interactions between two regions of DNA of at least this length.
In this work, we follow previous studies (Fulco al., 2019). calculating the frequency of contacts between a specific region (TSS) and all other regions, generating a Hi-C frequency distribution across the genome.
The Hi-C data were sourced from the 4D Nucleome project (Dekker et al. 2017) (K562: 4DNFI- TUOMFUQ; GM12878: 4DNFILUEGIHD).
mRNA half-life and promoter activity features. When predicting the CAGE values, following the implementation of Lin et al. 2202)), we use the promoter activity feature and mRNA half-life features supplementary for fair comparison and further improvement. The promoter activity feature is defined as the square root of the product of DNase-seq and H3K27ac signal values. The mRNA features include G/C contents, lengths of functional regions, intron length, and exon junction density within the coding region. Specifically, the features are
The trantransforeed z-score of the 5' UTR (untranslated region) length.
The log-transformed z-score of the CDS (coding sequence) length.
15

---

Published as a conference paper at ICLR 2025
Table 4: Performance on Gene Expression CAGE Prediction with Standard Deviation for Cell Type
K562.
MSE І
MAE 4
Pearson
Enformer
0.2920 + 0.0050
0.4056 + 0.0040
0.7961 + 0.0019
HyenaDNN
0.2265 0.0013
0.3497 + 0.0012
0.8425 + 0.0008
Mamba
0.2241 0.0027
0.3416 + 0.0026
0.8412 0.0021
Caduceus
0.2197 + 0.0038
0.3327 + 0.0070
0.8475 = 0.0014
Caduceus w/ signals
0.1959 + 0.0036
0.3187 + 0.0036
0.8630 + 0.0008
EPInformer
0.2140 + 0.0042
0.3291 + 0.0031
0.8473 = 0.0017
MACS3
0.2195 0.0023
0.3455 + 0.0018
0.8435 + 0.0013
Seq2Exp-hard
1863 0.0051
0.3074 + 0.0036
0.8682 + 0.0045
Seq2Exp-soft
0.1856 + 0.0032
0.3054 + 0.0024
0.8723 + 0.0012
Table 5: Performance on Gene Expression CAGE Prediction with Standard Deviation for Cell Type GM12878.
MSE |
MAE 4
Pearson
Enformer
0.2889 0.0009
0,4185 = 0.0013
0.8327 0.0025
HyenaDNA
0.2217 + 0.0018
0.3562 = 0.0012
0.8729 = 0.0010
Mamha
0.2145 0.0021
0.3446 + 0.0022
0.8788 0.0011
Caduceus
0.2124 + 0.0037
0.3436 + 0.0031
0.8819 + 0.0009
Caduceus w/ signals
0.1942 + 0.0058
0.3269 + 0.0048
0.8928 + 0.0017
EPInformer
0.1975 0.0031
0.3246 + 0.0025
0.8907
0.0011
MACS3
0.2340 0.0028
0.3654 + 0.0017
0.8634
0.0020
Seq2Exp-hard
0.1890 0.0045
0.3199 + 0.0040
0.8916 = 0.0027
Seq2Exp-soft
0.1873 + 0.0044
0.3137 + 0.0028
0.8951 = 0.0038
The ong-transformed z-score of the 3' UTR (untranslated region) length.
The GC content of the 5' UTR, expressed as the proportion of G and C bases.
The GC content of the CDS.
The GC content of the 3' UTR.
The oog-transformed z-score of the total intron length for gene.
The exon density within the open reading frame (ORF), reflecting the number of exon junctions per unit length of the ORF
A.4
EXPERIMENT SETUP
Here we present some hyperparameters values and their search space in Table 3
A.5
EXPERIMENT RESULTT
Based Table I and Table here we present the experimental results with standard deviation in Ta-
ble/4and and Table The results are from five runs of different random seeds: (2.22.222,2222,222222)
16


Reviews:
[{"review_id": "jKlrZhVwsk", "reviewer": "Reviewer_hbLE", "summary": "In this paper the authors have highlighted the importance of gene expression as the fundamental process in biology, and have introduced a framework for predicting gene expressions by leveraging causal relationships between DNA sequences, epigenomic signals and regulatory elements or motifs. They have provided viability of this approach over two cell types and also state that adding the \"extra info\" in form of epigenomic signals is not always useful and may act as noise in some prediction tasks. To agree on generalization of this approach it has to be tested on more cell types and diverse epigenomic data.", "questions": "Table 1 shows good comparison with previously available benchmark models and datasets. However, looking at EPInformer row, my question is did the authors run the EPInformer code or model over the CAGE prediction data? Because I only saw Pearson metric in the EPInformer paper. How were they able to get MSE, MAE, R2 numbers for comparison?\n\nIn the same table, GM12878 CAGE prediction numbers between EPInformer and Seq2Exp are exactly the same at two decimal places (not a significant improvement). This also indicates that the approach presented in the two papers is very similar. Neither one is better than the other. Originality in terms of problem definition and solution is also similar.\n\nAlso, some numbers do not match between the EPInformer row in this paper and the original EPInformer paper. For example, in the original EPInformer paper, figure 2C, Pearson coefficient for K-562 on CAGE-seq hold out test set is 0.848. But, the same info in Seq2Exp paper, table 1, says is 0.8517. How come?\n\nUpdate: Questions / Concerns were settled by reply from the authors. Thanks!", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "When I compare this paper with other benchmarks (except EPInformer) in the field and for the same problem definition - I would rate this paper as high in terms of originality & significance. Yes, this work shows that gene expression prediction is not a siloed process, and that there are causal relationships between DNA sequences, epigenomic data, regulatory elements, and surrounding causal / non-causal parts of the input data. Also, deep learning algorithms can learn from these relationships and perform better gene expression predictions.", "weaknesses": "This follows from my comments on the strengths of the paper. After reading the EPInformer paper (released in early 2024), my rating reduces. Was this idea a new approach? I respectfully decline. EPInformer highlights the same intent, experimental approach and even comes very close in performance benchmarks & standard metrics. Not sure if this paper has added anything new for the ICLR community? To me as a reader, it has only reinforced the claims of EPInformer. I would have been happier if they can show viability of this approach over multiple cell types. But, data availability may be a limitation here.\n\nUpdate: Thanks to the authors for reviewing my feedback and responding with relevant updates in the paper and their explanation. My concerns are settled. I would like to update my rating for this paper accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HejRtowXPJ", "reviewer": "Reviewer_KQDw", "summary": "The paper is aimed at predicting gene expression from DNA sequence. In addition to the sequence itself, it also incorporates epigenetic information at each based. The model consists of two MAMBA-based Caduceus networks: the first network is trained to estimate a sparse mask over the bases to select regulatory regions, and the second one uses the selected DNA regions to predict the expression of the gene.", "questions": "Details about the network training (and pre-training, if used), including architectural hyperparameters, as well as training procedure details (# steps, etc).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper showcases a successful application of the recently-proposed Selective State Space models (specifically, MAMBA-based Caduceus) on relatively long sequences (200k). \n\nThe model goes beyond most current approaches by combining sequence with epigenetic signaling in a single, jointly optimized architecture. The architecture is sound, and uses information bottleneck principle to promote sparsity of the mask for selecting sequence regions, and then a straight-through estimator to allow differentiability through the hard mask being produced by the first network.\n\nExperimental evaluation shows that the proposed architecture leads to improved prediction, in comparison with Transformer- and State-space-based models.", "weaknesses": "While the manuscript is mostly well written, it could use a more detailed explanation in several places, specifically related to:\n- gene expression Y: it is often just described as \"target variable Y\", or \"$Y \\in R$. From 5.1.4 we read that ultimately the model is applied to a 200kbp region around a specific target gene, and one can understand that Y is the expression of that gene; it would be helpful to provide this information clearly earlier in the paper. \n- model architecture: it also deferred to 5.1.4, and only described in one sentence (Caduceus-Ph). One would expect it to be in 4.1. entitled \"Model architecture\". Also, key details are missing: what are the specific hyperparameters (#layers, hidden dimension, etc.)? Is this a network trained de-novo for the expression task, or pre-trained (Caduceus has pretrained models of up to 131kbp, this one is 200kbp, if it was pre-trained by the authors, more details should be provided).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ckaWfgPYVe", "reviewer": "Reviewer_AbGL", "summary": "This paper describes a model, Seq2Exp, that predicts gene expression values from two inputs: DNA sequence and a corresponding matrix of epigenomic measurements.   A key component of the model is a binary masking strategy that aims to automatically identify relevant regulatory elements in the DNA sequence.  The model uses an information bottleneck approach to try to ensure accurate extraction of relevant sequence signals.  The method is compared to a variety of state-of-the-art methods and yields better performance according to a variety of measures.", "questions": "A priori, I would expect the true binary mask to be \"lumpy,\" i.e., groups of nearby base pairs should tend to be either relevant or irrelevant, reflecting the fact that, e.g., TFs tend to bind to regions rather than single base pairs.  How does your model handle this?\n\nSection 5.1.2 does not make clear whether the competing methods were trained using your cross-chromosome train/test splits.  Is that the case?  I think you should clarify this point in the paper and also discuss how it may impact your results.  If you trained from more or a different type of data than your competitors, the comparison might be unfair.  OTOH,  if your competitors trained on data from your test chromosomes, then it might be unfair in the other direction.\n\nI like the idea of Section 5.3, but I had trouble understanding exactly what was done here. MACS3 is a peak caller, so what is the model that predicts expression in this setting?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The main idea is well articulated and sensible.\n\nThe paper is very clearly written.\n\nThe related work section does a reasonable job of covering the recent, relevant literature.\n\nThe model is well formulated and modeling choices are well motivated in the text.\n\nThe empirical results are strong.", "weaknesses": "I think the sentence \"Gene expression prediction is one of the fundamental tasks in bioinformatics.\" should be supported with a citation or two from some of the seminal works in this area, e.g., Nir Friedman's 2000 paper on predicting gene expression from promoter sequences.\n\nThe task description should be expanded.  It's not clear, from the description, how the inputs are registered to the output value Y.  Is the idea that the length-L input window is centered on the TSS?  To my mind, it would be clearer to represent Y as a vector over genes, and then include some formalism to represent the indexing of genes relative to X_{seq} and X_{sig}.\n\nLine 420 (\"Furthermore, we incorporate additional features such as mRNA half-life and promoter activity ...\") is problematic because it sounds like this is an incomplete list.  For reproducibility, all inputs should be precisely described.  These features are not mentioned at all in Appendix A.3.\n\nI don't see any point in using both R and R^2 as two distinct performance measures.  One is just the square of the other.\n\nMinor point, but I think you should mention in the description of the baseline methods the size of the field of view of each one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TCIkBBubep", "reviewer": "Reviewer_XARj", "summary": "This paper deals with integrating genomic and epigenomic data to extract relevant regulatory DNA regions to predict gene expression, through the learning of a mask on DNA base pairs. The mask is then applied on the data, base-pair-position wise, and the masked data is used to train the predictor for gene expression. The paper describes the corresponding model in detail, and tests it alongside several baselines on genomic and epigenomic data.", "questions": "- What values were used for $\\alpha_3$, $\\beta_3$ in the experiments?\n- Hi-C data usually gives access to a contact matrix between regions of length at least 5,000 base pairs. The appendix mentions that the paper “calculat[es] the frequency of contacts between a specific region (TSS) and all other regions, generating a Hi-C frequency distribution across the genome.” (page 15) What is exactly concatenated to the remainder of epi/genomic data? A matrix of size #base pairs x #regions that contains the frequency of contacts between the region to which a base pair belongs and all other regions? Doesn’t the number of regions change across chromosomes? Or is it a fixed number in your implementation?\n- At page 8, “we incorporate additional features such as mRNA half-life and promoter activity from previous studies”. Where are those features added? To train the gene expression predictor $g_\\phi$? Are they also added to all other baselines, and if so, at which point in their model of gene expression? \n- Why don’t you use a neural network for also learning the parameters of the beta distribution for epigenomics ($\\alpha_2$, $\\beta_2$)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Quality: The submission seems technically correct and experimentally rigorous (however, I did not check the appendix in detail). The experimental results are convincing and appropriately compared to the numerous baselines.\n- Clarity: It is well-grounded in the literature. The paper is well written and the contribution is clear. \n- Originality: The paper proposes a novel decomposition of regulatory regions into three groups, that have different causal relationships to genomic and epigenomic data.", "weaknesses": "- Quality: The submission is not reproducible, as no code is provided (nor is the availability of the code discussed in the main paper nor in Appendix). There is no mention of hyperparameter tuning or value selection, especially for $\\alpha_3$ and $\\beta_3$, which parametrize the distribution of the soft mask on the regulatory regions. Performance is important for these methods, but I think runtimes across baselines and Seq2Exp should also be reported for practical purposes.\n- Clarity: Some parts are not clear, for instance, the inclusion of additional data in the experiments, or the use of Hi-C data (see questions). \n- Significance: The assumption of independence of selection of regulatory regions across base pair positions at fixed (epi)genomic data (assumption written at the bottom of page 4) might be a bit too strong for the real-life data (for instance, for CpG islands -although, no methylation peaks are considered in these papers). I understand that it simplifies the computations and allows for a tractable model training, but perhaps it should be discussed further in the paper. \n\nThe non-reproducibility and the unclear parts are the main reasons why I rate this paper 6 instead of 7-8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)