Extending Sequence Length is Not All You Need: Effective Integration of Multimodal Signals for Gene Expression Prediction
Authors: Zhao Yang
Yi Duan
Jiwei Zhu
Ying Ba
Chuan Cao
Bing Su
Venue: ICLR 2026 Oral (2026)
Keywords: dna language model
gene expression prediction
multimodal information integration
causal intervention
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Gene expression prediction, which predicts mRNA expression levels from DNA sequences, presents significant challenges. Previous works often focus on extending input sequence length to locate distal enhancers, which may influence target genes from hundreds of kilobases away. Our work first reveals that for current models, long sequence modeling can decrease performance. Even carefully designed algorithms only mitigate the performance degradation caused by long sequences. Instead, we find that proximal multimodal epigenomic signals near target genes prove more essential. Hence we focus on how to better integrate these signals, which has been overlooked. We find that different signal types serve distinct biological roles, with some directly marking active regulatory elements while others reflect background chromatin patterns that may introduce confounding effects. Simple concatenation may lead models to develop spurious associations with these background patterns. To address this challenge, we propose Prism, 
a framework that learns multiple combinations of high-dimensional epigenomic features to represent distinct background chromatin states and uses backdoor adjustment to mitigate confounding effects. Our experimental results demonstrate that proper modeling of multimodal epigenomic signals achieves state-of-the-art performance using only short sequences for gene expression prediction.

Published as a conference paper at ICLR 2026
EXTENDING SEQUENCE LENGTH IS NoT ALL YOU NEED: EFFECTIVE INTEGRATION OF MULTIMODAL SIG- NALS FOR GENE EXPRESSION PREDICTION
Zhao Yang1234 1 Duan Jiwei Zhu
Ying Ba
Chuan Cao² Bing Su
'Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
"Zhongguaneun Academy, Beijing, China
*Beijing Key Laboratory of Research on Large Models and Intelligent Governance
"Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE
ABSTRACT
Gene expression prediction, which predicts mRNA expression levels from DNA sequences, presents significant challenges. Previous works often focus on extending input sequence length to locate distal enhancers, which may influence target genes from hundreds of kilobases away. Our work first reveals that for current models, decrease performance. Even carefully designed
long sequence modeling algorithms only mitigate the performance degradation caused by long sequences. Instead, we find that proximal multimodal epigenomic signals near target genes prove more essential. Hence we focus on how to better integrate these signals, which has been overlooked. We find that different signal types serve distinct biological roles, with some directly marking active regulatory elements while others reflect background chromatin patterns that may introduce confounding effects. Simple concatenation may lead models to develop spurious associations with these background patterns. To address this challenge, we propose Prism, a framework that learns multiple combinations of high-dimensional epigenomic features to represent distinct background chromatin states and uses backdoor adjustment to mitigate confounding effects. Our experimental results demonstrate that proper modeling multimodal epigenomic signals achieves state-of-the-art performance using only short sequences for gene expression prediction.
INTRODUCTION
Understanding and predicting gene expression is fundamental to deciphering the complex regulatory mechanisms governing cellular functions (Pratapa al. 2020). Accurate gene expression predic-
tion enables breakthroughs across biomedicine (Mamoshina et al. 2016). from unraveling disease pathogenesis (Cookson et al. 2009 Emilsson et al.) 2008) and enabling personalized therapeutic strategies (Blass & Ot 2021 to guiding the design of specialized regulatory elements (Lal et al.
2024
Yang et al. 2025a).
However, accurately predicting gene expression presents significant challenges. First, gene expression depends on regulatory elements that can be located hundreds of thousands of base pairs (bps) away (Schoenfelder & Fraser 2019) (Figure (a)), which naturally requires models capable of
processing long DNA sequences (Figure I (b)) Avsec al. 202T Nguyen al., 2023 [Schiff et al.
2024 Su et al. 2025). Additionally, gene expression is a cell-type specific process (Shen-One Orr et al. 2010) that difficult to predict precisely using cell-shared DNA sequences alone, necessitating the integration of cell-type specific information such as histone modifications, chromatin accessibility.
and other multimodal epigenomic signals (Lin et al. 2024 Su et al. 2025) (Figure (c)).
* "Equal contribution.
I Work was done during Zhao Yang's internship at Zhongguancun Academy.
1 "oorreponndence bingsu@rue.cdu..o

---

Published as a conference paper at ICLR 2026
(a)
DNA ong raagge Interaction
(b) Current seg-eeqeence Models
M Multimodal Signals M Matler
2014
XB4
E1
TSS
Exhearr
SSM Longraauannt
118
Wesk
BAC:
Frender
Xleemmcyy
V'xx Biddenn
Blas
States
Degradation
(d)
Model Performance by Input. Length (e)
Cadeceus Performance
(1)
Cadeceus Performance
(Consistent TeiningTeettng)
(Pull Signals Triining Abblttdd Signal Testing)
0.875
0.620
0.86-
0.65
0002
0.8467
0.865
0.860
0.85-
0.0465
6.655
0.84-
0.850
0.83
0.70
0.6964
0.865
130
200000
0.82
Signel
NUCHE
w Dhoke
w/C
Spnadd
aet AB Signalc
0.5
Drop H3E27cc
Dtup DWake
Drop HI-C
Sequence Length
Figure 1: (a) Long-range regulatory interactions through chromatin looping. (b) Current long- sequence models suffer from technical limitations. (c) Multimodal epigenomic signals provide cell-type specific regulatory information. (d) Performance of Seq2Exp (Su et al. 2025) and Ca- duceus (Schiff et al. (024) with varying input sequence lengths. (c) Different signals show varying contributions. (f) Performance degradation when specific signals are removed during testing from model trained with all signals.
Previous works primarily focus modeling long sequences. However, through simple but insightful experiments, we demonstrate that these methods merely mitigate the performance degradation inherent in current long-sequence modeling paradigms (Figure dd) details in Section 2 In contrast, using short sequences already achieves excellent results, especially when combined with multimodal epigenomic signals. We attribute the effectiveness of short sequences to the fact that proximal epigenomic signals reflect the activity of distal regulatory elements through chromatin looping and spatial interactions (Plank Dean 2014). As shown in aa) although enhancers and genes are separated by large distances, some epigenomic signals near the gene can reveal the regulatory influence of these distal elements.
State-of-the-art (SOTA) methods (Lin et al. 2024: Schiff et al. 2024 Su et al. 2025 utilize epige-
nomic signals through simple concatenation (F (Figure I(c) without considering their distinet biological
roles. We conducted study characterizing the differential contributions of various epigenomic signals by training Caduccus (Schiff et al.) 2024) with DNA sequence alone and with individual signals (H3K27acc DNase-seq. Hi-C) or all combined. Figure (e) shows each signal improves performance, with H3K27ac providing the most substantial enhancement. This aligns with biological understanding: H3K27ac directly marks active regulatory elements (Creyghton et al. 2010),, func- tioning as a foreground signal, while DNase-seq and Hi-C serve as background signals indicating chromatin accessibility (Thurman et al.) (012) and organization (Rao al. 2014).. Models trained on all signals performed comparably to H3K27ac alone, indicating background signals provide limited incremental benefit beyond foreground signals.
Figure f reveals a critical paradox: removing background signals during testing from models trained on all signals causes severe performance degradation. While background signals provide minimal standalone improvement, models develop over-dependence during training. This asym- metric behavior indicates these background patterns introduce confounding effects. The underlying mechanism stems from spurious correlations in training data, where gene expression systematically co-occurs with open chromatin patterns, causing models to learn non-causal associations between accessibility and expression levels. However, gene expression occur independently of chromatin accessibility (Volpe et al. 2002), and our case study (Appendix demonstrates high expression in regions with limited accessibility, substantiating the spurious correlation hypothesis.
2

---

Published as a conference paper at ICLR 2026
To address these confounding effects, propose a simple yet effective approach, Prism (Proximal regulatory integration of signals for mRNA expression levels prediction), that learns multiple combinations of high-dimensional epigenomic features to represent distinct background chromatin states (Qiang et al. 2022). Each learned combination corresponds to a specific background state. We then apply backdoor adjustment (Pearl 2009) to perform causal intervention across these states, thereby mitigating confounding effects and enhancing the model's predictive performance.
We summarize our contributions here:
We challenge current approaches that use long sequence modeling for gene expression pre- diction, which, while biologically plausible, may not yield improvements due to limitations of present technical tools.
We systematically analyze the differential roles of various epigenomic signals and identify that background chromatin patterns may introduce confounding effects, leading models to learn spurious associations.
From causal perspective, propose Prism, approach that learns high-dimensional fea- ture combinations to represent background chromatin states and applies backdoor adjustment mitigate confounding effects.
Through extensive experimentation, we demonstrate the effectiveness of our approach, achieving state-of-the-ar performance using only short sequences through a simple and effective method.
2 CURRENT METHODS DO NOT BENEFIT FROM LONG SEQUENCE INPUT
Mainstream deep learning methods for gene expression prediction focus on extending model input length. However, since the context length that can influence gene expression is extremely long (up to IM bps (Avsec et al.] 2025)), quadratic-complexy Therefore, previous works have adopted alternative approaches beyond traditional Transformers, primarily falling into two categories.
Transformers cannot handle such sequences.
The first category comprises CNN-Transformer hybrid models, which first downsample long se- quences into low-resolution bins through convolutional neural networks (CNNs), then employ Transformers to model these low-resolution bins (Avsec al.] [021]] [inder al.. 2025| Avsec
et al. 2025). These works follow Enformer (Avsec et al. 2021) in performing 1 128-fold down- sampling, resulting the loss of single- resolution, which is sub-optimal for DNA data where single-base variations (Avsec et al.) 2025) can have profound biological impacts. Although Enformer performs well in multi-task prediction, [Su et al. (2025) revealed that it underperforms compared to single-nucleotide modeling approaches like Caduceus (Schiff et al. (0244 on specialized gene expression prediction tasks. Similarly, recent work focusing on personalizee iize gene expression prediction (Li al., 2025) demonstrated that these approaches perform worse than Caduceus when predicting gene expression in unseen individuals. Therefore, we conclude that for gene expression prediction tasks with specialized training data, maintaining single-nucleotide resolution is crucial.
Another class leverages neural networks with linear complexity, primarily the recently popular state space models (SSMs) (Gu & Dao 2023 Nguyen et al., [2023] [Schiff et al. [2024] Nguyen et al. (2024). which directly model long sequences at single-nucleotide resolution. Recently, Seq2Exp (Su et al. 2025) achieved SOTA results gene expression prediction by introducing learnable masks top of Caduceus (Schiff et al. 2024), whose motivation is to learn to focus SSMs on important regulatory elements, pushing SSM-based methods to SOTA performance.
In this work, we first challenge the prevalent approach of using linear-complexity SSMs for single- nucleotide resolution long sequence modeling (Schiff et af. 2024 Su al. (025). These methods typically evaluate their effectiveness on long sequences only. For instance, Seq2Exp (Su et al.) 2025) tested exclusively on 200K-length sequences and demonstrated superior performance over existing methods, thereby claiming enhanced long-sequence modeling capabilities. However, current SSMs merely offer computational efficiency advantages with linear complexity when processing long sequences, while their actual modeling performance remains questionable (Figure (b)). Specifically, (1) SSMs have fixed-size hidden states (Gu & Dao 2023), making difficult to memorize all information in long sequences. (2) Wang al. 2025)
indicates that SSMs exhibit a strong recency
3

---

Published as a conference paper at ICLR 2026
bias, meaning tokens in the sequence primarily interact with their nearby context. This contradicts the goal of gene expression prediction, which aims to model the relationship between target genes and distant regulatory elements.
Hence, we conducted a preliminary study to validate whether SSMs can truly handle long sequences ef- fectively. Specifically, we trained Caduceus (Schiff) al., 2024) and Seq2Exp (Su al., 2025) with vary- ing input lengths centered at the transcription start site (TSS) for gene expression prediction, completely following the experimental settings Su et al. (2025) except for sequence length. According to Figure d)) we observe that Caduceus's 's performance consistently declines after input lengths exceed 2k. Seq2Exp, de- spite its carefully designed Marning-to-mass mecha- nism for filtering unimportant regions, doesn't I show a clear downward trend, but its performance with 200k input length remains essentially comparable to using just 500 bps. Figure 2 (raw data in Table 4 demon- strates that the Seq2Exp model trained on 200k se- quences maintains nearly identical performance even when input sequences are shortened to 2.5k during the testing phase, suggesting that even Seq2Exp trained on long sequences fundamentally relies only on proximal information. Therefore, rather than extending sequence length, we focus on better leveraging multimodal epigenomic signals longstanding overlooked direction for enhancing prediction performance.
2.5
0.3
AMRE 0.034
5.2
2055
Sequencce Staio)
Figure 2: Shortening input length at test time.
METHOD
3.1
PROBLEM FORMULATION
Given gene sequence X T2........ æL], where for each i E (1,2.... L}, E IR represents {A.T, C, G), and L denotes the
the one-hot encoding of a nucleotide base from the sequence length surrounding the gene's TSS (Lin et al. 2024 Su at. 2025). For each X, there are associated multimodal epigenomic signals S [81, 82, S... where 8; E IRd with d representing the number of epigenomic signals. Our approach first employs a signal encoder go : .Rdd + L with parameters A to map the raw epigenomic signals S into nigher-dimensional feature space H = go (S). where d' represents the dimensionality of this enriched representation following (Su et al. 2025). We then use predictor network ha :(T< (RR4x4, RLxdd + R with parameters & that integrates both sequence information X and encoded epigenomic features to predict gene expression levels Y E IR. To optimize our model parameters {0, Φ). we define the following objective function:
L t (h ( X, go (S)), Y).
(1)
where b denotes the smooth LI loss (Huber loss) following Su et al. (025
3.2 STRUCTURAL CAUSAL MODEL
From the previous analysis, we observed that models may learn spurious associations with background epigenomic signals. To conceptualize this con- founding issue, we formalize the problem using Structural Causal Model (SCM) shown in Figure 3 where nodes represent data variables and directed edges represent hypothesized relationships. For clarity, we omit X from the graph, though our model ultimately uses both X and H.
We first explain our definition of confounder C. In Section I we categorize H3K27ac as foreground signal and DNase-seq/Hics as background signals based on biological priors. However, this categorization overly simplistic. H3K27ac alone cannot fully capture causal effects, as incorporating additional signals improves performance (Figure ()). Similarly, background signals cannot be directly defined confounders. Instead, define the confounder as abstract concept: background chromatin
Figure 3: The SCM.
4

---

Published as a conference paper at ICLR 2026
88
S
HIR37
DNose
IBC
Y
X
Figure 4: Architecture of Prism. Epigenomic signals S are processed by two encoders: a signal encoder ge extracts high-dimension epigenomic features H, while confounder encoder In learns distinct weights representing the confounder C. A final predictor he uses these weighted features along with the DNA sequence to make prediction.
states, which represent complex combinations of multiple epigenomic signals. This aligns with approaches like ChromHMM (Ernst & Kellis 2017). which combinatorial patterns of epigenomic signals to define chromatin states across the genome. The specific functional implementation of C' is detailed in Subsection 3.4
This definition is inspired by works in computer vision (Zhou et al. 2016) Yue et al. 2020) Qiang et al. 2022). where confounders (representing image background information) are modeled combinations of figh--dmmensioal semantic feature representations. Specifically, RGB images (analogous to our raw signals S E IR (xxd) are encoded into high-dimensionn spaces (analogous to our H E R'xd') ) where different linear combinations of features can represent various background contexts (Zhou et al. 2016 Qiang et al. 2022). Next, we explain the meaning of the edges in our SCM (Figure
H ++Y Y. High-dimensional epigenomic features H contain comprehensive regulatory information that directly influences gene expression Y.
H C Y. The confounding pathway where background chromatin state simultaneously affects both the observed epigenomic features H and expression levels Y. For instance, globally active chromatin regions often exhibit both high accessibility signals and high expression, creating correlations that may not reflect gene-specific regulation directly.
3.3 CAUSAL INTERVENTION VIA BACKDOOR ADIUSTMENT
An effective prediction model should capture the direct regulatory relationship + Y rather than spurious correlations through the confounding pathway H t Y. However, standard approaches optimize P(Y|H) which conflates both pathways. Our goal is to estimate the interventional distri- bution P(Y|do( (H)) (Pearl et 2016) that isolates the direct causal effect by controlling for back- ground chromatin states C. The do operator represents an intervention that sets H while removing its dependency confounders, enabling isolation of the direct causal effect. We stratify the confounder C' into distinct background chromatin states: C' (C1,C,, Cn where n is hyperparameter. Using backdoor adjustment, formulate: P(Y = PHY|H, C(C C.). For computational tractability, we assume C' follows uniform distribution (Qiang et al. 2022)
P(C = C.)
3.4 FUNCTIONAL IMPLEMENTATION
To functionally instantiate the confounder C, we draw inspiration from methods in computer vision that model background context using learnable weights (Qiang et al. 2022). We introduce confounder encoder 9w : 4xd - xd with parameters w, which processes the raw epigenomic signals to generate set of learnable weight vectors A [a1, .2...., an]. Each vector a, E R represents distinct background chromatin state C, by applying unique weighting scheme across the d' dimensions of the encoded epigenomic features. These weights are gene-wise rather than ssittion-wise reflecting the assumption that background regulatory patterns consistent across
5

---

Published as a conference paper at ICLR 2026
given gene region. For example, weight vector might learn to emphasize chromatin accessibility signals, while another might prioritize features related to 3D chromatin organization.
This data-driven approach allows the model to capture the complex nature of background confounding effects without relying on overly simplistic biological priors. With this implementation, we can compute the interventional distribution from the backdoor adjustment formula by stratifying across these learned background states. Since we assume the DNA sequence is independent of the epigenomic features H/
(Su al. 2025). we include it directly in the predictor:
Y = X, =
Σ P C,) Σ hs ( (2)
where @ denotes element-wise multiplication. Each term h H o a;) represents prediction under specific background context C₁. More details about the functional implementation of he- including the signal-sequee sequnnce fusion strategy and the interventional prediction computation, are provided in Appendix F3
We incorporate this interventional prediction as regularization term (Qiang et al. 2022). forming a second loss component that encourages the model to be robust different background chromatin states:
L2 = LH
Σ 1=1
hx H aa)), Y ))
17
3.5 TRAINING OBJECTIVE
To ensure our model learns a meaningful and diverse set of background chromatin states, we should prevent the weight vectors {a:} from collapsing into single pattern. We introduce uniform loss function [Wang & Isola 2020) that encourages the weight vectors to be distinct from each other. This loss penalizes similarity between background representations, promoting diversity in the learned weights:
Lg log e2p a. 21)
where a; aa/l|ail22 is the L2-normalized weight vector and t is temperature parameter that controls the sharpness of the penalty.
Our final training objective combines the standard prediction loss, the intervention- based regulariza- tion, and the uniform diversity loss:
L1 aL2 + BLs,
where a and 3 are hyperparameters controlling the relative importance of the intervention regular- ization and the uniform diversity constraint, respectively. The complete algorithm workflow for our Prism framework is provided in Appendix E
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. To evaluate gene expression prediction, we adopt Cap Analysis of Gene Expression (CAGE) values as our prediction proxy, in line with established approaches (Avsec et al.
2021 Lin
et al. 2024) Su et al. 2025). Our study focuses on two well-characterized human cell lines: K562
and GM12878. We use CAGE measurements obtained from the ENCODE (Consortium et al. 201
Following the experimental framework established previous studies (Lin et al. 2024 Su ct al. 2025). evaluate model across 18.377 protein-coding genes.
For input data, we utilize both DNA sequences and epigenomic signals. The DNA sequences derived from the human genome HG38 project, while the epigenomic signals were carefully selected (Su et al. 2025) to capture different aspects of gene regulation: H3K27ac marks histone
acetylation at active enhancers and promoters. DNase-seq measures chromatin accessibility in

---

Published as a conference paper at ICLR 2026
genomic regions, often coinciding with but not causally determining regulatory elements. Hi-C quantifies contact frequencies between genomic positions and the target TSS, processed using the ABC pipeline (Fulco et al., 2019). Like DNase-seq, we categorize Hi-C as background signal representing the broader chromatin environment rather than specific regulatory elements.
Furthermore, we incorporate additional features such mRNA half-life and promoter activity, which are taken from previous studies (Lin et al.] 2024 Su et al. 2025 These features are simply concate- nated to the final linear predictor and are not part of our core modeling approach for epigenomic signals.
Baselines. We benchmark our Prism against the following baselines: Enformer (Avsec et al. 2021
a CNN-Transformer hybrid architecture designed to predict epigenomic signals and gene expression from sequences, here used solely for CAGE prediction; HyenaDNA (Nguyen et al. 2002)), Mamba (Gu & Dao 2023), and Caduceus (Schiff et al. 2024), three recently developed DNA foundation models modeling capabilitties through SSMs as prediction backbones; EPInformer (Lin et al. 2024), which extends the Activity-By-Conte
leveraging efficient Long-sequent
(ABC) model (Fulco et al. 2019) by utilizing DNase-seq peaks to define potential regulatory regions and applying attention mechanisms to aggregate enhancer signals; and Seq2Exp (Su et al. 2025), recent SOTA method that applies information bottleneck principles to learn regulatory element masks, available in hard (binary) and soft (continuous) variants. We also include Caduceus w/signal, which incorporates epigenomic signals directly into Caduceus's encoder, and MACS3 (Zhang et al.] 20088, which differs from 2eq by using MACS3-identified regulatory elements instead of learned masks. Most baseline models process raw DNA sequences from the input region, while EPInformer operates on potential enhancer candidates extracted based on DNase-seq measurements following the ABC model (Fulco et al. 2019).
Evaluation Metrics. We assess model performance using three metrics following Su et al. (2025): Mean Squared Error (MSE) for measuring prediction variance with emphasis on larger errors; Mcan Absolute Error (MAE) for quantifying average prediction deviation expression units; and Pearson Correlation for evaluating how well models capture expression patterns and gene rankings regardless of absolute scale. These metries together provide a balanced assessment of both prediction accuracy and pattern preservation capabilities.
Implementation Details. We partition datasets by chromosome for training, validation, and testing, following [Su et al. (2025). Specifically, chromosomes 3 and 21 serve the validation set, while chromosomes 22 and X are reserved for testing. The inclusion of chromosome X provides a more stringent evaluation of model robustness due its distinct biological characteristics compared to autosomes.
Our signal encoder ge is implemented as simple linear layer (Su et al. 2025), while the confounder encoder Iw utilizes a lightweight ID-CNN, with details in Appendix F For the predictor hgg we adopt Caduceus (Schiff et al. 2024) our backbone model, following Seq2Exp (Su et al., (025). Notably, we maintain the same training hyperparameters as in Seq2Exp (Su et al. 2025). Further performance gains could likely be achieved through hyperparameter fine-tuning specific our approach. We use the smooth L1 loss (Huber loss) as our prediction loss function, while the best model is selected based on the MSE metric on the validation set following (Su et al. (2025). All experiments were conducted on NVIDIA A40 and A100 GPUs. While most baseline models process inputs of length 200k, our Prism implementation operates on sequences of 2k bps. Additional implementation details can be found in Appendix F
4.2 RESULTS OF GENE EXPRESSION PREDICTION
Table presents performance results across all methods for the K562 and GM12878 cell types, respectively. All baseline results are directly cited from Seq2Exp (Su et al. 2025) to ensure fair comparison. Additionally, all results reported include the mean and standard deviation from five runs using different random seeds: {2, 22, 222, 2222, 22222] following Su et al. 20255) The best-performing method for each metric is highlighted in bold, with the second-bes best underlined. Notably, our Prism consistently outperforms the previous SOTA Seq2Exp-soft across all datasets and metrics. Among the six total metrics, only K562's ' MAE and Pearson correlation show improvements less than standard deviation, while all other metrics demonstrate robust improvements exceeding
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance on Gene Expression CAGE Prediction with Standard Deviation for Both Cell Types.
K562
GM12878
MSE
MAE 1
Pearson ?
MSE
MAE L
Pearson f
Enformer
0.2920 +: 0.0050
0.4055 1: 0.0040
0.7961 1: 0.0019
0.2689 1: 0.0009
0.4185 1: 0.0013
0.8327 1: 0.0025
HyenaDNA
0.2265 = 0.0013
0,3497 + 0.0012
0,8425 : 0.0008
0.2217 = 0.0018
0.3562 = 0.0012
0.8729 + 0.0010
Mamba
0.2241 0.0027
0.3416 1: 0.0026
0.8412 1: 0.0021
0.2144
0.0021
0.3446
0.0022
0,8788 1: 0.0011
Caduccus
0.2197 = 0.0038
0,3327 + 0.0070
0,8475 4 0.0014
0,2124 = 0.0037
0.3436 = 0.0031
0.8819 + 0.0009
EPInformer
0.2140
0.0042
0.3291 1. 0.0031
0.8473
0.0017
0.1975
0.0031
0.3246
0.0025
0.8907 1. 0.0011
MACS3
0.2195 = 0.0023
0,3455 + 0.0018
0.8435 + 0.0013
0.2340 + 0.0028
0.3654 + 0.0017
0.8634 + 0.0020
Caduceux w/ signals
0.1959
0.0036
0,8630
0.0008
0.1942 0.0058
0.3269
0.0048
0.0017
0.1863
0.0051
0.3074 + 0.0036
0.8682 + 0.0045
0. 1890 + 0.0045
0.3199 + 0.0040
0.8916 + 0.0027
Seq2Exp- soft
0.1856 0.0032
0.3054
0.0024
0,8723
0.0012
1873
0.0044
0.3133
0.0028
0.0038
Prism
0.1789 0.0041 0.3000 + 0.0058
0.8751 + 0.0036
0.1759 + 0.0054
0.3054 + 0.0048
0.9016 + 0.0024
Table 2: Hyperparameter sensitivity analysis for Prism on the K562 cell line. We evaluate the model's performance while varying (a) the number of background states n, (b) the intervention loss weight a, and (c) the diversity loss weight B.
(b) Sensitivity on a
(a) Sensitivity on n
MSE
MAE
Pearson 1
MSB
MAE +
Pearson 1
0.1
0.1829 1 0.0065 0.3037 +: 0.0078
0.8725 +: 0.0030
1.0
0.1789 + 0.0041 0.3000 0.0058 0.8751 + 0.0036
0.1863 0.0035 0.3092 + 0.0050 0.8713
0.0023
10.0
0.1916 1 0.0055
0.3119 0.0071 0.8709 + 0.0029
0.1891 1 0.0047
0.3084 1 0.0039
0.8676 1 0.0032
2 0.1789 + 0.0041
0,3000 + 0.0058 0.8751 = 0.0036
3 0.1818 + 0.0091
0.3018 1 0.0090 0.8739 + 0.0031
(c) Sensitivity B
4
0.1762 + 0.0071
0.2961 + 0.0070
0.8780 + 0.0028
MSE
MAR
Pearson
5
0.1788 + 0.0062
0.2996 = 0.0071 0.8752 + 0.0030
0.1
0.1789 # 0.0056 0.2993 0.0037 0.8757 = 0.0038
6
0.1857
1 0.0078
0,3057 1 0.0047
0.8737 1 0,0022
1.0
0.1789 1 0.0041
0,3000 + 0.0058
0.8751 + 0.0036
10.0
0.1836 + 0.0120 0.3027 + 0.0123
0.8748 = 0.0036
one standard deviation. These results provide strong evidence that our approach achieves new SOTA performance in gene expression prediction.
4.3 HYPERPARAMETER SENSITIVITY ANALYSIS
Our method introduces several hyperparameters: the number of background chromatin states n, and coefficients and 3 that balance the loss components in our training objective (Equation (5). We conducted sensitivity analysis on the K562 cell line, with results (Also averaged results from five runs, here only the mean values are shown) presented in Table 2
Our analysis of n shows that while performance peaks = 4, configurations with n 2 2 substantially outperform the n baseline, validating our intervention; we select 2 to balance performance and efficiency. For the intervention weight a, we found performance is optimal at 1.0 and degrades when either disabled (a ===00 0) or set too high (a = 10.0). This confirms its role as auxiliary regularizer, consistent with prior work (Qiang et al. 2022). Finally, the diversity constraint proves to be robust. The model's s performance is nearly identical for B 0.1 and B 1.0, and shows only slight degradation even with a large weight of B = 10.0.
4.4 ANALYSIS OF LEARNED WEIGHTS
To understand how our model represents the confounder C, we visualize the weights learned by the confounder encoder 9w (Figure 55. The analysis reveals two key properties. First, we observe strong intra-gene diversity: for given gene, the two learned weight vectors (a) and (2) are distinct and often complementary, confirming that our model learns non-redundant representations for each confounder stratum. Second, we find evidence of inter-gent structural similarity. The overall intensity of the learned weights is clearly gene- specific, reflecting each gene's 's unique local epigenomic context. Despite this variation in magnitude, the relative pattern between the two states is remarkably consistent across different genes, suggesting the model learns a generalizable strategy-Such as an
8

---

Published as a conference paper at ICLR 2026
INSG00000006777
anten ALBAMM
Index of
1.0
0.5
16
32
48
64
8O
96
112
128
index of feature dimensian
ENSC00000001497
an M4009
index of C
-1.
0.5
0.0
index of feature dimension
00000008036
anten W.BieM
Index of
-10
0.5
16
32
48
64
80
96
112
128
index of feature dimension
Figure 5: Visualization of learned confounder weights (a1, .2) for three sampled genes.
"activating" versus a "suppressive" state state-whick then adapts to each gene's local context, These structured representations support the validity of our causal framework.
4.5 PARAMETER OVERHEAD
Our confounder encoder is designed to lightweight while delivering substantial perfor- mance improvements. We compare the ad- ditional parameters introduced by Prism and Seq2Exp (Su et al.] (0225) relative to the base Caduceus (Schilf al. (004)). As shown in Ta- 3 Prism adds only 11K trainable parameters the base model. Our lightweight confounder encoder 9u introduces minimal parameter over- head, whereas Seq2Exp's mask generator causes its parameter count to double compared to Caduceus. Notably, approach outperforms Seq2Exp across all metrics while maintaining an almost unchanged parameter count compared to Caduceus.
Table 3: Parameter comparison between models.
Model
Trainable Parameters
Caduceus Seq2Exp
574K
1.IM
Prism
585K
RELATED WORK
Sequenceeto-ffucctoo models are designed to predict functional genomic signals directly from DNA sequences. DeepSEA (Zhou & TTroyankkayal 2015) established this approach by utilizing convolutional neural networks (CNNs) to extract sequence features for multi-task prediction. The field has evolved through architectural innovations and expanded training datasets (Kelley al.) 2018 Zhou et al. 2018 Chen et al.) 2022). Currently, Enformer (Avsec et al. (021) l) represents the leading method, achieving exceptional performance through its hybrid Transtormer-CNN architecture. While these models simultaneously predict various outputs, including epigenomic signals and gene expression levels, they typically lack specialized mechanisms for leveraging epigenomic data to enhance expression prediction, specifically, treating all prediction targets parallel outputs rather than considering their biological interdependencies.
Unsupervised DNA foundation models leverage the successful paradigm of unsupervised pre- training established in natural language processing, DNABERT (Ji et al. 2021] was the first to adapt this approach to genomics, applying BERT-like Devlin et al. (2019) techniques to learn transferable DNA representations. Subsequent models have expanded upon this foundation (Zhou et al. 2024
Dalla-Torre et al. 2024 Li al. 2024 Sanabria et al. 2024). In parallel, generative frameworks like Evo (Nguyen et al. 2024) have emerged (Nguyen et al. 2023
Brixi et al. 2025), enabling functional element design applications (Linder al. 2025 Yang et al. 2025a). Despite these advances, such

---

Published as a conference paper at ICLR 2026
models' effectiveness for gene expression prediction remains limited due their exclusive reliance on DNA sequence information, without incorporating the critical epigenomic context that modulates gene activity.
Gene expression predictioneepreeentt represents a fundamental challenge bioinformatics (Segal et al. 2002). Early approaches like Enformer (Avsec et al., 2021), Borzoi (Linder et al. 2025 and SPACE Yangg et al. 2025b) attempted to predict gene expression directly from DNA sequences, facing inherent limitations, while GraphReg (Karbalayghareh al. 2022) enhanced performance incorporating epigenomic information through graph attention networks to model physical interactions between genomic regions. More recent methods have progressed toward integrating both sequence and epigenomic information, with Creator (Li et al. (022) and EPInformer (Lin et al. (024) demonstrating improved performance through this combined approach. However, these models typically rely on pre- pre-identified regulatory elements, overlooking potential contributions from unannotated regions. Seq2Exp (Su et al., 2025) addressed this limitation through an end-to-end, data-driven methodology that simultaneously learns to identify relevant regulatory elements and predict expression with epigenomic guidance. Despite these advances, current research tends to focus predominantly on modeling distal regulatory elements through long sequence architectures, rather than optimizing the utilization of biologically interrelated epigenomic signals that directly influence gene regulation.
CONCLUSION
This work reveals critical challenge in gene expression prediction: while previous methods focus on modeling longer sequences, current technical paradigms suffer from inherent performance degradation with extended sequence length. Instead, we discovered that proximal epigenomic signals crucial, but complex background chromatin states may introduce confounding effects, creating spurious correlations in models. Building on these insights, we propose Prism, lightweight framework that achieves state-of of titt-offthe-arr gene expression prediction performance through effective integration of multimodal epigenomic signals using only short sequences while adding minimal computational overhead.
ETHICS STATEMENT
We acknowledge that we have read and adhered to the ICLR Code of Ethics.
If the reviewers or the community raise any ethical concern about our work, we are ready to address them transparently and responsibly.
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we detail the complete pipeline of our Prism framework using pseudocode in the appendix. We also provide an exhaustive documentation of the experimental setups, covering computing infrastructure, random seeds, and optimization details such as learning rate schedules and batch sizes. Furthermore, the robustness of our results thoroughly validated through systematic evaluations, detailed ablation studies, and hyperparameter sensitivity analyses presented throughout the paper. Code available at https://gthub.com/aaggghhet h 130//rrssm
ACKNOWLEDGEMENTS
This work supported in part by the National Natural Science Foundation of China No. 62376277, Public Computing Cloud, Renmin University of China, and fund for building world-class universities (disciplines) of Renmin University of China.
REFERENCES
Žiga Avsec Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinng Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene
10

---

Published as a conference paper at ICLR 2026
expression prediction from sequence by integrating long-range interactions. Nature methods, 18 (10):1196-1003 2021.
Ziga Avsec, Natasha Latysheva, Jun Cheng, Guido Novati, Kyle R Taylor. Tom Ward, Clare Bycroft, Lauren Nicolaisen, Eirini Arvaniti, Joshua Pan, et al. Alphagenome: advancing regulatory variant effect prediction with a unified dna sequence model. bioRxiv, pp. 2025-06, 2025.
Eryn Blass and Patrick Ott. Advances in the development of personalized neoantigen-base therapeutic cancer vaccines. Nature reviews Clinical oncology, 18(4):215-229, 2021.
Garyk Brixi. Matthew G Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel Gonzalez, Samuel H King, David B Li, Aditi T Merchant, et al. Genome modeling and design across all domains of life with evo 2. BioRxiv, pp. 2025-02, 2025.
Kathleen M Chen, Aaron Wong, Olga G Troyanskaya, and Jian Zhou. A sequence- based global map of regulatory activity for deciphering human genetics. Nature genetics, 54(7):940-949) 2022.
ENCODE Project Consortium et al. An integrated encyclopedia of dna elements in the human genome. Nature, 489(7414):57, 2012.
William Cookson, Liming Liang, Gonçalo Abecasis, Miriam Moffatt, and Mark Lathrop. Mapping complex disease traits with global gene expression. Nature Reviews Genetics, 10(3):184-19, 2009.
Menno P Creyghton, Albert W Cheng, G Grant Welstead, Tristan Kooistra, Bryce W Carey, Eveline J Steine, Jacob Hanna, Michael A Lodato, Garrett M Frampton, Phillip A Sharp, et al. Histone h3k27ac separates active from poised enhancers and predicts developmental state. Proceedings of the National Academy of Sciences, 107(50):21931-21935 2010.
Hugo Dalla-Torre, Liam Gonzalez, Javier Mennoza-Revillle Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida. Hassan Sirelkhatim, al. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, pp. 1-11, 2024.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Valur Emilsson, Gudmar Thorleifsson, Bin Zhang, Amy Leonardson, Florian Zink, Jun Zhu, Sonia Carlson, Agnar Helgason, G Bragi Walters, Steinunn Gunnarsdottir, et al. Genetics of gene expression and its effect on disease. Nature, (22(718))423-4444 2008
Jason Ernst and Manolis Kellis. Chromatin-state discovery and genome annotation with chromhmm. Nature protocols, ((12):247--2492, 2017.
Charles P Fulco, Joseph Nasser, Thouis Jones, Glen Munson, Drew T Bergman, Vidya Subramanian, Sharon Grossman, Rockwell Anyoha, Benjamin R Doughty, Tejal A Patwardhan, et al. Activity- by-contact model of enhancer prommotrrppromtter regulation from thousands of crispr perturbations. Nature genetics, 51(12):1664-166 2019.
Albert Gu and Tri Dao. Mamba: Linear-ting sequence modeling with selective state spaces, arXiv preprint arXiv:23/2.00752 2023.
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: encoder representations from transformers model for dna-language in genome. Bioinformatics, 37 (15):2112-2120,, 2021.
pre-trained bidirectional
Alireza Karbalayghareh, Merve Sahin, and Christina S Leslie. Chromatin interaction -ware gene regulatory modeling with graph attention networks. Genome Research, 32(5):930 9444,,94444 2022.
David R Kelley, Yakir A Reshef, Maxwell Bileschi, David Belanger, Cory Y McLean, and Jasper Snoek. Sequential regulatory activity prediction across chromosomes with convolutional neural networks. Genome research, 2855)773--750, 2018.

---

Published as a conference paper at ICLR 2026
Avantika Lal, David Garfield, Tommaso Biancalani, and Gokcen Eraslan. Designing realistic regulatory dna with autoregressive language models. Genome Research, 34(9):1411-1420, 2024.
Shumin Li, Ruibang Luo, and Yuanhua Huang. Assessing large-scale genomic language models in predicting personal gene expression: promises and limitations. bioRxiv, pp. 2025-07, 2025.
Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, Cheng Tan, Jiangbin Zheng. Yufei Huang, and Stan Li. Vqdna: Unleashing the power of vector quantization for multi-species genomic sequence modeling. In Forty-first International Conference on Machine Learning. 2024.
Yongge Li, Fusong Ju, Zhiyuan Chen, Yiming Qu, Huanhuan Xia, Liang He, Lijun Wu, Jianwei Zhu, Bin Shao, and Pan Deng. Creator: zero-shot cis-regulatory pattern modeling with attention mechanisms. Genome Biology, 24(1):266, 2023.
Jiecong Lin, Ruibang Luo, and Luca Pinello. Epinformer: a scalable deep learning framework for gene expression prediction by integrating promoter-enhancer sequences with multimodal epigenomic data. bioRxiv, pp. 2024-08, 2024.
Johannes Linder, Divyanshi Srivastava, Han Yuan, Vikram Agarwal, and David R Kelley. Predicting ma-seq coverage from dna sequence as unifying model of gene regulation. Nature Genetics, pp. 1-13, 2025.
Polina Mamoshina, Armando Vieira, Evgeny Putin, and Alex Zhavoronkov. Applications of deep learning in biomedicine. Molecular pharmaceutics, 1355):44451444, 2016.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36:43177-43201, 2023.
Eric Nguyen, Michael Poli, Matthew G Durrant, Brian Kang, Dhruva Katrekar, David B Li, Liam J Bartie, Armin W Thomas, Samuel H King, Garyk Brixi, et al. Sequence modeling and design from molecular genome scale with evo. Science, 386(6723) 2024.
Judea Pearl. Causal inference in statistics: An overview. 2009.
Judea Pearl, Madelyn Glymour, and Nicholas Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
Jennifer L Plank and Ann Dean. Enhancer function: mechanistic and genome-wide insights come together. Molecular cell, 55(1):5-14, 2014.
Aditya Pratapa, Amogh P Jalihal, Jeffrey N Law, Aditya Bharadwaj. and TM Murali. Benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data. Nature methods, 17(2):147-1544 2020.
Wenwen Qiang, Jiangmeng Li, Changwen Zheng, Bing Su, and Hui Xiong. Interventional contrastive learning with meta semantic regularizer. In International conference on machine learning, pp. 18018-18030 PMLR, 2022.
Suhas SP Rao, Miriam H Huntley, Neva Durand, Elena K Stamenova, Ivan D Bochkov, James T Robinson, Adrian L Sanborn, Ido Machol, Arina D Omer, Eric S Lander, et al. 3d map of the human genome kilobase resolution reveals principles of chromatin looping. Cell, 159(7): 1665-1680, 2014.
Melissa Sanabria, Jonas Hirsch, Pierre M Joubert, and Anna R Poetsch. Dna language model grover learns sequence context in the human genome. Nature Machine Intelligence, 6(8):91 1933, 2024.
Yair Schiff, Chia Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Ca- duceus: Bi-directional equivariant long-range dna sequence modeling. In International Conference on Machine Learning, pp. 43632-43648, PMLR, 2024.
Stefan Schoenfelder and Peter Fraser. Long-range enhancer promootr contacts in gene expression control. Nature Reviews Genetics, 20(8)437-455 2019.
12

---

Published as a conference paper at ICLR 2026
Eran Segal, Yoseph Barash, Itamar Simon, Nir Friedman, and Daphne Koller. From promoter sequence to expression: a probabilistic framework. In Proceedings of the sixth annual international conference on Computational biology, pp. 263-272, 2002.
Shai S Shen-Or, Robert Tibshirani, Purvesh Khatri, Dale L Bodian, Frank Staedtler, Nicholas M Perry, Trevor Hastie, Minnie M Sarwal, Mark M Davis, and Atul J Butte. Cell type-specific gene expression differences in complex tissues. Nature methods, 7(4):287-289) 2010.
Xingyu Su, Haiyang Yu, Degui Zhi, and Shuiwang Ji. Learning to discover regulatory elements for gene expression prediction. The Thirteenth International Conference on Learning Representa- tions, 2025.
Robert E Thurman, Eric Rynes. Richard Humbert, Jeff Vierstra, Matthew T Maurano, Eric Haugen, Nathan C Sheffield, Andrew Stergachis, Hao Wang, Benjamin Vernot, et al. The accessible chromatin landscape of the human genome. Nature, 489(7414):75-82 2012.
Toni A. Volpe, Catherine Kidner, Ian M. Hall, Gang Teng, Songtao I. Grewal, and Robert A. Martienssen. Regulation of heterochromatic silencing and histone h3 lysine-9 methylation by mai. Epub 2002 Aug 22.
Science, (885558))11833-1855 Sep 2002. doi: 10. 11.1116/60111
Peihao Wang, Ruisi Cai, Yuehao Wang, Jiajun Zhu, Pragya Srivastava, Zhangyang Wang, and Pan Li. Understanding and mitigating bottlenecks of state space models through the lens of recency and over-smoothing. In The Thirteenth International Conference on Learning Representations, 2025.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In International conference machine learning, pp. 9929-9939, PMLR, 2020.
Zhao Yang, Bing Su, Chuan Cao. and Ji-Rong Wen. Regulatory DNA sequence design with reinforcement learning. In The Thirteenth International Conference Learning Representations, 2025a.
Zhao Yang, Jiwei Zhu, and Bing Su. Space: Your genomic profile predictor is a powerful dna foundation model. In International Conference on Machine Learning, pp. 71648-71667, PMLR, 2025b.
Zhongqi Yue, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. Interventional few-shot learning. Advances in neural information processing systems, 33:2734-2746, 2020.
Yong Zhang, Tao Liu, Clifford A Meyer, Jérôme Eeckhoute, David S Johnson, Bradley E Bernstein, Chad Nusbaum, Richard M Myers, Myles Brown, Wei Li, al. Model-based analysis of chip-seq (macs). Genome biology, 9:1-9, 2008.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Jian Zhou and Olga G Troyanskaya. Predicting effects of noncoding variants with deep learning- -ased sequence model. Nature methods, 12(10):931-934, 2015.
Jian Zhou, Chandra L Theesfeld, Kevin Yao, Kathleen M Chen, Aaron K. Wong, and Olga G Troyanskaya. Deep learning sequence- based ab initio prediction of variant effects on expression and disease risk. Nature genetics,
2018
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana V Davuluri, and Han Liu. Dnabert-2: Efficient foundation model and benchmark for multi-species genomes. In The Twelfth International Conference Learning Representations, 2024.
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
We used large language models (LLMs) only to aid or polish the writing of this manuscript. They were not involved in idea generation, methodological design, experiments, or analysis. All scientific content was created and verified by the authors, who take full responsibility for the final text.
13

---

Published as a conference paper at ICLR 2026
B SHORTENING INPUT SEQUENCE LENGTH AT TEST TIME
In Figure T (d) of Section D we have confirmed that training with longer sequences from scratch does not provide additional benefits. Further, we aim to investigate whether shortening the input length at test time would decrease the performance of model trained longer sequences. Specifically, tested the Seq2Exp-soft model (Su et al. 2025 trained on 200k sequences to evaluate if reducing context during inference affects performance. As shown in Table found that Seq2Exp, despite being trained on 200k inputs, shows minimal performance degradation when the input length is reduced from 200K 2.5K during testing. This suggests that Seq2Exp fails effectively utilize long-context information even during training, indicating that the model does not genuinely leverage the extended sequence information it was provided.
Interestingly, however, there is significant performance drop when inputs are shortened to 2,500 tokens, with particularly sharp decline observed below 2,100 tokens. We attribute this behavior to an implementation detail in Seq2Exp: the model forcibly prevents the central 2,000-bp region from undergoing masking (this constraint was not mentioned in the Seq2Exp paper but can be found in their GitHub repository), effectively forcing the model to focus predominantly the central 2,000 bp and proximal regulatory information.
Based these observations and comparing with Figure has a much smaller impact on model performance than epigenomic signals. Removing epigenomic signals during testing substantially hurts performance, while shortening sequence length has minimal effect. This finding motivates our focus on modeling epigenomic information effectively.
can conclude that input context length
Table Performance Seq2Exp (Su al. the K562 cell line.
2025
when testing with shortened input sequences on
Input Length
MSE
MAE L
Pearson
200000
0.1856
0.3054
0.8723
10000
0.1855
0.3074
0,8751
8000
0.1864
0.3082
0.8747
3000
0.1943
0.3134
0.8698
2500
0.1996
0.3174
0.8674
2100
0.2301
0.3471
0.8603
2070
0.3639
0.2464
0.8576
2050
0.3848
0.2686
0.8540
2040
0.4017
0.2855
0.8521
2030
0.4248
0.3093
0.8496
2020
0.4634
0.3543
0.8429
2010
0.5371
0.4506
0.8291
2000
0.6485
0,6183
0,8084
C EXPERIMENTAL DATA OF TABLE
We provide comprehensive numerical results corresponding to Figure I in the main text, including complete performance metrics and ablation studies.
C.1 SEQUENCE LENGTH SENSITIVITY
Table 5 compares the performance stability of Seq2Exp and Caduceus across different input lengths.
C.2
EPIGENOMIC SIGNAL CONTRIBUTIONS
Table demonstrates that combining all epigenomic signals yields optimal performance, with H3K27ac showing the strongest individual impact.
Model available http:///wwwp...tooocc//////////r
14

---

Published as a conference paper at ICLR 2026
Table Performance comparison with varying input lengths (left: Seq2Exp Caduceus (Schiff et al. 2024
(Su et 2025 right:
Length
MAE
MSE
Pearson
Length
MAE
MSE
Pearson
100
0.3394
0.2233
0.8441
100
0.3385
0.2200
0.8449
500
0.3096
0.1879
0.8744
500
0.3096
0.1889
0.8716
2000
0.3150
0.1971
0.8678
2000
0.3036
0.1831
0.8747
5000
0.3098
0.1949
0.8703
5000
0,3170
0.1941
0.8692
10000
0.3088
0.1897
0.8719
10000
0.3235
0.2029
0.8550
Table 6: Caduceus performance with different epigenomic signal configurations
Configuration
MSE
MAE
Pearson
No signals
0.2163
0.3325
0.8485
+H3K27ac
0.1873
0.3080
0.8628
+DNase
0.2089
0.3227
0.8497
+Hi-C
0.2135
0.3264
0.8530
All signals
0.1886
0.3079
0,8652
C.3 REMOVE SIGNALS TEST TIME
Table 7 reveals critical signal dependencies. Removing H3K27ac during testing from model trained on all signals degrades performance most severely (22.3% MAE increase), while Hi-C removal has minimal effect (4.7% MAE increase).
Table 7: Performance degradation from signal removal (trained with all signals)
Condition
MAE
MSE
Pearson
Drop H3K27ac
0,5653
0.6115
0.6964
Drop DNase
0.3890
0.2962
0.8189
Drop Hi-C
0.3548
0.2280
0.8467
Baseline (all signals)
0.3078
0.1886
0.8652
D CASE STUDY AND QUANTITATIVE EVIDENCE OF WIDESPREAD BACKGROUND CONFOUNDERS
D.1 QUANTITATIVE PREVALENCE LONG-RANCE INTERACTIONS
To statistically validate the ubiquity of long-distance chromatin interactions in the K562 and GM12878 cell lines core premise of our work we conducted two-fold analysis. This quantitative evidence establishes that the case study Section D.2 representative of genome-wide eggnal-to-oiis. challenge.
First, we analyzed promoter-centric Hi-C contact data. For each gene's TSS, we examined its vector of Hi-C contact frequencies, defining significant interaction any contact with signal strength greater than 0.01. We classified an interaction as "long-range" if the genomic distance from the TSS exceeded 50kb. Second, to specifically quantify connections putative regulatory elements, we analyzed pre-computed Promoter- Promotcr-Enhancer Enhancer (P-E) linkages from the Setivittyby-Conttal (ABC) least distal enhancer (>50kb away).
model, identifying all genes connected
The results, presented in Table & of both genomes. Nearly all genes (~~99%) exhibit numerous long- rangrange contacts, with median of nearly 200,000 potential interaction partners per gene. The ABC model data further confirms that virtually all genes linked to least one distal enhancer. This creates significant signal-
demonstrate that disttnis interactions are a ubiquitous feature
15

---

Published as a conference paper at ICLR 2026
to-noise problem, the vast number of interactions indicated by Hi-C data cannot all be causally determinative of gene expression, thus acting as background confounders.
Table Statistical Summary of Hi-C Long-Raan Range Interactions K562 and GM12878 Cell Lines
Statistic
K562
GM12878
% of genes with promoter-interactng
98.9%
99.3%
% genes with long-range (>50kb) promoter' % of genes linked distal enhancer via ABC model
98.9%
99.3%
100.0%
100.0%
Median number of manggraage partners per gene' Median distance of long-range interactions (kb)'
199,899
199,899
49,707.0
49,939.0
Statistics derived from promoler-centric Hi-C contact vectors. Statistics derived from pre-computed ABC model P-E links.
D.2 QUALITATIVE CASE STUDY
The statistical prevalence of interactions motivates our hypothesis that many of these signals act as confounders rather than direct regulators. To qualitatively support this, we present a representative case (Figure 0 at genomic locus (Entrez ID: ENSG00000080561)
In this region, both DNase (chromatin accessibility) and Hi-C (3D spatial proximity) signals exhibit broad, high activation. Despite this permissive chromatin environment, the marker of active regulatory elements, H3K27ac, shows no enrichment. Consequently, the gene expression level remains low (0.6021).
This case demonstrates that high background signal activity alone is insufficient to drive gene expression. The absence of H3K27ac indicates that key regulatory elements inactive, resulting in minimal transcriptional output despite strong accessibility and spatial contacts.
Gane Expression: 0.8021
HOK27AC
DR0TE
Figure 6: A representative genomic locus (Entrez ID: ENSG000008005)) where DNase and Hi-C signals are broadly active, but H3K27ac shows no enrichment. Despite strong chromatin accessibility and spatial contacts, gene expression remains low (0.6021). This supports the hypothesis that such pervasive background signals (quantified in Table 8) act as confounders rather than causal regulators.
D.3 CONCLUSION AND MOTIVATION FOR OUR METHOD
Together, the quantitative data and the qualitative case reinforce the necessity of disambiguating causal foreground signals (like H3K27ac) from pervasive background confounders (like broad DNase and Hi-C signals) when modeling gene expression. This pervasive signal-to-noise problem directly motivates approach of explicitly modeling background signals through a Structural Causal Model and applying backdoor adjustment to correct for their confounding effects, thereby improving both interpretability and prediction accuracy.
16

---

Published as a conference paper at ICLR 2026
E ALGORITHM WORKFLOW
Here provide the complete algorithm workflow for our Prism framework in Algorithm [ The algorithm initializes three neural networks: the signal encoder 9ø. the predictor network hgg and the confounder encoder Mw- During training, we compute both standard and interventional predictions, then optimize the model using three objectives: prediction loss L1. intervention loss L2, and uniform diversity loss C3-
Algorithm 1 Interventional Framework for Gene Expression Prediction (Prism)
Require: Gene sequence X, epigenomic signals S, gene expression Y, hyperparameters oz, 8,B, Trained model parameters 0, p.w. Initialize parameters 0, Φ, w randomly.
Ensure:
2: while not converged do
// Forward Pass
4: H t go(5) {Encode epigenomic signals into features)
5:
Y t ho(X, H) (Make standard prediction)
6:
{ai...., an) +- (S) {Learn confounder weights representing C}
7:
1- Interventional Prediction via Backdoor Adjustment
8:
Yau +.in (X, @ a,) {Apply backdoor adjustment)
9:
11 Loss Computation
10:
L1 + Eu(Y, Y) (Standard prediction loss (Huber loss)
11:
L2 + fu(Yoo. Y) [Intervention loss (Huber loss)}
12:
á + as/ ||:lll 2 for all (L2-normalize weight vectors}
La log [Eu, exp(2t a ā 21) (Uniform diversity loss}
14:
C Li + +L BLs (Total objective)
15: 1/ Backward Pass
Update 0, σb using gradient descent C.
17: end while
return 8, 0
MORE IMPLEMENTATION DETAILS
F.1
TRAINING SETTINGS
Our training framework is implemented using PyTorch Lightning. All training-related hyperparame- ters were adopted directly from Seq2Exp (Su et al. 2025), which means we did not perform extensive parameter tuning for our specific approach. Consequently, there is potential for further performance improvements through careful hyperparameter optimization. The complete set of hyperparameters used our experiments is presented in Table
Table 9: Hyperparameter values following Seq2Exp Su et al. 2025
Hyperparameters
Values
Layers of backbone
4
Hidden dimensions
128
Max training steps
50000
Batch size
Learning rate
5c-4
Scheduler strategy
CosineLR with Linear Warmup le-5
Initial warmup learning rate
Min learning rate
1e-4
Warmup steps
5,000
Validation model selection criterion
validation MSE
17

---

Published as a conference paper at ICLR 2026
F.2 IMPLEMENTATION DETAILS OF CONFOUNDER ENCODER
Our confounder encoder In is implemented as a lightweight ID-CNN that maps raw epigenomic signals S E IR Lxd to weight vectors A E Rexd'' The architecture consists of three-layer CNN followed by projection layer:
Layer 1:
Conv (in men_chhnnelsd, channnesl,, kemnel_sizeet followed by BatchNorm, ReLU, and MaxPool (kernelssize=4)
Layer 2: Conv ID (in mo__channel=-.,. out chaa_nccaalls66,.. kernel_size=5) wou..chaamelssl.kkemeliiieeeeee followwd followed by BatchNorm,
ReLU, and MaxPool (Kernel_size=4))
Layer Conv ID ((n_channels=16, out_cchanels=32, kernel_size=3) followed by Batch-
Norm, ReLU, and MaxPool (kernel size=44
Global Pooling: ddaptiveAvvPPollID(( followed by Flatten
Projection: Linear layer mapping the flattened features (32 dimensions) to n dd dimensions
The progressive reduction in sequence length through max pooling operations (by a factor of 64 in total) efficiently captures patterns at different genomic scales while significantly reducing the computational overhead. After obtaining the raw weights, we apply sigmoid activation function to constrain the values between and 1, making them suitable for weighting the epigenomic signals via the Hadamard product operation. This lightweight design adds minimal parameters to the overall model while effectively modeling the background epigenomic regulatory patterns. The entire encoder requires only 11K parameters, which is negligible compared to the backbone model's parameter count.
F.3 IMPLEMENTATION DETAILS OF THE PREDICTOR
In practice, both the DNA sequence X and the weighted epigenomic signal O ag are first indepen- dently projected to the model's 's hidden dimension d' via separate linear layers (the signal encoder go and the sequence input projection, respectively), and then combined via element- addition before being fed into the Caduceus backbone.
Furthermore, when computing the interventional prediction Ya. (Equation 3). rather than av- eraging n separate end-to-end predictions, we average the intermediate backbone representa- tions across the n contexts before applying the shared prediction head. Formally, Ydo h.hd - body (X, @ a:)). where 11 denotes the Caduceus backbone and h,head denotes @
the prediction head. This representation averaging is standard and computationally efficient ap- proximation, and closely approximates the formulation in the main text, with the two being exactly equivalent when the prediction head is linear.
G THE EFFECT OF PRE-TRAINING
Our main experiments follow Seq2Exp Su et al. (2025). where all models are trained from scratch without pre-training. To further investigate whether DNA model pre-training benefits gene expression prediction, we conducted experiments using pre-trained Enformer (pre-trained on 200k sequences) and Caduceus (pre-trained on 131k sequences) with signals, as shown in Table TO to examine the effectiveness of long-context pre-training.
Overall, we find that pre-training provides consistent improvements. For Enformer, the improvement marginal. For Caduceus, which was pre-trained on 131k sequences, loading pre-trained weights before fine-tuning 200k gene expression prediction tasks yields substantial improvements, with Pearson correlation notably improving by 0.0113. However, the effect of pre-training mirrors that of Seq2Exp only mitigate the performance degradation caused by extended sequence length, rather than making long-context models superior to short-context ones.
When only 2k input length, pre-training also provides some improvement, but this im- provement relatively modest. The absolute MSE improvement is particularly negligible. We also experimented with loading pre-trained Caduceus weights for Prism training and found that
18

---

Published as a conference paper at ICLR 2026
Table 10: The effect of pre-training
Model
Metric
From Scratch
Pre-trained
MSE + Pearson
0.2920 + 0.0050
0.2913 ( 0.0007) + 0.0209
Enformer
0.7961 + 0.0019
0.7983 (f 0.0022) = 0.0229
MSE +
0.1863 + 0.0035
0.1858 (L 0,0005) = 0,0082
Caduceus w/signals (2k input)
Pearson
0.8713 + 0.0023
0.8759 (f 0.0046) = 0.0042
MSE +
0.1959 +: 0.0036
0.1897 (L 0.0062) = 0.0026
Caduceus w/signals (200k input)
Pearson
0.8630 + 0.0008
0.8743 (f 0.0113) = 0.0030
MSE +
0.1789 + 0.0041
0.1795 (T 0.0006) = 0.0061
Prism
Pearson
0.8751 +: 0.0036
0.8774 (1 0.0023) + 0.0018
achieves stable improvements while Prism continues to maintain state-of-the-ar performance. Notably, pre-training significantly improves Pearson correlation, while MSE shows minimal change.
Therefore, our conclusion is that long- con-context pre-training can substantially improve long- contextt capabilities, but this improvement only mitigates the performance degradation inherent to long-context models, while providing only marginal improvements for short-context models.
H EXTENDED ANALYSIS WITH ADDITIONAL EPIGENOMIC SIGNALS
To comprehensively evaluate the effectiveness of different epigenomic signals in gene expression prediction, we conducted additional experiments on the K562 cell line using signals beyond the three primary ones (H3K27ac, DNase-seq. and Hi-C) employed in our main analysis following Lin et al. (2024):
Su et al. (025).
H.1 ADDITIONAL SIGNAL DESCRIPTIONS
We incorporated three additional epigenomic signals with distinct biological functions:
H3K4me3 (ENCODE ID: ENCFF405ZDL).: A histone modification signal that specifically marks active promoter regions with high precision, complementing H3K27ac which marks both promoters and enhancers with broader coverage.
DNase footprint (ENCODE ID: ENCSR000EOT): High-resolution protein-DNA binding footprints derived from DNase-seq data, identifying exact transcription factor binding sites within accessible chromatin regions through computational algorithms.
ChIA-PET (ENCODE ID: ENCFF278RFG), Protein mediated chromatin interaction data that captures functionally relevant long-range contacts, providing more targeted information compared to genome-wide Hi-C interactions.
Signal processing followed standard protocols: H3K4me3 and ChIA-PET used direct bigwig signal values, while DNase footprint regions from bigbed annotations were encoded as binary signals for annotated regions, elsewhere).
H.2 INDIVIDUAL. SIGNAL ANALYSIS
Table 11 presents the performance of Caduceus with individual signals. Most signals demonstrate improvements over the no-signal baseline, with H3K4me3 showing the most substantial enhancement (MSE: 0.1801, Pearson: 0.8781). ChIA-PET showed degraded performance, likely due to high noise levels in the raw data. DNase footprint performed comparably DNase-seq, suggesting limited additional information content despite higher theoretical resolution. The superior perfor- mance of H3K4me3 and H3K27ac aligns with their roles direct indicators of active regulatory elements, supporting our categorization as foreground signals with stronger causal relationships to gene expression.
expression
19

---

Published as a conference paper at ICLR 2026
H.3 COMPOSITIONAL SIGNAL EFFECTS
Table 12 examines the effects of combining multiple signals. Adding DNase footprint to the initial three signals provides minimal improvement, consistent with derivation from DNase-seq data. However, incorporating H3K4me3 yields substantial performance gains actoss all metrics.
Most notably, Prism with H3K4me3 integration achieves the best performance (MSE: 0.1719, representing a 0.0137 improvement over Seq2Exp baseline). This demonstrates that Prism's 's causal intervention framework maintains robust improvements even when strong individual signals like H3K4me3 present, suggesting that the method effectively disentangles genuine regulatory signals from confounding background effects.
H.4 KEY FINDINGS
Our extended analysis reveals several important insights: First, signals with direct regulatory roles (H3K4me3, H3K27ac) provide greater predictive value than background accessibility signals. Second, computationally derived signals like DNase footprint offer limited additional information beyond their source data. Third, Prism consistently outperforms baseline approaches across different signal combinations, validating the robustness of our causal intervention framework. These findings support the importance of careful signal selection and highlight the potential for further improvements through strategic integration of complementary epigenomic data types.
Table 11: Single Signal Input (Caduceus w/signals (2k input))
Signal
MSE 4
MAE 1
Pearson f
No Signal
0.2215 + 0.0086
0.3342 + 0.0081
0.8502
0.0026
H3K27ac
0.1986 + 0.0059
0.3179 = 0.0054
0.8645
0.0037
DNase-seq
0.2207 +: 0.0060
0.3342 + 0.0085
0.8530
0.0037
Hi-C
0.2202 + 0.0045
0.3330 + 0.0064
0.8489
0.0039
H3K4me3
0.1801 + 0.0079
0.3084 + 0.0099
0.8781 = 0.0018
ChIA-PET
0.2262 + 0.0062
0.3387 + 0.0060
0.8422
0.0059
DNase footprint
0.2186 + 0.0073
0.3300 + 0.0045
0.8523 = 0.0044
Table 12: Effect of Compositional Signal Input (Models with 2k input)
Model Configuration
MSE L
MAE L
Pearson
Caduceus w/signals
(initial signals)
0.1863 = 0.0035
0.3092 +: 0.0050
0.8713 1: 0.0023
Caduceus w/signals
(initial signals + DNase footprint)
0.1870 = 0.0059
0.3092 1 0.0058
0.8703 0.0026
Caduceus w/signals
(initial signals + H3K4me3)
0.1789 = 0.0122
0.3067 + 0.0117
0.8804 1: 0.0101
Caduceus w/signals
(initial signals DNase footprint H3K4me3)
0.1762 = 0.0054
0.3072 1: 0.0031
0.8837 1 0.0024
Prism
(initial signals)
0.1789 - 0.0041
0.3000 + 0.0058
0.8751 +: 0.0036
Prism
(initial signals + DNase footprint)
0.1794 = 0.0064
0.2996 1: 0.0055
0.8752 1: 0.0042
Prism
(initial signals + H3K4me3)
0.1719
0.0070
0.2969
0.0049
0.8839 0.0035
Prism
(initial signals + DNase footprint H3K4me3)
0.1730 = 0.0055
0.3005 + 0.0051
0.8850 - 0.0020
20

---

Published as a conference paper at ICLR 2026
1 CROSS-CELL GENERALIZATION
Our main experiments follow EPInformer (Lin al. 2024) and Seq2Exp (Su al. 2025) training separate models for each cell type. To evaluate whether single model can generalize across cell types, we conducted mixed-trainine experiment combining K562 and GM12878. Specifically, during training, for each gene we randomly sample either its K562 or GM12878 epigenomic signals and corresponding expression value. The Table 13 below shows that this mixed model achieves comparable performance sell-fype-ppecific models.
Table 13: Performance of mixed- mixed-training
Dataset
Model
MSE 1
MAE 4
Pearson
K562
Seq2Exp (cell-specific)
0.1856 = 0.0032
0.3054 = 0.0024
0.8723 = 0.0012
K562
Prism (cell-specific)
0.1789 = 0.0041
0.3000 = 0.0058
0.8751 = 0.0036
K562
Prism (mixed-training)
0.1875 # 0.0085
0.3084 = 0.0077
0.8662 = 0.0049
GM12878
Seq2Exp (cell-specific)
0.1873 = 0.0044
0.3137 = 0.0028
0.8951 = 0.0038
GM12878
Prism (cell-specific)
0.1759 = 0.0054
0.3054 = 0.0048
0.9016 = 0.0024
GM12878
Prism (mixed-training)
0.1759 = 0.0038
0.3027 = 0.0041
0.9012 = 0.0032
J PERFORMANCE ACROSS DIFFERENT INPUT LENGTHS
We have explored how performance changes with varying sequence lengths. In Table 14 below, we present the Pearson correlation results for Caduceus w/signal (Schiff et al. 2024). Seq2Exp (Su et al. 2023). and Prism on K562 across different input lengths ranging from 100 to 10,000 bp.
Table 14: Pearson across different input lengths on K562
Input Length
Caduceus w/signal
Seq2Exp-soft
Prism (a = 1)
Prism (a = 2)
100
0.8488 = 0.0042
0.8492 = 0.0064
0.8493 = 0.0056
500
0.8719 = 0.0043
0.8694 # 0.0051
0.8726 0.0045
2000
0,8713 = 0.0023
0.8643 = 0.0088
0.8751 = 0.0036
5000
0.8662 0.0035
0.8675 0.0035
0.8690 0.0037
10000
0.8614 = 0.0059
0.8699 # 0.0032
0.8661 # 0.0027
0.8699 # 0.0023
As shown in Table 14. Prism exhibits a similar trend Caduceus, with performance beginning to decline when input length increases beyond 2000-5000 bp. While Seq2Exp maintains relatively robust performance across lengths, their reported results at 200k only match the performance of Caduceus at 500-2000 bp, validating our claim that Seq2Exp merely mitigates the performance degradation caused by extending sequence length Caduceus. Prism achieves the best performance among all three models across lengths from 100 5000 bp. We hypothesize that input length increases, the confounding effects of background signals become stronger. To test this, we experimented with increasing the intervention loss weight 2 when the input length reaches 10,000 bp, and observed performance improvement.
K PERFORMANCE ON H1 CELL LINE
We conducted additional experiments the HI cell line (which only appeared in Seq2Exp's rebuttal stage but was not included in their final camera-ready version). The results are shown in Table [5] below.
21

---

Published as a conference paper at ICLR 2026
Table 15: Performance HI cell line
Method
MSE
MAE 4
Pearson ?
Caduceus w/signal (2k input)
0.2751 + 0.0104
0.3929 = 0.0103
0.6681 = 0.0137
Scq2Exp-soft (our reproduction) Prism
0.2784 0.0064
0.3957 = 0.0045
0.6595 + 0.0089
0.2642 0.0060
0.3817 = 0.0044
0.6844 0.0078
On HI, Seq2Exp performs worse than the Caduceus baseline, while Prism consistently achieves SOTA results across all metrics. These results on a third cell line further demonstrate the consistent improvements of our approach.
L LEARNING CONFOUNDER WEIGHTS WITHOUT SUPERVISION
Unsupervised learning of chromatin states well-established approach in genomics, such as ChromHMM (Ersst & Kellis 2017). ChromHMM defines states based combinatorial patterns of epigenomic marks without explicit labels, which are subsequently mapped to biological chromatin states through expert annotation. Similarly, although our confounder weights are learned without supervision, Figure 5 in our manuscript shows that the model captures structured, gene-specific patterns rather than random noise. This suggests the model may learn meaningful latent states driven by the prediction task, which could potentially be validated with the assistance of biological experts like ChromHMM (Ernst Kellis 2017
We distinguish our approach from naive data augmentation. The fundamental difference is that our weights are learnable and gene-dependent, whereas naive augmentation relies on random perturbations. While random augmentation is effective in domains like computer vision, applying it to biological signals carries the risk of destroying critical information.
To validate this distinction, we first conducted dropout experiment the Caduceus baseline, where input signals are randomly discarded during training. Here, we define the retention rate as the proportion of signals preserved (i.e.. 1- dropout probability). We observed that performance degrades significantly as the retention rate decreases. As shown in Table[[6]ellow, keeping only 70% of signals results in notable performance drop, and keeping 50% degrades the MSE further to 0.2248. In contrast, we further evaluated our approach by using the learned weights to element-wise multiply the raw high-dimensional signals for prediction. We found that the average weight values on the K562 test set is approximately 0.35, which corresponds to an average retention rate of approximately 35%. However, unlike the random dropout baseline, our model maintains robust performance despite this high sparsity. This demonstrates that the learnable weights are meaningful, selectively preserving essential biological information.
Table 16: Comparison of dropout and learned weight methods
Model
MSE I.
MAE
Pearson 1
Caduceus w/ Dropout (rate = 0.9)
0.1874 + 0.0074
0.3062 + 0.0064
0.8702 + 0.0026
Caduceus w/ Dropout (rate = 0.7)
0.2059 0.0075
0.3199 0.0010
0.8625 0.0041
Caduceus w/ Dropout (rate = 0.5)
0.2248 # 0.0115
0.3428 # 0.0119
0.8446 + 0.0109
Prism w/ Learned Weights (rate 2s 0.35)
0. 0834 0.0092
0.3032 0.0083
0.8745 0.0061
22


Reviews:
[{"review_id": "ipLCkC182d", "reviewer": "Reviewer_a7M2", "summary": "This paper introduces Prism, a framework that accounts for the effects of confounders in epigenomic signals when predicting gene expression.  Specifically, it employs a three-layer 1D CNN as the confounder encoder, which processes the epigenomic signals to generate a set of confounder weight vectors.  In parallel, a single linear layer serves as the signal encoder, producing the corresponding signal weights. These two components are then combined, together with the DNA sequence features, to predict gene expression levels.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of confounder components for the gene expression prediction and their connection to biological intuition is important.\nAs it completes the current casual relationship formulation of the epigenomic signal.\n\n2. The observation regarding the sequence length required for CAGE prediction is interesting and biologically reasonable. \nThe provided experiments support such observation on the K562 cell for Gene Expression CAGE Prediction.\nI still have some doubts about whether a shorter sequence length is universally applicable to gene expression prediction tasks, or if it is specific to the datasets used in this study.\nIn other words, does 2k sequence is enough for all the gene expression prediction tasks beyond Gene Expression CAGE Prediction on the K562 and GM12878 cell?\nIf not, how to select the suitable length for diverse task?\n\n3. The experimental results looks good with the introduction of the confounder components. Overall, the paper writing is clear.", "weaknesses": "The overall framework appears well designed and complete, and I have no further comments regarding potential improvements.\nMy remaining concern lies in how to determine the appropriate sequence length for different prediction tasks. \nFurthermore, if the goal is to train a unified model for general gene expression prediction, it would be helpful to clarify how the model can adapt to varying sequence length requirements across different genes or datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AOjVLVJH0w", "reviewer": "Reviewer_jLJ9", "summary": "**Problem (P)**\n\nThe paper is motivated by the issues that arises during multi-modal long-context genomic modelling. Specifically,  \n1. said models merely mitigate the performance degradation inherent in current long-sequence modelling paradigms. They do not fix it. Figure 1 d shows this. \n2. the epigenomic signals are interdependent and play different roles. Naive modelling, with all of the signals, may render the model to learn spurious relations, the effects of which are demonstrated in figure 1 e/f.\n\n**Solution (S)**\n\nTo mitigate the confounding effects Prism is introduced, an approach that: \n1. learns high-dimensional feature combinations to represent background chromatin states, and \n2. makes a prediction for each state and averages them (backdoor adjustment) to reduce confounding. Training uses three losses: prediction, intervention (on the averaged prediction), and a uniformity loss that promotes diversity in the learned weights.\n\n**Contributions (C)**\n1. The authors challenge the focus on long-sequence modeling for gene expression, showing it does not necessarily help with current tools. \n2. They analyze roles of signals and point out that background chromatin patterns can confound models.\n3. Introduce and evaluate Prism. The new method beats all baselines while operating at a 2kbp genomic context, while most baselines utilise 200kbp.\n\n**Experimental Setting (E)**\n\n1. Inputs: DNA sequence, sequence-wide H3K27ac, Hi-C, and DNase-seq data. \n2. Baselines: Enformer, HyenaDNA, Mamba, Caduceus, EPInformer, Seq2Exp (hard and soft), Caduceus w/ signals, and MACS3 variant.\n3. Metrics: MSE, MAE, Pearson.", "questions": "Kindly address the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation **P**. The paper picks upon a prevalent issue in long-context DNA sequence modelling. The authors narrow down on the key-issue and validate it experimentally. \n2. Within gene expression prediction, using latent background-state weights + uniform backdoor averaging is relatively novel.\n3. While most baselines are trained and reported at 200k bp, Prism runs at 2k bp and still beats prior SOTA (Table 1). This supports their claim that better multi-modal integration can offset long-context modelling.\n4. Ample baselines are explored and the evaluation metrics seem fine.", "weaknesses": "1. Prism completely discards long-range sequence information by design, operating on only 2kbp. This is presented as a strength, but I believe that this is also a fundamental limitation. The model cannot discover regulatory elements or sequence variations beyond its 2kbp window unless their effects are already captured by the provided proximal epigenomic signals. Have the authors explored how the metrics change when we increase the context? Why was 2k chosen?\n\n2. Results in Table 1 are based on only two human cell lines, K562 and GM12878. While these are standard benchmarks, gene regulation is notoriously complex and cell-type specific. The model's SOTA performance may not hold across a wider, more diverse set of cell types or tissues. With the current breadth of exploration I feel that the proposed method has a limited experimental scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3K66x6C9Ya", "reviewer": "Reviewer_oNqu", "summary": "The paper challenges the dominant narrative that using models with longer DNA sequences improves gene expression prediction (e.g., Enformer,AlphaGenome). The authors show that relatively short sequence lengths are often sufficient to preserve most of the predictive performance. They demonstrate that multimodal epigenomic signals (such as histone modifications and chromatin accessibility) carry rich cell-type--specific information. However, naïve integration of these signals introduces confounding effects, as background chromatin patterns correlate with gene expression. To address this issue, the authors propose a causal framework that learns multiple background chromatin states and applies backdoor adjustment to isolate true causal effects of regulatory signals. The paper reports improved gene expression prediction performance in two of the most well-characterized cell lines, outperforming the models it builds upon, i.e. Caduceus and Seq2Exp.", "questions": "- In the experimentation in Table 1, it is not clear to me the length of the sequences the model is using when computing the results in each row. Is Prism using a smaller genome sequence length when surpassing the SOTA models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a clear and well-supported argument that genomic sequence models do not significantly benefit from longer input sequences, a strong claim that is convincingly demonstrated through extensive experimentation (specifically Table 12). The results showing the impact of epigenetic markers are compelling and supported by thorough ablation studies that highlight the individual contribution of each signal type. The analysis of the confounding effect is insightful, and the proposed solution using backdoor adjustment is both conceptually well-suited to the biological problem, offering a principled way to disentangle causal relationships from spurious correlations.", "weaknesses": "My concerns regarding the efficiency of the proposed approach. As shown in the hyperparameter sensitivity analysis (Section 4.3), the variation in performance when tuning the parameters \\alpha and \\beta appears minimal, suggesting limited sensitivity to these design choices. Similarly, the number of background states n has only a minor impact on results, as even the case n=0 in Table 2a performs comparably well. This raises questions about how essential the proposed causal intervention mechanism truly is for achieving the reported improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a0SymbjGo4", "reviewer": "Reviewer_uN9e", "summary": "This paper suggests that in the problem of gene expression prediction, sequence length may not be the key factor for prediction performances. Instead, leveraging epigenomic signals can better predict the gene expression.", "questions": "The following points are intended as open discussions \n- Following the weakness part, how do the authors think that adding additional external information could help the model learn more interpretable confounder representations $a_i$? \n- The current analysis only focuses on one task: gene expression prediction. I wonder whether the authors think the same causal regularization principles could generalize to broader DNA signal modeling tasks?. Would the authors expect similar trends to hold in those settings?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The idea itself is very interesting. While many works model DNA sequences to show off their long-sequence ability, modeling long sequences in DNA itself may not be helpful to many genomics related works. \n- The experiments are comprehensive to support the paper claims.", "weaknesses": "While the proposed $L_2$ objective is conceptually appealing, it is not fully clear how the model can learn meaningful confounder weights $a_i$ without any external supervision, but only based on epigenomic signals $S$. I can understand the performance gain from data augmentation perspective (e.g., to avoid overfit), but it’s not fully clear to me how the model can get the real confounder. Could the author elaborate it more clearly?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)