SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model
Authors: Zhao Yang
Jiwei Zhu
Bing Su
Venue: ICML 2025 poster (2025)
Keywords: DNA Foundation models
mixture of experts
genomic profile prediction
DNA
biology
Primary area: applications->health_medicine

Abstract: Inspired by the success of unsupervised pre-training paradigms, researchers have applied these approaches to DNA pre-training. However, we argue that these approaches alone yield suboptimal results because pure DNA sequences lack sufficient information, since their functions are regulated by genomic profiles like chromatin accessibility. Here, we demonstrate that supervised training for genomic profile prediction serves as a more effective alternative to pure sequence pre-training. Furthermore, considering the multi-species and multi-profile nature of genomic profile prediction, we introduce our **S**pecies-**P**rofile **A**daptive **C**ollaborative **E**xperts (SPACE) that leverages Mixture of Experts (MoE) to better capture the relationships between DNA sequences across different species and genomic profiles, thereby learning more effective DNA representations. Through extensive experiments across various tasks, our model achieves state-of-the-art performance, establishing that DNA models trained with supervised genomic profiles serve as powerful DNA representation learners.

SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model
Zhao Yang 123
Jiwei Zhu Zhu 123 Bing Su
123
Abstract
terson & Liu, 2023; Trajanoska et al., 2023), and protein engineering (Gosai al., 2024; Yang et al., 2025).
Inspired by the success of unsupervised pre- training paradigms, researchers have applied these approaches to DNA pre- pre-trrining. However, we argue that these approaches alone yield subop- timal results because pure DNA sequences lack sufficient information, since their functions are regulated by genomic profiles like chromatin ac- cessibility. Here, we demonstrate that supervised training for genomic profile prediction serves as more effective alternative pure sequence pre-training. Furthermore, considering the multi- species and multi-profile nature of genomic pro- file prediction, we introduce our Species-Proffe Profile Adaptive Collaborative Experts (SPACE) that leverages Mixture of Experts (MoE) to better cap- ture the relationships between DNA sequences across different species and genomic profiles, thereby learning more effective DNA represen- tations. Through extensive experiments across various tasks, our model achieves state-of-the- art performance, establishing that DNA models trained with supervised genomic profiles serve as powerful DNA representation learners. The code is available at https: : //github. com/ 2hwJiweil11/SPACR
Due the complexity of DNA sequences, gaining clear understanding of DNA is not easy. Inspired by the suc- cess of unsupervised pre-training paradigms in NLP, such as masked language modeling (Devlin et al., 2019) (MLM) and next-token prediction (Brown et al., 2020) (NTP), several DNA foundation models (DFMs) have recently emerged following similar pre-training approaches. DFMs (Ji et al., 2021; Zhou et al., 2024; Dalla-Torre et al., 2024; Nguyen et al., 2024a;b) aim to learn transferable representations by leveraging large-scale pre- pre-training on massive DNA se- quences, which can be adapted for exploring the functions and mechanisms of DNA. Through fine-tuning, these mod- els have demonstrated promising performance tasks such regulatory element identification, splice site recognition, and epigenetic modification prediction.
Although DFMs have made some progress, we argue that solely applying unsupervised pre-training techniques to DNA sequences alone cannot learn high-level semantic rep- resentations with strong generalization capabilities (Tang et al., 2023: Tang & Koo, 2024). Natural language se- quences themselves completely. In contrast, the functional roles of DNA se- quences in organisms are far more complex and are regu- lated by numerous genomic profiles in cell-type-specific manner (Fu et al., 2025), including various epigenetic mod- ifications (Portela & Esteller, 2010), chromatin accessibil- ity (Tan et al., 2023), and transcription factor binding (Pe- terson & Liu, 2023). Therefore, without incorporating these additional biological contexts, DFMs pre-trained only on raw DNA sequences may struggle to generalize to diverse cell environments or provide meaningful insights into the underlying biology.
express their meaning relatively
1. Introduction
DNA sequences, encoded by four nucleotide bases (A, C, G, and T), serve as biology's fundamental language that carries genetic instructions. Understanding the syntax and regula- tory grammar of this molecular language is promising for diverse applications, including disease diagnosis (Kernohan & Boycott, 2024; Sermon al., 2004), drug discovery (Pe-
Given that DNA's functional roles are regulated by various biological factors beyond sequence alone, we revisit super- vised genomic profile prediction models (GPPMs) ternative to unsupervised DFMs for learning DNA sequence representations. These models (Zhou & Troyanskaya, 2015; Kelley et al., 2018; Zhou et al., 2018; Chen et al., 2022; Avsec et al., 2021) are trained to predict experimentally measurable genomic profiles that directly encode regulatory and functional information in cell-type-speeifee manner. By learning map sequences these biologically meaning-
Equual contribution 'Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China Beijing Key Labora- tory of Research on Large Models and Intelligent Governance 'Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE, Correspondence to: Bing Su <bingsu @ edc.edd.cn..
al-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
ful profiles that reflect the complex regulatory mechanisms described above, supervised models may capture more func- tionally relevant representations compared to unsupervised DFMs that rely solely on sequence patterns, Such biological context-aware representations could be better aligned with downstream genomic applications, such as identifying regu- latory elements or understanding gene expression patterns. While Dalla-Toree al. (2024) has demonstrated through preliminary experiments that Enformer (Avsec et al., 2021) can learn effective DNA representations, there has been systematic study on improving the representation learning capabilities of GPPMs.
profiles, thereby learning more effective DNA repre- sentations compared to current GPPMs, which simply employ shared encoder and independent prediction heads.
Through extensive experiments across wide range of tasks, our SPACE achieves state-of-the-an (SOTA) per- formance, demonstrating that supervised pre-training for genomic profile prediction serves as a more ef- fective and powerful alternative to pure sequence pre- training.
2. Related Work
However, current GPPMs employ oversimplified architec- tures, using a shared encoder for DNA sequences from different species and independent prediction heads for dif- ferent genomic profiles. This design has two major limi- tations. First, the species-shared encoder fails to capture species- characteristics, as regulatory mechanisms and their influences often vary across species (Karollus et al., 2024). These distinct features are crucial for under- standing subtle genomic variations and context-dependent expression patterns. Second, genomic profile prediction inherently involves multiple interrelated tasks (Fu et al., 2025), as different profiles influence each other and are of- ten regulated by common mechanisms. The independent prediction heads, however, prevent the model from captur- ing these cross-profile dependencies and their variations across species.
Supervised genomic profile models are trained to predict functional genomic profiles from DNA sequences (Kathail et al., 2024). DeepSEA (Zhou & Troyanskaya, 2015) pio- neered this paradigm by leveraging convolutional neural net- works (CNNs) to extract DNA sequence features for multi- task prediction. Subsequent works (Kelley et al., 2018; Zhou et al., 2018: 8: Chen et al., 2022) have continued to advance this direction through either more advanced architectures or larger-scale training data. Enformer (Avsec et al., 202 I), widely recognized as the SOTA method, achieved superior prediction performance through hybrid Transformer- architecture. While these methods primarily focus on ab initio prediction of genomic profiles from DNA sequences and directly utilize these profiles for downstream tasks such as variant effect prediction, few studies (Dalla-Tonee et al., 2024) have explored whether their intermediate represen- tations capture meaningful biological patterns. Moreover, these models, which typically adopt a shared encoder cou- pled with independent profile prediction heads, have not thoroughly explored more effective architectural designs that could potentially enhance both prediction performance and representation learning.
To effectively model both cross-species and cross- cross-profile relationships, we introduce our Species-Profile Profile Adaptive Collaborative Experts (SPACE), which consists of two key components: (1) a species aware encoder module and (2) a profile-grouped enhancement decoder module, both built upon Mixture of Experts (MoE). The species-aware encoder employs sparse routing to dynamically balance species-specific and shared biological features, while the profile-grouped enhancement decoder uses dual-gated ex- pert weighted aggregation to capture the intricate depen- dencies between different genomic profiles. This design enables our model to effectively learn both species-specifts patterns and shared regulatory mechanisms across profiles.
Unsupervised DNA foundation models draw from the suc- cess of unsupervised pre-training in NLP. DNABERT (Ji et al., 2021) pioneered this approach, maintaining nearly identical training methods to BERT (Devlin et al., 2019) while adapting the tokenization scheme to 6-mers (Ce- likkanat et al., 2024) for DNA sequences. Subsequent works have continued along this direction, employing either MLM (Zhou et al., 2024; Dalla-Torre et al., 2024; Sanabria et al., 2024) or NTP (Nguyen et al., 2024a;b) as unsuper- vised training objectives. Although these methods have made effective optimizations in terms of training data, model architectures, and tokenization strategies, they still adhere to the assumption that unsupervised pre-training on pure DNA sequences alone is sufficient for learning effective representations. Moreover, there has been little system- atic comparison between these models and genomic profile prediction models in terms of their representation learning capabilities.
The major contributions of this paper include:
We revisit the supervised pre-training paradigm for DNA sequence foundation models through genomic profile prediction as the pre-training objective, demon- strating how funetion-related biological contextual in- formation can be effectively encoded into the learned representations.
We propose SPACE, a novel architecture that lever- ages MoE to better capture the relationships between DNA sequences across different species and genomic
2

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
Inputs (Atemraiin tulchwe brom (S))
Enformer Modules
Refinement
Module
Consalination
Colegotication
Add
Mulppication
Linser Projection
Basai Prediction
+
absle
Hiskien abel of Six
Sequenoo Gate
Specias embedding te
Prottle-Specitic Expert-Solecte Groups
Si Sp-ppeefff
Sar Spe-Speeific
Seep
Gatina
Rege Rupraseertatinn
C
MHABlook
Gross-Spacies Shared Export Pool
Expert
Expert
Expart
Expert
Species
Specins Gate
Cross-Profle Shared Expert Pool
Embodding
Expert
Expert
Expert
Expert
y
Represenzaion of S, sequence
Representation of Say sequente
Profile-Groped Enhaancament
Oudeane Frediction
Obaal
Enhanced profiles prediction St..
Enhanced prediction
Figure 1. Overview of SPACE architecture. It processes the input DNA sequence with three stages: (1) spatial compression and local context aggregation via a CNN-based aggregation module; (2) latent representation learning via a species-aware sparse MoE-based encoding module; (3) multi-profile prediction decoder via the dual-ggted expert weighted prediction enhancement module. The detailed structures of the encoding module and the dual-layer gated prediction enhancement module are shown in the left and right, respectively.
The MoE framework conditional computation tech- nique that selectively activates different expert networks for different inputs through sparse routing (Jacobs et al., 1991; Shazeer et al., 2017). In Transformer-based large language models (LLMs), MoF. is typically applied to feed- forward networks (FFNs) to achieve better parameter effi- ciency while maintaining model capacity (Fedus et al., 2022; Jiang al., 2023; Liu al., 2024). This adaptive routing mechanism is particularly well-suited for genomic mod- eling task, as it enables the model to dynamically balance between learning species-specific patterns and shared bio- logical features, while also capturing the complex dependen- cies between different genomic profiles. Following common practice in Transformer architectures, we also implement MoE by replacing the FFNs our model.
3.2. Overview
To better capture cross-species and cross-profile represen- tations, present SPACE. As illustrated in Figure 1, our architecture consists of three key stages: (1) CNN-based Lo- cal Context Aggregation following Enformer (Avsec et al., 2021); (2) Species- aware Transformer Encoder and (3) Profile-Grouph Enhancement Decoder for genomic profile prediction.
3.3. Local Context Aggregation
Given an input DNA sequence Ima we first follow En- former (Avsec al., 2021) to compress and aggregate the raw nucleotides through ID-CNNs, generating hidden states hm E [..dd at 128bp resolution, where L denotes the com- pressed sequence length and d, is the hidden dimension.
3. Method
3.4. Species-aware Encoder
3.1. Problem Formulation
Previous approaches to cross-species modeling (Kelley, 2020; Avsec et al., 2021) typically employ a shared encoder for all species. lacking fine-grained modeling of species relationships. To address this limitation, we propose a novel cross-species modeling framework consisting of Species- specific Embedding and Cross-species MoB layers.
Consider DNA sequences from M species (Si....,Sy SM... For each sequence Em from species Sm. we predict Cm genomic profile values. train with interleaved batches across all M species facilitate cross-species knowledge transfer (Kelley, 2020; Avsec al., 2021). Through this supervised pre-trate pre-training, the learned representations pected to capture rich biological and regulatory information.
Species-specific embedding. We augment the aggregated
3

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
hidden states hm with trainable species-specific embed- ding 6m IRdd by concatenation. The combined rep- resentation then passes through D transformer layers with our Sparse Cross-species MoE for further transformation. This design is analogous to the source tokens used in recent language models (Jiang et al., 2023), where document-level embeddings are prepended to provide explicit context about the content source. In our case, the species-specific em- bedding serves an explicit signal to guide the model in distinguishing and handling species-specific characteristics. Among DNA models, Karollus et al. (2024) employed similar approach.
After the encoding stage, we obtain the sequence represen- tation y E TR. Lxdx that captures both species-specific and shared biological features. Note that the sequence length remains L as we exclude the species embedding token from the final representation, retaining only the original sequence positions for downstream genomic profile prediction.
3.5. Profile-goouped Enhancement Decoder
Current GPPMs treat profile prediction as independent multi- tasks, overlooking inherent relationships between genomic profiles. This limitation motivates our decoder design based on the following biological principles:
Cross-species MoE. Furthermore, we introduce a sparse MoE encoding module that enables adaptive species-aware representation learning through dynamic parameter routing. For the M species (.., SM }. each MoE layer consists of two core components: (1) set of N shared expert networks [Ex...,Exx] and (2) M species-specific gating networks {GI, G.. where each Gm is associated with species S.. to dynamically weight expert contributions based on species-specifie patterns.
Biological principles for profile relationships. Genomic profile prediction should account for two key biological relationships: (P1) Evolutionary conservation: Shared reg- ulatory mechanisms exist across homologous profiles in different species (Schmidt et al., 2010). (P2) Functional in- terdependencies: Different genomic profiles often share reg- ulatory mechanisms and exhibit mutual influences, c-g., re- gions with high gene expression typically exhibit increased chromatin accessibility (Fu et al., 2025).
For an aggregated hidden state fim from species S., the output representation ym is computed as:
To leverage these biological insights. we propose dual- layer gated prediction enhancement module that enables systematic knowledge sharing across profiles. For clarity, we present the formulation for single species Sm, and omit the subscript m in subsequent notation.
hm MHAttention((Cm. hml))
N
Gm (fom)
Eu (hmm)
Vm
the k-th value of (ho(h))))
Genomic profiles can be categorized based on their experi- mental assays: for instance, DNase and ATAC-seq measure chromatin accessibility, while CAGE quantifies gene ex- pression levels. Profiles from the same experimental type typically share similar functional mechanisms, enabling knowledge transfer within each category. Given Q distinct profile types (Ti, Tq) with specific biological interpreta- tions, for the DNA sequence representation y e IR xxd, and the species embedding E TRdh,, the enhancement module operates through the following sequential steps.
where cm E Rd denotes the species embedding vector, and [-] represents concatenation, Atm is the hidden state after attention. The gating weights are computed through:
Gm(hmn)) Softmax (Topk (o(fm)) + Rmoise)
where g(-) is the gating function and Rnoise is noise injec- tion term for training stability enhancement (Fedus et al., 2022). Moreover, to explicitly guide expert networks in learning both conserved and species-specific patterns, we introduce an sepert-species mutual information loss inspired by Mod-Squad (Chen al., 2023). We maximize the mutual information between species identity S and expert selection E to encourage species-speciff expert specialization:
Profile categorization for initial predictions. We first perform a linear projection on y to obtain the initial base prediction Obase, which represents the final profile predic- tions from previous GPPMs (Kelley, 2020; Avsec al., 2021) that do not incorporate biological insights. Based on biological priors, Orase is categorized into Q independent parts ....,,o)}, as follows.
LM = MI((S E) = -H(S) - H(E) - H(S,E) E)
M.
N
P(Sm) log P(Sm) + Σ P(En) log P(E,)
Obase = (Linear(y)) (y))
n=l
(4)
N
(o! (OOmee)
Σ
Σ P(Sm: F.n) log P(Sms F,,,,
where dout denotes the dimension specifying the total num- ber of genomic profiles (i.c., dour equals Cm for species S,.). The category operator (b-) is constructed based on domain- specific biological knowledge, which decomposes the base prediction into Q profile types (o @ ) where 04 E R
(3) where Sm denotes the species probability and E,, represents the selection weight of each expert. The detailed derivations are provided in Appendix A.

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
corresponds to biological profile type T.. with dy indicating the number of profiles categorized to Tq.
3.6. Training Objective
Following Enformer (Avsec et al., 2021), adopt the Pois- son negative log- oggliikeihood the primary loss function. To further refine species-award aware expert selection in Section 3.4 by maximizing mutual information between species propor- tion and expert activations, we introduce an auxiliary mutual information loss. The composite loss is defined as:
Dual-gated expert weighted aggregation. Each dimension of 6 represents the base predicted sequence for specific profile track. To capture shared regulatory patterns, we em- ploy cross-profile-type shared experts [E where each expert Ex R2 - Fd,x1. enhances the catego- rized base prediction 0º.
D
For adaptive expert selection, we introduce dual-gated mechanism with two complementary components. The first layer (group-level gating) addresses principle (P1) by dynamically assigning weights to R expert groups based on species and sequence context, capturing evolutionary conservation patterns. The group weights 6 are computed as:
Liotal = CPoisson C' MI(S;
(8)
where 0.01 controls the mutual information regulariza- tion strength, D denotes the number of transformer layers, S represents the species identifier, and Ed indicates the shared expert pool at layer d, the Poisson loss Poossoon is mathematically formulated in Appendix A.2.
G = SSotmmx (iuueces (e) + Sequence (Poo)(y)) (5)
4. Experiments
where G Grpecies (.) and ( map Rd + R², weight- ing the R expert- selected groups from species and sequence perspectives, respectively. Here, Pool(-) denotes global av- erage pooling applied along the sequence length L and q indicates the profile type. The second layer (expert-level gating) addresses principle (P2) by selecting specific ex- perts based on profile prediction patterns, capturing func- tional interdependencies between profiles. For each group r and profile type q. we define the expert selection func- tion (09) E K where (o represents the weight of the k-th expert within the r-th group, determined by the base prediction patterns. The computation of C follows similar gating mechanism as in Equation 2.
4.1. Experiment Setup
Pre-training dataset. The pre-training dataset aligns with that used in Enformer (Kelley, 2020; Avsec et al., 2021), containing DNA sequences and corresponding genomic profiles for human and mouse genomes. Both species shared four conserved profile types: chromatin accessibility (DNase/ATACC ATAC-se)), transcription factor binding (TF ChIP- seq), histone modifications (Histone ChIP-seq). and tran- scriptional activity (CAGH). The number of profiles varies among different profile types in different species, with de- tailed dataset specifications provided in Appendix B.
Implementation details. Our model was pre-trained using supervised genomic profile prediction, maintaining the same prediction targets and genomic intervals as implemented in Enformer (Avsec et al., 2021). For rrosssspecies joint mod- eling, implemented an alternating training strategy using eight NVIDIA A40 GPUs. Training proceeded for 50,000 steps (approximately 8 days) with global batch size of 64, achieved through 8 gradient accumulation steps (1 sample per GPU). Optimization employed Adam W (Loshchilov & Hutter, 2019) with an initial learning rate of x 10-4, lin- early ramped from 0 during the first 5,000 steps followed by cosine decay. Gradient norms were clipped 0.2 to maintain stability.
Combining both gating layers, the enhanced prediction for profile type T, is formulated as:
R
2 entanced
G Group weight
G2(09)& Fo(
Expert weight
(6)
where the dual-gated mechanism first weights relevant ex- pert groups through G7 (based on species and sequence context), then within each selected group, chooses appropri- ate experts through (o")) (based on prediction patterns).
The final predictions are computed through connections between enhanced and base predictions:
4.2. Nucleotide Transformer Downstream Tasks
Oenhanced Vt (fo (Oenhanced (alaasooe))
Q
We conducted rigorous benchmarking against the suite of 18 genomic datasets established in NT (Dalla-Torre et al., 2024), encompassing three fundamental task cate- gories: (1) histone modification marker prediction, (2) cis- regulatory element annotation, and (3) splice site recog- nition. Following the evaluation protocol from NT, we employed the Matthews Correlation Coefficient (MCC)
T Genhanced:
Ofiul Obuse
where (P() is the inverse operator of (--), concatenating the enhanced predictions across different profile types, and the residual connection ensures stable training.
5

---

SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model
Table MCC performance of Nucleotide Transformer downstream tasks. This benchmark includes three categories downstream tasks, comprising a total of 18 datasets derived from human samples. The term 'NT downstream tasks' will be used to refer to these tasks.
Chromatin profiles
Model
H2AFZ
H3K27ac
H3K27mc3
H3K36me3
H3K4mel
H3K4me2
DNABERT-2
0.490 + 0.013
0.491 + 0.010
0.599 + 0.010
0.637 +: 0.007
0.490 + 0.008
0.558 + 0.013
HyenaDNA-32KB
0.467 0.012
0.421 1 0.010
0.550 1 0.009
0.553
0.011
0.423 1 0.016
0.515 1 0.018
NT-Humanneef (500M)
0.465 + 0.011
- 1.010
0.589 1 0.009
0.594 1: 0.004
0.468 1 0.007
0.527 1 0.011
NT-1000G (500M)
0.464 0.012
0.458 = 0.012
0.591 1 0.007
0.581 0.009
0.466 1 0.006
0.528 1 0.011
NT-1000G (2.5B)
0.478 + 0.012
0.486 = 0.023
0.603 + 0.009
0.632 +: 0.008
0.491 + 0.015
0,569 + 0.014
NT-Multispecies (2.5B)
0.503 0.010
0.481 = 0.020
0.593 + 0.016
0.635 0.016
0.481 + 0.012
0.552 1 0.022
Enformer
0.522 0.019
0.520 0.015
0.552 + 0.007
0.567
0.017
0.504 0.021
0.626 + 0.015
SPACE
0.548 0.005
0.547 + 0.007
0.586 + 0.010
0.602 + 0.005
0.543 + 0.009
0.640 + 0.007
Chromatin profiles
Regulatory elements
Model
H3K4me3
H3K9ac
H3K9mc3
H4K20mel
Enhaneers
Enhancers(types)
DNABERT-2
0,646 1 0.008
0.564 1 0.013
0,443 1 0.025
0.655 1 0.011
0.517 + 0.011
0,476 1 0.009
HyenaDNA-32KG
0.603 0.020
0.487 1 0.025
0.419 1 0.030
0.590 0.007
0.476 1 0.021
0.445 0.009
NT-HumanRef (500M)
0.622 + 0.013
0.524 0.013
0,433 1 0.009
0.634
0,515 + 0.019
0.477 + 0.014
NT-1000G (500M)
0.609 0.011
0.515 0.018
0.01
0.505 1 0.009
0.459 0.011
NT-1000G (2.5B)
0.615 + 0.017
0.529 = 0.012
0,483 0.013
0.659 :: 0.008
0,504 1 0.009
0.469 + 0.005
NT-Multispecies (2.5B)
0.618 0.015
0.527 = 0.017
0.447
0.650
0.527 0.012
0,484 + 0.012
Enformer
0.635 0.019
0.593
0.020
0.453 + 0.016
0.606
0.016
0.614 0.010
0.573 0.013
SPACE
0.661 0.025
0.635 + 0.016
0.490 + 0.011
0.650 + 0.011
0.631 + 0.007
0.583 + 0.008
Regulatory elements
Splicing
Model
All
NoTATA
TATA
Donors
Acceptors
All
DNABERT-2
0.754 + 0.009
0.769 = 0.009
0,784 + 0.036
0.837 + 0.006
0.855 + 0.005
0.861 + 0.004
HyenaDNA-32KB
0.698 0.011
0.729 = 0.009
0.666 + 0.041
0.808 0.009
0.907 0.018
0.915 0.047
NT-HumanRef (500M)
0.734 + 0.013
0.738 + 0.008
0.831 + 0.022
0.941 + 0.004
0.939 + 0.003
0.952 + 0.003
NT-1000G (500M)
0.727 0.004
0.743 + 0.012
0.855 + 0.041
0.041
0.933 + 0.007
0.939 0.004
0.952 0.004
NT-1000G (2.5B)
0.708 + 0.008
0.758 + 0.007
+ 0.030
0.952 + 0.004
0.956 + 0.004
0.963 + 0.001
NT-Multispecies (2.5B)
0.761 0.009
0.773 + 0.010
0.944 0.016
0.958
0.003
0.964
0.003
0.970 + 0.002
Enformer
0.745 0.012
0.763 = 0.012
0.793 + 0.026
0.749 1 0.007
0.739 + 0.011
0.780 + 0.007
SPACE
0.764 = 0.012
0.776 = 0.011
0.838 + 0.028
0.942 + 0.006
0.902 + 0.004
0.906 + 0.003
as the primary performance metric across all tasks to en- sure methodological consistency. The formal definition of MCC, along with its theoretical properties, is comprehen- sively detailed in Appendix A.3. Our comparative anal- ysis includes both unsupervised pre-training approaches (DNABERT (Ji et al., 2021), DNABERT2 (Zhou et al., 2024), and NT (Dalla-Torre al., 2024)) and supervised baseline (Enformer (Avsec et al., 2021)). In alignment with NT's methodology, we implemented 10-fold cross- validation with fixed random seeds (0-9) and early stopping based on validation performance. All benchmark perfor- mance metrics for the compared models in downstream tasks are directly sourced from the original experimental results reported in NT, ensuring consistent evaluation proto- cols and dataset configurations. As detailed in Table 1, our model achieves SOTA performance on 11 out of 18 predic-
tion tasks. Notably, this superior performance persists even when compared to the parameter-intensive NT-Multispecies variant (2.5B parameters), demonstrating that our supervised pre-training paradigm enables the acquisition of more robust DNA sequence representations. Moreover, our architectural improvements consistently outperform Enformer's 's original implementation across all tasks, empirically confirming the effectiveness of our modules. The specific details and com- plete results of the tasks are presented in Appendix C.
4.3. Cross-species Validation GUE Benchmark
To rigorously evaluate the cross-species generalization ca- pacity of our architectural refinements to Enformer, we employed the GUE benchmark (Zhou et al., 2024). While the benchmark encompasses 7 prediction tasks across 4 taxonomic groups, our experimental design strategically pri-
6

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
Table Comparison Results with Enformer on the GUE Benchmark. MCC was used in EMP tasks, while Fl-score was employed the CVC task. The upward arrow (1) denotes SPACE's relative improvement over Enformer in novel species.
Epigenetic Marks Prediction
Model
H3
H3K144c
H3K36me3
H3K4mel
H3K4me2
Enformer
70.65
37.87
42.41 54.82 (1 12.41)
34.00
29.65
SPACE
79.53 (1 8.88)
54.12 16.25)
50.92(1 16.92)
43.80 (1 14.15)
Epigenetic Marks Prediction
Virus
Model
H3K4me3
H3K79me3
H3K9ac
H4
H4ac
Covid
Enformer
22.19 49.47 (1 27.28)
55.69 66.93 (1 11.24)
49.35
76.32
32,90
61.33
SPACE
59.29 (1 9.94)
81.25 (t 4.93)
53.09 (? 20.19) 70.26 (1 8.93)
oritizes yeast and viral genomic contexts evolutionarily distant lineages characterized by marked nucleotide-level divergence from the mammalian species used during model training. These phylogenetically distinct evaluations in- clude Epigenetic Mark Prediction (EMP) across 10 yeast datasets and COVID Variants Classification (CVC) in viral genomes. Following the evaluation protocol established in DNABERT2 (Zhou et al., 2024), we adopted the MCC for EMP and the Fi-score for CVC.
we referenced the CNN results from Genomic Benchmarks. The results are presented in the Table 3.
4.5. Analysis of the MoE architecture
Analysis of the specics-aware encoder module. As shown in Figure 2a, this part reveals the implicit species- characteristic learning mechanism through visual analysis of expert selection frequencies in the final Transformer layer from the Enformer test dataset. In our experiments, we adopt a 4-expert architecture with a top-3 selection mechanism (A: 3). Quantitative analysis demonstrates significant functional differentiation in the expert system: Expert I and Expert 3 exhibit species-speciffe learning capabilities for single species (human and mouse), while Expert 0 and Expert 2 primarily participate in cross-species conserved feature extraction. This hierarchical division successfully achieves spatial decoupling of species-specifie representa- tions and evolutionarily conserved features, providing an interpretable solution for multi-species joint modeling.
As quantified in Table 2, our refined architecture demon- strates significant improvements over the original Enformer in these tasks. This systematic evaluation provides empirical evidence that our architectural modifications enhance cross- species generalization capabilities, particularly in identify- ing evolutionarily conserved regulatory features compared to Enformer's 's baseline implementation. Cross-architecture benchmarking against DNABERT2 and other established baselines (Appendix D) confirms the universality of these improvements, with non- non-Enformer baseline results rigor- ously reproduced from DNABERT2's original experimental protocol to ensure methodological consistency. All evalua- tions strictly adhered to benchmark specifications, including standardized train-test splits and hyperparameter configura- tions, to maintain reproducibility and fairness.
A Analysis of the profile-grouped enhancement decoder module. In our experiments, we employed 8 cross-pre pross-profile- type shared experts (K- = 8) with 2 expert-selected groups per profile type (R = 2). where each group dynamically integrated the top 3 most contributory experts through a dual- gated expert weighted aggregation. Through hierarchical weighting, we derived final expert selection probabilities. Normalized expert selection frequencies across Enformer test datasets were systematically quantified and visualized in Figure 2b, revealing distinct biological patterns.
4.4. Genomic Benchmarks
To further validate the capabilities of our model, per- formed extended benchmarking using the Genomic Bench- marks (Grešová et al., 2023) dataset, which represents the only mainstream benchmark encompassing species be- yond those investigated in our previous experiments, in- cluding Human-or-we worm classification and Drosophila en- hancer classification. Following a methodology similar to Caduceus (Schiff et al., 2024), we evaluated Enformer and SPACE, adopting the bascline model results reported in that paper. It is worth noting that Caduceus did not measure the enhancer prediction task for Drosophila melanogaster, so
Notably, TF binding (TF ChIP-seq) and histone modifica- tions (Histone ChIP-seq) profiles exhibited high expert spe- cialization, reflecting their biologically inherent complex- ity: TF binding involves combinatorial interactions among diverse transcription factor families, while histone modi- fications require interpretation of multilayered epigenetic codes govemed by cooperative nost-translationn modifica- tions. Conversely, Chromatin accessibility (DNase/ATAC-

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
Table The results on the Genomic Benchmarks datasets. The results for Drosophila enhancers were obtained from Genomic Benchmarks (Grešová et al., 2023), while all other baseline results were sourced from the Caduceus model (Schiff' et al., 2024).
Mouse
Demo
drosophila
Model
Enhancers
Coding VS. Intergenomic
Human VS. Worm
Enhancers
CNN
0.715 0.087
0.892 + 0.008
0,942 0.002
0.586
HyenaDNA
0.780 = 0.025
0,904 + 0.005
0.964 + 0.002
Mamba
0.743 0.054
0.904 + 0.004
0.967 + 0.002
Caduceus-PH
0.754 0.074
0.915 + 0.003
0.973 0.001
Caduceus-PS
0.793 0.058
0,910 + 0.003
0.968 + 0.002
Enformer
0.835
0.012
0.913 + 0.001
0,958 0.001
0.613 1 0.005
SPACE
0.905 0.010
0.922 + 0.001
0.967 + 0.014
0.721 + 0.016
Human
Model
Enhancers Cohn
Enhancer Ensembl
Regulatory
OCR Ensembl
Nontata Promoters
CNN
0,702 + 0.021
0,744 + 0.122
0.872 0,005
0.698 = 0.013
0,861 0.009
HyenaDNA
0.729 + 0.014
0,849 + 0.006
0.869 0.012
0.783 = 0.007
0.944 0.002
Mamba
0.732 + 0.029
0.862 + 0.008
0.814 0.211
0,815 + 0.002
0,933 0.007
Caduccus-Pe PH
0.747 + 0.004
0.893 + 0.008
0.872 + 0.011
0.828 = 0.006
0.946 + 0.007
Caduceus-PS
0,745 + 0.007
0,900 + 0.006
0.873 = 0.007
0.818 = 0.006
0,945 = 0,010
Enformer
0.723 + 0.001
0.844 : 0.001
0.903 + 0.001
0.876 = 0.001
0.878 + 0.002
SPACE
0.769 + 0.006
0,919 + 0.014
0,944 0.002
0.854 = 0.001
0,940 + 0.002
seq signals) and transcription initiation (CAGE signals) profiles showed substantial expert overlap with differen- tial weighting, mirroring their mechanistic and positional interdependence. Chromatin accessibility creates permis- sive 3D topological environments essential for transcription initiation, with transcription start site -aSS)aassociated pro- moter and enhancer regions exhibiting substantial spatial overlap with chromatin accessibility domains. This "func- tional dependency-spall
1/3 of those used by Enformer, yet we were still able to achieve good results on the training objective tasks.
4.7. Ablation Study
We conducted ablation experiments with a half-scale model (hidden dim=768) on five configurations: (1) baseline with- out the prediction-enhanced decoder, (2) decoder replace- ment with a parameter-matchhe MoE layers with standard FFNs in the encoder, (4) addi- tional removal of species embeddings from configuration (3), and (5) our complete dual-module architecture. The results are presented in the Table 4. SPACE demonstrates superior performance across most tasks, with the notable exception of the TATA box dataset (see Table 14)- - due to its exclusive focus simple sequence motifs rather than complex regulatory mechanisms. This indicates that while our decoder doesn't 't directly boost chromatin profile pre- diction accuracy, the MoE architecture implicitly models cross- cross-profile regulatory dependencies, offering significant advantages for tasks requiring integrated profile understand- ing. Cross-species evaluation on the GUE benchmark (yeast and virus tasks, detailed in Table 15) further demonstrates that the MLP-based decoder variant exhibits substantially weaker generalization to new species compared to SPACE's enhancement decoder architecture
MLP, (3) substitution of
coupling" drives the prediction en- hancement decoder module to develop coordinated feature extraction strategies. These findings collectively demon- strate that divergent genomic profiles converge on shared regulatory architectures and engage in reciprocal regulatory interactions- a phenomenon that fundamentally informed our development of the defile-gruued enhancement de- coder to systematically leverage such interconnectivity.
4.6. Comparative Analysis with Enformer in Gene Expression Prediction
We conducted a comparative analysis based on the core task of the baseline Enformer model, which aims to predict hu- man and mouse genomic profiles 128-bp resolution from input DNA sequences. We computed the average Pearson correlation coefficients across all positions for genomic pro- files in the test set and performed stratified visualization by species and profile types, as illustrated in Figure 2c, The results demonstrate that our approach significantly enhances the prediction accuracy for mouse genomic profiles while maintaining the prediction performance for human genomic profiles. It should be noted that our training steps were only
5. Limitations
This work has limitations in data coverage and model scale. First, SPACE has only been trained on two species. While
8

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
CAD
GAROPAA
27.98
40:90
30.63
0.50
Nouse
6m
1.4
6m
15000077
ST2095413
(b)
Enformer
(c)
Figure 2. Expert selection visualizations and prediction results, (a) Visualization of expert selection in the final cross-species MoE. (b) Expert selection in the profile-grone grouuee enhancement decoder module. Pearson correlation coefficients on the test set. Each point represents the average correlation of predicted genomic profiles across all 128-bp binned genomic positions.
all positions per profile
Table 4. Ablation Studies NT downstream tasks and GUE benchmarks. The results include the average outcomes of the three major categories of downstream tasks and average results the EMP tasks the CVC task in GUE benchmark experiments.
NT
GUE
Model
Chromatin
Regulatory
Splicing
EMP
CVC
SPACE w/o decoder
0.5674
0.7054
0.8977
0.5339
0.6866
SPACE w/o decoder w/ MLP
0.5651
0.6920
0.9020
0.5153
0.6783
SPACE w/o encoder
0.5653
0.7022
0.8887
0.5346
0.6846
SPACE w/o encoder w/o species emb
0.5692
0.6986
0.8957
0.5322
0.6856
SPACE
0.5705
0.7024
0.9077
0.5368
0.6889
this study demonstrates the advantages of our design, we believe extending ditional data becomes available (Vandereyken al., 2023). Second, constrained by computational resources, our model (588M, sparse-activated) is significantly smaller than the largest variant of NT (2.5B, dense) (the detail in Ap- pendix H. Given the observed scaling laws DFMs (Dalla- Torre et al., 2024; Nguyen al., 2024a), we believe increas- ing model scale would lead to improvements.
that our supervised pre-training with the proposed SPACE sets new standard for DNA representation learning, paving the way for future developments in DFMs.
species would yield benefits ad-
Impact Statement
This paper presents work whose goal is to advance the field the intersection of Machine Learning and Genomics. There are many potential societal consequences of work in this interdisciplinary area, none of which feel must be specifically highlighted here.
6. Conclusion
In this work, we demonstrate that supervised pre-training via genomic profile prediction serves as a morceffective and targeted alternative to pure sequence pre-training for DNA foundation models. To distinguish unique characteristies of different species and profiles while fully utilizing the transferrable knowledge among species and profiles, we introduce SPACE that provides biological insights into the model architecture. Through extensive evaluations, we show
A cknowledgements
This work was supported in part by the National Natural Sci- ence Foundation of China No. 62376277 and No. 61976206. the Public Computing Cloud of Renmin University of China, and the Fund for Building World-Class Universities (Disci- plines) of Renmin University of China.

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
References
Gosai, S. J., Castro, R. I., Fuentes, N., Butts, J. C., Mouri, K., Alasoadura, M., Kales, S., Nguyen, T. T. L., Noche, R. R., Rao, A. S., et al. Machine-guides design of cell- type-targeting cis-regulatory elements. Nature, pp. 1-10, 2024.
Avsec, Z., Agarwal, V., Visentin, D., Ledsam, J. R., Grabska- Barwinska, A., Taylor, K. R., Assael, Y., Jumper, J., Kohli, P., and Kelley, D. R. Effective gene expression predic- tion from sequence by integrating long-range interactions. Nature methods, (910):116-11233 2021.
Grešová, K., Martinek, V., Čechák, D., Simeček, P., and Alexiou, P. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24 24(1):25, 2023.
Benegas, G., Albors, C., Aw, A. J., Ye, C., and Song, Y. S. A dna language model based on multispecies alignment predicts the effects of genome-wide variants. Nature Biotechnology, pp. 1-6, 2025.
Jacobs, R. A., Jordan, M. I., Nowlan, J., and Hinton, G. E. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 87, 1991. doi: 10.1 1a.1166/nncc.199 190///cco.199...... 1991.3.1.799
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Ji, Y., Zhou, Z., Liu, H., and Davuluri, R. V. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioin- formatics, 37(15):2112-212) 2021.
Celikkanat, A., Masegosa, A. R., and Nielsen, Re- visiting k-mer profile for effective and scalable genome representation learning. In The Thihtt-eighth Annual Con- ference Neural Information Processing Systems, 2024.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S.,, Casas, D. d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.., et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
Chen, K. M., Wong, A. K., Troyanskaya, O. G., and Zhou, J. A sequence- seaeence-bbeed global map of regulatory activity for deciphering human genetics. Nature genetics, 54(7): 940-949, 2022.
Karollus, A., Hingerl, J., Gankin, D., Grosshauser, M., Kle- mon, K., and Gagneur, J. Species-aware dna language models capture regulatory elements and their evolution. Genome Biology, 25(1):83, 2024.
Chen, Z., Shen, Y., Ding, M., Chen, Z., Zhao, H., Learned- Miller, E. G., and Gan, C. Mod-squad: Designing mix- tures of experts modular multi-task learners. In Pro- ceedings of the IEEE/CVF Conference Computer Vi- sion and Pattern Recognition, pp. 11828-11837, 2023.
Kathail, P., Bajwa, A., and loannidis, N. M. Leveraging genomic deep learning models for non-coding variant effect prediction. arXiv preprint arXiv:2411.11158. 2024.
Darre-Torre, H., Gonzalez, L.., Mendoza- Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F,, Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et Nucleotide transformer: building and evaluating robust foundation models for human genomies. Nature Methods, pp. 1-11, 2024.
J.,
Kelley, D. R. Cross-species regulatory sequence activity prediction. PLoS computational biology, 16(7))e1088050, 2020.
Kelley, D. R., Reshef, Y. A., Bileschi, M., Belanger, D., McLean, C. Y., and Snoek, Sequential regulatory ac- tivity prediction across chromosomes with convolutional neural networks. Genome research, 28(5):739 2018.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume (Long and Short Papers). pp. 4171-4186, 2019.
Kernohan, K. D. and Boycott, K. M. The expanding diag- nostic toolbox for rare genetic diseases. Nature Reviews Genetics, 25(6):401-415, 2024.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-exxpert language model. arXiv preprint arXiv:2405.0444 0444444 2024.
Fu. X., Mo, S., Buendia, A., Laurent, A. P., Shao, A., Alvarez-Torre M. d. M., Yu, T., Tan, J., Su, J., Sagatelian, R., et al. A foundation model of transcription across human cell types. Nature, pp. 1-9, 2025.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019. URL https: h/tp://////eeeiie.. net pet./forumiid-Bee 6RicC77767
10

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
Marin, F. I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., and Boomsma, W. Bend: Benchmarking dna language models on biologically meaningful tasks. In The Twelfth International Conference on Learning Representations, 2024.
Tang, Z. and Koo, P. K. Evaluating the representational power of pre-trained dna language models for regulatory genomics, march 2024, 2024.
Tang, Z., Toneyan, S., and Koo, K. Current approaches genomic deep learning struggle to fully capture human genetic variation. Nature Genetics, 55(12):2021-202) 2023.
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G. et al. Sequence modeling and design from molecular to genome scale with evo. Science, 386(6723); eado9336, 2024a.
Trajanoska, K., Bhérer, C., Taliun, D., Zhou, S., Richards, J. B., and Mooser, V. From target discovery clinical drug development with human genetics. Nature, 620 (7975):737-745, 2023.
Nguyen, E.., Poli, M.. Faizi, M., Thomas, A., Wornow, M., Birch-Sykes, C., Massaroli, S., Patel, A., Rabideau, C., Bengio, Y., et al. Hyenadna: Long-range genomic se- quence modeling at single nucleotide resolution. Ad- vances in neural information processing systems, 36, 2024b.
Vandereyken, K., Sifrim, A., Thienpont, B., and Voet, T. Methods and applications for single-cell and spatial multi- omics. Nature Reviews Genetics, 24(8):494-513 2023,
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P. Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer. S., von Platen, P., Ma, C., Jer- nite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pp. 38 38-45, Online, October 2020. Association for Compu- tational Linguisties. URL. ht.ps:: /www///wwww.... acl web. org/ //a/athholgyy/// 2oolooyyy/200000 emnlp demos. 6.
Peterson, A. A. and Liu, D. R. Small-molecule discovery through dna-encoded libraries. Nature Reviews Drug Discovery, 22(9):699-722, 2023.
Portela, A. and Esteller, M. Epigenetic modifications and human disease. Nature biotechnology. 28(10):0057-1068, 2010.
Sanabria, M., Hirsch, J., Joubert, P. M., and Poetsch, A. R. Dna language model grover learns sequence context in the human genome. Nature Machine Intelligence, 6(8): 911-923, 2024.
Yang, Z., Su, B., Cao, C., and Wen, J.-R. Regulatory DNA sequence design with reinforcement learning. In The Thirteenth International Conference on Learning Repre- sentations, 2025.
Schiff, Y., Kao, C.-H., Gokaslan, A., Dao, T., Gu, A., and Kuleshov. V. Caduceus: Bi-directional equivari- ant long-range dna sequence modeling. arXiv preprint wrXiv22403.03234, 2024.
Zhou, J. and Troyanskaya, O. G. Predicting effects of noncoding variants with deep learning- based sequence model. Nature methods, 12(10):93--934 2015.
Schmidt, D., Wilson, M. D., Ballester, B., Schwalie, C., Brown, G. D., Marshall, A., Kutter, C., Watt, S., Martinez- Jimenez, C. P., Mackay, S., et al. Five-vertebrate chip-seq reveals the evolutionary dynamics of transcription factor binding. Science, 328(5981 (18(5591)110361100 10:106-1040, 2010.
Zhou, J., Theesfeld, C. L., Yao, K., Chen, K. M., Wong, A. K., and Troyanskaya, O. G. Deep learning sequence- based ab initio prediction of variant effects on expres- sion and disease risk. Nature genetics, 101-):1 2018.
Sermon, K., Van Steirteghem, A., and Liebaers, I. Preim- plantation genetic diagnosis. The Lancel, 363(9421): 1633-1641, 2004.
Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri, R. V., and Liu, H. Dnahert-2: Efficient foundation model and benchmark for multi-species genomes. In The Twelfth International Conference on Learning Representations, 2024.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, J. Outrageously large neural mixture-of-exxper layer. 2017.
networks: The sparsely
arXiv preprint
Tan, J., Thenksr-Tauris, N., Rodriguez- Rodriguez-Hermaez, Hernaez, J., Wang. E., Sakellaropoulos, T., Boccalatte, F., Thandapani, P., Skok, J.. Aifantis, I., Fenyö. D., et al. Cell-type-specific prediction of 3d chromatin organization enables high- throughput in silico genetic screening. Nature biotechnol- ogy, 41): 140- 1150, 2023.

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
A. Derivation of Mathematical Formulations for Key Functions
A.I. Mutual Information Analysis
The Mutual Information defined in Equation (3) is:
LI - MT(S; = -
M
P(Sm) P(Sm)) + Σ P(En) log P(En)
M P(Sm: E,) log P(Sm, En).
ΣΣ
where Sm denotes the species probability and En, represents the selection weight of each expert.
We split the formulae to analyse them separately. The mutual information decomposition exhibits three fundamental components:
Species Entropy:
M P(S.) log P(S,) = H()).
This term represents the inherent diversity of species distribution in training data. As P(S:) constitutes a fixed prior, (S) remains constant during optimization.
Expert Diversity Regularization:
Σ Pog = H(E).
Maximizing this entropy term encourages balanced utilization of experts, preventing expert collapse where few experts dominate computations. Formally, this ensures:
lim HlE)- N
P(E,)
Vj.
Conditional Specialization Objective:
M
N
P(S. log E,) ES,
Minimizing this joint entropy (equivalent to maximizing (S( (.S,)) E)) sharpens the conditional distribution (EE))))). thereby promoting:
jf arg max; G (æ:)
lim PSE,(S.) -)
0 otherwise
This objective ensures that, for given species, the model preferentially activates a fixed subset of & experts.
In this way, the sparse MoE-based encoding module encourages different expert combinations to handle different species, while some shared experts the pool can capture common knowledge across species.
A.2. Poisson negative log-likelihood
The Poisson negative lig-liiel function is defined as
N
L Poisson
Σ
(p: - (p Inp:;
N
12

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
whree p denotes the prediction vector and represents the target vector.
A.3. Matthews Correlation Coefficient (MCC)
The Matthews Correlation Coefficient (MCC) is a statistically rigorous metric for evaluating classification models. Its definition and generalization multi-class problems are formally outlined below.
Binary Classification Case For binary classification, let TP, TN.FP, and FN denote the counts of true positives, true negatives, false positives, and false negatives, respectively. The MCC is defined as:
TN. - FP- FN EN)(TN
MCC
FNJ(NN + FP)(TN +FN)
Here, and FN correspond to entries in the confusion matrix for two classes.
Multi-class Classification Case
For K-class classification (K 2 2), be the K K confusion matrix, where Ciy represents the number of samples
from class predicted class j. The MCC generalizes to:
ΣK
K Σ 1=1
EmEl
CubCim CRICmk
MCC
k=
E EK
CR
ΣK
Cmm) mif
EK ΣK
Cik CI mzf
Σ Ckm
This formulation quantifies the covariance between all class pairs, ensuring robustness imbalanced data distributions.
The MCC ranges in [-1, 1], where 1, 0, and -1 correspond to perfect prediction, random guessing, and total disagreement, respectively.
B. Pre-training Dataset
Table Genomic Dataset Statistics
Species
Train
Val
Test
Sequence Length
Human
34,021
2,213
1,937
131,072 bp
Mouse
29,295
2,209
2,017
131,072 bp
Our model pretrained the same dataset Enformer (Avsec et al., 2021), with detailed composition statistics provided in Table 5. To address the pronounced species imbalance between human and mouse genomic data, implemented balanced batch sampling through randomized minority-class augmentation, ensuring equal representation of both species in every batch. This strategy mitigates species bias while preserving sequence diversity through stochastic resampling.
The dataset comprises DNA sequences paired with genomic profiles as prediction targets. These genomic profiles are categorized into four functional classes: chromatin accessibility (DNase/ ATAC-seq), transcription factor binding (TF ChIP-seq), histone modifications (Histone ChIP-seq), and transcriptional activity (CAGE). The species-specific distribution of profile types is quantified in Table 6, which details the number of available tracks per category for each organism.
Table 6. Distribution of Genomics profiles
species
DNase/ATA
TF ChIP
Histone ChIP
CAGE
Total
Human
684
2131 308
1860
638
5313
Mouse
228
750
357
1643
13

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
C. Nucleotide Transformer Downstream Tasks Revised
C.1. Datasets
The benchmark dataset comprises 18 downstream tasks originally proposed in Nucleotide Transformer (NT) (Dalla- co ctt.a/aaaaa.ecctettttteeeeeeeeee htt.ss/sttteeeeeeccleeeattoot These tasks establish unified genomies benchmarking frame-
Torre al., 2024), accessible via
trans forme rwt__oosstteemtttsssseeeeeee work encompassing both binary and multi-class classification challenges. All data exclusively derived from human samples, organized into three biologically meaningful categories: Chromatin Profiles, Regulatory Elements and Splicing. The complete dataset composition, including sequence numbers, class distributions and sequence length statistics, is detailed in Table 7.
Table 7. Details of the NT downstream tasks
Task
Number train sequences
Number of test sequences
Number of labels
Sequence length
promoter all
30,000
1,584
2
300
promoter_ta
5,062
212
2
300
promoter no tata enhancers
30,000
1,372
2
300
30,000
3,000
2
400
enhancers types
30,000
3,000
3
400
splice sites all
30,000
3,000
3
600
splice.sites_acceptoo
30,000
3,000
2
600
pplice_sites_dongg _donor
30,000
3,000
2
600
H2AFZ
30,000
3,000
1,000 2
1,000
H3K27ac
30,000
1,616
2
H3K27mc3
30,000
3,000
2
1,000
H3K36me3
30,000
3,000
2
1,000
13K4me1
30,000
3,000
2
1,000
H3K4me2
30,000
2,138
2
1,000
H3K4me3
30,000
776
2
1,000
НЗК9ac
23,274
1,004
2
1,000
H3K9me3
27, 27,438
850
2
1,000
H4K20me1
30,000
2,270
2
1,000
C.2. Implementation
We maintained identical hyperparameter configurations across all tasks. Our systematic hyperparameter search included learning rates of 5 x 10-5,3 3 10-5, and 5 10-1, combined with batch sizes of 8, 16. and 32. Through empirical validation, we identified the optimal configuration employing a learning rate of 5 x 10-5 with batch size 8. The training protocol utilized the AdamW optimizer (Loshchilov & Hutter, 2019) over epochs, while retaining default parameter settings from the HuggingFace Transformer Trainer implementation (Wolf et al., 2020).
C.3. Results
The complete benchmark results the downstream tasks for NT are presented in Table 8. All baseline results are sourced from NT (Dalla-Tore Torre et al., 2024). Performance per task was calculated as the median the 10 cross- validation folds (# standard deviation). The best results for each task are highlighted bold.
14

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
Table 8. Complete Renchmark Results of Nucleotide Transformer Downstream Tasks
Chromatin profiles
Model
H2AFZ
H3K27ac
H3K27mc3
H3K36mc3
H3K4me1
H3K4me2
BPNet (original)
0.473 0.009
0.296 0.046
0.543 + 0.009
0.548
0.009
0.436
0.008
0.427 0.036
BPNet (large)
0.487 + 0.014
0.214 1 0.037
0.551 1: 0.009
0.570 + 0.009
0.459 + 0.012
0,427 + 0.025
DNABERT-2
0.490 0.013
0.491 1 0.010
0.599 1 0.010
0.637 0.007
0.490 0.008
0.558 0.013
HyenaDNA-IKS
0.455 + 0.015
0.423 - 0.017
0.541 0.018
0.543 1: 0.010
0.430 1 0.014
0,521 + 0.024
HyenaDNA-32KB
0.467 0.012
0.421 = 0.010
0.550 1 0.009
0.553 0.011
0.423 1 0.016
0.515 0.018
NT-HumanRef (500M)
0.465 + 0.011
0.457 = 0.010
0.589 0.009
0.594 0.004
0.468 + 0.007
0,527 + 0.011
NT-1000G (500M)
0.464 0.012
0.458 = 0.012
0.007
0.581 0.009
0.466 0.006
0.528 1 0.011
NT-1000G (2.5B)
0,478 = 0.012
0.486 = 0,023
0.603
0.632
0.491 0.015
0,569 1069:0000 0.014
NT-Multispecies (2.5B)
0.503 0.010
0.481 = 0.020
0.593 0.016
0.635
0.481 = 0.012
0.552 - 0.022
Enformer
0.522 0.019
0.520
0.015
0.552 + 0.007
0.567
0.017
0.504 0.021
0.626 0.015
SPACE
0.548 0.005
0.547 0.007
0.586 + 0.010
0.602 + 0.005
0.543 + 0.009
0.640 + 0.007
Chromatin profiles
Regulatory elements
Model
H3K4me3
H3K9ac
H3K9me3
H4K20me1
Enhancers
Enhancers(types)
BPNet (original)
0,445 + 0.047
0.336 + 0.034
0.298 + 0.030
0.531 + 0.025
0.488 + 0.009
0.449 + 0.006
BPNet (large)
0.445 0.049
0.298 0.033
0.234 + 0.037
0.525 0.038
0.492 0.008
0.454 0.008
DNABERT-2
0.646 + 0.008
0.564 + 0.013
0.443 + 0.025
0.655 + 0.011
0.517 + 0.011
0.476 + 0.009
HyenaDNA-1KK
0.596 0.015
0.484 0.022
0.375 + 0.026
0.580 + 0.009
0.475 0.006
0.441 0.010
Hy2naDNA-322KB
0.603 + 0.020
0.487 + 0.025
0.419 + 0.030
0.030
0.590 + 0.007
0.476 + 0.021
0.445 + 0.009
NT-HumanRef (500M)
0.622 0.013
0.524 0.013
0.634 0.009
+ 0.009
0.515 0.019
0.477 0.014
NT-1000G (500M)
0.609 0.011
0.515 = 0.018
0.415 + 0.019
+.010
0.505 + 0.009
0.459 + 0.011
NT-1000G (2.5B)
0.615 0.017
0.529 0.012
0.483 +
0.659 0.008
0.504 + 0.009
0.469 + 0.005
NT-Multispecies (2.5B)
0.618 0.015
0.527 - 0.017
0.447 + 0.018
0.650
0.527 + 0.012
0.484 1 0.012
Enformer
0.635 0.019
0.593 = 0.020
0.453 + 0.016
A.006
0.614 - 0.010
0.573 + 0.013
SPACE
0.661 0.025
0.635 0.016
0.490 + 0.011
0.650
0.631 = 0.007
0.583 + 0.008
Regulatory elements
Splicing
Model
All
NoTATA
TATA
Donors
Acceptors
All
BPNet (original)
0.696 0.026
0.717 = 0.023
0.848 1 0.042
0.859 + 0.038
0.793 + 0.072
0.920 + 0.014
BPNet (large)
0.672 = 0.023
0.672 = 0.043
0,826 : 0.017
0.925 0.031
0.865 + 0.026
0.930 : 0.021
DNABERT-2
0.754 0.009
0.769 = 0.009
0.784 + 0.036
0.837 : 0.006
0.855 = 0.005
0.861 0.004
HyenaDNA- IKB
0.693 0.016
0.723 0.013
0.648 0.044
0.815 + 0.049
0.854 = 0.053
HyenaDNA-32KB
0.698 0.011
0.729 = 0.009
0.004
0.808 + 0.009
0.907 = 0.018
0.915 1015=000 0.047
NT-HumanRef (500M)
0.734 0.013
0.738 = 0.008
0.941 + 0.004
0.939 0
0.952 0.003
NT- (500M)
0.727 0.004
0.743 = 0.012
0.041
0.933 + 0.007
0.939 = 0.004
0.952 + 0.004
10-0 (2.5B)
0.708 0.008
0.758 = 0.007
0.030
0.952 + 0.004
0.956 - 0.004
0.963 + 0.001
NT-Multispecies (2.58)
0.761 0.009
0.773 = 0.010
+
0.958 + 0.003
0.964 + 0.003
0.970 + 0.002
Enformer
0.745 1 0.012
0.763 1 0.012
0.793 1 0.026
0.749 1 0.007
0.739 1 0.011
0.780 1 0.007
SPACE
0.764 0.012
0.776 0.011
0.838 1 0.028
0.942 0.006
0.902 0.004
0.906 0.003
D. GUE
D.I. Dataset
GUE is a comprehensive benchmark for genome understanding consising of 28 distinct datasets across tasks and species, downloaded from https: : : /tthbbbttbb.. Ctom/MMAICS--A -AADDDSE-LBADDARRTTT 2. The complete dataset composition, including sequence numbers, class distributions and sequence length statistics, is detailed Table
15

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
Table 9, The Composition of GUE Datasets
Species
Task
Num. Datasets
Num. Classes
Sequence Length
Core Promoter Detection
3
2
70
Transcription Factor Prediction
5
100
Human
Promoter Detection
3
300
Splice Site Detection
400
Mouse
Transcription Factor Prediction
5
100
Yeast
Epigenetic Marks Prediction
10
500
Virus
Covid Variant Classification
1000
D.2. Implementation
Building upon DNABERT2's downstream task hyperparameter framework, we systematically evaluated learning rates from 5 x 10-6, 6, x 10-5, 6 x 10-5, 7 10-5, 8 x 10-5, x 10-1 while maintaining a consistent batch size of 32 across all tasks. Task-specific learning rates were empirically determined through validation set performance. The optimization process employed the AdamW algorithm (Loshchilov Hutter, 2019) with 10,000 training steps, while retaining default parameter configurations from the HuggingFace Transformer Trainer implementation (Wolf et al., 2020).
Table 10. The results on the GUE datasets
Epigenetic Marks Prediction
Model
H3
H3K 14ac
H3K36me3
H3K4mel
H3K4me2
H3K4me3
DNABERT (3-mer)
74.15
42.07
48.49
42.95
31.34
28.92
DNABERT (4-mer)
73.03
41.88
48.03
41.06
30.66
25.31
DNABERT (5-mer)
73.40
40.68
48.29
40.65
30.67
27.10
DNABERT (6-mer)
73.10
40.06
47.25
41.44
32.27
27.81
NT-SOOM-human
69.67
33.55
44.14
37.15
30.87
24.06
NT-500M-1000g
72.52
39.37
45.58
40.45
31.05
26.16
NT-2500M-1000g
74.61
44.08
50.86
43.10
30.28
30.87
NT-2500M-multi
78.77
56.20
61.99
55.30
36.49
40.34
DNABERT-2
78.27
52.57
56.88
50.52
31.13
36.27
DNABERT-2
80.17
57.42
61.90
53.00
39.89
41.20
Enformer
70.65
37.87
42.41
34.00
29.65
22.19
SPACE
79.53
54.12
54.82
50.92
43.80
49.47
D.3. Results
The results the GUE datasets are presented in Table 10 and Table 11. In accordance with the implementation protocol of DNABERT2 (Zhou et al., 2024), all benchmark tasks utilized the Matthews Correlation Coefficient (MCC) for performance evaluation, with the singular exception of viral sequence analysis where F1-score metrics were employed. The notation DNABERT2 specifically denotes the model variant that underwent additional masked language modeling (MLM) pre- training on the training sets of the Genomic Understanding and Evaluation (GUE) benchmark, as detailed in the DNABERT2 methodology.
16

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
Table 11. The results the GUE datasets.
Epigenetic Marks Prediction
Promoter Detection
Model
H3K79me3
H3K9ac
H4
H4ac
all
notata
tata
DNABERT (3-mer)
60.12
50.48
78.27
38.60
90.44
93.61
69.83
DNABERT (4-mer)
59.77
51.44
78.28
36.40
89.54
92.65
66.78
DNABERT (5-mer)
59.61
51.11
77.27
37.48
90.16
92.45
69.51
DNABERT (6-mer)
61.17
51.22
79.26
37.43
90.48
93.05
61.56
NT-50OM-human
58.35
45.81
76.17
33.74
87.71
90.75
78.07
NT-500M-1000g
59.33
49.29
76.29
36.79
89.76
91.75
78.23
NT-2500M-1000g
61.20
52.36
79.76
41.46
90.95
93.07
75.80
NT-2500M-multi
64.70
56.01
81.67
49.13
91.01
94.00
79.43
DNABERT-2
67.39
55.63
80.71
50,43
86.77
94.27
71.59
DNABERT-2
65.46
57.07
81.86
50.35
88.31
94.34
68.79
grover
86.42
92.3
59.77
Enformer
55.69
49.35
76.32
32.90
85.68
92.92
69.63
SPACE
66.93
59.29
81.25
53.09
91.90
94.23
79.13
Transcription Factor Prediction (Human)
Core Promoter Detection
Model
1
2
3
all
notata
tata
DNABERT(3-mer
67.95
70.90
60.51
53.03
69.76
70.92
69.82
78.15
DNABERT(4-mer)
67.90
73.05
59.52
50.37
71.23
69.00
70.04
74.25
(5))ERT((-mer)
66.97
69.98
59.03
52.95
69.26
69.48
69.81
76.79
(6)
66.84
70.14
61.03
51.89
70.97
68.90
70.47
76.06
NT-500M-huma
61.59
66.75
53.58
42.95
60.81
63.45
64.82
71.34
NT-500M-
63.64
70.17
52.73
45.24
62.82
66.70
67.17
73.52
NT-2550MM-1000g
66.31
68.30
58.70
49.08
67.59
67.39
67.46
69.66
NT-2500M-multi
66.64
70.28
58.72
51.65
69.34
70.33
71.58
72.97
DNABERT-2
71.99
76.06
66.52
58.54
77.43
69.37
68.04
74.17
DNABERT-2
69.12
71.87
62.96
55.35
74.94
67.50
69.53
76.18
grover
65.76
67.9
61.62
48.26
74.68
63.58
66.75
60.57
Enformer
69.42
72.76
77.88
66.41
81.89
60.94
66.46
46.21
SPACE
69.02
76.49
76.45
66.08
82.91
68. 18
68.04
79.23
Transcription Factor Prediction (Mouse)
Virus
Splice
Model
1
2
3
Covid
Splice
42.31
79.10
69.90
55.40
41.97
62.23
84.14
DNABERT(4-met
49.42
79.95
72.62
51.79
44.13
59.87
84.05
(NA
42.45
79.32
62.22
49.92
40.34
50.46
84.02
DNABERT(6-mer)
44.42
78.94
71.44
44.89
42.48
55.50
84.07
NT-500M--uumn
31.04
75.04
61.67
29.17
29.27
50.82
79.71
NT-500M-1000g
39.26
75.49
64.70
33.07
34.01
52.06
80.97
NT-2500M-1000g
48.31
80.02
70.14
42.25
43.40
66.73
85.78
NT-2500M-mullt
63.31
83.76
71.52
69.44
47.07
73.04
89.35
DNABERT-2
56.76
84.77
79.32
66.47
52.66
71.02
84.99
DNABERT-2
64.23
86.28
81.28
73.49
50.80
68.49
85.93
grover
84.35
Enformer
67.15
81.56
85.99
67.88
44.03
61.33
81.55
SPACE
65.94
84.91
90.30
86.72
50.66
70.26
87.48
17

---

SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model
E. Results on BEND Benchmark
Table 12. Results on all tasks of BEND
Genomic Tasks
Method
Chromatin accessibility
Histone modification
CpG Methylation
Variant effects (expression)
Variant effects (disease)
Expert method
0.85 BASSET
0.74 BASSET
0.93 BASSET
0.70 DEEPSEA
0.56
DEEPSEA
Fully supervised
ResNet
CNN
0.75
0.76
0.84
Pre-trained
ResNet-LM
0.82
0.77
0.87
0.35
0.55
AWD-LSTM
0.69
0.74
0.81
0.53
0.45
NT-H
0.74
0.76
0.88
0.55
0.48
NT-MS
0.79
0.78
0.92
0.54
0.77
NT-1000G
0.77
0.77
0.89
0.45
0.49
NT-V2
0.80
0.76
0.91
0.48
0.48
DNABERT
0.85
0.79
0.91
0.60
0.56
DNABERT-2
0.81
0.78
0.90
0.49
0.51
GENA-LM BERT
0.76
0.78
0.91
0.49
0.55
GENA-LM BigBird
0.82
0.78
0.91
0.49
0.52
HyenaDNA large
0.84
0.76
0.91
0.51
0,45
HyenaDNA tiny
0.78
0.76
0.86
0.47
0.44
GROVER
0.82
0.77
0.89
0.56
0.51
GPN-MSA
0.97
SPACE
0.89
0.81
0.92
0.51
0.49
Experiments in the main paper involve complete fine-tuning of our Encoder parameters. In this section, we validate effectiveness of SPACE's embeddings on the Bend Benchmark (Marin al., 2024) (i.e., directly using frozen SPACE embeddings for downstream tasks). All our experimental settings strictly follow the official configurations of Bend (Marin et al., 2024). The results shown in Figure 12. We observe that SPACE achieves SOTA performance chromatin accessibility, histone modification, and CpG methylation tasks, with chromatin accessibility surpassing the second-best method by 0.04. It is worth noting that although our supervised pre-training tasks also include chromatin accessibility and histone modification representations, which may provide potential advantages, their data processing approaches are not entirely identical. However, on the two variant effects tasks, SPACE shows limited effectiveness, similar to most DFMs. We hypothesize that masked language modeling tasks may necessary to achieve good performance on variant effect prediction (Benegas al., 2025).
F. Genomic Benchmarks
F.I Dataset
Genomic Benchmarks currently comprises nine datasets focusing on regulatory elements (promoters, enhancers, and open chromatin regions) from three model organisms: Homo sapiens (human), Mus musculus (mouse), and Caenorhabditis elegans (nematode). All data were downloaded from https: : /ttaa:///iiiii //gitubb com com/ML-BB /////MLBBiin Biii-Bionf-o-ooo Seloinfo-CEITE//eeeeee CEITEC/genome benchmard rks. The detailed composition of these datasets is presented in Table 13.
18

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
Table 13. Composition of Genomie Benchmarks.
Name
sequences
classes
Class ratio
dummy dummy-_mouse_enhaneeeee mmw.sueeehhancccrssssmmnii
1210
2
1.0
demo_ comoocoding.voo vs. intergenomic_segy
100000
2
1.0
aato__hman_or__oon
100000
2
1.0
drosophila enhancers stark
6914
2
1.0
27791
2
1.0
Sannannenhaneerss _ensembl
154842
2
1.0
human wwweeennleeeeeeeaaeeeeeeeeeeeeee
289061
3
1.2
36131
2
1.2
human_od comannocr.ccm .ersemblll
174756
2
1.0
F.2. Implementation
We systematically evaluated learning rates 10 x 10 5,- 10 10 7 10 10-5.3 10 and batch sizes 8, 16, 32, 64. The optimal learning rate and batch size for each task were determined through validation set performance experiments. The optimization process employed the Adam W algorithm (Loshchilov Hutter, 2019) with 3 training epochs, while maintaining the default parameter configuration from the HuggingFace Transformer Trainer implementation (Wolf al., 2020).
G. Ablation Study
SPACE demonstrates comparable or superior performance decoder-removed variant in 14/18 tasks, with 11/18 tasks still outperforming even when replaced by parameter-matched MLP. Notably, for regulatory element elassification tasks, SPACE achieves better results in 4/5 datasets, with the only exception being the TATA box dataset-which primarily examines sequence motifs of TATA boxes and does not require complex regulatory mechanism understanding. This suggests that while our decoder does not explicitly improve direct chromatin profile prediction accuracy, the MoE architecture implicitly captures cross-profile regulatory interactions by modeling their dependencies. This capability provides critical advantages for tasks requiring integrated understanding of multiple profiles, such as regulatory element prediction.
H. Model Parameter Counts
We present the parameter counts of SPACE and its ablation variants in Table 16. The SPACE (large) configuration represents primary model with complete architectural components for comparative analysis, while the other variants correspond to reduced-scale models specifically designed for ablation studies. These smaller models employ 131 KB input sequences with a compressed hidden dimension of 768 and operate under a batch size of 32.
Table 16. Model Parameter Counts of SPACE and its ablation variants
SPACE (large)
SPACE w/o enhancement
SPACE w/o species MoE
SPACE (small)
param counts hidden dim
588.75M
150.96M
105.19M
183. 19M
15336
768
768
768
should be particularly noted that, based the sparse architecture design of the MoE, model activates only partial subset of parameters during single forward computation. This selective parameter activation mechanism makes the number of effective parameters actually involved in the computation significantly lower than the total number of parameters in the model, thus significantly reducing the computational resource consumption while maintaining the model capacity.
19

---

SPACE: Your Genomic Profile Predictor is Powerful DNA Foundation Model
Table 14. Ablation study on NT downstream tasks.
Chromatin profiles
Model
H2AFZ
H3K27ac
H3K27me3
H3K36me3
H3K4me1
H3K4me2
SPACE w/o decoder
0.535
0.514
0.567
0.593
0.520
0.604
SPACE w/o decoder w/ ML.
0.551
0.528
0.577
0.580
0.534
0.637
SPACE w/o encoder
0.540
0.521
0.569
0.579
0.506
0.625
SPACE w/o encoder and specics cmb
0.551
0.518
0.566
0.585
0.519
0,622
SPACE
0.556
0.529
0.579
0.593
0.516
0.612
Chromatin profiles
Regulatory elements
Model
H3K4me3
H3K9ac
H3K9me3
H4K20me
Enhancers
Enhancers(types)
SPACE w/o decoder
0.661
0.601
0.452
0.627
0.598
0.563
SPACE w/o decoder w/ MLP
0.668
0.589
0.451
0.636
0.601
0.558
SPACE w/o encoder
0.627
0.585
0.461
0.637
0.612
0.564
SPACE w/o encoder and species
0.654
0.588
0.454
0.635
0.596
0.563
SPACE
0.637
0.582
0.457
0.644
0.607
0.564
Regulatory elements
Splicing
Model
All
NoTATA
TATA
Acceptors
All
Donors
SPACE w/o decoder
0.752
0.773
0.841
0.873
0.884
0.936
SPACE w/o decoder w/ MLP
0.743
0.750
0.808
0.883
0.886
0.937
SPACE w/o encoder
0.738
0.769
0.828
0,864
0.869
0.933
SPACE w/o encoder and species emb
0.739
0.767
0.828
0,869
0.876
0.942
SPACE
0.763
0.776
0.802
0.898
0.884
0.941
Table 15. Ablation study GUE henchmarks.
Epigenetic Marks Prediction
Model
H3
H3K14ac
H3K36me3
H3K4me1
H3K4me2
SPACE w/o dec
76.76
46.75
50.09
39.56
34.80
SPACE w/o dec w/ MLP
75.59
45.17
48.21
39.70
34.81
SPACE w/o enc SPACE w/o enc and species emb
76.16
48.78
49.14
37.57
34.08
76.94
48.77
42.46
43.01
34.33
SPACE
76.40
50.76
49.18
41.30
32.83
Epigenetic Marks Prediction
Virus
Model
H3K4me3
H3K79me3
H3K9ac
H4
H4ac
Covid
SPACE w/o dec
34.85
57.85
55.38
79.78
49.05
68.66
SPACE w/o dec w/ MLP
34.26
58.94
56.36
78.81
43.49
67.83
SPACE w/o enc
36.84
63.44
56.63
77.17
50.78
68.46
SPACE w/o enc and species emb
37.13
63.84
56.27
78.29
51.14
68.56
SPACE
37.74
61.10
57.06
79.33
51.05
68.89
20


Reviews:
[{"review_id": "1W3W0golcG", "reviewer": "Reviewer_ceSM", "summary": "The paper introduces SPACE, a supervised DNA foundation model that predicts genomic profiles (e.g., chromatin accessibility) to learn effective DNA sequence representations. The authors argue that unsupervised DNA foundation models (DFMs) lack biological context, leading to suboptimal generalization. SPACE addresses this via a Species-Profile Adaptive Collaborative Experts architecture, which combines a species-aware encoder (using Mixture of Experts, MoE) and a profile-grouped decoder to capture cross-species and cross-profile dependencies. \n\nGenerally, key contributions include advocating supervised genomic profile prediction as a superior pre-training objective and proposing a biologically inspired MoE-based architecture.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive feedback. We supplemented with additional experiments which are presented in https://anonymous.4open.science/r/charts-CDE6. We address each key issue and outline manuscript improvements below.\n## Q1:Generalization Across Species and Tasks\nWe concur that evaluating generalization capacity is paramount for supervised DNA foundation models.To comprehensively address this concern, we performed extended benchmarking using the Genomic Benchmarks dataset, which represents the only mainstream benchmark encompassing species beyond those investigated in our previous experiments, including Human-or-worm classification and Drosophila enhancer classification. The results are presented in Section 1.3 of supplementary materials. SPACE achieves robust performance on these tasks, demonstrating cross-species generalizability. While it doesn't achieve SOTA on the human-or-worm classification, it significantly outperforms Enformer.\n## Q2:Architectural Complexity vs. Performance Gain\nSPACE's architecture is biologically motivated:\n* ​Species-aware Encoder: Explicitly models shared and species-specific regulatory mechanisms to enable cross-species knowledge transfer.\n* ​Profile-Grouped Decoder: Captures mutual regulatory influences among distinct genomic profiles.\n\nSPACE consistently improves performance across ​most tasks while maintaining biological interpretability. And additional ablation studies on module effectiveness are provided in *Q3 of Reviewer oB6U*.\n## Q3:Missing Baselines\nWe acknowledge this limitation and resolved in *​Q2 of Reviewer ueQp*.\n## Q4:Novelty and Contribution Clarity\nWe consider both aspects as key contributions:\n\nAs the first work to rigorously explore supervised pretraining for DNA foundation models, SPACE achieves SOTA over NT on most benchmarks despite using significantly fewer parameters and less training data. This validates the effectiveness of supervised pretraining for genomic tasks. While computational limitations and the scarcity of curated multi-species, multi-omics datasets prevent scaling SPACE to the extremes of NT or Evo, our work highlights supervised pretraining as a promising scale-up direction for future research.\n\nThe Species-Aware Encoder and Profile-Grouped Decoder collectively enable substantial improvements over Enformer, particularly in cross-species tasks and new biological applications(e.g., splice site prediction). The detailed experimental results presented in the Appendix D consistently demonstrate that our proposed module achieves superior performance compared to Enformer. The MoE design enhances interpretability: expert selection frequencies align with foundational biological principles. While we acknowledge the reviewer’s concern about limited downstream benchmarks, we have validated SPACE on additional benchmarks, including Genomic Benchmarks and BEND (see *​Q2 of Reviewer ueQp*).\n## Q5:keys for developing DNA LMs\nFirst, based on the scaling laws observed in NT [1] and Evo [2], expanding both model parameter size and training data scale — following practices in the LLM field — is critical for developing DNA language models.\n\nSecond, increasing the model's ​context window length is theoretically essential to prevent information loss by accommodating full DNA sequences. However, current DNA models have not yet empirically validated this advantage. As noted in Evo 1.5 [3], training with 8K context lengths outperforms 131K configurations. While longer input lengths technically allow processing full DNA sequences, they currently lack consistent performance improvements.\n\nAdditionally, most existing DNA LMs scale training data by aggregating reference genome datasets, which inherently fail to capture mutation patterns. This explains why DNA LMs underperform Protein LMs [5] in ​zero-shot variant effect prediction [4]. Protein LMs benefit from more diverse training data (e.g., multiple sequence alignments), suggesting that incorporating population-level mutation data into DNA LM pretraining — as proposed in [6] — holds significant promise.\n\nFinally, as emphasized in our work, **​genomic profiles** are indispensable for advancing DNA LMs. DNA sequence functionality is regulated through cell-type-specific interactions with diverse genomic profiles (e.g., chromatin accessibility, histone modifications). These profiles collectively shape chromatin environments that govern gene expression patterns across cellular contexts.\n\nReferences:\n1. Nucleotide Transformer: building and evaluating robust foundation models for human genomics\n2. Sequence modeling and design from molecular to genome scale with Evo\n3. Semantic mining of functional de novo genes from a genomic language model\n4. GPN-MSA: an alignment-based DNA language model for genome-wide variant effect prediction\n5. Evolutionary Scale Modeling: Pretrained language models for proteins\n6. Pre-training Genomic Language Model with Human Variants for Better Understanding Functional Genomics", "claims_and_evidence": "The claims:\n1. Supervised pre-training > unsupervised DNA LMs:\n2. SPACE’s architecture improves over GPPMs \n\nHowever,\n1. Since the method uses supervised pre-training, it's very important to evaluate generalization capability. Authors demonstrate that SPACE is better than Enformer on yeast and viral genome in Tab. 2, but it would be more promising to demonstrate generalization across more species on more tasks. \n2. The newly proposed model architecture is rather complex compared to transformers or state-space models, while the performance improvement is not so significant.", "methods_and_evaluation_criteria": "**Methods**: The MoE design for species-specific and shared features is biologically motivated (species-specific regulatory mechanisms). The profile-grouped decoder leverages functional dependencies (e.g., chromatin accessibility ↔ transcription).\n\n**Evaluation**: Benchmarks (NT, GUE) and metrics (MCC, Pearson correlation) are standard.\n\nHowever,\n- Benchmarking on GUE and NT only is largely behind the development of DNA LMs. For example, BEND [3]  develops more biological important tasks that many DNA LMs fail.\n- BEND [3] also shows that many DNA LMs cannot outperform simple supervised models trained from scratch using ResNet or CNN. SPACE is also trained supervisely, and it's also important to consider supervised expert models.\n- Importance baselines like Caduceus [1] and Evo [2] are missing\n\n[1] Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling\n\n[2] Evo: DNA foundation modeling from molecular to genome scale\n\n[3] BEND: Benchmarking DNA Language Models on biologically meaningful tasks", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "As mentioned above, importance baselines like Caduceus [1] and Evo [2] are missing\n\n[1] Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling\n\n[2] Evo: DNA foundation modeling from molecular to genome scale", "other_strengths_and_weaknesses": "- The novelty of this paper is unclear. \n    - If it's demonstrating supervised pretraining is a better choice for DNA LMs, the performance improvement is not so significant. \n    - If it's proposing a new architecture applying MoE onto DNAs with cross-species and genomic profile design for the gate, it's important to focus on comprehensive downstreams for benchmarking, rather than simply adopting GUE and NT (see the next point on more benchmarks)", "other_comments_or_suggestions": "N/A", "questions_for_authors": "What do you think is important for developing DNA LMs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Luk8fwTKMS", "reviewer": "Reviewer_qYoF", "summary": "The paper claims that self pretraining alone in DNA is not a good prior to later generalize for downtream tasks. Instead this paper revisits Genomic Profile Prediction Models (GPPMs) such as Enformer the are trained to directly to predict genome profiles. The paper proposear a further refinement on Enformer model, incorporating a specie-aware encoder and profile-grouped decoder. Experimentation shows that the refined version of Enformer surpases its original in several tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback. We appreciate your acknowledgment of our work’s potential contributions to multi-genome foundation models. We supplemented with additional experiments which are presented in https://anonymous.4open.science/r/charts-CDE6. Below, we address your points to clarify and strengthen the manuscript:  \n\n## 1. Response to the Question  \n**Reviewer’s Question**:  \nAre the authors using a pretrained model from Enformer or performing pretraining from scratch? \n\n**Response**:\nSPACE is **trained from scratch** on genomic profile prediction tasks without initializing weights from Enformer. We sincerely appreciate this insightful inquiry, as it inspires us that initializing each expert FFN's weights with Enformer's pretrained parameters could be a promising direction for future exploration.\n\n## 2. Significance of Improvements \n**Reviewer’s Observation**:  \nAlthough the improvement over Enformer is not very significant in most cases, the experimentation is sufficient. Perhaps, the most notable improvement is in the underrepresented mouse experiments.  \n\n**Response**:  \nOur improvements yield **significant gains** over Enformer in cross-species generalization and new genomic tasks, demonstrating that SPACE learns more robust DNA representations. As detailed in Table 2 of the main text, we systematically quantify SPACE’s cross-taxa generalization superiority over Enformer. This is further empirically validated through the Drosophila melanogaster enhancer classification task (Section 1.3 of supplementary materials). Notably, on splice prediction benchmarks, SPACE demonstrates marked improvements of ​**0.0593** on the GUE splice dataset and an average ​**0.161** across NT splice tasks (Table 7 and Table 10 of main text). These quantitative enhancements provide rigorous evidence of SPACE’s enhanced generalization capacity on both evolutionarily divergent species and functionally distinct genomic tasks.\n\nThe strong performance on underrepresented species (e.g., mouse) highlights SPACE’s ability to generalize in **data-scarce scenarios**—a strength enabled by its species-aware encoder. By decoupling shared and species-specific regulatory mechanisms via MoE, SPACE effectively adapts to species with limited training data.", "claims_and_evidence": "The paper claims that the architecture of current GPPMs is not optimal in the sense that the encoder is shared for different species and the prediction heads are independent.", "methods_and_evaluation_criteria": "The paper proposes a new architecrure of GPPM with (1) a specie-aware encoder and (2) profile-grouped decoder following MoE in both cases.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimentation is performed in 2 datasets: The NT dowstream tasks and the GUE benchmark. In both cases, the model seems to improve the performance of Enformer and in some cases becomes SoTA. Although the improvement over Enformer is not very significant in most cases, the experimentation is sufficient. Perhaps, the most notorious improvement is in the underrepresented mouse experiments.", "relation_to_broader_scientific_literature": "The contribution of this paper might help in designing multi genome foundational models.", "essential_references_not_discussed": "The paper references are correct and up to date.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Are the authors using an initial pretained model from Enformer or are you performing the pre-training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "slqCWZn7oz", "reviewer": "Reviewer_oB6U", "summary": "This paper introduces a novel approach using a genomic profile prediction models (GPPMs) as a foundation model in biological sequence modeling. The authors propose SPAC which leverages species Mixture-of-Experts (MoE) and an enhanced category operator to improve predictive performance. They conduct ablation studies to analyze the necessity of individual components and demonstrate SPACE  outperforms alternatives in existing benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for highlighting areas where our manuscript can be improved. We supplemented with additional experiments which are presented in https://anonymous.4open.science/r/charts-CDE6. Below, we address your concerns and outline revisions to strengthen the paper.  \n## Q1:Category Operator\nThe category operator leverages biological priors derived from the **Enformer dataset**, which groups genomic profiles into four categories based on distinct measurement technologies:  \n1. Transcription factor (TF) chromatin immunoprecipitation and sequencing (ChIP–seq)  \n2. Histone modification ChIP–seq  \n3. DNase-seq or ATAC-seq  \n4. CAGE tracks  \n\nThese categories reflect critical biological distinctions. For example, **chromatin accessibility** (DNase/ATAC-seq) and **transcription initiation** (CAGE signals) are functionally linked: chromatin accessibility establishes permissive 3D topological environments necessary for transcription initiation, with transcription start site (TSS)-associated promoters and enhancers overlapping spatially with accessible chromatin domains. This biological rationale guides expert specialization in the decoder—profiles with strong interdependencies (e.g., CAGE and ATAC-seq) share experts, while distinct profiles (e.g., TF ChIP-seq) activate specialized experts.  \n## Q2:Necessity of the components\nThe evaluation of DNA language models (LMs) typically exhibits substantial variance, and no existing DNA LM achieves absolute dominance across all datasets. For instance, DNABERT2 adopted mean ranking as its primary metric in their publication. We note that while the ablation results in Table 3 of our original paper report averaged outcomes, the complete results in Appendix E (Table 11) demonstrate robust improvements over Enformer across nearly all evaluated tasks. To further validate the architecture, we conducted targeted ablation studies, which demonstrate that:\n\n• The **species-aware MoE encoder** enables cross-species knowledge transfer.  \n• The **profile-grouped decoder** captures category-specific regulatory mechanisms.  \n\nThese components are biologically grounded and critical for generalizability, particularly in data-scarce scenarios. To further strengthen our claims, we have included and analyzed new ablation results in Q3.\n## Q3:Ablation Studies\nAdditional ablation studies are documented in Section 2 of our supplementary materials. Section 2.1 presents comprehensive ablation results on Nucleotide Transformer downstream tasks. All ablation models were pre-trained with their hidden dimension parameters halved, as detailed in Section 4.6 of the main text.\n\nSPACE demonstrates comparable or superior performance to the decoder-removed variant in ​14/18 tasks, with ​11/18 tasks still outperforming even when replaced by a parameter-matched MLP. Notably, for regulatory element classification tasks, SPACE achieves better results in ​4/5 datasets, with the only exception being the ​TATA box dataset—which primarily examines sequence motifs of TATA boxes and does not require complex regulatory mechanism understanding. This suggests that while our decoder does not explicitly improve direct chromatin profile prediction accuracy, the ​MoE architecture implicitly captures cross-profile regulatory interactions by modeling their dependencies. This capability provides critical advantages for tasks requiring integrated understanding of multiple profiles, such as regulatory element prediction.\n\nTo further validate cross-species generalization, we evaluated ablation variants on the GUE benchmark (yeast and virus tasks, detailed in Section 2.1). Results reveal that the MLP-based decoder variant shows markedly weaker generalization to novel species compared to SPACE with its enhancement decoder architecture.\n## Q4:Qualitative Analysis and Interpretability  \nIn **Section 4.4 of main text**, we analyze expert specialization patterns:  \n• **Encoder**: Two shared experts capture cross-species mechanisms, while one species-specific expert adapts to unique genomic contexts.  \n• **Decoder**: Complex profiles (e.g., TF ChIP-seq) activate dedicated experts, whereas interdependent profiles (e.g., CAGE and ATAC-seq) co-activate shared experts—aligning with their biological relationships.  \n\nWe agree that deeper case studies could strengthen interpretability and will prioritize this in revisions.  \n## Q5:Broader Impact\nWe have re-examined the viability of ​supervised learning paradigms for DNA foundation models and proposed specifically designed optimizations. This approach effectively addresses the core limitation of traditional unsupervised pre-training — ​inadequate modeling of biological functional associations. We believe these insights could have profound implications for advancing research in ​genomic language models (gLMs).", "claims_and_evidence": "- Overall, I think this is a solid paper providing new perspectives of using a GPPM as a foundation model.\n- The paper claims that SPACE, with its category operator and species MoE, significantly improves structured prediction performance. However, the ablation studies (Table 3) show only marginal performance differences between the full model and its variants, raising concerns about the necessity of these components. The authors do not provide sufficient discussion on the results of these ablations, making it difficult to assess their importance.\n- The paper states the category operator decomposes the base prediction into Q profile type based on domain-specific biological knowledge. It's unclear what they meant by \"domain-specific biological knowledge\" and how they are incorporated.", "methods_and_evaluation_criteria": "Evaluation is performed using standard benchmarks, but the paper lacks a qualitative analysis of why certain components contribute to performance improvements.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "- The ablation studies lack interpretability; Table 3 alone is insufficient to justify the necessity of different components.\n- Additional case studies or qualitative insights would enhance understanding of why specific design choices matter.", "relation_to_broader_scientific_literature": "The key contributions are very specific to the application, and would be difficult for broader scientific community to benefit from this paper.", "essential_references_not_discussed": "Not applicable.", "other_strengths_and_weaknesses": "Not applicable.", "other_comments_or_suggestions": "Not applicable.", "questions_for_authors": "- What specific biological knowledge is incorporated into the category operator, and how does it influence model decisions?\n- Given the marginal performance differences in Table 3, how do the authors justify the necessity of the species MoE and enhanced components?\n\n---\n***Post-Rebuttal***\nI'm updating the score from Weak Reject to Weak Accept based on the authors' clarifications.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SUh1eoVAHK", "reviewer": "Reviewer_ueQp", "summary": "The paper challenges the recent paradigm of self-supervised genomic language models (gLMs), which are trained on the DNA sequence alone, and proposes supervised genomic profile prediction (GPP) task as a more effective pre-training method. The motivation stems from the number and complexity of different factors (epigenetic modifications, chromatin accessibility patterns etc.) that determine the function of the DNA, in addition to the raw sequence. To overcome the limitations of current genomic profile prediction models (GPPMs) in learning the relationship between DNA sequence across species and genomic profiles, the authors implement Species-Profile Adaptive Collaborative Experts (SPACE), which extends Enformer by preprending species embedding token and adding two Mixture of Experts (MoE)-based modules: (1) Transformer Encoder with MoE layer replacing the FFN component including a shared pool of experts and a per-species gating function, to allow species-aware weighing of expert contribution  (2) MoE-based decoder to decode the genomic profiles. MoE learning is employed with standard practices of noisy top-k selection to ensure sparsity. In addition, mutual information loss is used, which encapsulates load balancing between experts (to avoid collapse) and species diversity in training data, while encouraging species aware preference to specific expert subset.  SPACE is evaluated on two benchmarks that are commonly used in gLMs literature (GUE and Nucleotide Transformer’s downstream tasks) and compared to three gLM baseslines (HyenaDNA, DNABERT2, Nucleotide Transformer and its variants) and a GPPM popular baseline (Enformer).\n\n## Update after rebuttal\nThe authors have addressed my questions. I thus keep my initial rating and lean to accept this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. Additional experiments are presented in https://anonymous.4open.science/r/charts-CDE6. We respond your concerns below.\n## Q1:Performance on Splicing Tasks and GUE Dataset\nWe clarify two key points:\n* SPACE ​outperforms DNABERT2 across all splicing benchmarks (see Table 7 and Table 10). While SPACE does not yet surpass Nucleotide Transformer (NT) in absolute terms, it achieves ​substantial gains over Enformer:\n   * ​GUE splice task: 0.8748 (SPACE) vs. 0.8593 (DNABERT2), 0.0593 improvement (relative to Enformer)\n   * ​NT splice tasks: Average 0.917 (SPACE) vs. 0.851 (DNABERT2), ​0.161 average improvement (relative to Enformer)\n* Our analysis shows that SPACE underperforms DNABERT2 only on a subset of Epigenetic Marks Prediction tasks in the GUE benchmark. This can be attributed to SPACE encountering a novel species (yeast) in these tasks. Notably, SPACE demonstrates significant improvements over Enformer under the same conditions, which provides strong evidence for its generalization capability.\n## Q2:Comparison to gLMs Baselines\n* Grover: Directly referenced Grover's results on the GUE dataset. As Grover used only human-annotated datasets, we limited our comparisons accordingly (in section 1.1 of supplementary materials). Our results demonstrate significant outperformance over Grover across all these tasks. Furthermore, we fine-tuned Grover using the official hyperparameters on the revised NT downstream tasks (in section 1.2 of supplementary materials).\n* Caduceus: All baseline comparisons were derived from Caduceus' reported performance on genomic benchmarks (as showed in their paper). Notably, Caduceus did not evaluate the Drosophila task, consequently, the CNN baseline results were extracted from genomic benchmarks (showed in ​Supplementary Materials Section 1.3).\n* GPN-MSA: Compared SPACE against GPN-MSA on mutation effect prediction tasks within the BEND benchmark.While SPACE underperforms GPN-MSA, we emphasize that ​SPACE uses only reference sequences during inference, aligning with standard foundation model evaluation protocols. In contrast, GPN-MSA requires ​multi-sequence alignments (MSAs) as input, introducing an unfair advantage for this baseline.\n* evo: While we acknowledge Evo's contributions, integrating its results during rebuttal faced challenges due to:(1) Evo's 7B parameters exceed the largest baseline capacities (NT-2.5B), and current resources cannot support timely replication;(2) No official benchmark data exists for Evo; (3) Evo focuses on generation tasks, lacking results on mainstream benchmarks.\n## Q3:Data Availability Challenges\nWe acknowledge that raw sequence data is more accessible than genomic profiles. However: (1) Multi-omic data is increasingly available, and we've demonstrated SPACE's advantages using only limited data, suggesting future improvements as these resources expand; (2) While profile data is needed during pre-training, downstream applications require only sequence input since profile understanding is embedded in the model parameters, unlike GPN-MSA which requires additional MSA input for both pre-training and inference.\n## Q4:Increasing capacity by decoder and species embedding\nWe appreciate your suggestion. We have incorporated this consideration in our new ablation studies, with the results presented in Section 2 of the supplementary materials. The corresponding analysis is provided in *Q3 of Reviewer oB6U*.\n## Q5:Task Performance\nYes, we hypothesize that SPACE demonstrates superior performance on tasks requiring modeling of gene regulatory relationships. However, for purely sequence-based tasks, SPACE may underperform compared to gLMs pretrained on extensive sequence data, as its training data is more limited in this regard.\n## Q6: Other Comments\nWe fully agree with the reviewer's perspective and will soften our claims. Our work primarily validates the potential of supervised pre-training for DNA foundation models, having already achieved SOTA. Both our results and those of GPN-MSA demonstrate that reference sequence pre-training alone is insufficient. However, we think that integrating extra information as supervision signals is particularly meaningful, as it enables downstream tasks to rely solely on raw sequences while implicitly embedding biological knowledge into model parameters, whereas GPN-MSA requires additional inputs. In the protein LM field, recent work has similarly adopted MSA information as supervision [1]. [2] has also demonstrated the advantages of using AlphaFold2's supervised-trained backbone for pLMs. Therefore, we believe incorporating additional information as supervision signals provides advantages for foundation models.\n\nDetailed discussion on long sequence modeling is in our response to *Q5 of Reviewer ceSM*.\n\nReferences:\n\n[1]Evolution-inspired loss functions for protein representation learning\n\n[2]Exploring evolution-aware &-free protein language models as protein function predictors", "claims_and_evidence": "The authors present two main claims:\n- Supervised GPP is a more effective  are more effective for pre-re than self-supervised gLMs that are trained on raw DNA sequences\n- A MoE-based GPPM architecture as a mitigation for inferior performance of previous models\n\nWhile the motivation for challenging the current sequence-only self-supervised gLMs paradigm makes sense, there are several issues that needs to be addressed:\n- While SPACE consistently outperforms the selected baselines on chromatin profiles and regulatory elements prediction tasks (Table 3), it shows inferior performance on splicing tasks and some prediction tasks in the GUE dataset (Table 3; Tables 9-10), where Nucleotide Transformer (NT) and DNA-BERT2 show better performance. The authors suggest the model capacity of NT as a possible reason for its better performance in some cases, but DNA-BERT2 is on the same order of magnitude as SPACE and employs a simple Transformer Encoder architecture. \n- In addition, the comparison to gLMs is missing recent strong baselines including GPN-MSA, EVO and Caduceus\n\nThe MoE modules are a novel extension to GPPMs (and gLMs) to the best of my knowledge and the authors also present results on emerging expert capabilities (specific and cross-species). However, it is not completely clear if improvement is gained simply thanks to additional model capacity or to the MoE addition. This is important to understand as MoE training at scale can introduce complexities due to load balancing and bandwidth bottlenecks.", "methods_and_evaluation_criteria": "The authors employ well accepted criteria on common benchmarks for evaluating gLMs", "theoretical_claims": "The paper does not consist of any theoretical proofs", "experimental_designs_or_analyses": "The paper employs standard and well accepted experimental design and analysis for gLMs as well as an ablation study which examines the contribution of the main architectural components (Table 3 in main text, Section E in the supplementary). The Results section is well structured and is supplemented with additional results in the appendix of the paper.", "relation_to_broader_scientific_literature": "The paper makes two main contributions:\n- Pointing out the difference between natural language, where the sequence typically includes all information to determine its function, to DNA, where additional factors collaboratively determine the phenotype, and by that, challenging the paradigm of many gLMs, which are trained on raw DNA sequences without additional information using self-supervised tasks (next or masked token prediction). \n- Proposing supervised GPPMs as an alternative to self-supervised gLMs and extending a current popular baseline with MoE architecture, which outperforms several gLM baselines", "essential_references_not_discussed": "The comparative analysis in the paper is missing some recent competitive gLMs baselines, including:\n- Caduceus [1] \n- Grover [2]\n- Evo [3]\n- GPN-MSA [4]\n\n[1] Schiff, Y.,et al. Caduceus: Bi-directional equivariant long-range dna sequence modeling. ICML’24\n\n[2] Sanabria, M. et al. DNA language model GROVER learns sequence context in the human genome. Nat Mach Intell 6, 911–923 (2024).\n\n[3] https://doi.org/10.1038/s42256-024-00872-0[1] Eric Nguyen et al. Sequence modeling and design from molecular to genome scale with Evo. Science 2024.DOI:10.1126/science.ado9336\n\n[4] Gonzalo et al. \"GPN-MSA: an alignment-based DNA language model for genome-wide variant effect prediction.\" bioRxiv (2024): 2023-", "other_strengths_and_weaknesses": "Strengths:\n- Presenting an interesting discussion, which challenges popular paradigm on self-supervised gLMs \n- Introducing a novel MoE-based GPPMs, which outperforms several gLMs on commonly used benchmarks.\n- The paper is well written and well structured\n\nWeaknesses:\n- Leading gLMs are missing from the comparison and the model does not consistently outperform selected baselines, which weakens the main claim of the paper\n- Key claim does not address the challenge of collecting the supervised signal. While using the raw sequence alone has its disadvantages it is readily available, and related information such as conservation metrics were shown to greatly improve performance (GPN-MSA).\n- The exact contribution of MoE (as opposed to simple addition of capacity) is not completely clear", "other_comments_or_suggestions": "1. Perhaps it is worth considering a “softer” claim, which points to the importance of adding information beyond the raw sequence (either as a prior or as a supervision signal) as a means to improve current genomic language models. This is supported by the results but does not necessitate a clear and consistent superiority over self-supervised gLMs. In addition, it is worth discussing challenges such as modelling long range interactions.\n\n2. It is worth adding comparison, or at the very least references, to recent gLMs (see previous comments)\n\n3. Minor - running title is incorrect (still using the default template string I think)", "questions_for_authors": "1. Have you tried simply increasing the capacity of Enformer by adding a decoder module and adding species embedding (instead of MoEs)?\n\n2. Do you have an intuition to why SPACE outperforms the selected gLMs in some tasks and less so in others?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes SPACE, a supervised genomic profile prediction approach that improves upon purely sequence-based DNA language models. Through a Mixture-of-Experts design, SPACE incorporates species-specific and profile-specific modules to capture diverse regulatory mechanisms and shows strong generalization across species. The authors addressed reviewer concerns by providing new experiments on additional benchmarks and clarifying biological motivations for each architectural component. While some gaps were noted—such as missing baselines and relatively modest gains on certain tasks—the authors demonstrated meaningful improvements over Enformer and showed that including genomic profile supervision helps the model learn more robust DNA representations, particularly for complex regulatory tasks. Overall, the approach and supplementary results convincingly argue that supervised training with genomic profiles can yield a competitive and biologically grounded DNA foundation model, and I recommend acceptance.
Decision: Accept (poster)