Omni-DNA: A Genomic Model Supporting Sequence Understanding, Long-context, and Textual Annotation
Authors: Zehui Li
Vallijah Subasri
Yifei Shen
Dongsheng Li
Wentao Gu
Guy-Bart Stan
Yiren Zhao
Caihua Shan
Venue: NeurIPS 2025 poster ()
Keywords: DNA
Genomics
Health
Representation Learning
Primary area: machine_learning_for_sciences
TL;DR: a genomic sequence model that unifies long-context reasoning and sequence interpretation with state-of-the-art performance.

Abstract: The interpretation of genomic sequences is crucial for understanding biological processes. To handle the growing volume of DNA sequence data, Genomic Foundation Models (GFMs) have been developed by adapting architectures and training paradigms from Large Language Models (LLMs). Despite their remarkable performance in DNA sequence classification tasks, there remains a lack of systematic understanding regarding the training and task-adaptation processes of GFMs. Moreover, existing GFMs cannot achieve state-of-the-art performance on both short and long-context tasks and lacks multimodal abilities. By revisiting pre-training architectures and post-training techniques, we propose **Omni-DNA**, a family of models spanning 20M to 1.1B parameters that supports sequence understanding, long-context genomic reasoning, and natural-language annotation. **Omni-DNA** establishes new state-of-the-art results on 18 of 26 evaluations drawn from Nucleotide Transformer and Genomic Benchmarks. When jointly fine-tuning on biologically related tasks, **Omni-DNA** consistently outperform existing models and demonstrate multi-tasking abilities.  To enable processing of arbitrary sequence lengths, we introduce **SEQPACK**—an adaptive compression operator that packs historical tokens into a learned synopsis using a position-aware learnable sampling mechanism, enabling transformer-based models to process ultra-long sequences with minimal memory and computational requirements. Our approach demonstrates superior performance on enhancer-target interaction tasks, capturing distant regulatory interactions at the 450kbp range more effectively than existing models. Finally, we present a new dataset termed **seq2func**, enabling Omni-DNA to generate accurate and functionally meaningful interpretations of DNA sequences, unlocking new possibilities for genomic analysis and discovery.

Omni-DNA: A Genomic Model Supporting Sequence Understanding, Long-context, and Textual Annotation
Zehui Lile?, Vallijah Subasri Yifei Shen',
Dongsheng Li², Wentao Gu²,
Guy-Bart Stan Yiren Zhao Caihua Shan
'Imperial College London Miirosoff Research
Vector Institute *University Health Network
"Correspondence to: 2e162228ic..c... ac. uk, caihua.
caihua. shan@mi comm
"Co-corresponding authors: ...esfg.ssnaeeee stan@imperial..cmmm ac uk, a.zhao@imperial ac. uk}
Abstract
The interpretation of genomic sequences is crucial for understanding biological processes. To handle the growing volume of DNA sequence data, Genomic Foundation Models (GFMs) have been developed by adapting architectures and training paradigms from Large Language Models (LLMs).. Despite their remarkable performance in DNA sequence classification tasks, there remains a lack of systematic understanding regarding the pre-training and task-adaptation processes of GFMs. Moreover, existing GFMs cannot achieve state-of-the-atr performance on both short and long-context tasks and lack multimodal abilities.
By revisiting pre- training architectures and bost-training techniques, we propose OMNI-DNA, a family of models spanning 20M to 1.1B parameters that supports sequence understanding, long-context genomic reasoning, and natural- notation. Omni-DNA establishes state-of-the-art results 18 of 26 evaluations drawn from Nucleotide Transformer and Genomic Benchmarks. When jointly fine- tuning biologically related tasks, Omni-DNA consistently outperforms existing models and demonstrates multi-tasking abilities. Furthermore, we introduce SEQ- PACK, adaptive compression mechanism that enables efficient long-context mod- eling by summarizing historical tokens through position-aware learnable sampling. This allows cansformer-based models process ultra-long genomic sequences with minimal memory and computational overhead. Leveraging SEQPACK, Omni- DNA excels at enhancer-target interaction prediction, capturing distal regulatory ef- fects over 450kbp. Finally, we present SEQ2FUNC, a newly constructed dataset that empowers Omni-DNA generate accurate and functionally meaningful interpreta- tions of DNA sequences, opening new avenues for genomic analysis and discovery.
an-
Introduction
As genomic data has been expanding exponentially over recent decades [19], Genomic Foundation Models (GFMs) have emerged as an essential paradigm to model genomic sequences and complete various tasks, including variant effect prediction [21, 44], sequence generation [39], and functional classification [10]. Most GFMs follow "pretrain + task-specific finetune" paradigm: first pretraining on unlabeled DNA sequences, and finetuning for each specific task.
While GFMs have demonstrated utility on specific genomic tasks [13, 47], they still exhibit important limitations: (i) Limited optimization of architectural components for genomic tasks: Prior work has explored design choices including tokenization (DNABERT [17] vs. DNABERT-2 [49]), posi- tional encodings (NT NTv2 [10]), and genomic-specific modules (GPN-MSA [5], Caduceus [30]),
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

but a more comprehensive, task-specific search of architectures and pretraining strategies tailored to genomic inductive biases still lacking. (ii) Limited exploration of multi-task synergies: Many approaches fine-tune on a single downstream task, missing the opportunity to exploit shared biological signals and achieve cross-task performance gains, despite evidence of such benefits in genomic assay prediction (e.g., AlphaGenome [3], Enformer [2]). (iii) Insufficient multimodal capability: Most ex- isting GFMs restrict vocabularies to nucleotide base pairs, predefined labels, or learned subwords via BPE tokenization, limiting their ability to handle tasks that require mapping DNA sequences to func- tional annotations in natural language. While ChatNT can process joint (DNA, text) inputs, it employs separate tokenizers and two-stage architecture consisting of NTv2 [10] the DNA encoder and Vicuna-7B 7B [38] as the decoder, resulting in less unified multimodal integration. (iv) Context-length adaptivity challenges: Transformer-bet based models remain constrained by the quadratic cost of self- attention, limiting their scalability to long-context genomic tasks such as enhancer- prediction.
To bring DNA sequence modeling closer to the breakthroughs that large language models have enabled in natural language processing, we comprehensively explored the design space of GFMs, from pre-training architectures and objectives to post-training techniques, and developed several targeted improvements. To this end, we propose Omni-DNA (Figure 1), a family of GFMs ranging from 20M to I.IB parameters that enables multitask learning, long-context genomic reasoning, and natural-language annotation. We then outline how Omni- Mani-DNA addresses these challenges.
Optimized backbone (Section 2). Early GFMs such as DNABERT [17]. DNABERT-2 [49] and Nucleotide Transformer [10] rely on masked language modeling within BERT framework; recent models, including HyenaDNA [24], BioxL.TTM [32], DNAGPT [46], and MegaDNA [36]. have shifted to causal language modeling. For the scaling of data and models, Omni-DNA is also decoder- only Transformer trained with causal language modeling on 300 billion nucleotides from RefSeq [33]. Performing an extensive architecture sweep, we further analyze and ablate normalization (LayerNorm vs. RMSNorm) and positional encoding (RoPE vs. ALiBi) to select the most robust configuration.
Unified multi-task finetuning (Section 3.1). In the post-training phase, the standard approach is to append a classification head (an MLP) and fine-tume the model independently on each downstream dataset. Using this conventional setup. Omni-DNA achieves of-th- art performance on 18 of 26 tasks in both the Nucleotide Transformer [9] and GB benchmarks [13]. We further extend this approach by grouping biologically related tasks and performing joint fine-tuning. The multitask fine-tuning achieves substantial improvements across multiple tasks and obtains the highest overall average accuracy. It demonstrates that Omni-DN, captures transferable biological knowledge during pre-training and exhibits the synergistic effect in the fine-tuning.
Since Omni-DNA is auto-regressive, it can naturally accommodate new tokens as instructions or labels. This enables the use of instruction-followwe enrich the model's 's vocabulary and enrich the model's vocabulary for more flexible genomic reasoning.
techniques, such as supervised fine-tuning, to
Effective vocabulary expansion (Section 3.2). Because most GFMs are pre-trained on unlabeled genomes, their vocabularies are typically restricted to four canonical nucleotides (A, T, C, G) plus a handful of special tokens [9]. In contrast, LLMs routinely operate with vocabularies exceeding 10 sub-wort units [25, 41]. This restricts GFM's capacity to both comprehend and generate sequence-leve semantics, and conduct the multi-model tasks. To address this, we analyze how to mitigate the distribution shift of vocabulary when adding unseen tokens during post-training. We successfully create a large-scale dataset of text-function mapping, which enables Omni-DNA to directly generate natural functional descriptions of given DNA sequences.
Scalable long-context reasoning (Section 3.3). Our final improvement focuses on extending the model's effective context length. Prior approaches such Caduceus [30], HyenaDNA [24], Evo [22] and MegaDNA [36] replace the standard Transformer with state-space architectures, hierarchical compression,
convolution-basee operators achieve sub-quadratic complexity. However, the whole genome includes long repetitive regions and short meaningful elements (e.g., genes and regulatory regions). These methods do not explicitly exploit such sparsity, limiting their effectiveness at the whole-genome scale. To address this gap. we introduce SEQPACK, a post-training context-extension operator that learns compress historical tokens into a compact synopsis. Specifically, SeqPack adaptively compresses a DNA sequence of length N into an embedding of length L (L & N), reducing the complexity from (NV2) to O(L)). By learning to ignore "junk" regions [27] and retain only informative segments, SeqPack enables efficient ultra-long genomic reasoning while preserving critical positional information.
2

---

TEG
Omni-DNA Model Pretraining
Downstream Tasks
39
Regulatory Element
(B) DNA Function
Pretraining Datasel
AutoRegrossive Transformer
Classification
Annolation Generation
from NCBI RefSeq
Promoter
XXX
Enhancer
Others
Function
Long Context Adoptation with SeqPack
E' R
<
(C) Enhancer Target
(D) Genetic Disease
Sample &Cmmrres
Prediction
Variants Type Classification
Enhancer
== (eI,...,e)) Rxd
Input Sequence E
Target
Figure I: Overview of the Omni-DNA framework. Omni-DNA combines large-scale autoregres- sive pretraining on 300 billion nucleotides with post-training compression via SeqPack enable efficient long-contex, DNA modeling. The pretrained model is evaluated separately across diverse downstream tasks, including (A) regulatory element classification, enhancer-target interaction prediction, and (D) disease type classification based on genetic variants.
(B) DNA function annotation, (C)
Revisiting DNA Sequence Pretraining Paradigms
2.1 Pretraining Setup
Omni-DNA is pre-trained with causal next-token prediction objective on a 3000billion-
corpus drawn from NCBI's multi-species assemblies [33]. Data deduplication procedures described in Appendix C. Following Zhou et al. [48], we adopt Byte-Pair Encoding (BPE) [35] with initial vocabulary of 4096. Each model processes 250-token contexts (R/ 1kbp) in batches of 384 sequences across 8 x A 100-40 GB GPUs for 800k steps. The optimization is performed by Adam W (31 = 0.9, 32 0.95, weight-decay=0.1) with linear warm-up of 5k steps to peak learning rate 3 x 10- followed by linear decay to 3 x 10-5, We also employ mixed precision (bf16 matmuls, fp32 norm stats), global gradient clipping at 1.0, and PyTorch FSDP layer sharding.
2.2 Model Architecture
Table 1: Architecture of pretrained Amni-DNA models.
Following Table 1, Omni-DNA adopts decoder-only Transformer back-
# Params
Layers
Heads
dmmodel
Layer Norm
Pos. Emb.
bone [14], but is not a naive transplant of NLP models. We conduct three
20M
8
256
RMSNorm
RoPE
60M
8
$12
RMSNorm
RoPE
kinds of ablation studies: normaliza-
116M
12
768
Bias-Free LN
RoPE
tion layer (bias-free LayerNorm [4]
116M
12
768
RMSNorm
RoPE
vs. RMSNorm [45]), positional en-
116M
12
12
768
Bias-Free LN
ALiBi
coding (ALiBi [29] vs. RoPE [40]),
116M
12
12
768
RMSNorm
ALiBi
and parameter budget. The optimizer, learning-rate scheduler, and tokenizer are kept identical.
300M
16
1024
RMSNorm
RoPE
700M
16
1536
RMSNorm
RoPF
IB
16
2048
Bias-Fres LN
RoPF
2.3 Empirical Insights
The combination of RoPE and Bias-Free LayerNorm deliver the best trade-off between compute efficiency and downstream accuracy, whereas ALiBi consistently converges ~ 15% slower and lags on both promoter and enhancer tasks.
Finding 1:
Explanation. As shown in Figure 2 (Left), all variants exhibit lower training loss with increasing FLOPs, but those using RoPE (green, red) descend more steeply than their ALIBI counterparts (blue, orange). When transferred to the performance of downstream tasks in Figure 2 (Middle and Right), such as TATA-box recognition and enhancer-activity prediction, RoPE+Biar Bias-Feae LayerNorm attains 0.967 + 0.002 and 0.594 + 0.006 accuracy, surpassing all other models. These results suggest that relative phase encoding (ROPE) captures long-range DNA dependencies more effectively than the linear bias of ALIB1, especially when coupled with the unbiased normalizer.

---

With the same compute larger models tend to obtain lower test loss; however, lower test loss does not guarantee higher downstream accuracy
Finding 2:
Explanation. Figure (Left) follows the similar compute-loss scaling law [18] extended from LLMs: with the same compute budget, larger models attain lower test loss (cross-entropy). Yet Figure (Right) reveals decoupling between losses and downstream performance: and enhancer tasks saturate once the loss reaches 4.1, while promoter and splice tasks remain near-ceiling even for the smallest models (The downstream tasks refer to 18 tasks from Dalla- Torre al. [9]; finetuning details are in Section 4.1.) This mismatch cautions against relying solely on the loss proxy for downstream biological performance.
eestone-modificatinn
Compute vs Loes
Promoter TATA Task Ascuracy
Enhancer Task facuracy
0.95
Model
AL MLBI-AMM
AL saram
0,800
0.99
BEPE Hhooypmm
Acuuncy
6.4
IRVS
8.80
0.57
0.555
61
0.500
10
AuBiH pian.
A-SUARMS
Nutt-tion gara
N. N.BeNon p5aTh
ALEU-RMM
RePE-1BA5 TOAS
Compute FLOFS)
Figure 2: ablation pretraining architectures. Left: Test loss as a function of cumulative pre-training compute (FLOPs, log scale) for four 116M -arammreete variants that combine ALIB1 or RoPE positional biases with biee-free LAYERNORM (Non-param) param) or RMSNORM. Middle: Promoter TATA site classification accuracy Right: Enhancer activity prediction accuracy. Bars show + std over three fine-tuning runs.
Compute Loss
Test Loss Task Accuracy
4.5
Task
Hizzna
Erhwrcer
700M
Promoter
1B
Spiie
Average
4.0
Computia (FLOPs)
10
Final Test Loss
4.2
4,3
4.5
Figure 3: Compute-Loss Scaling and Its Decoupling from Downstream Accuracy. Left: Pro-training loss decreases predictably with total compute as model size increases from 20M to IB parameters, following an approximate power law. Right: Final test loss poorly predicts the fine-tuned performance four representative tasks (Histone, Enhancer, Promoter, Splice).
3 Adaptive Post-Training Strategies for GFMs
Here we first explore the new multi-task setting in Section 3.1. Subsequently, we propose novel approaches for vocabulary expansion and long-context compression in Section 3.2 and Section 3.3.
3.1 Single-fask vs. Multi-task Setting
Current fine-tuning in GFMs for a specific task typically attaching a linear head (an MLP) to the hidden state of the last token from the final layer, which is then used predict the class. In the single-task setting, the model is fine-tuned separately for each individual task. In contrast, for the multi-task setting, first group & related tasks, and jointly fine-tune these linear heads.
Moreover, since Omni-DNA is auto-regressive, an alternative fine-tuning approach is to introduce new tokens instructions and labels and apply instruction-followe techniques, such as supervised fine-tuning (SFT). For multiple tasks, we add task-specific prompt tokens and class label tokens, enabling the model to Iearn k tasks simultancously.

---

3.2 Vocabulary Expansion
To enable the model to handle prompts, answer tokens, or label words that are absent from the pre-training alphabet during SFT. the original vocabulary Vo must be expanded to V = V. U V., where V represents the newly introduced symbols (e.g., task tags, response delimiters, class labels).
Vocabulary distribution shift problem. Let 44-1 E Id be the decoder state at step t-1 and E = [eo lVEV & [dd the output-embedding matrix whose columms are the token vectors ev. With the expanded vocabulary, the softmax becomes
exp(hi-1en exp(hi-1eas (Par)
(0 Z1.-1)
(I)
+
Σ exp((1e-1en)
For pre-existing token Iz E Vo, its probability under the original softmax would have been Po (ox m1a-1) exp(1i 1/2. Comparing with Equation yields the shrinkage factor:
old p= x1e-11) Pe (re x1a-1) x 7+ z'
Z old (Ze x11c-1) x 1 2'/Z
1
(2)
Thus every pre-existing token probability is decayed by 1/(1 + /2) shift that can slow conver- gence and, if unaddressed, cause catastrophic forgetting.
To mitigate distribution shift, we integrate two techniques: (1) NEFTune [16]: isotropic Gaussian noise is added all token embeddings during SFT, acting as dropout in representation space and discouraging the model from over-relying on a few high-frequency additions. (2) Key-token
replication: for classification tasks, we duplicate each label token a times (we use a 5) within the target string. This boosts its relative frequency, ensuring that gradients from the label position dominate those from task prompts. practice, we also keep [V_ to the minimum required.
3.3 SEOPACK: Long-Context Sequence Compression
Let IIN (Z),..... uN) denote DNA sequence of length N >>> Lnnt. where Lnat is the native context length of a pretrained backbone fo (typically 1-16k tokens). We tokenize DNA sequences into k-mers and apply the embedding table $ V + R to obtain the embedding matrix E = ((xI.)) E R xd The objective of SEQPACK is learn differentiable compression operator
1 - gK kxd K < Leat
which can identify and preserve the most informative K tokens based on their positions and content before the computation-heat layers.
3.3.1 Position-Aware Learnable Sampling
To realize the compression operator Cr,, we estimate the token importance by trainable temperature vector 0 E R.N. In detail, the importance weight of each position defined by
exp(O:)
m
(4)
Σ exp(O,)
Because hard sampling is non-differentiable, employ the Gumbel-Softmax trick [15]:
logit, = log T2 + 9t. 9: Gumbel(0, 1),
(5)
followed by top-K operator that keeps the highest-scoring indices. Concatenating the selected rows of E with a fixed suffix of the most recent K tokens yields the compressed representation E' = (E), guaranteeing that the downstream backbone receives at most K tokens.
3.3.2 Geometric Splitting
Applying global top-K over N tokens costs O(N log K) time, which is untenable for ultra-long sequences. To avoid this bottleneck, we introduce a geometric splitting scheduler sequence into exponentially growing segments and performs local sampling within each segment.
that partitions the
5

---

Algorithm SEQPACK compression layer (inference mode)
Require: Embeddings E E _X*e the number of preserved tail tokens Kuil, the total preserved tokens K, the
learned temperature vector 0
Ensure: Compressed sequence E' E R Rid
1: H - N Kuil -1
> history length (excluding e and tailed tokens)
2: B <- K- Kuil
> history budget
3: if I then
return
no compression needed
5: end
6: Segment Ku. into geometric blocks {S; J.
7: for all St do
8: Compute the sampling budget b; via Equation
9: Compute the importance weight zj via Equation
10: Sample obtain compressed output E& based b; and n.
end for
12:
E' - Fi :.....;; EM ex Kaa:N]
13:
return E"
Given input embeddings E E Nxd and compression target K, we divide the sequence into M = [V] segments with exponentially increasing sizes. The starting point of these segments e M - is
Pi+1 + min IN
..N.NN-. -Pp)
(6)
where the boundary conditions are p1 = 1 and PM++1 = N + 1. For each segment S, Elpepees 1/+ allocate sampling budget based on the length:
b max (1.1..1- K)
N
We then sample 6, tokens from each segment according to the segment norrentzeo importance weights following Equation 4:
exp(0,) Σ Kkmp exp(O.)
E [pt, 1],
(8)
From each segment, we obtain sampled E and the final compressed output is E" ... 1 To further analyze the efficiency of geometric splitting, we present Proposition 1 with the proof in Appendix B, which establishes the linear-time complexity of the sampling process.
Proposition 1 (Linear-time compression under geometric splitting). Let E Nx be token embed- dings and let K < Lnat be the target context length. Using the geometric splitting scheduler defined above, the expected time to obtain the compressed sequence E' E RxKxd is O(N).
3.3.3 Compression with Positional Bias Preservation
We implement SEOPACK as a lightweight PyTorch layer that is inserted once before the first attention block during post-training. To preserve positional inductive biases, we exclude the first token and the final Kma tokens from compression. This design is motivated by the attention sink phenomenon [42], where attention to the first token can dominate attention scores. In DNA-pretrained transformers, we observe similar sink effects at both the beginning and end the sequence (Appendix G). The overall procedure is described in Algorithm 1.
4 Experiments
We benchmark Omni-DNA integrated with SEQPACK against leading genomic foundation models across four types of representative tasks. (i) Gene regulatory element classification: identify functional regions that control gene expression; (ii) Pathogenic variant effect prediction: assess the functional impact of genetic mutations linked to disease; (iii) Enhancer-targee gene interaction mapping: resolve the complex spatial relationships between regulatory elements and their target genes; and (iv) Dequence-conditiones functional interpretation generation: a novel task introduced in this work, supported by carefully curated dataset derived from high-quality NCBI genome assemblies.

---

Table 2: Comparison of Omni-DNA with existing GFMs 18 NT downstream tasks. The family of Omni-DNA obtains the best average result. In splice tasks, Omni-DNA performs lower than NT models but still surpasses other models.
Metric
CADUCEUS-PR
(1
CADUCEUS-PS (1.9M)
DNABERT2 (117 M)
(500M)
NTV2
NT2.5B (2.5 B)
HyessDNA [1.6 M)
Ommi-DNA (116 M)
Omai-DNA B)
0.799
0.784
0.814
0.779
0.818
0.022
H3NT4AC
0.631 0.026
0.541 +-0.212
0.516= 0.028
0.021
0.550
0.612 0.083
0.655
0.077
H3K3MES
0.00 0.129
0.009 0.13
0.591
0020
0.825 = 0.013
0.632
0.613 # 0.041
0.861
= 0.033
0.686
0.002
H3K4MKI
0.533 + 0.039
D.488 +-0 102
0.511
0.008
0.550
- 0.021
0.559
0.512 0.024
0.577
0.063
0.617 + 0.000
HEK4ME2
0.487 1 1..1770
0.388 1 0.101
0.330
0.0
0.319 - 1.0015
0.328
0. 15455 0.095
0.576
0.000
0.547
0.006
H3K4ME3
0.544 0.045
0:400 0.2200
0.352
0.077
0.410
0.053
0.421
0.549
0.056
0.587
0 0.222
0.642
0.001
H3K79MES
0.077
0,676 0.0025
0.613
0.825
0.026
0.642
0.672 0.048
0.718
0.752
0.007
HESKAC
0.622 0.030
0.004 0.018
0.5422 0.029
0.575
0.581
0.051
0,058
0.701
0.002
0.11 0.022
0.789 0.020
0.706-- 0.027
0.799 0.025
0.822
0.044
0.802
0,822
0.005
H4AC
0,021 0.0664
0.625 0 0.240
0.363
0.041
0.235 = 0.002
0.501
0.564
0.033
0.663
0.029
0.632
0.001
Eahonner
0.546 0.077
0.401 0.006
0.516 0.008
0.548 + 0.144
0.580
0.51T + 0.117
0,503
= 0.006
0.580
0.018
Eahaneer Types
0.430 0.054
0.416 + 0.005
0.423
0.051
0.424 + 0.132
0.474
0.386 0.185
0.408
0.001
0.492
0.023
Promeler: ALL
0.970 0,004
0.967 1.0.004
0,971 0.006
0.976 0.006
0.974
0.960 0.005
0.975
0.002
0.975
0.001
Promoler: NONTATA
0.0.00
0.972
0.950
0.001
Promoter: TATA
0.016
0.567 0.015
0.955 0.021
0.964
0.941 0.040
0.967
0.002
Spliee All
0.940 0.027
0.927 0.021
0.933 0.000
0.983
0.008
0,983
0.956 0.011
0.921
0,941
0.003
Splice Acceptor
0.5377 + 0.023
0.506 + 0.077
0.975 0.000
0.981
= 0.011
0.990
0.958 0.010
0.968
= 0.002
0.972
0.001
Splice Donor
0,948 + 0.025
0.874 + 250
0. 0.063-
0.006
0.985
+ 0.022
0.984
0. 0.940
+ 0.024
0.961 = 0.007
0.963 = 0.001
Average
0.715
0.689
0.079
0.608
0.708
0.707
0.755
0.767
Table 3: Performance of Omni-DNA across 8 tasks in the genomic benchmark. Omni-DNA (116M) achieves the highest average.
Task Name
CNN (264K)
HYENADNA (436K)
CADUCEUS -PH (470K)
DNABERT2 (117M)
Omni-DNA (116M)
MOUSE ENHANCERS
0.715 0.087
0,780 0.025
0.754 0.074
0.792 = 0.031
0.799 : 0.004
CODING VS. INTERGENOMIC
0,892 0.008
0.904 0.005
0,915
0.003
0,949 0.002
0.942 0.010
HUMAN VS. WORM
0.942 + 0.002
0.964 + 0.002
0.973 + 0..01
0.975 + 0.002
0.976 + 0.001
HUMAN ENHANCERS COHN
0.702 ::02:::00 0.021
0.729 1 0.014
0.747
0.004
0.714 + 0.025
0.738 = 0.002
HUMAN ENHANCER ENSEMBL
0.122
0.849 0.006
0,893 0.008
0.891 0.051
0.919 0.021
HUMAN REGULATORY
0.872 + 0.005
0.869 + 0.012
0.872 + 0.011
0.852 + 0.024
0.895 + 0.012
HUMAN OCR ENSEMBL
0.698 0.013
0.783 0.007
0.828 0.006
0.789 + 0.012
0.791
= 0.001
HUMAN NONTATA PROMOTERS
0.861 0.009
0.944 0.002
0.007 0.946
0.912 0.013
0.968
0.013
Average
0.803
0.853
0.866
0.859
0.879
4.1 Gene Regulatory Element Classification
Nucleotide Transformer Downstream Nucleotide Transformer Downstream [9] includes 18 tasks both binary and multi-class classification tasks for epigenetic marker [28], promoter [26]. enhancer and splice site prediction. Closely following the evaluation metrics from Caduceus [31], Matthews Correlation Coefficient (MCC) is used for histone marker and enhancer, while Fi-score is used for splice and promoter classification. To compare Omni-DNA against Nucleotide Transformer V2(NTv2) [9], DNABERT2 [48], HyenaDNA [23], Caduceus [31]. We perform 10-fold cross- validation to select the hyperparameter for each model as detailed in Appendix D, and the detailed task-specific results for NT-Downstream tasks in Table 2. Omni-DNA (1B) and Omni-DNA (1 16M) achieve the best average performance of 0.767 and the second of 0.755, and they achieve superior suserre
performance on 13 out of 18 benchmark tasks, surpassing competing methods. For the remaining five tasks, Omni-DNA (IB) ranks second and third in Promoter:AL and Promoter:Non the three splice site classification tasks, Omni-DN models perform lower than NT models but still surpass other models.
In
Genomic Benchmark Genomic Benchmark (GB) [13] contains eight DNA regulatory element classification tasks, similar to NT downstream tasks, with additional tasks on species classification and gene regulatory element classification on mouse. We compare model performance of Omni-DNA (116M) with DNABERT2, CADUCEUS-PH, and HymnaDNA, and show the results in Table 3. Omni-DNA (1 (116M) is ranked first in five out of eight tasks and second in the remaining three, while achieving the highest average score. Compared to DNABERT-2, which has a similar model size, Omni-DNA (1 (116M) outperformed DNABERT-2 in seven out of eight tasks.
7

---

Multi-tasking, We extend pretrained Omni-DN (116M) and NTv2 (500M) models multi-task setting and compare their performance with their single tasks pounter-parts. In particular, 10 related tasks from NT Downstream 19]-H3, H3K14AC, H3K36ME3, H3K4MEI, H3K4MEE, H3K4ME3, H3K79ME3, H3K9AC, H4, and H4AC-are consid- ered. These tasks are interconnected due to the bi- ological correlation between acetylation and methy- lation effects [28]. The objective here is to train model capable of addressing all 10 tasks with single fine-tuning.
(Sinle-Tees) (SingleTTas)
HIKANEL
HEKAMEE
EKLAAC
H3K4HE
Two approaches are compared: Multi-head fine- tuning attaches task-specific linear heads to a shared Transformer backbone. Next-Token prediction aug- ments the tokenizer with three label tokens {0, 1, and ten task identifiers {task) ... taskio); the model autoregressively emits the label token, casting classification language modelling.
H3KZANE:
HAC
HOKSAC
Figure 4: Multi-task fine-tuning boosts accu- racy. Classification MCC on nucleosome- target (NT) tasks. Multi-head outperform NTP multi-tasking, but both surpass single- task baselines.
Figure 4 shows Omnii-DN (multitask, multi multi-head) and N7V2 imulittakkmullt-hhedd multi-head) strictly outperform their single-task counterparts on all tasks, while OMni-DNA imulti-taskNNT NTP) also achieves higher accuracy half of the tasks. We hereby summarize the results as following finding.
Jointly fine-tuning on biologically related tasks consistently outperforms inde- pendent single-task fine-tuning.
Finding 3:
4.2 Enhancer-Target Interaction Prediction
Task Enhancers are distal regulatory elements that can reside kilobases to megabases from the genes they activate, in contrast to promoters that typically lie immediately upstream of a transcription start site. We adopt the curated benchmark from Cheng et al. [7] originated from [11, 12, 34], which contains 2,602 experimentally validated enhancer-target pairs plus matched negative controls. Each sample is a 450 kb genomic window, and the data are split 8:1:1 1 into training, validation, and test sets. The task is binary classification: does the window contain an effective enhancer-targe interaction? Because accurate prediction requires modelling dependencies across hundreds of kilobases, this task intrinsically more challenging than the shorter-context tasks assessed in the previous section.
Models Because 450kb exceeds the context limits NTv2 and DNABERT2, both models ran out of memory (OOM) and are therefore omitted. For Omni-DNA (116M) we raise the native context length to 2048 tokens to accommodate longer sequences. Additional long-range baselines include HYENADNA (medium-450K) and CADUCEUS (PH/PS). We also report expert "upper- bound" model, Activity-Bi Activity-By-Contee By--Cnttcct (ABC), which leverages DNase-seq, ChIP-seq, and Hi-C matrices, and a lightweight CNN baseline from Cheng et al. [7]. The performance is measured by AUROC.
Table 4: Enhancer- Target Se-
quence Classification
Results For HYENADNA, CADUCEUS, and Omni-DNA,
we append single linear classification head and mean-pool the final hidden states over the sequence length. Models are fine-tuned for at most 20 epochs; learning rate and batch size are selected via a small grid search (see Table 8). As summarized in Table 4, Omni-DNe attains AUROC of 0.895, outperforming all neural baselines and closing much of the gap to the assay- rich ABC upper bound. In Appendix H, we provide additional ablation studies on the computational overhead and compression ratio of SeqPack.
Model
AUROC
ABC (expert)
0.926
CNN
0.797
HyenaDNA
0.828
CADUCEUS -P
0.826
CADUCEUS-PS
0.821
Omni-DNA
0.895

---

Table Context-length sensitivity (left) and variant-type classification (right), Left: F1 score of Omni-DNA on LC as function of input length. Right: Fl score (1) on LC and ICCs.
0.9
HyenaDNA
DNABERT2
NTv2
Omni-DNA
HyenaDMA* Omni-DNA
LC
0.83=0.02
0.71 0.03
0.75 0.04 0.75=
0.93:10.01 0.89:10.01
ICCs
0.78
0.49:00.22
1024
Dataset Generation from NCBI Ref Seq
Evaluation Pipeline
Human
Lnc RNA 2789 doeen't encode proteins
Input: ATGGACG...
Ground Truth
Model Prediction
Chimpanzee
ATGC
Hsa-miR-4200 -466
Horsa
but reguisting various biological processes
regulating gene expression
found in cbickens
Genomes of 20 species
300,000 DNA, Function Pairs
LLM (Evaluator)
LLM (Annotator)
Prompt: Decide Type of Transcnptiin function desoriptin
ATAC
Lnc_ RNA 2789
Extend short Armolsiion
Prompt: describe functiooss of givan gene/ma prociselly and concisely
Sequencee
Short Function Annotation
MIRNA
MRNA
Figure 5: SEQ2FUNC construction and evaluation pipeline. Genomic windows extracted from 20 NCBI species are clustered by transcriptional evidence and passed to a LLM that produces concise, expert-style function descriptions. After agreement filtering and manual vetting the process yields 300k cenguuence-fucctio pairs covering 10 functional classes.
4.3 Genetic Disease Variant-Type Classification
Moving beyond binary pathogenicity prediction [21], address the more challenging task of directly inferring the disease category from a single-nucleotide variant (SNV), encoded as the triple (reference sequence, mutated sequence, label). We consider two clinically salient groups: (i) Lung Cancer (LC) with 11 disease types, and (ii) Inherited Cardiac Conditions (ICCs) with 1108 disease types.
Starting from the full Clin Var release (14810 unique diseases and ~33 M variants), variants annotated with any of the LC ICCs labels marked positive; negatives drawn at random from the remaining diseases. The resulting training sets contain 7,876 LC variants and 532,621 ICC variants. Each example is therefore triple (r, m, y). where is the reference sequence, m the mutated sequence, and E {0, }} denotes the absence or presence of the target disease class.
The left of Table 5 shows that extending the context length improves performance up to 1024bp, with negligible gains thereafter. We fine-tune Omni-DNA (116M), DNABERT2, NTv2 (500M), and MlyenaDNA-medium (450K) based on the procedure in Appendix E. The FI scores are summarized in the right of Table 5, where Omni-DNA delivers the strongest performance on both disease groups.
4.4
Sequence-to-Functioo Generation
Publicly available functional labels are sparse, making automatic interpretation of raw DNA sequences challenging. To bridge this gap, we release SEQ2FUNC, corpus of 300k high- quality (sequence, function) pairs (Figure 5). Starting from coarse annotations in the NCBI multi- species assemblies [6]. we prompt state-of-the-art art LLM generate refined, natural- -lagugge descriptions. Each entry is validated through agreement between two independent LLM passes, with low-confidence cases discarded. The final dataset spans 20 species and 10 broad functional categories, with detailed statistics provided in Appendix F. We follow an 80/ 10/10 110 train/validation/test split.
Dataset.
Task and training protocol. Given a DNA sequence, the objective is to generate a textual de- scription for its functional annotation. We first extend the Omni-DNA vocabulary from 4096 DNA tokens to 8173 tokens (adding 4077 wordpieces) and then fine-tune with next-token prediction

---

Model Input: What is the function of this sequenoe7GGCTG...TTII
Ground Truth: Olfactory receptor 5A2-like (OR5A2L) is a member of the olfactory receptor gene family, which is involved in the detection of volatile substances, contributing to smell sense.
Omni-DNA:
The mRNA olfactory receptor family subfamily member 109 (RRB1109) encodes protein that is part of the offactory receptor family. These receptors are involved in the detection of odorant molecules and and play a crucial role in the sense of smell.
Demonstration of Omni-DNA's functional description generation. Given a DNA sequence, Omni- could generate natural language description for functional annotations.
Figure 6:
the SEQ2FUNC training set, consisting of concatenated [DNA]] [description] sequences, We use AdamW and NEFTune train for up to 10 epochs with early stopping on validation perplexity. An ALM-based evaluator maps each generated description to one of 10 functional classes, allowing quantitative comparison with ground truth via weighted F1 score and Matthews correlation coefficient (MCC). Since existing genomic models cannot produce free-form text, we benchmark against GPT--4 zero-shot mode (used prompts are shown Table 9), with a random guesser as a lower bound.
Results. The qualitative examples in Figure and Appendix F show that Omni-DNA @ @T pro- duces fluent and biologically plausible explana- tions and correctly identifies an unseen olfactory- receptor locus. Moreover, the quantitative re- sults in Table 6 show that Omni-DNA outper- forms GPT-4o in zero-shot generation by 7.1 % in weighted Fl and 38.2 % in MCC, substan- tially exceeding the random baseline.
Table 6: Performance on the SEQ2FUNC test set.
Model
Weighted F1
MCC
Omni-DNA @FT
0.730
0.367
GPT-4o (zero-shot)
0.659
-0.015
Random Guess
0.483
0.008
5 Conclusion
In this paper, we introduced Omni-DNA, a framework for exploring optimal strategies in pre-training and task adaptation for genomic modeling. Empirical analysis identified that the combination of non-parametric bias-free layer normalization with RoPE positional embeddings delivers the best trade-off between computational efficiency and downstream accuracy. We further demonstrate that multi-task fine-tuning on biologically related tasks substantially enhances performance. To extend the applicability of Omni-DNA, we develop SEQPACK, enabling efficient processing of ultra-long DNA sequences, and presentEEQFUuNG; which empowers Omni-DN to annotate DNA sequence functions using natural language. Across range of downstream tasks, including regulatory element classification, enhancer-targe interaction prediction, genetic disease variant classification, and DNA function annotation generation, Omni-DN, achieves Mfte-of-thhe-ee performance. We envision Omni- DNA as a step for scaling and generalizing genomic language models toward broader biological tasks.
Acknowledgments and Disclosure of Funding
Zehui Li is grateful to his family-Jinghai Li, Huiqing Wang, and Yiqiu Sun for their support.
References
[1] Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, and Thomas Hofmann. Dynamic context pruning for efficient and interpretable autoregressive transformers. Advances in Neural Information Processing Systems, 36:65202-6527 2023.
[2] Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska- Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley, Effectiive gene expression prediction from sequence by integrating long-range interactions. Nature methods,
2021.
10

---

[3] Ziga Avsec, Natasha Latysheva, Jun Cheng, Guido Novati, Kyle R Taylor, Tom Ward, Clare Bycroft, Lauren Nicolaisen, Eirini Arvaniti, Joshua Pan, et al. Alphagenome: advancing regulatory variant effect prediction with a unified dna sequence model. bioRxiv, pages 2025-06, 2025.
[4] Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv::667.0000
2016.
[5] Gonzalo Benegas, Carlos Albors, Alan J Aw, Chengzhong Yc, and Yun S Song. Gpn-msa: an aligmment-basee dna language model for genome- wide variant effect prediction. bioRxiv, pages
2023-10, 2024.
[6] Richa Tanya Jeff Dennis A Colleen Evan Devon J Rodney St Agarwala Barrett Beck Benson Bollin Bolton Bourexi, Richa Agarwala, Tanya Barrett, Jeff Beck, Dennis A. Benson, et al. Database resources of the national center for biotechnology information. Nucleic Acids Research,
2017. 46:D8- - 113, 2017.
[7] Wenduo Cheng. Zhenqiao Song, Yang Zhang, Shike Wang, Danqing Wang, Muyu Yang, Lei Li, and Jian Ma. Dnalongbench: A benchmark suite for long-range dna prediction tasks. bioRxiv,
pages 2025-02 01, 2025.
[8] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025.
[9] Hugo Dalla-Torre, Liam Gonzalez. Javier Lendoza-Revilla Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, et al. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, pages 1-11, 2024.
[10] Hugo Dalla-torre, Liam Gonzalez, Javier Mendoza Revilla, Nicolás López Carranza, Adam Hen- ryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Hassan Sirelkhatim, Guil- laume Richard, Marcin J. Skwark, Karim Beguir, Marie Lopez, and Thomas Pierrot. The nucleotide transformer: Building and evaluating robust foundation models for human genomics. bioRxiv, 2024.
[11] Charles P Fulco, Joseph Nasser, Thouis R Jones, Glen Munson, Drew T Bergman, Vidya Subra- manian, Sharon R Grossman, Rockwell Anyoha. Benjamin R Doughty, Tejal A Patwardhan, et al. -byivity-by-contact model of enhancer-promoce regulation from thousands of crispr perturbations. Nature genetics. 512))): (00)::664-11000 1664-1669, 2019.
[12] Molly Gasperini, Andrew J Hill, José Melissa D Zhang. Dana Jackson, Anh Leith, Jacob Schreiber, William S Noble, et al. A genome- wide framework for mapping gene regulation via cellular genetic screens. Cell, 176(1):377-390.
Beth Martin, Seungsoo Kim,
2019.
[13] Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Šimetek, and Panagiotis Alexiou. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24(1):25, 2023.
[14] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402. 00.000088, 2024.
[15) Iris AM Huijben, Wouter Kool, Max B Paulus, and Ruud JG Van Sloun. A review of the gumbel- max trick and its extensions for discrete stochasticity in machine learning. IEEE transactions
on pattern analysis and machine intelligence, 45(2): 1353-13353113111 2022.
[16] Neel Jain, Ping-yeh Chiang. Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, et al. Neftune: Noisy embeddings improve instruction finetuning, arXiv preprint arXiv:2310.05914 2023.
[17] Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V. Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. bioRxiv, 2020, [18] Jared Kaplan. Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar Xrxiv:2001. 200.00000.....1. 2020.
[19] W Lathe, J Williams, M Mangan, and D Karolchik. Genomic data resources: challenges and promises. Nature Education, 1(3):2, 2008
[20] Guihong Li, Duc Hoang, Kartikeya Bhardwaj. Ming Lin, Zhangyang Wang, and Radu Mar- culescu. Zero-shot neural architecture search: Challenges, solutions, and opportunities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[21] Zehui Li, Vallijah Subasri, Guy-Ban Bart Stan, Yiren Zhao, and Bo Wang. Gv-rep: A large-scale dataset for genetic variant representation learning. Advances in Neural Information Processing Systems, 37:134701 134720, 2024.
11

---

[22]
Eric Nguyen, Michael Poli, Matthew G. Durrant, Armin W. Thomas, Brian Kang, Jeremy Sullivan, Madelena Ng. Ashley Lewis, Aman Patel, Aaron Lou, Stefano Ermon, Stephen A. Baccus, Tina Hermandez-Bousard Christopher Ré, Patrick D. Hsu, and Brian I.. Hie. Sequence modeling and design from molecular to genome scale with evo. bioRxiv, 2024.
[23] Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch- Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: I Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024.
[24] Eric D Nguyen, Michael Poli, Marjan Faizi, Armin W. Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton M. Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, and Christopher Ré. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. ArXiv, 2023.
[25] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, et al. Gpt-4 technical report. 2023.
[26] Mhaned Oubounyt, Zakaria Louadi, Hilal Tayara, and Kil To Chong. Deepromoter: robust promoter predictor using deep learning, Frontiers in genetics, 10:286, 2019. Alexander F Palazzo and T Ryan Gregory. The case for junk dna. PLoS genetics,
[27]
10(5)=e 2014.
[28] Dmitry K Pokholok, Christopher T Harbison, Stuart Levine, Megan Cole, Nancy M Hannett, Tong Ihn Lee, George W Bell, Kimberly Walker, P Alex Rolfe, Elizabeth Herbolsheimer, et al. Gideme-wide map of nucleosome acetylation and methylation in yeast. Cell, 122(4):517-527, 2005.
[29] Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv: 108. 12409, 2021.
[30] Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao. Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. ArXiv, hss/2403.032344 2024.
[31] Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshow. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234 2024.
[32] Niklas Schmidinger, Lisa Schneckenreiter, Philipp Seidl, Johannes Schimunek, Pieter-Jan Hoedt, Johannes Brandstetter, Andreas Mayr, Sohvi Luukkonen, Sepp Hochreiter, and Gün- ter Klambauer. Bio-xIstm: Generative modeling, representation and in-context learning of biological and chemical sequences. arXiv preprint arXiv:2411.0 1.04 2024.
[33] Conrad L Schoch, Stacy Ciufo, Mikhail Domrachev, Carol L Hotton, Sivakumar Kannan, Rogneda Khovanskaya, Detlef Leipe, Richard Meveigh, Kathleen O Neill, Barbara Robbertse, et al. Ncbi taxonomy: a comprehensive update on curation, resources and tools. Database, 2020:baaa062, 2020.
[34] Daniel Schraivogel, Andreas R Gschwind, Jennifer H Milbank, Daniel R Leonce, Petra Jakob, Lukas Mathur, Jan O Korbel, Christoph A Merten, Lars Velten, and Lars M Steinmetz. Targeted perturb-seq enables genome-scale genetic screens in single cells. Nature methods, 17(6))629-
635, 2020.
[35] Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508. 07909, 2015.
[36] Bin Shao. A long-context language model for deciphering and generating bacteriophage genomes. bioRxiv, 2024.
[37] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.08. 0300,, 2024.
[38] Tianyuan Shi, Fanqi Wan, Canbin Huang, Xiaojun Quan, Chenliang Li, Ming Yan, and Ji Zhang. Progressive fusion of large language models. arXiv preprint arXiv:2408. 04998, 2024. [39] Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. arXiv preprins 24X.:24422008844 2024.
Profuser:
[40] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. [41] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint N.Xv:2307.00288. 2023.
12

---

[42]
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 2023.
[43]
Zhao Yang. Bing Su, Chuan Cao, and Ji-Rong Wen. Regulatory dna sequence design with reinforcement learning. arXiv preprint 0rXiv:2503.0798/ 2025.
[44]
Huixin Zhan, Jason H Moore, and Zijum Zhang. A disease-specific language model for variant pathogenicity in cardiac and regulatory genomics. Nature Machine Intelligence, pages 1-11,
2025.
[45] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32. 2019.
[46] Daoan Zhang, Weitong Zhang. Yu Zhao, Jianguo Zhang, Bing He, Chenchen Qin, and Jianhua Yao. Dnagpt: A generalized pre-trained tool for versatile dna sequence analysis tasks. arXiv preprint arXiv:2307 00668.0688 2023.
[47] Jian Zhou and Olga Troyanskaya. Predicting effects of noncoding variants with deep learning- based sequence model. Nature methods. 12( 2(10):931995) 2015.
[48] Zhihan Zhou, Yanrong Ji. Weijian Li, Pratik Dutta, Ramana Davuluri. and Han Liu. Dnabert-
2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint
arXiv:2306. 15006, 2023.
[49] Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana V. Davuluri, and Han Liu, Dnabert-
2: Efficient foundation model and benchmark for multi-species genome. ArXiv, abs/2306.
2023.
13

---

A Limitations
Despite these advances, limitations exists. Naive approach to explore the architecture choices through pretraining could be costly. Future research could streamline architecture optimization using automated techniques such as Zero-Shot NAS [20]. Additionally, integrating reinforcement learning for post-training methods could enhance efficiency and address the generalizability issue with supervised fune-tuning [8], although effective reward design remains challenging [37, 43].
B Proofs of Propositions
Proof. We bound the cost of constructing the compressed sequence E' under the geometric splitting scheduler.
(a) Segment construction. The sequence is partitioned into M = [loga N| segments whose lengths grow geometrically as |S,] < 2 Forming the segment boundaries therefore requires (M) = O(log N) operations.
(b) Local sampling. Let by (1 < i S .M be the sampling budget of segment S, Selecting b indices with a heap-based top-b; (or an equivalent QuickSelect routine) costs O([S: log b;) time. Because the scheduler guarantees b, S |S,] and, in practice, b, histories, we may upper-boume the per-segment cost by D[[S,] (S])).
< |S:] for long
(c) Total cost. Summing over all segments,
M OlS. log = (i2 Σ
M
(i-1) 1)).
Since EM 2 = N and (i 1) s 15 the series is dominated by the linear term:
M
Σ
2 (i 1) (N).
Hence the overall expected running time ((1)+ N) ((V), in contrast to the O(N log K) required by global top-K selection.
Table 7: Comparison of pretraining setup between Omni-DNA DNABERT2, and NT-transformer.
Omni-DNA (six sizes)
DNABERT2 (116M)
NT-transformer (S sizes)
Layer norm type
non peeeparrantcc/MMSNonn
parametric
parametric
Positional embeddings
RoPE
ALiBi
RoPE
Attention variant
full
Blases
none
in both LN and Attention sequential GEGLU 128
in both IN and Attention sequential SwiGLU
Block type
sequential SwiGLU
Activation
Sequence length (In Tookens)
250
1000
Batch size (Across GPUs)
384
4096
512
Total Steps
800000
150000
16, 000
Warmup steps
5000
16,000
Peak LR
3.0E-04
5.0E-04
1.0E-04
Minimum L.R
3.0E-05
5.0F-05
Weight decay
0.1
lc-5
Betal
0.9
0.9
0.9
Beta2
0.95
0.98
0.999
Epsilon
1.0E-05
1.0E-05
1.0E-08
LR schedule
linear
lincar
linear
Grudient clipping
global 1.0
Training Precision
Mxied (bfl6 for most operations)
Training Framework
Pytorch Built-in FSDP
Pytorch HuggingFace trainer 8GPUs 14 days SGPUs 14 days Nvidia RTX 208011 GPUs.
Training Duration (min)
8 GPUs 1 days
128 GPUs*1days
Training Duration (max)
GPUs days
128 GPUs 22dayy
GPU Types
1 Nvidia GPU 40GB
A100 Nvidia GPU
14

---

C Detailed Deduplication Process
C.1 Handling.Now Non-(A, T, G, CJ Characters
The raw genome data from the reference genome includes additional characters beyond (A, T, G, C), which represent ambiguities gaps. These characters are defined follows:
N: Represents any nucleotide (A, T, G. or C).
R: Represents purines (A or G).
Y: Represents pyrimidines (C or T).
Other characters (e.g.. W, K, M, etc.) represent specific nucleotide subsets or unknown bases.
These characters were removed during preprocessing to retain only the core nucleotide sequences (IA, T, G, CJ), ensuring consistency and facilitating the deduplication process.
C.2
Chunk-Based Deduplication
The deduplication procedure was performed as follows:
1. Chunking: The genome was divided into non-overlapping chunks of 1024 base pairs.
2. Exact Matching: Identical sequences across the dataset were identified and removed.
3. Efficiency: This step utilized hashing techniques and optimized string comparison algo- rithms to handle the large dataset efficiently.
D Finetuning with Classification Head
Basic Setup We closely follow the setup from Caduceus [31] for evaluating Omni-DNA. The fol- lowing models are compared: Omni-DNA, DNABERT-2 [48], NT-Transformer [9], HyenaDNA [23], and the Caduceus models [31].
For NT Downstream tasks, we use maximum fine-tuning epoch 20, while for the Genomic Benchmark (GB) tasks, we use a maximum of 10 epochs. Both NT Downstream and GB tasks include training set and a test set. For hyperparameter search, 10% of the training set reserved as validation set.
Hardware & Framework All fine-tuning is conducted single NVIDIA A100 40GB GPU. We utilize the Hugging Face Trainer API for full-size fine-tuning of the pretrained checkpoints. Models loaded using the AutoModel lutoooodeFFoSSeeuenccche FowSequenceelasssitf....eeeeeeee catiiniiiiiiiiiiinoommoii class, which automatically adds a linear layer on top for sequence classification.
Our experimental results align with those reported in Caduceus [31]. For consistency, report statistics from [31] for two Caduceus models, DNABERT-2, NT-Transformer 500M, and HyenaDNA in NT Downstream tasks, as well CNN, HyenaDNA, and CADUCEUS-PH in GB tasks. For NT2.5B, we use the results from its original paper [9].
We replicate experiments for the following models: NT50M, NT100M, NT250M, along with six Omni-DNA models on 18 NT Downstream tasks. Additionally, we fine-tune DNABERT-2 and Omni-DNA 116M on the eight Genomic Benchmark tasks.
Hyperparameters 10-fold cross validation is performed for each model decide the best hyperpa- rameters provided in table 8.
Additional Notes which align with those of [31], and the results reported in the original paper [23], both on NT downstream and GB tasks. Our replication follows our own setup, yielding results consistent with [31]. Upon reviewing the code within the container image provided by [23], we identified additional techniques such as Exponential Moving Average (EMA) and data augmentation that were employed, which may account for the discrepancy. In this work, we use HyenaDNA results obtained without these techniques the baseline.
We observe a performance gap in HyenaDNA between our replication results,
15

---

Table 8: Finetuning hyperparameters.
Hyperparameter
Peak learning rate
[8 10-10,11
10-5,6
Per device train batch size
[8,16,32]
Max gradient norm
1.0
Weight decay
Adam f
0.9
Adam Bg
0.999
10-6
Optimizer
ADAMW
E
Genetic Disease Variant Type Classification
Finetuning procedure. For every backbone (NTv2, Omni-DNA, DNABERT2, HyenaDNA) we wrap the pretrained model with lightweight classification head that follows the implementation in algorithm Each sample encoded as pair of gepuences-refereen (ref) and alternate (alt). Both sequences are passed independently through the frozen backbone to obtain the last-layer hidden states.
Encoder backbones: we take the first-token representation ([CLS]).
Decoder backbones: we take representation of the last non-padding token in each sequence.
The vectors are optionally transformed by a learnable pooler (a linear layer followed by tanh). We then compute their difference, Ah = hat heer and feed it to single fully connected layer that produces class logits. Cross-entropy loss is minimised for most 100 epochs; coarse grid search (summarised in table 8) is used to select learning rate, batch size, and weight decay.
F Functional Annotation Generation
Dataset Construction Process The Seq2Func dataset constructed using genomic sequences sourced from the NCBI Reference Sequence (RefSeq) database, incorporating sequences from 20 different species. The selected species and their genome assemblies include:
GCF-000001405.40, human
GCF-000001635.27, mouse
GCF-036323735.., rat
GCF-028858775.22 Chimpanzee
GCF-018350175.1, cat
GCF-011100685.1, dog
GCF-000696695.11 burgud
GCF-000003025.6, pig
GCF-016772045.2, sheep
GCF-016699988.2, chicken
GCF-005200985.1111 fly
GCF-000002035.6. Zebrafish
GCF-029289425.2, Pygmy Chimpanzee
GCF-029281585.2, Western gorilla
GCF-028885655.2, Pongo abelii
GCF-028885625.2,, Bornean orangutan
GCF-000339765.1. Rhesus monkey
GCF-037993035.11 Macaca fascicularis
Chinese hamster
16

---

Algorithm 2 Differential Classification Wrapper
1: procedure INTTALIZEWRAPPERMMMMR C, decoder, usePooler)
2:
d 4- -idden-iddde__izz
fallback to moddal if absent
3: if usePooler then
4: Pooler Linear(d, d) tanh
5: else
6: Pooler identity
7: end if
8: Head <- Lineer(dd C)
9: return W = (Masse Pooler, Head, decoder)
10:
end procedure
11:
procedure FORWARD(W, x m xal, m y)
13:
HIT +- MMusee (xte md ( lastHidden H +- -Mbase (x.". m lastHidden
14:
15:
if W.dcodeeer then
16:
h <- LASTTOKEN H mm h LASTTOKEN H m alt)
17:
18:
else
he halt
H H
[CLS] token
20:
21:
end if
22:
h <- Pooler(h
h Poller(h
24:
Ah t h hes
25:
Head(Ah)
26:
if * Ø then
L CrossEntropy(00. v)
28:
return {L, 5}
29:
else
30:
return ú
31:
end
32:
end procedure
GCF-041296265.1 horse
The construction process follows these key steps:
Sequence Extraction and Functional Annotation
The raw dataset includes a diverse set of RNA sequences with different functional annota- tions.
The original dataset consists of the following RNA types and their counts:
mRNA: 220397
IRNA: 20674
snRNA: 18996
snoRNA: 15577
IncRNA: 14291
miRNA: 12238
primary-tramm transcript: 8644
IRNA: 6225
transcript: 3860
ncRNA: 477
guide-RNA RNA: 143
antisense- Batieense-RNA:22222 28
17

---

RNase-P-RNA: 9
V-gene-segment: 7
ScRNA: 7
Y-RNA: 6
Telomerase-RNAy,
vault-RNA: 4
SRP-RNA: 4
MRR-RN--P--N:::: 3
C-gene-segment: 2
D-gene-segment
To refine the dataset, only seven functional RNA types are retained:
mRNA
IRNA
SIRNA
SnoRNA
IneRNA
miRNA
-RRNA
Annotation Enhancement Using a Large Language Model (LLM)
A LLM Annotator employed to extend and refine functional annotations.
The model is prompted with: Describe the functions of the given gene/RNA precisely and concisely."
For example, an initial annotation like Lnc RNA 2789" is expanded into:
"It is Lne RNA 2789 that doesn't encode proteins but regulates various biological processes."
Final Dataset Composition
The final dataset consists of 300,000 DNA-function pairs, where each DNA sequence is paired with an enhanced functional description.
These functionally annotated sequences high-quality input for machine learning models.
Usage
Prompt
Used by LLM (Annotator) for ex- tending the short annotation to de- tailed function annotation
(System Prompt) You are helpful assistant that answers functions of given gene/rna precisely and concisely.
Used by LLM (Evaluator) for de- ciding the type of DNA given its function description
(System Prompt) You are helpful assistant that de- termines the type of RNA based on the given function description. When deciding the functoin. Your answer should to only be one of the I'mRNA', 'tRNA', 'snRNA', 'snoRNA', 'IncRNA', 'miRNA', TRNA'
Given to GPT4u @ zeroshot for per- forming zeroshot DNA function pre- diction
(System Prompt) You are a helpful assistant that deter- mines the type of RNA based on the given DNA sequence. Your answer should only be one of the mRNA, IRNA, snRNA, snoRNA, IncRNA, miRNA, rRNA
Table Prompt Used during dataset construction, evaluation and zeroshot prediction
18

---

Ground Truth
GPT4o
OmniDNA@H @ft
(Input) of length 1000)
(DNA
mRNA:CD209, also known as DC-SIGN (Dendritic Cell Specific Intercellular adhesion molecule-3-rrabbing Noninte- grin). is a type I trans- membrane protein primarily expressed on dendritic cells and some other immune cells. It functions as receptor that binds and internalizes pathogens, including viruses and bacteria, thereby facil- itating their presentation to T cells and playing a critical role in the immune response CD209 is involved in the recog- nition of mannose-rich lig- ands, mediating cellular adhe- sion, and influencing various aspects of immune activation and tolerance. Additionally, it can modulate inflammatory responses and has been im- plicated in HIV-1 and other pathogen infections.
The provided DNA sequence likely represents genomic re- gion involved in regulatory or coding functions. Based on its composition, it may con- tain promoter elements, cod- ing regions, untranslated re- gions (UTRs), or repetitive se- quences.
C-C motif chemokine lig- and 22 ( CCL22 ) a chemokine involved in immune responses It primarily at- tracts and activates T cells particularly regulatory T cells Tregs ), and plays a role in modulating inflammation and
immune responses CCL22
produced by various cell types, including macrophages and dendritic cells and im- plicated in various diseases in- cluding autoimmune disorders and cancer.
Table 10: Comparison of Ground Truth, GPT4o, and OmniDNA@ft in RNA type determination (Example I)
G Attention Sink Visualization
To investigate the emergence of attention sinks in our model, we visualize the self-attention scores across several transformer layers during inference. Specifically, we load the Omni-DNA Maai--DN-1110 -16M model and generate a random DNA sequence length 100. This sequence is tokenized and passed through the model without applying causal attention mask, as many downstream genomic tasks require bidirectional context.
We attach an attention probe to each transformer block, which records the defimax-normallin attention weights during the forward pass. In Figure 7, we show the attention matrices for four layers (LO, L1, L-2, L-1), each across four attention heads.
The visualization reveals distinct patterns: Layer 1, the attention is diffuse without clear sink; in Layer 2, we observe a strong attention sink at the START token; and in the last two layers, both the START and END tokens receive significant attention, suggesting the model learns to aggregate global information around these special positions in deeper layers.
H
Additional Ablation Study On SeqPack Algorithm
Additional Ablation on the SeqPack Parameter with Different Compression Ratios To ablate
the parameter of SeqPack compression ratios (c Loriginal and assess SeqPack's effectiveness TTpruned
against established methods, we adapted Dynamic Context Pruning (DCP) [1] to work with Omni- DNA 116M on the Fnhancer-Targel Prediction task. The results demonstrate SeqPack's substantial advantages.
19

---

Ground Truth
GPT4o
OmniDNA@R @ft
(Input) GAGGTAGTAGATIIIINAG
miR -3596d is microRNA found in rats (Rattus norvegi- cus). It is involved in the regulation of gene expres- sion at the post-rranscriptional level, primarily by binding to complementary sequences in messenger RNAs (mRNAs),
The DNA sequence GAG-
gga - miR 65 77 Sp a mi- croRNA ( miRNA found in chickens GGallu gallus ). Mi- croRNAs are small non cod- ing RNA molecules that play a crucial role in regulating gene expression by binding to com- plementary sequences on target messenger RNAs ( (RRNA ),
GTAGTAGATTGTATAG could serve various functions depending on its genomic context. Generally, it could be: 1. A regulatory element If found in promoter or enhancer regions, it may influence gene expression. 2. A binding site It could be recognition site for transcription factors, such as ETS famill proteins (which recognize GGAA/T motifs). 3. A coding non-coding region - within a gene, it might encode peptide or function as a non-coding RNA. 4. A structural or repeat element - It could contribute chromatin organization or genome stability.Fov precise functional annotation, its location within genome and experimental validation would be needed.
leading to their degradation
leading to their degradation
or inhibition of translation While specific target genes may vary, miRNAs like mo-miR- 3596d generally play roles in various biological processes such as development, cell dif- ferentiation, proliferation, and apoptosis. Specific roles or tar- gets for mo--iiR-3596d would require experimental validation or further studies to elucidate its functions in rat biology.
or inhibition of translation Specifically gga: mii 65 77 - may be involved in var- ious biological processes in- cluding development, differen- tiation and response stress although specific target genes and functions may require fur- ther study
Table 11: Comparison of Ground Truth, GPT4o, and OmniDNA@ft in RNA type determination (Example 2)
At extreme compression ratios (220x, 150x), DCP performance degrades severely, barely outper- forming random baselines, while SeqPack maintains strong performance (AUROC > 0.89). Even at moderate compression (50x), SeqPack outperforms DCP (0.907 vs. 0,880 AUROC).
The Computational Overhead of SeqPack Compressing inputs (25,600- 819,200 tokens) to fixed 4,096 with SeqPack, we report (i) SeqPack operator time and (ii) Omni-DNA-II forward time in seconds on the compressed sequence. As input length grows 32x, SeqPack time rises only 3.56x (inference: 0.00898 to 0.03197 s) and 5.95x (training: 0.015 to 0.091 s). Forward time remains effectively flat. SeqPack's share of end-to-end time remains low and increases gently with the compression ratio (6.25x x to 200x): 1.2% to 4.0% (inference) and 4.3% to 21.6% (training).
20

---

Ground Truth
GPT4o
OmniDNA @ft
(Input) TTCTTCCCCTTCNNGGGGAACCCTCTTTCAAT
TGGANTGACCCCMMGGGGGNNTTGGGGGGGCC TAACCAAGGTTAAAACCCTTTTTTTTCCCCCC
nRNA:U6 spliceosomal RNA
The given DNA sequence
U6 spliceosomal RNA is a small nuclear RNA snRNA that plays a crucial role in the splicing of pre mRNA in eu- compo-
is small nuclear RNA
likely encodes a functional ge- netic element, such as protein- coding region or a regulatory sequence. To determine cise function, will need to an- alyze for potential coding re- gions, motifs, or regulatory ele- ments.
(snRNA) that plays an es- sential role in the splicing of pre-mRNA in eukaryotic cells. It is core component of the spliceosome, the complex responsible for the removal of introns from pre-mRNA transcripts. U6 engages in base-pairing interactions with other spliceosomal RNAs and pre-mRNA, facilitating the assembly and function of the spliceosome. Its primary roles include catalyzing the cleavage of introns and the ligation of exons during the splicing process. U6 is also involved in the activation of the catalytic core of the spliceosome, making crucial for proper gene expression.
pre-
karyotic cells nent of the spliceosome the complex responsible for remov- ing introns from pre mRNA and joining exons together. U6 RNA is involved in the cat- alytic activity of the spliceo- some and is essential for the recognition of splice sites and the formation of the active site for splicing It also interacts with other snRNAs.
Table 12: Comparison of Ground Truth, GPT4o, and OmniDNA@ft in RNA type determination (Example 3)
Table 13: Performance comparison between SeqPack and DCP under various compression ratios the Enhancer-Target Prediction task.
Model
AUROC (+ SE)
Compression Ratio
ABC (expert)
0.926
CNN
0.803 + 0.022
SeqPack
0.894 = 0.015
220
SeqPack
0.899 = 0.009
150
SeqPack
0.907 0.011
50
DCP
0.502 + 0.089
220
DCP
0.632 + 0.010
150
DCP
0.880 + 0.012
50
Table 14: Inference-time and Training-time analysis of SeqPack compression across varying input lengths. For inference, SeqPack operator time grows sublinearly with sequence length, while forward time remains stable. At training time, SeqPack accounts for an increasing but still moderate fraction of total time as sequence length increases.
Length
SeqPack (s)
Forward (s)
SeqPack %
25,600
0.00898 (=00000)
0.77257 (=.0033)
1.1
51,200
0.01032
(+0.001)
0.76120
(=.001)
1.3
102,400
0.01225
(=0.001)
0.77986
(=0.003)
1.5
204,800
0.01508
(=0.001)
0.76666
(=0.001)
1.9
409,600
0.02050
(+-0.001)
0.76665
(+0.001)
2.6
819,200
0.03197
(+00.001)
0.76514
(+0.003)
4.0
21

---

H0
WH
WP2
HS
142
149
0.5
020
020
020
0.5
09:00
0.4
0.4
0.4
0.0
0.3
03
02
0.4
02
0.1
0.
0.0
2010
14g
340
035
000
0000
025
0.3
0.15
0.0
0.10
0.15
0.10
0.05
0.1
0.05
0.8
0.6
0.6
0.4
0.2
02
2000
0.4
0.6
0.8
0.6
300
419
459
070
0.4
0.
02
0.2
02
02
0.1
wil
Figure 7: Attention patterns across layers and heads in Omni-DNA. Fach row corresponds to different transformer layer (LO, L1, L-2, L-1), and each column to a different attention head (HO-H3). For layer 1, attention relatively diffused with no clear sink. At layer 2, we observe strong attention sink at position 0 (START). In the last two layers, both the START and END tokens exhibit sink-like behavior, suggesting global aggregation mechanisms emerging in deeper layers.
Length
SeqPack (s)
Forward (s)
SeqPack %
25,600
0.01523
0.33756 (10.00))
(0.002
4.3
51,200
0.01793
0.32607 0.0.00)(10.000
(=0.002)
5.2
102,400
0.02339
(#0.001)
0.34088
(=0.001)
6.4
204,800
0.03389
(10.001)
0.32727
(=0.002)
9.4
409,600
0.05290
(+0.001)
0.32809
(=0.002)
13.9
819,200
0.09067
(+0.001)
0.32850
(=0.002)
21.6
NeurIPS Paper Checklist
1. Claims
22

---

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly list the section return for each claim made in the introduction (Section 1).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see appendix A for the limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: For each theoretical results, we either provide the proof inline section 3.3 or in the appendix section 3.
23

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully describe the pretraining settings in table and the additional details about finetuning for each tasks are included in appendix D and algorithm 2.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

reproducible, they Answer: [Yes]
Justification: We provide the full code we use for performing the downstream tasks in the supplementary. Some of the data and pretrained model could be large share but provide instructions on how to obtain the data in the supplementary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co
guuides/CodeSubmi
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. wttpccccpppcccccccccccccccccccct wwwpc/gguss/.eeeeeee SCt///oodeiissseeeeeeeee ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please the justification for question Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We conduct each experiment for multiple times to obtain the statistical signifi-
cance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
25

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We declare the use of computational resource for each experiment. Mostly on A A100 Nvidia GPU 40GB*8 for pretraining, and A100 Nvidia GPU 80GB*1 for finetuning.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't It make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttp:///ee.i../eeeeee cc wctppbccc//aaecccccccccccccceeet htt.ee/ethiiiiiieeeeeeeeeaaaaaae
Answer: [Yes]
Justification:
We confirm we conform all the items in NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We briefly discuss the positive contributions of our research to the broader community. To the best of our knowledge, does not pose any negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We confirm we have properly cited the work, on which we perform the benchmarking and used in any other form.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We include structure description file for the code in the README along with the code itself in the supplementary.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
28

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We do not use LLM as important, original, or non-standard components for core method development.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : /ttp///nss.////// tweurrps.cc/ooofeeeeeeeee.e wee.aeeeeneee22///////// 2LLM/LLM for what should or should be described.
29


Reviews:
[{"review_id": "8MY2M4XYT9", "reviewer": "Reviewer_rvnr", "summary": "This paper introduces Omni-DNA, a family of decoder-only transformer models (from 20M to 1.1B parameters) designed to be a comprehensive Genomic Foundation Model (GFM). The authors aim to address four key limitations in existing GFMs: limited architecture exploration, a single-task fine-tuning paradigm, a lack of multi-modal (textual annotation) capabilities, and the inability for a single model to excel at both short- and long-context tasks.", "questions": "See Cons.", "limitations": "The authors briefly mention that their architecture exploration could be costly and that reinforcement learning could be a future direction. However, they do not adequately address the limitations of the current work. The primary limitation is the immense complexity and cost of the entire framework, which makes it very difficult for other researchers to reproduce or build upon. Furthermore, the paper does not discuss the potential for cascading errors: errors in the LLM-generated SEQ2FUNC dataset could lead to a model that learns to generate incorrect functional annotations. The reliance on LLMs for both dataset creation and evaluation is a significant methodological limitation that is not fully acknowledged.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Pros\n- The paper tackles several important challenges in genomic modeling simultaneously, from architecture design to long-context reasoning and multi-modality. The evaluation is extensive, covering 26 tasks from the Nucleotide Transformer and Genomic Benchmarks, in addition to long-context and generation tasks.\n\n- The work makes a concerted effort to move beyond the standard \"pre-train, fine-tune per task\" paradigm. The demonstration of successful multi-task fine-tuning and the introduction of the SEQ2FUNC dataset for sequence-to-text generation are significant steps toward more versatile and useful genomic models.\n\n## Cons\n- The architecture ablation (RoPE vs. ALiBi, RMSNorm vs. LayerNorm) explores known components, and the conclusion that \"ROPE + Bias-Free Layernorm delivers the best trade-off\"  is an empirical finding for this domain, not a fundamental architectural innovation.\n\n- The multi-task fine-tuning approaches (multi-head and instruction-style next-token prediction) are standard techniques in NLP. While their application to genomics is valuable, it is not a novel methodological contribution.\n\n- The core idea of SEQPACK—learning to compress a sequence by selecting important tokens—is conceptually similar to various token pruning, sequence compression, and \"attention sink\" aware methods in NLP literature. The paper does not adequately compare SEQPACK to these related works, making its specific novelty hard to assess.\n\n- The paper presents Omni-DNA as a holistic framework, which makes it difficult to disentangle the source of performance gains. For instance, in the benchmarks in Table 2 and Table 3, is the superior performance due to the optimized backbone, the larger pre-training dataset (300 billion nucleotides ), the specific pre-training recipe, or a combination? The paper lacks clear ablation studies that would isolate the impact of each \"targeted improvement.\"\n\n- The SEQ2FUNC dataset is a cornerstone of the paper's multi-modal claims, but its creation process is concerning. The dataset is generated by prompting an LLM to \"generate refined, natural-language descriptions\" from coarse annotations. The quality control relies on two LLM passes agreeing and rejecting \"low-confidence cases\".", "was_revised": false, "final_justification": "", "rebuttal": "We appreciated for your feedback. We have provided additional experiment results in response to your questions, which include further ablations on pretraining scheme and data to the downstream model performance and an comparison to the other sequence pruning methods. In the following, we also discuss the quality of SEQ2FUNC dataset and the significance of multi-tasking. \n\n### Q1. Does the performance gain of Omni-DNA come from pretraining recipe, data or optimized backbone?\n\nWe acknowledge that our holistic framework approach made it challenging to isolate individual contribution sources. To address this concern and provide the requested ablation analysis to disentangle the key factors contributing to Omni-DNA's performance. Our original paper included ablations for architectural choices (model size, normalization layers, and positional encoding). To isolate the remaining factors you identified, we designed two controlled experimental sets using the 116M-parameter Omni-DNA.\n\n**Experiment 1**: Pre-training Objective Analysis\n\n- Masked Language Modeling (MLM) using HuggingFace's default flow and configurations [7]\n- Next-Token Prediction (NTP) using our original training framework\n- This directly isolates the impact of our pre-training recipe choice\n\n**Experiment 2**: Corpus Scaling Analysis\n\n- Systematic evaluation across 30B, 60B, 90B, and 120B nucleotides\n- This quantifies the contribution of dataset scale independent of other factors\n\nThese controlled ablations reveal two findings: (1) For Promoter and Enhancer Classification tasks NTP consistently outperforms MLM across all data scales, but for splice classification MLM wins  (2) Performance scales with data size; This indicates that Omni-DNA's performance is potentially attributed to synergy of the pre-training recipe, architecture and data scale.\n\n| Pre-training Objective | Data size (B nt)| Splice All$\\uparrow$ | Promoter$\\uparrow$ | Enhancer$\\uparrow$ |\n|------------------------|------------------|------------|----------|-------------------------|\n| (Original Omni-DNA 116 M) | 300 | 0.927 ± 0.025 | 0.973 ± 0.002 | 0.593 ± 0.005 |\n| NTP                    | 120 | 0.910 ± 0.018 | 0.970 ± 0.008  | 0.585 ± 0.001 |\n| MLM                    | 120 | 0.935 ± 0.017 | 0.968 ± 0.001 | 0.580 ± 0.002 |\n| NTP                    | 90  | 0.862 ± 0.020 | 0.965 ± 0.002 | 0.572 ± 0.009 |\n| MLM                    | 90  | 0.905 ± 0.019 | 0.962 ± 0.005 | 0.560 ± 0.004 |\n| NTP                    | 60  | 0.864 ± 0.022 | 0.958 ± 0.004 | 0.555 ± 0.001 |\n| MLM                    | 60  | 0.880 ± 0.021 | 0.943 ± 0.001 | 0.550 ± 0.004 |\n| NTP                    | 30  | 0.820 ± 0.025 | 0.945 ± 0.009 | 0.535 ± 0.002 |\n| MLM                    | 30  | 0.840 ± 0.024 | 0.940 ± 0.004 | 0.530 ± 0.005  |\n\n### Q2.  SEQPACK and other token pruning methods used in NLP\n\nWe acknowledge there are existing token pruning methods in NLP. However, SEQPACK's unique way of compressing the input sequence with geometric splitting schema is not used by existing token pruning methods in NLP. Below, we highlight these factors and provide additional experimental results. We will add these into a new Appendix section in our revised paper later to illustrate key differences.\n\n\n**Key Differentiating Factors:** \nSEQPACK operates at significantly higher compression ratios (c = $\\frac{L_{original}}{L_{pruned}}$) than typical NLP pruning methods. In the Enhancer-Target Prediction task, SEQPACK achieves compression ratios of c = 100-220, whereas most existing pruning methods (e.g., LazyLLM[1], Dynamic Context Pruning in NeurIPS 2023[2]) typically operate at $c = 4$ to $5$ in their original NLP applications. Additionally, many existing token pruning methods are designed specifically for generation tasks (i.e., decoding)[3][4] rather than sequence understanding (i.e., the full-sequence encoding), limiting their applicability to our genomic prediction benchmarks.\n\nTo directly assess SEQPACK's effectiveness against established methods, we adapted Dynamic Context Pruning (DCP) to work with Omni-DNA 116M on the Enhancer-Target Prediction task. The results demonstrate SEQPACK's substantial advantages:\n\n| Model        | AUROC (± SE) | Comp. Ratio |\n|--------------|--------------|-------------|\n| ABC (expert) | 0.926  |  |\n| CNN          | 0.803 ± 0.022 |  |\n| **Seq-Pack** | 0.894 ± 0.015 | 220 |\n| **Seq-Pack** | 0.899 ± 0.009 | 150 |\n| **Seq-Pack** | **0.907 ± 0.011** | 50 |\n| **DCP** | 0.502 ± 0.089 | 220 |\n| **DCP** | 0.632 ± 0.010 | 150 |\n| **DCP** | 0.880 ± 0.012 | 50 |\n\nAt extreme compression ratios (220×, 150×), DCP performance degrades severely, barely outperforming random baselines, while SEQPACK maintains strong performance (AUROC > 0.89). Even at moderate compression (50×), SEQPACK outperforms DCP (0.907 vs 0.880 AUROC). These results highlight SEQPACK's unique capability to maintain genomic sequence understanding at compression levels that render existing NLP methods ineffective.\n\n### Q3. Quality of LLM-based Dataset\nWe appreciate the concern about training on LLM-generated text. In our pipeline, the LLM is not asked to infer function from a sequence. Instead, for each record we first retrieve structured fields from NCBI (e.g., molecule type and canonical gene/product names). The LLM is then constrained to rephrase those fields into a readable description to minimize the hallucination. An example is shown below. \n\nExamples: \n> NCBI Annotation: mRNA, regulator of G-protein signaling 20\n\n> LLM Annotated: RGS20 mRNA encodes a regulator of G-protein signaling that acts as a GTPase-activating protein (GAP) for Gα subunits—especially Gαz and Gαi—thereby accelerating GTP hydrolysis and terminating GPCR signaling (dampening pathways such as μ-opioid receptor signaling).\n\nQuality control includes a dual-pass agreement check (two independent generations must match on key entities). These controls aim to ensure the rewrite remains a faithful, human-readable rendering of the underlying NCBI fields rather than a speculative annotation. For the test split, we recruited a genomic domain expert to examine 10% of the dataset (around 2700 records) to cross-validate on the quality, and each of the selected records does not have hallucination. \n\nLLM-based weak/synthetic labeling—now routine in NLP [5], multimodal learning [6], and clinical AI [7] —can provide reliable, performance-boosting supervision when properly constrained and filtered. We extend this to genomics by converting structured database fields into readable supervision to train task-ready models at scale.\n\n### Q4. The Novelty of the Multi-Tasking\nWhile we acknowledge that multi-task learning techniques are established in NLP, our contribution lies in demonstrating their unexplored potential in genomic foundation models.\n\n**Key Contribution:**\nOur work reveals that multi-task learning can achieve remarkably high performance on standard genomic benchmarks—a finding not systematically explored in prior genomic foundation model research. This identification of multi-tasking's effectiveness in genomics represents an important empirical contribution to the field. Applying mature techniques to a new domain is valuable when it yields new evidence and usable practice. Recent work in genomics (e.g., AlphaGenomep[8]) likewise achieves strong results by emphasizing training strategy over architectural novelty. Our experiments reinforce this trend: careful multi-tasking—not a new model class—drives the improvements, offering the community actionable guidance for more effective GFMs.\n\n### Q5. \"limitation is the immense complexity ... difficult for other researchers to reproduce\"\nWe acknowledge the concern about reproducibility and have prioritized accessibility in our implementation. To ensure broad adoption and ease of use, we provide comprehensive resources that integrate seamlessly with mainstream frameworks. \n\n**Accessibility Features:** All components of our framework—including pretrained models, SEQPACK implementation, and training code—are fully compatible with standard libraries such as Hugging Face Transformers[9] and TRL[10]. This represents a significant advantage over existing DNA foundation models that often require substantial adaptation or custom implementations.\n\n**Ease of Reproduction:** Researchers can reproduce our results or build upon our work with minimal setup. The model can be loaded and fine-tuned for sequence understanding tasks with just a few lines of standard code, or directly used as a DNA generation model without modification. This streamlined approach removes typical barriers to entry that have limited adoption of genomic foundation models. As detailed in our supplementary materials, we provide the complete codebase. If accepted, the codebase would be hosted as an open-source repository for full reproducibility, and we will keep track of the issues. And we will release all pretrained model weights, and detailed documentation. \n\n[1] Lazyllm: Dynamic token pruning for efficient long context llm inference. arXiv 2024.\n\n[2] Dynamic context pruning for efficient and interpretable autoregressive transformers. NeurIPS 2023.\n\n[3] Saliency-driven dynamic token pruning for large language models. arXiv 2025.\n\n[4] H2o: Heavy-hitter oracle for efficient generative inference of large language models. NeurIPS 2024.\n\n[5] Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv 2022.\n\n[6] Visual Instruction Tuning. arXiv 2023.\n\n[7] Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework.  JMIR Medical Informatics 2025\n\n[8] AlphaGenome: advancing regulatory variant effect prediction with a unified DNA sequence model. bioRxiv 2025.\n\n[9] Huggingface's transformers: State-of-the-art natural language processing. arXiv 2019\n\n[10] TRL: Transformer Reinforcement Learning"}, {"review_id": "jO8NmChZu4", "reviewer": "Reviewer_kHNe", "summary": "The paper proposes a genomic foundation model (GFM), Omni-DNA, architecture that seeks to improve upon existing GFMs by expanding the vocabulary and extending the context via the SeqPack operator. Omni-DNA is also fine-tuned via a multi-task approach to leverage shared knowledge across tasks.\n\nThe authors first establish the optimal Omni-DNA architecture through ablation studies on the performance of the pre-trained model with various types of normalization layers, positional encoding and parameter budget. Subsequently, multi-task learning is enabled by adding extra tokens to the vocabulary and SeqPack is used to compress long contexts. The latter two innovations are incorporated during the post-training phase of Omni-DNA training.\n\nOmni-DNA is evaluated against existing GFMs on benchmarks from the Nucleotide Transformer paper, and the Genomic  Benchmark. Additional experiments on the task of enhancer-target prediction and genetic disease variant-type classification demonstrate the long-range capability of Omni-DNA. Finally, the task of sequence-to-function generation is shown as an example of the capacity of Omni-DNA to support textual outputs.", "questions": "**Questions:**\n\n1.\tSeqPack compresses a sequence of length K into a much smaller embedding of length L. Does this lead to a loss of resolution, thereby limiting OmniDNA’s sensitivity to very short sequence tasks, such as those related to single nucleotide polymorphisms (SNPs)?\n2.\tIn Table 3, Caduceus-PH seems to perform comparably to Omi-DNA despite having 300x fewer parameters. Could the additional performance shown by Omni-DNA simply be a result of the much larger parameter space it explores?\n3.\tWhat do you mean by “output all tasks” in line 235?\n4.\tWhen conducting multi-task training, does the choice of tasks impact the benefit one could derive from this approach? Intuitively, one would expect more performance gain if related tasks are learned together.\n\n**Suggestions:**\n\nThere are several typos that should be corrected.\n1.\tLine 14: “outperform” -> “outperforms”\n2.\tLine 84: “Omni-DNA are” -> “Omni-DNA is”\n3.\tLine 85: “data are” -> “data is”\n4.\tLine 211: “HynnaDNA”  “HyenaDNA”\n\nAdditionally, I recommend moving the discussion of empirical insights to after the results section. In its current form, there is little connection between the empirical insights and any of the ensuring discussion.", "limitations": "The authors have not addressed the limitations or potential negative societal impact of their work. In isolation, I do not there are clear negative social impacts arising from this work, but if the downstream tasks are of a sensitive nature, concerns regarding malicious use and user privacy naturally follow. This is especially true due to the multi-task paradigm, where private information from the dataset for one task may “leak” into the embeddings used to perform another task.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 4, "strengths_and_weaknesses": "The paper addresses a well-established shortcoming of several existing GFMs in that they are unable to support long context windows due to the poor scaling of attention. In addition to that, the idea of jointly fine-tuning on multiple tasks is an intriguing one as it allows for the transfer of knowledge across related tasks; it is less clear if this benefit would be realized if the selected tasks are known to arise from entirely separate biological phenomena/pathways.\n\nThe use of ablation studies to justify the design choices made in Omni-DNA is also appreciated, as it enables the reader to appreciate the reasoning used to arrive at the proposed GFM. The use of well-established genomic benchmarks during the evaluation of Omni-DNA also serves as a fair comparison. However, it isn’t clear if the increased performance of Omni-DNA models stems from the proposed innovations or the significantly higher parameter counts. \n\nHowever, I see several issues with some of the claims made in this paper. These are detailed below:\n\n1.\tIn Section 1, the authors claim that “existing works usually adopt the backbone of language models”. While this was true of early GFMs, newer GFMs have innovated on these early models, in ways reminiscent of the improvements outlined in this work.\n2.\tOn line 39, the authors state that “the state-of-the-art models on short context cannot do long-context tasks.” It is unclear if they are referring to which models they are referring to here, as several models such as Mamba and HyenaDNA, are capable of handling very long context lengths.\n3.\tOn lines 63-65, it is stated that most GFMs are restricted to ternary alphabets. This is incorrect as it ignores tokenization, which is a key area of development in GFMs.\n4.\tOn line 75, the authors motivate their use of SeqPack due to the whole genome containing long “junk” regions. I would be wary of resorting to such a description as non-coding regions of the DNA have been shown to contain information, such as regulatory elements and useful motifs. In fact, this might suggest that such an approach renders Omni-DNA to be less useful for tasks that deal with non-coding regions of genomic sequences.\n\nAdditionally, the writing of the paper has several issues which should be addressed.\n\n1.\tWhile I appreciate the inclusion of the ablation studies in Section 2.2, the introduction and subsequent discussion of those studies comes across as abrupt, instead of being used to motivate the design choices made in Omni-DNA.\n2.\tSection 3.3.1 is difficult to follow.\n3.\tWhile it is admirable that Omni-DNA is capable of mapping sequences to functions (Section 4.4), this does lend weight to the argument that Omni-DNA is a more capable GFM than existing models. Moreover, the baseline there is an NLP model (GPPT-4o), which one would not expect to do well on this task anyway.", "was_revised": true, "final_justification": "While I think this work makes significant contributions to the area of GFMs, I have reservations about the pre-filtering of supposed junk regions and the vastness of the proposed model.", "rebuttal": "Thank you for the detailed feedback. Below, we first address each of your questions, then clarify Section 3.3.1 and present a revised Contributions section aligned with your suggestions.\n\n### Q1. SeqPack compresses a sequence of length K into a much smaller embedding of length L. Does this lead to a loss of resolution, thereby limiting OmniDNA’s sensitivity to very short sequence tasks?\n\nWe intentionally do not apply SeqPack during pretraining. Instead, SeqPack is an optional, task-configurable module used at fine-tuning time. This separation preserves a fully resolution-retaining base model while allowing practitioners to choose compression only when it benefits a specific task.  \n\nFor tasks that require base-level fidelity (e.g., SNP-focused tasks), SeqPack’s compression ratio ($\\frac{L_{original}}{L_{pruned}}$) can be set to 1× (i.e., no compression). In this setting, there is no information loss, so sensitivity to short motifs is unchanged. For long-sequence tasks, when inputs exceed the model’s native context window (due to memory/time constraints), SeqPack enables efficient processing with minimal performance loss by compressing long contexts while preserving salient information. This allows Omni-DNA to handle long-range tasks within practical computational budgets.\n\nTo demonstrate SeqPack's effectiveness compared to other context length compression algorithms, we evaluated it against Dynamic Context Pruning (DCP) by Anagnostidis et al.[1] on the Enhancer–Target Prediction task. In the table below, we applied DCP to OmniDNA-116M and compared its performance with SeqPack. DCP performed significantly worse than SeqPack across compression ratios ranging from 100 to 200, barely outperforming the random baseline model.\n\n\n| Model        | AUROC (± SE) | Comp. Ratio |\n|--------------|--------------|-------------|\n| ABC (expert) | 0.926  |  |\n| CNN          | 0.803 ± 0.022 |  |\n| **Seq-Pack** | 0.894 ± 0.015 | 220 |\n| **Seq-Pack** | 0.899 ± 0.009 | 150 |\n| **Seq-Pack** | **0.907 ± 0.011** | 50 |\n| **DCP** | 0.502 ± 0.089 | 220 |\n| **DCP** | 0.632 ± 0.010 | 150 |\n| **DCP** | 0.880 ± 0.012 | 50 |\n\n\n### Q2. In Table 3, Caduceus-PH seems to perform comparably to Omi-DNA despite having 300x fewer parameters. Could the additional performance shown by Omni-DNA simply be a result of the much larger parameter space it explores?\n\nThis performance comparison can be explained at multiple levels. First, the fundamental architectural differences between Mamba and Transformers (state space models vs. dense self-attention) play a crucial role. According to the original Mamba papers [2][3], Mamba-based models are computationally efficient, demonstrating advantages with lower computational requirements (smaller models and reduced training). However, when scaling both Mamba and Transformer architectures to billion-parameter scale, empirical results [4] show that large Mamba-based models (at Billion Scale) trained on fewer than 1 trillion tokens perform significantly worse than transformer-based models on sequence understanding tasks. This motivates the use of transformers as the backbone in this work. Second, Caduceus-PH benefits from Reverse Complementarity (RC) data augmentation during training. This augmentation technique enables the model to learn effectively with substantially fewer parameters by leveraging the inherent symmetry in DNA sequences.\nNotably, when comparing OmniDNA to other transformer-based models like DNABERT2, OmniDNA consistently demonstrates superior performance across most tasks, suggesting that the performance gains extend beyond simply having more parameters and reflect architectural and training improvements.\n\n\n\n### Q3. What do you mean by “output all tasks” in line 235?\n\nWe thank the reviewer for pointing this out, it is a typo. Line 235 should be \"Omni-DNA (multitask,multi-head)\nand NTv2 (multitask,multi-head) outperform their single-task counterparts on all tasks\".\n\n### Q4. Does the choice of tasks impact the benefit one could derive from this approach?\n\nYes, the choice of tasks would impact the benefits derived from this approach. In the multitasking framework presented in this work, a shared model backbone processes all tasks, followed by lightweight task-specific linear MLPs. When tasks are too dissimilar or contradictory to each other, the shared backbone may struggle to learn useful hidden representations.\nHowever, this limitation can be addressed by incorporating more expressive task-specific decoders. Such decoders would provide greater flexibility for the model to adapt its representations to diverse tasks, potentially enabling effective joint learning among seemingly unrelated tasks. \n\n### W1. The position of Section 2.2  \nWe are happy to move the position of ablation studies in Section 2.2 for a better flow. Thank you for the suggestion on that.\n\n### W2. Additional Baselines for sequences to functions \nWe have added ChatNT[5] as an additional baseline for sequence-to-function tasks. ChatNT employs NTv2 500M as its DNA sequence encoder and a 7B language model as the language decoder, processing (DNA/RNA, text) inputs to generate textual outputs. Following the methodology from the original paper, we fine-tuned ChatNT using our Seq2Func training split. ChatNT@zeroshot represents direct application to the function prediction task without fine-tuning, while ChatNT@ft is the fine-tuned version trained on our evaluation split for 10 epochs.\nThe results demonstrate that ChatNT, despite being specifically designed for handling (text, sequence) inputs and generating textual outputs, does not perform well on function prediction tasks. In zero-shot evaluation, although the model was instructed to output function descriptions, it consistently produced a fixed formatted response: \"This RNA exhibits a motif of \\<number\\>.\" The fine-tuned version (ChatNT@ft) provided more reasonable answers but still lagged behind OmniDNA's performance.\nThese results show that even compared to ChatNT, which uses separate tokenizers and models for DNA and text processing, OmniDNA presents a simpler yet more effective solution for sequence-to-function prediction.\n\n| Model              | Weighted F1 | MCC      |\n|--------------------|-------------|----------|\n| Omni-DNA@ft         | **0.730**   | **0.367** |\n| chatnat@ft         | 0.701           | 0.220        |\n| chatnt@zeroshot    | 0.479           | 0.001       |\n| GPT-4o (zero-shot) | 0.659       | -0.015   |\n| Random Guess       | 0.483       | 0.008    |\n\n### W3. Clarification on Section 3.3.1\n    \nIf we directly use the top-K selection method, we get hard binary choices (either a token is selected or not) and this creates a problem: since the selection is discrete, there's no gradient information flowing back through the selection process. We use Gumbel–Softmax to build a soft, differentiable selector $y_{\\text{soft}}$ from the scores, then apply a straight-through estimator: the forward pass uses hard one-hot indices for behavior, while the backward pass uses the soft probabilities for gradients. Thus, the model learns from a smooth relaxation even though the forward selection is discrete. At inference, we switch to ordinary discrete selection (sampling or top-k).\n\n```\nInputs\n X: (B, L, D) token embeddings\n weights: (L,) non-negative scores (normalized internally)\n k: number of tokens to keep\nOUTPUT: selected_token_embeddings(B,k,D)\n\nIF training_mode:\n    // Gumbel-Softmax for differentiable selection\n    logits = expand(log(weights), to=(k,L))\n    gumbel = sample_gumbel_noise(shape=(k,L))\n    y_soft = softmax(logits + gumbel )\n    // Straight-through: hard forward, soft backward\n    y_hard = one_hot(argmax(y_soft))\n    y_st = detach(y_hard - y_soft) + y_soft\n    // Apply selection and return\n    RETURN einsum('kl,lbd->kbd', y_st, transpose(sequence))\n```\n\n### W4. Clarification on \"Junk Regions\" and SeqPack's Treatment of Non-coding DNA\nWe appreciate the reviewer's concern regarding our terminology and its implications for non-coding regions. To clarify, our use of \"junk regions\" specifically refers to repetitive sequences and regions with no currently identified functional significance, not regulatory elements or other functionally important non-coding DNA. As explicitly stated in line 75, regulatory elements are not categorized as junk regions in our framework.\nImportantly, SeqPack does not impose any algorithmic bias against non-coding regions. The algorithm operates in a task-agnostic manner during the packing process, preserving sequence information regardless of coding status. During task-specific fine-tuning, the model learns to identify and weight functionally relevant elements—including those in non-coding regions—based on their importance for the particular downstream task. This approach allows Omni-DNA to maintain sensitivity to regulatory elements, enhancer sequences, and other functional non-coding elements when they are relevant to the prediction task. We acknowledge that our terminology could be clearer and will revise it to avoid potential misinterpretation.\n\n### W5. Writing (Lines 31–40)\nThank you for the suggestion. We have revised lines 31–40 to improve clarity, concision, and flow. For the full updated rationale, please see our response to Reviewer MarC (Q10).\n\n[1] Anagnostidis, Sotiris, et al. \"Dynamic context pruning for efficient and interpretable autoregressive transformers.\" NeurIPS 2023\n\n[2] Dao, Tri,  et al.  \"Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.\" ICML 2024\n\n[3] Gu, Albert,  et al.  \"Mamba: Linear-time sequence modeling with selective state spaces.\"  COLM 2024.\n\n[4] Waleffe, Roger, et al. \"An empirical study of mamba-based language models.\"  arXiv 2024. \n\n[5] de Almeida, Bernardo P., et al. \"A multimodal conversational agent for DNA, RNA and protein tasks.\" Nature Machine Intelligence 2025"}, {"review_id": "A2RdGHLGap", "reviewer": "Reviewer_nFBa", "summary": "This manuscript introduces the Omni-DNA. a family of models spanning 20M to 1.1B parameters that supports sequence understanding, long-context genomic reasoning, and natural-language annotation.\n\nOmni-DNA is a decoder only structure for modelling genetic information. The main contribution of this manuscript lie in three aspects: \n1. Multi-Task Learning to improve performance. \n2. Vocabulary expansion to allow model directly generate natural functional descriptions of given DNA sequences. \n3. SeqPack to combine multiple nucleotide to a single token to allow model operates with long sequences.", "questions": "The results of HyenaDNA is significantly lower than the values reported in the original paper. Please explain.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nThe authors have developed models in different sizes to evaluate the effectiveness of the proposed method.\n\nThe proposed method excels at the chromatin tasks in the NT Benchmark, highlighting the effectiveness of the multi-task learning. \n\n## Weakness\n\n### Technical soundness of the proposed method. \n\nMLM and NTP represents the two paradigm of self-supervised learning: discriminative and generative. \nNo paradigm is superior than another. \nFor example, Vision Transformer have been used in CV for years, but no one uses autoregressive model for extracting visual features. This is simply because of the nature of the data. \nNature language is Turing-complete and scalable. Unfortunately, genome is not. \n\nPeople write new sentences everyday, like what the authors have done, what I am currently doing, and what comment the AC will give. They are easily accessible on the internet. However, the human genome is static. While mutation happens all the time, the technology we have cannot capture them, and we are unable to put them into database. From this point of view, genome is more like pictures. \n\n### Losing nucleotide level precision\n\nThe SeqPack can effectively reduce the number of tokens in a sequence. But this is at the cost of losing single nucleotide resolution -- which is crucial for many gnomic tasks. \n\n### Unsatisfactory results on cCRE and splicing\n\nThe results on cis-regulatory elements and splicing sites is rather unsatisfactory. \n\n### Unclear writing \n\nThe authors mention that it is possible to use LM head for multi-task learning. \n\nWhat heads does the proposed method use in this paper? Have the authors evaluated the performance of different heads?", "was_revised": true, "final_justification": "> Why 4?\n\nThe Seqpack algorithm can be very important to long context windows in Genetic Foundation Models. \n\n> Why not lower score? \n\nThe Seqpack method can be of interest to the border research community. In experiments, it has demonstrated satisfactory performance. \n\n> Why not higher score? \n\nMy concerns in cCRE and splicing remain. For an advanced method like Omni-DNA, it should not be *modest*, but rather *outstanding*.", "rebuttal": "We appreciate your feedback on the paper. Below, we provide detailed responses to your questions.\n\n### Q1. MLM vs. NTP as pretraining objective\n    \nWe agree with your view if one considers the genome as an object that needs to be modelled as a whole. However, there is no de-facto paradigm for pretraining genomic foundation models. Some recent models, such as Evo[1] and Evo2[2], adopt next-token prediction (NTP), while others, such as DNABERT2 and NTv2, use masked-language modeling (MLM); as do others for genomic foundation models, e.g., contrastive learning in Orthrus [3].  Our work does not depend on a specific pretraining objective, and the choice is largely orthogonal to our contribution. \n\nTo address this point, we added a small ablation as shown below: we pretrain the same backbone (Omni-DNA 116M) with either NTP or MLM under the same token budget and model size, then fine-tune without a causal mask for the discriminative downstream tasks (so both settings see bidirectional context at fine-tuning time). We observe task-dependent differences: NTP matches or exceeds MLM on several tasks (Promoter and Enhancer), while MLM is competitive on others (Splice); importantly, our main conclusions are unchanged. \n\n\nA practical side note is that NTP-pretrained models also retain generative capability, which can be convenient for sequence design use cases; however, this is not central to our claims.\n\n| Pre-training Objective | Data size (B nt)| Splice All$\\uparrow$ | Promoter$\\uparrow$ | Enhancer$\\uparrow$ |\n|------------------------|------------------|------------|----------|-------------------------|\n| (Original Omni-DNA 116 M) | 300 | 0.927 ± 0.025 | 0.973 ± 0.002 | 0.593 ± 0.005 |\n| NTP                    | 120 | 0.910 ± 0.018 | 0.970 ± 0.008  | 0.585 ± 0.001 |\n| MLM                    | 120 | 0.935 ± 0.017 | 0.968 ± 0.001 | 0.580 ± 0.002 |\n| NTP                    | 90  | 0.862 ± 0.020 | 0.965 ± 0.002 | 0.572 ± 0.009 |\n| MLM                    | 90  | 0.905 ± 0.019 | 0.962 ± 0.005 | 0.560 ± 0.004 |\n| NTP                    | 60  | 0.864 ± 0.022 | 0.958 ± 0.004 | 0.555 ± 0.001 |\n| MLM                    | 60  | 0.880 ± 0.021 | 0.943 ± 0.001 | 0.550 ± 0.004 |\n| NTP                    | 30  | 0.820 ± 0.025 | 0.945 ± 0.009 | 0.535 ± 0.002 |\n| MLM                    | 30  | 0.840 ± 0.024 | 0.940 ± 0.004 | 0.530 ± 0.005  |\n\n\n\n\n### Q2. Losing nucleotide level precision\n\nWe intentionally do not apply SeqPack during pretraining. Instead, SeqPack is an optional, task-configurable module used at fine-tuning/inference time. This separation preserves a fully resolution-retaining base model while allowing practitioners to choose compression only when it benefits a specific task.  \n\nFor tasks that require base-level fidelity (e.g., SNP-focused tasks), SeqPack’s compression ratio can be set to 1× (i.e., no compression). In this setting, there is no information loss, so sensitivity to short motifs is unchanged. For long-sequence tasks, when inputs exceed the model’s native context window (due to memory/time constraints), SeqPack enables efficient processing with minimal performance loss by compressing long contexts while preserving salient information. This allows Omni-DNA to handle long-range tasks within practical compute budgets.\n\nTo demonstrate SeqPack's effectiveness compared to other context length compression algorithms, we evaluated it against Dynamic Context Pruning (DCP) [4] on the Enhancer–Target Prediction task. In the table below, we applied DCP to OmniDNA-116M and compared its performance with SeqPack. DCP performed significantly worse than SeqPack across compression ratios ranging from 100 to 200, barely outperforming the random baseline model.\n\n\n\n| Model        | AUROC (± SE) | Comp. Ratio |\n|--------------|--------------|-------------|\n| ABC (expert) | 0.926  | — |\n| CNN          | 0.803 ± 0.022 | — |\n| **Seq-Pack** | 0.894 ± 0.015 | 220 |\n| **Seq-Pack** | 0.899 ± 0.009 | 150 |\n| **Seq-Pack** | **0.907 ± 0.011** | 50 |\n| **DCP** | 0.502 ± 0.089 | 220 |\n| **DCP** | 0.632 ± 0.010 | 150 |\n| **DCP** | 0.880 ± 0.012 | 50 |\n\n\n### Q3. Unsatisfactory results on cCRE and splicing\n    \n    \nWe appreciate the concern. Our reading is that this comment refers to the NT downstream benchmarks in the Table 2 of the original manuscript.\n    \nOverall, Omni-DNA attains the best score on 13/18 tasks. The remaining tasks which Omni-DNA does not performs the best are related to promoter (cCRE-related) and splicing categories.\n\nThe promoter classification task includes two subtasks: \n\n- NONTATA: Omni-DNA-116M trails the top model (NTv2-500M) by 0.1%.\n- ALL promoters: Omni-DNA-1B is 0.2% shy of the best result.\n\nThese deltas are marginal and within typical variance; they do not indicate a substantive weakness. For splicing tasks, Omni-DNA-1B achieves an average of 0.959 across the three splicing tasks, ranking just behind NT-2.5B and NTv2-500M, and ahead of all other baselines. Given reported standard deviations, the differences are modest rather than “unsatisfactory.” \n    \nThe evidence shows consistent, near-SOTA performance on promoter/cCRE and strong top-tier results on splicing, alongside state-of-the-art performance on the majority of tasks. We therefore believe the characterization “unsatisfactory” is not supported by the data.\n        \n    \n    \n### Q4. What heads does the proposed method use in this paper? Have the authors evaluated the performance of different heads?\n\n\nAs stated in section 3.1 (L125–128), we attach a lightweight 1-hidden-layer MLP (“linear-MLP”) head per task on top of the model’s sequence representation, following prior GFMs (e.g., NTv2, DNABERT2). Concretely, with last-token $r \\in \\mathbb{R}^d$, the head is, $z = W_2\\sigma(W_1 r + b_1) + b_2 \\in \\mathbb{R}^{C}, \\qquad \\hat{y} = \\mathrm{softmax}(z)$, where $C$ is the number of classes. As we update the transformers' weights during fine-tuning; increasing head complexity did not yield performance gains in our checks (linear vs. 1-hidden-layer vs. deeper), so we keep this lightweight head for all results.\n\n\n### Q5. Results with HyenaDNA\nWe notice the gap between the original Hyena paper and the reproduction experiments, hereby providing a detailed discussion of the problem in the Appendix section E. In summary, the difference arises from two factors. 1) Evaluation Settings: our evaluation setting follows Caducess [5], which allows a maximum fine-tuning epoch of 20 for NT Downstream tasks and 10 epochs for Genomic Benchmark; while the original paper allow the max finetuning epoch to a few hundreds.  2) the original HyenaDNA uses the EMA (Exponential Moving Average), while our evaluation does not use it. Note that the reported results in our manuscript are consistent with the prior work, such as Caducess[5] and the follow-up work[6]. \n\n\n[1] Nguyen, Eric, et al. \"Sequence modeling and design from molecular to genome scale with Evo.\" Science 2024\n\n[2] Brixi, Garyk, et al. \"Genome modeling and design across all domains of life with Evo 2.\" bioRxiv 2025\n\n[3] Fradkin, Philip, et al. \"Orthrus: towards evolutionary and functional RNA foundation models.\" bioRxiv 2024\n\n[4] Anagnostidis, Sotiris, et al. \"Dynamic context pruning for efficient and interpretable autoregressive transformers.\" NeurIPS 2023\n\n[5] Schiff, Yair, et al. \"Caduceus: Bi-directional equivariant long-range dna sequence modeling.\" ICML 2024\n\n[6] Wu, Wei, et al. \"GENERator: a long-context generative genomic foundation model.\" arxiv 2025"}, {"review_id": "Q9kjyIPhRR", "reviewer": "Reviewer_UqSE", "summary": "This paper presents Omni-DNA, a new autoregressive DNA language model that includes a novel method of long context compression. The model is trained on genome assemblies from NCBI. It is fine-tuned by adding additional tokens or multi-head fine-tuning. It is benchmarked on Gene regulatory element classification, Pathogenic variant effect prediction, Enhancer–target gene interaction mapping, and Sequence-conditioned functional interpretation generation.", "questions": "1. For enhancer-target prediction, did you mask the intervening bases between the enhancer and the promoter as in (https://openreview.net/forum?id=opv67PpqLS)? If so, how was this masking accomplished?\n\n2. I am very confused by the genetic disease variant-type classification. Conventionally, models are evaluated by their ability to classify disease-causing variants from negative control variants, which have no phenotypic effect. The assumption is that the models should be able to identify variants that change protein sequence or alter conserved regulatory sequences, vs. variants that have no impact. Here, the study seems to have taken the surprising route of classifying variants that cause one disease from variants that cause a different disease. All of these variants are pathogenic, so what is the model classifying exactly? My guess is that it is learning the genes near the variants, and has learned that some genes are associated with a particular disease - in which case it is not really evaluating whether the model has learned properties of variants. Please clarify the purpose of this task or draw the negative controls from non-pathogenic variants.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The model achieves SOTA performance on classification tasks which is impressive. The task of generating textual descriptions of DNA sequences also appears very useful and original, adding a novel capability to the current foundation models in the field. However, the remaining evaluations are confusing and could use some clarification or editing (see below).", "was_revised": true, "final_justification": "I am satisfied with the explanation of masking strategy. \n\nI remain unconvinced by the evaluation of sequence models based on their ability to link pathogenic variants to a disease - it still seems to me that this is likely to reflect memorization that certain regions of the genome are disease-associated, rather than the model understanding the properties of the variants themselves (e.g. predicting their effect on gene expression). However, seeing that the authors have added a non-pathogenic variant baseline where Omni-DNA also shows the highest performance addresses this comment.\n\nAs such, I maintain the previous rating and the recommendation to accept.", "rebuttal": "Thank you for your positive feedback of and constructive suggestion for our manuscript. Below, we address each of your questions and comments in detail.\n\n### Q1. Did you mask the intervening bases between the enhancer and how was this masking accomplished?\nWe adopt the curated benchmark from Cheng et al. [1] and use the same masking strategy to mask the intervening enhancers. In the original implementation, the positions of the intervening enhancers in K562 cell lines are recorded in a text file as a blacklist. This file includes the chromosome, start, end, and strand of the enhancers that need to be masked. During data loading with selene_sdk (a python library), the masked positions are replaced with [0.25, 0.25, 0.25, 0.25]. In our case, we replace these positions with 'N' for Omni-DNA to consume.\n\n\n### Q2. Genetic disease variant-type classification\n\n**Objective:** Conventional training on benign vs. pathogenic labels is valuable but inherently coarse: it often fails to capture nuanced mutation mechanisms—such as how mutation direction varies across diseases and how these differences map to distinct disease phenotypes within the same gene. Our objective in this task is to move beyond generic assessments of deleteriousness and enable the model to learn disease-related patterns that reflect fine-grained functional changes. Disease-type labels serve as a form of weak supervision for learning mutation directionality—something that a simple benign-vs-pathogenic objective cannot reliably disentangle.\n\nIn real-world workflows (e.g., cardiac clinics), clinicians need to prioritize variants specific to the target disease in the form of patient reports, rather than flagging all pathogenic variants across conditions. Current practice relies heavily on rule-based gene–disease associations from resources such as OMIM[2] and the Clinical Genomic Database (CGD)[3] as disease-focused filters. We demonstrate that by fine-tuning general sequence models with disease-type labels, we can rapidly recover these associations and highlight the most relevant variants for the disease of interest—thereby aligning model outputs with established clinical triage pipelines.\n\n\n**Draw the negative controls from non-pathogenic variants:** Nevertheless, we are happy to include the additional results in the appendix, based on sampling negatives from non-pathogenic variants, as shown below. Specifically, we sample the negatives from benign variants with an AF_TGP (allele frequency from the 1000 Genomes Project) less than 0.01, using the same version of ClinVar. The classification F1 is generally higher than that of the disease-type classification task (as shown in Table 5 of the manuscript).\n\n\n| Task | HyenaDNA | DNABERT2 | NTv2 | Omni-DNA |\n|---|---:|---:|---:|---:|\n| **LC** | 0.85 ± 0.02 | 0.73 ± 0.03 | 0.77 ± 0.04 | 0.92 ± 0.01 |\n| **ICCs** | 0.80 ± 0.08 | 0.62 ± 0.02 | 0.76 ± 0.00 | 0.91 ± 0.01 |\n\n**What does the model learn:** \nIn order to examine whether the model learns disease–variant associations by memorizing gene identity or other non-coding regions, we performed a simple ablation study for lung cancer. In this setup, genes were masked with 'N' (based on the human gene annotation file), and the model was still able to achieve reasonable performance. Note that implicitly including gene position is not necessarily undesirable, as this aligns with how hospitals typically prioritize variants using rule-based systems.\n\n| Model      | HyenaDNA     | DNABERT2     | NTv2         | Omni-DNA     |\n|------------|--------------|--------------|--------------|--------------|\n| LC         | 0.83±0.02    | 0.71±0.03    | 0.75±0.04    | 0.93±0.01    |\n| LC (MaskGene)   | 0.79±0.01    | 0.60±0.04    | 0.68±0.02    | 0.87±0.00|\n\n\n\n\n[1] Cheng, Wenduo, et al. \"Dnalongbench: a benchmark suite for long-range dna prediction tasks.\" bioRxiv 2025\n\n[2] https://www.omim.org/  OMIM (Online Mendelian Inheritance in Man)\n\n[3] https://research.nhgri.nih.gov/CGD/  CGD (Clinical Genomic Database)"}, {"review_id": "eeAQTuCc7o", "reviewer": "Reviewer_MarC", "summary": "This paper presents a new family of genomic foundation models, OmniDNA. The authors conduct extensive comparisons against widely used baselines on several downstream tasks. The authors also present a novel algorithm, Seqpack, for extending context length of existing pre-trained models.", "questions": "1. The authors discuss several approaches for extending pre-training vocabularies (Lines 142-149). Which approach worked best / is used in the experiments?\n2. The motivation for SeqPack is that $O(N \\log K)$ is too expensive. Ultimately SeqPack costs $\\Theta(N)$, is the savings here indeed that significant? I would have thought the $\\log K$ factor would not be that untenable?\n3. In Section 3.3.1, the gumbel softmax trick is used to maintain differentiability, but it is then composed with top-k selection. Doesn’t applying top-k destroy differentiability?\n4. Can the authors confirm that all the models from Section 4.2 (Table 4) have 450kbp input sequences. Additionally, would it be possible to provide error bars for Table 4?\n5. What is the positive class ratio of the final datasets used in the experiments from Section 4.3.", "limitations": "See weaknesses.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\nThe experimental work in this paper is extensive and the consistently superior results of the OmniDNA models are promising. The novel Seqpack algorithm is a nice contribution and the ablations on mult-task training and the positional embeddings are useful for practitioners in the field. The breadth of baselines is well curated and the experimental details provided to reproduce the experiments are well organized and extensive. The natural language interaction tasks are novel and interesting.\n\n### **Weaknesses**\n\n1. I disagree with the framing of several of the limitations and contributions.\n    - Specifically, claiming that there has not been exploration of the design space (Limitation 1) seems inaccurate to me. Several models have explored the effect of tokenization (e.g., DNABERT 1 vs 2, and the base pair tokenization of Hyena and Caduceus) and have employed different positional embedding strategies (e.g., NTv1 vs NTv2). Additionally, Caduceus, in particular, created custom extensions of Mamba-based modules specifically for genomic tasks. The authors claim that rather than the “plug-and-play” of existing NLP modules from existing GFMs, they optimize their backbone for genomics. However, I think one could argue that they performed a search over the same modules that other GFMs have employed. I believe that a reframing would be more accurate, improve readability, and would by no means detract from the extensive experiments and useful results of Finding 1 in the paper.\n    - In a similar vein, I disagreed somewhat with the motivation of limitation 2, in the sense that arguing for the need of multi-task training from the lens of expensive storing of weights is not reflective of the reality of how these models will be used or the true bottlenecks that practitioners face with the current scale of GFMs. Here too, more accurately situating the experiments and results of Finding 3 in the rich literature of multi-task training (even in the context of biological models, e.g. Enformer) would be better, in my opinion. This by no means takes away from the fact that exploring multi-task downstream fine-tuning of GFMs is indeed a contribution of this work.\n    - Lastly for limitation 4, the authors state that previous GFMs cannot adapt to varying context lengths, but this claim needs to be supported either with references from the literature or by pointing to specific experimental results from this work that support the claim.\n2. The takeaways from Finding 2 are unclear to me. Several works have already indicated that the NT tasks are in fact not ideal for understanding GFMs (e.g., Kirill, et al [1]). Additionally, it is possible that these tasks are too “easy” and can be saturated. Lastly, unless I am misreading it, I think Figure 3 does in fact show an important correlation between pre-training loss and downstream performance.\n3. I think the authors could include a more robust discussion about what is making OmniDNA work. For example:\n    - Is data curation an important factor?\n    - Is the fact that the authors are using a transformer-based AR model vs BERT-style bi-directional one important (e.g., NTv2 and DNABERT are transformers but not AR, Hyena is AR but not a transformer; so is the claim of OmniDNA that the correct recipe is AR + transformer)?\n    - Is the combination of positional embedding + layer norm a key driver? Specifically, these design decisions are already explored / used in previous models (e.g. NTv2 uses RoPE).\n4. An ablation exploring the efficacy of SeqPack vs existing context extension mechanisms or even the naive approach of simply providing pre-trained models with longer sequences would be useful and convincing. Additionally, exploring the hyperparameters of SeqPack would be informative\n5. Additionally, the computational overhead of SeqPack is not well described or quantified. Adding this would be useful.\n6. Some minor typos:\n    - Line 174: “Understand the…” should be “To understand the…”?\n    - Line 235: “strictly output” should be “strictly outperform”?\n\n\n---\n\n[1] Vishniakov, Kirill, et al. \"Genomic foundationless models: Pretraining does not promise performance.\" bioRxiv (2024): 2024-12.", "was_revised": true, "final_justification": "Authors addressed my concerns and provided extensive additional experimental results", "rebuttal": "We sincerely appreciate your detailed feedback and constructive remarks on our paper. Below, we provide a detailed response to your questions and comments, along with additional ablation results from your suggestion.\n\n### Q1. The authors discuss several approaches for extending pre-training vocabularies (Lines 142-149). Which is used in the experiments?\n\nThere are two tasks which introduce new vocabulary to the Omni-DNA: Next-Token Prediction approach for 10 tasks (section 4.1 Multitasking) and Sequence-to-Function Generation. The key-token replication is only used for multi-tasking; while NEFTune (i.e., adding isotropic Gaussian noise) is used for both tasks. \n\n### Q2. ...the $logK$ factor would not be that untenable?\n\nWhile the theoretical difference between $O(N\\log K)$ and $\\Theta(N)$ might appear marginal, the practical efficiency gains of our method are impactful due to the following reasons:\n\n- 1. As SeqPack needs to be integrated into both finetuning and inference, when N is very large, logK*N could lead to differences as well. \n- 2. Memory footprint is another concern. When we perform global sampling, we need to maintain relatively large tensors in memory, while the splitting approach allows the sequential/parallel processing of each segment.\n- 3. Implicit prior, by encouraging the compressor to sample from each segment, we are enforcing an implicit uniform prior, which could help to avoid the potential issue of collapsing the learned mass on a single region. \n\n\n### Q3. Doesn’t applying top-k destroy differentiability?\n\nIf we directly use the top-K selection method, we get hard binary choices (either a token is selected or not) and this creates a problem: since the selection is discrete, there's no gradient information flowing back through the selection process. We use Gumbel–Softmax to build a soft, differentiable selector $y_{\\text{soft}}$ from the scores, then apply a straight-through estimator: the forward pass uses hard one-hot indices for behavior, while the backward pass uses the soft probabilities for gradients. Thus, the model learns from a smooth relaxation even though the forward selection is discrete. At inference, we switch to ordinary discrete selection (sampling or top-k).\n\n\n```\n// Inputs\n// X: (B, L, D) token embeddings\n// weights: (L,) non-negative scores (normalized internally)\n// k: number of tokens to keep\nOUTPUT: selected_token_embeddings(B,k,D)\n\nIF training_mode:\n    // Gumbel-Softmax for differentiable selection\n    logits = expand(log(weights), to=(k,L))\n    gumbel = sample_gumbel_noise(shape=(k,L))\n    y_soft = softmax(logits + gumbel )\n    \n    // Straight-through: hard forward, soft backward\n    y_hard = one_hot(argmax(y_soft))\n    y_st = detach(y_hard - y_soft) + y_soft\n    \n    // Apply selection and return\n    RETURN einsum('kl,lbd->kbd', y_st, transpose(sequence))\nELSE:\n    // Standard multinomial sampling for inference\n    indices = multinomial(weights, k, no_replacement=True)\n    RETURN sequence[:, sort(indices), :]\n\n```\n\n\n### Q4. Can the authors confirm that all the models from Section 4.2 (Table 4) have 450kbp input sequences. Additionally, would it be possible to provide error bars for Table 4?\nYes, we confirm that all the models are evaluated with the input sequence of length 450,000 bp. We use the original benchmark from [1]. \nFollowing your suggestion, we ran the code from the original benchmark to obtain the new table with error bars as shown below, and we will integrate this into the paper. \n\n\n| Model        | AUROC (± SE) |\n|--------------|-------------|\n| ABC (expert) | 0.926  |\n| CNN          | 0.803 ± 0.022 |\n| HyenaDNA     | 0.826 ± 0.014 |\n| CADUCEUS-PH  | 0.821 ± 0.023 |\n| CADUCEUS-PS  | 0.814 ± 0.020 |\n| **Omni-DNA** | **0.894 ± 0.015** |\n\n\n### Q5.What is the positive class ratio of the final datasets used in the experiments from Section 4.3.\n\nThe positive: negative ratio is 1:1. \n\n\n### Q6. Additional ablation on the SeqPack Parameter with different Compression Ratio\n\n\nTo abalate the parameter of SEQPACK compression ratios (c = $\\frac{L_{original}}{L_{pruned}}$) and assess SEQPACK's effectiveness against established methods, we adapted Dynamic Context Pruning (DCP)[2] to work with Omni-DNA 116M on the Enhancer-Target Prediction task. The results demonstrate SEQPACK's substantial advantages.\n\n\n| Model        | AUROC (± SE) | Comp. Ratio |\n|--------------|--------------|-------------|\n| ABC (expert) | 0.926  | — |\n| CNN          | 0.803 ± 0.022 | — |\n| **Seq-Pack** | 0.894 ± 0.015 | 220 |\n| **Seq-Pack** | 0.899 ± 0.009 | 150 |\n| **Seq-Pack** | **0.907 ± 0.011** | 50 |\n| **DCP** | 0.502 ± 0.089 | 220 |\n| **DCP** | 0.632 ± 0.010 | 150 |\n| **DCP** | 0.880 ± 0.012 | 50 |\n\nAt extreme compression ratios (220×, 150×), DCP performance degrades severely, barely outperforming random baselines, while SEQPACK maintains strong performance (AUROC > 0.89). Even at moderate compression (50×), SEQPACK outperforms DCP (0.907 vs 0.880 AUROC). \n\n### Q7. The computational overhead of SeqPack\n\n\nCompressing inputs (25,600–819,200 tokens) to a fixed 4,096 with SeqPack, we report (i) SeqPack operator time and (ii) Omni-DNA-1B forward time in seconds on the compressed sequence. As input length grows 32×, SeqPack time rises only 3.56× (inference: 0.00898 to 0.03197 s) and 5.95× (training: 0.015 to 0.091 s). Forward time is effectively flat. SeqPack’s share of end-to-end time remains low and increases gently with the compression ratio 6.25× to 200×: 1.2% to 4.0% (inference) and 4.3% to 21.6% (training).\n\n**Inference**\n| length | SeqPack  |  Forward  | SeqPack_percent% |\n|---:|---:|---:|---:|\n| 25,600 | 0.00898 (±0.000) | 0.77257 (±0.003) | 1.1 |\n| 51,200 | 0.01032 (±0.001) | 0.76120 (±0.001) | 1.3 |\n| 102,400 | 0.01225 (±0.001) | 0.77986 (±0.003) | 1.5 |\n| 204,800 | 0.01508 (±0.001) | 0.76666 (±0.001) | 1.9 |\n| 409,600 | 0.02050 (±0.001) | 0.76665 (±0.001) | 2.6 |\n| 819,200 | 0.03197 (±0.001) | 0.76514 (±0.003) | 4.0 |\n\n**Training**\n| length | SeqPack  |  Forward  | SeqPack_percent% |\n|---:|---:|---:|---:|\n| 25,600 | 0.01523 (±0.000) | 0.33756 (±0.002) | 4.3 |\n| 51,200 | 0.01793 (±0.000) | 0.32607 (±0.002) | 5.2 |\n| 102,400 | 0.02339 (±0.001) | 0.34088 (±0.001) | 6.4 |\n| 204,800 | 0.03389 (±0.001) | 0.32727 (±0.002) | 9.4 |\n| 409,600 | 0.05290 (±0.001) | 0.32809 (±0.002) | 13.9 |\n| 819,200 | 0.09067 (±0.001) | 0.32850 (±0.002) | 21.6 |\n\n### Q8. Additional Ablations on why Omni-DNA works\nWe generally believe the transformer's learning capability is the reason why the model works. To see the effect of the training paradigm (BERT vs. NextTokenPrediction), the additional ablations are conducted, with detailed results presented in our response to Q1 of reviewer 1(rvnr).  We could see that the BERT pretraining, in terms of the data source, we found that the number of training samples is a key. But NTP training provides the model with the natural ability for sequence generation, which is generally preferred and has substantial downstream use cases. This is the reason why we use the NTP as the default training scheme. But due to the limit of computational resources, we may not be able to explore all the architectural choices. \n\n### Q9.  Finding 2\nFinding 2 is to emphasize that the lower loss **does not guarantee** better performance in downstream tasks. We can see several bumps in Figure 3, which means such relations are not guaranteed. We should be cautious about using the loss as the only criterion for selecting the best model.\n\n### Q10. Regarding a few suggestions of the introduction section.\n\nWe appreciate your kind suggestion. The below is the reframed paragraph to replace line 31-40 of the original manuscript. \n\nWhile GFMs have demonstrated utility on specific genomic tasks, they still exhibit important limitations: **(i) Limited optimization of architectural components for genomic tasks:** Prior work has explored design choices including tokenization (DNABERT vs. DNABERT-2), positional encodings (NT vs. NTv2), and genomic-specific modules (GPN-MSA, Caduceus), but a more comprehensive, task-specific search of architectures and pretraining strategies tailored to genomic inductive biases is still lacking. **(ii) Limited exploration of multi-task synergies:** Many approaches fine-tune on a single downstream task, missing the opportunity to exploit shared biological signal and achieve cross-task performance gains, despite evidence of such benefits in genomic assay prediction (e.g., AlphaGenome, Enformer). **(iii) Insufficient multimodal capability:** Most existing GFMs restrict vocabularies to nucleotide base pairs, predefined labels, or learned subwords with the BPE tokenizer, limiting tasks that require mapping DNA sequences to functional annotations in natural language (DNA2Func). While ChatNT can process (DNA, text) inputs, it uses separate tokenizers and a two-stage stack consisting of NTv2 as the DNA encoder and Vicuna-7B as the decoder, resulting in less unified handling of multimodal information. **(iv) Context-length adaptivity challenges:** Transformer-based models (e.g., DNABERT-2, NTv2) remain constrained by the quadratic cost of self-attention. This limits the application of transformer-based models to long-context tasks such as enhancer–target prediction. \n\n[1] Cheng, Wenduo, et al. \"Dnalongbench: a benchmark suite for long-range dna prediction tasks.\" bioRxiv 2025.\n\n[2] Anagnostidis, Sotiris, et al. \"Dynamic context pruning for efficient and interpretable autoregressive transformers.\" NeurIPS 2023."}]

Author rebuttal:

Meta-review: This paper introduces Omni-DNA, a family of genomic foundation models (GFMs) that support sequence understanding, long-context reasoning, and natural-language annotation. The authors propose a holistic approach that includes architectural optimizations, a novel compression operator (SEQPACK) for handling ultra-long sequences, and a multi-task fine-tuning strategy. The model achieves state-of-the-art results on 18 out of 26 tasks from established genomic benchmarks and demonstrates strong performance in long-range regulatory tasks and sequence-to-function generation. While Reviewers rvnr  initially raised valid concerns regarding the novelty of individual components and the potential information loss from the SEQPACK compression operator, the authors' extensive rebuttal and commitment to additional experiments have addressed these issues.

The work is deemed acceptable because its holistic approach—delivering state-of-the-art performance across a wide range of tasks, enabling long-context reasoning, and introducing a novel multimodal capability—provides significant practical utility and a valuable foundation for future research. The overall strengths and the community value of the released models and benchmarks outweigh the initial reservations.
Decision: Accept (poster)