Title: Omni-DNA: A Genomic Model Supporting Sequence Understanding, Long-context, and Textual Annotation
Venue: NeurIPS 2025
Accepted: yes
Keywords: DNA

Genomics

Health

Representation Learning
Abstract: The interpretation of genomic sequences is crucial for understanding biological processes. To handle the growing volume of DNA sequence data, Genomic Foundation Models (GFMs) have been developed by adapting architectures and training paradigms from Large Language Models (LLMs). Despite their remarkable performance in DNA sequence classification tasks, there remains a lack of systematic understanding regarding the training and task-adaptation processes of GFMs. Moreover, existing GFMs cannot achieve state-of-the-art performance on both short and long-context tasks and lacks multimodal abilities. By revisiting pre-training architectures and post-training techniques, we propose **Omni-DNA**, a family of models spanning 20M to 1.1B parameters that supports sequence understanding, long-context genomic reasoning, and natural-language annotation. **Omni-DNA** establishes new state-of-the-art results on 18 of 26 evaluations drawn from Nucleotide Transformer and Genomic Benchmarks. When jointly fine-tuning on biologically related tasks, **Omni-DNA** consistently outperform existing models and demonstrate multi-tasking abilities.  To enable processing of arbitrary sequence lengths, we introduce **SEQPACK**—an adaptive compression operator that packs historical tokens into a learned synopsis using a position-aware learnable sampling mechanism, enabling transformer-based models to process ultra-long sequences with minimal memory and computational requirements. Our approach demonstrates superior performance on enhancer-target interaction tasks, capturing distant regulatory interactions at the 450kbp range more effectively than existing models. Finally, we present a new dataset termed **seq2func**, enabling Omni-DNA to generate accurate and functionally meaningful interpretations of DNA sequences, unlocking new possibilities for genomic analysis and discovery.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new family of genomic foundation models, OmniDNA. The authors conduct extensive comparisons against widely used baselines on several downstream tasks. The authors also present a novel algorithm, Seqpack, for extending context length of existing pre-trained models.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\nThe experimental work in this paper is extensive and the consistently superior results of the OmniDNA models are promising. The novel Seqpack algorithm is a nice contribution and the ablations on mult-task training and the positional embeddings are useful for practitioners in the field. The breadth of baselines is well curated and the experimental details provided to reproduce the experiments are well organized and extensive. The natural language interaction tasks are novel and interesting.\n\n### **Weaknesses**\n\n1. I disagree with the framing of several of the limitations and contributions.\n    - Specifically, claiming that there has not been exploration of the design space (Limitation 1) seems inaccurate to me. Several models have explored the effect of tokenization (e.g., DNABERT 1 vs 2, and the base pair tokenization of Hyena and Caduceus) and have employed different positional embedding strategies (e.g., NTv1 vs NTv2). Additionally, Caduceus, in particular, created custom extensions of Mamba-based modules specifically for genomic tasks. The authors claim that rather than the “plug-and-play” of existing NLP modules from existing GFMs, they optimize their backbone for genomics. However, I think one could argue that they performed a search over the same modules that other GFMs have employed. I believe that a reframing would be more accurate, improve readability, and would by no means detract from the extensive experiments and useful results of Finding 1 in the paper.\n    - In a similar vein, I disagreed somewhat with the motivation of limitation 2, in the sense that arguing for the need of multi-task training from the lens of expensive storing of weights is not reflective of the reality of how these models will be used or the true bottlenecks that practitioners face with the current scale of GFMs. Here too, more accurately situating the experiments and results of Finding 3 in the rich literature of multi-task training (even in the context of biological models, e.g. Enformer) would be better, in my opinion. This by no means takes away from the fact that exploring multi-task downstream fine-tuning of GFMs is indeed a contribution of this work.\n    - Lastly for limitation 4, the authors state that previous GFMs cannot adapt to varying context lengths, but this claim needs to be supported either with references from the literature or by pointing to specific experimental results from this work that support the claim.\n2. The takeaways from Finding 2 are unclear to me. Several works have already indicated that the NT tasks are in fact not ideal for understanding GFMs (e.g., Kirill, et al [1]). Additionally, it is possible that these tasks are too “easy” and can be saturated. Lastly, unless I am misreading it, I think Figure 3 does in fact show an important correlation between pre-training loss and downstream performance.\n3. I think the authors could include a more robust discussion about what is making OmniDNA work. For example:\n    - Is data curation an important factor?\n    - Is the fact that the authors are using a transformer-based AR model vs BERT-style bi-directional one important (e.g., NTv2 and DNABERT are transformers but not AR, Hyena is AR but not a transformer; so is the claim of OmniDNA that the correct recipe is AR + transformer)?\n    - Is the combination of positional embedding + layer norm a key driver? Specifically, these design decisions are already explored / used in previous models (e.g. NTv2 uses RoPE).\n4. An ablation exploring the efficacy of SeqPack vs existing context extension mechanisms or even the naive approach of simply providing pre-trained models with longer sequences would be useful and convincing. Additionally, exploring the hyperparameters of SeqPack would be informative\n5. Additionally, the computational overhead of SeqPack is not well described or quantified. Adding this would be useful.\n6. Some minor typos:\n    - Line 174: “Understand the…” should be “To understand the…”?\n    - Line 235: “strictly output” should be “strictly outperform”?\n\n\n---\n\n[1] Vishniakov, Kirill, et al. \"Genomic foundationless models: Pretraining does not promise performance.\" bioRxiv (2024): 2024-12.\n\nQUESTIONS:\n1. The authors discuss several approaches for extending pre-training vocabularies (Lines 142-149). Which approach worked best / is used in the experiments?\n2. The motivation for SeqPack is that $O(N \\log K)$ is too expensive. Ultimately SeqPack costs $\\Theta(N)$, is the savings here indeed that significant? I would have thought the $\\log K$ factor would not be that untenable?\n3. In Section 3.3.1, the gumbel softmax trick is used to maintain differentiability, but it is then composed with top-k selection. Doesn’t applying top-k destroy differentiability?\n4. Can the authors confirm that all the models from Section 4.2 (Table 4) have 450kbp input sequences. Additionally, would it be possible to provide error bars for Table 4?\n5. What is the positive class ratio of the final datasets used in the experiments from Section 4.3.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Omni-DNA, a new autoregressive DNA language model that includes a novel method of long context compression. The model is trained on genome assemblies from NCBI. It is fine-tuned by adding additional tokens or multi-head fine-tuning. It is benchmarked on Gene regulatory element classification, Pathogenic variant effect prediction, Enhancer–target gene interaction mapping, and Sequence-conditioned functional interpretation generation.\n\nSTRENGTHS AND WEAKNESSES:\nThe model achieves SOTA performance on classification tasks which is impressive. The task of generating textual descriptions of DNA sequences also appears very useful and original, adding a novel capability to the current foundation models in the field. However, the remaining evaluations are confusing and could use some clarification or editing (see below).\n\nQUESTIONS:\n1. For enhancer-target prediction, did you mask the intervening bases between the enhancer and the promoter as in (https://openreview.net/forum?id=opv67PpqLS)? If so, how was this masking accomplished?\n\n2. I am very confused by the genetic disease variant-type classification. Conventionally, models are evaluated by their ability to classify disease-causing variants from negative control variants, which have no phenotypic effect. The assumption is that the models should be able to identify variants that change protein sequence or alter conserved regulatory sequences, vs. variants that have no impact. Here, the study seems to have taken the surprising route of classifying variants that cause one disease from variants that cause a different disease. All of these variants are pathogenic, so what is the model classifying exactly? My guess is that it is learning the genes near the variants, and has learned that some genes are associated with a particular disease - in which case it is not really evaluating whether the model has learned properties of variants. Please clarify the purpose of this task or draw the negative controls from non-pathogenic variants.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript introduces the Omni-DNA. a family of models spanning 20M to 1.1B parameters that supports sequence understanding, long-context genomic reasoning, and natural-language annotation.\n\nOmni-DNA is a decoder only structure for modelling genetic information. The main contribution of this manuscript lie in three aspects: \n1. Multi-Task Learning to improve performance. \n2. Vocabulary expansion to allow model directly generate natural functional descriptions of given DNA sequences. \n3. SeqPack to combine multiple nucleotide to a single token to allow model operates with long sequences.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe authors have developed models in different sizes to evaluate the effectiveness of the proposed method.\n\nThe proposed method excels at the chromatin tasks in the NT Benchmark, highlighting the effectiveness of the multi-task learning. \n\n## Weakness\n\n### Technical soundness of the proposed method. \n\nMLM and NTP represents the two paradigm of self-supervised learning: discriminative and generative. \nNo paradigm is superior than another. \nFor example, Vision Transformer have been used in CV for years, but no one uses autoregressive model for extracting visual features. This is simply because of the nature of the data. \nNature language is Turing-complete and scalable. Unfortunately, genome is not. \n\nPeople write new sentences everyday, like what the authors have done, what I am currently doing, and what comment the AC will give. They are easily accessible on the internet. However, the human genome is static. While mutation happens all the time, the technology we have cannot capture them, and we are unable to put them into database. From this point of view, genome is more like pictures. \n\n### Losing nucleotide level precision\n\nThe SeqPack can effectively reduce the number of tokens in a sequence. But this is at the cost of losing single nucleotide resolution -- which is crucial for many gnomic tasks. \n\n### Unsatisfactory results on cCRE and splicing\n\nThe results on cis-regulatory elements and splicing sites is rather unsatisfactory. \n\n### Unclear writing \n\nThe authors mention that it is possible to use LM head for multi-task learning. \n\nWhat heads does the proposed method use in this paper? Have the authors evaluated the performance of different heads?\n\nQUESTIONS:\nThe results of HyenaDNA is significantly lower than the values reported in the original paper. Please explain.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a genomic foundation model (GFM), Omni-DNA, architecture that seeks to improve upon existing GFMs by expanding the vocabulary and extending the context via the SeqPack operator. Omni-DNA is also fine-tuned via a multi-task approach to leverage shared knowledge across tasks.\n\nThe authors first establish the optimal Omni-DNA architecture through ablation studies on the performance of the pre-trained model with various types of normalization layers, positional encoding and parameter budget. Subsequently, multi-task learning is enabled by adding extra tokens to the vocabulary and SeqPack is used to compress long contexts. The latter two innovations are incorporated during the post-training phase of Omni-DNA training.\n\nOmni-DNA is evaluated against existing GFMs on benchmarks from the Nucleotide Transformer paper, and the Genomic  Benchmark. Additional experiments on the task of enhancer-target prediction and genetic disease variant-type classification demonstrate the long-range capability of Omni-DNA. Finally, the task of sequence-to-function generation is shown as an example of the capacity of Omni-DNA to support textual outputs.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper addresses a well-established shortcoming of several existing GFMs in that they are unable to support long context windows due to the poor scaling of attention. In addition to that, the idea of jointly fine-tuning on multiple tasks is an intriguing one as it allows for the transfer of knowledge across related tasks; it is less clear if this benefit would be realized if the selected tasks are known to arise from entirely separate biological phenomena/pathways.\n\nThe use of ablation studies to justify the design choices made in Omni-DNA is also appreciated, as it enables the reader to appreciate the reasoning used to arrive at the proposed GFM. The use of well-established genomic benchmarks during the evaluation of Omni-DNA also serves as a fair comparison. However, it isn’t clear if the increased performance of Omni-DNA models stems from the proposed innovations or the significantly higher parameter counts. \n\nHowever, I see several issues with some of the claims made in this paper. These are detailed below:\n\n1.\tIn Section 1, the authors claim that “existing works usually adopt the backbone of language models”. While this was true of early GFMs, newer GFMs have innovated on these early models, in ways reminiscent of the improvements outlined in this work.\n2.\tOn line 39, the authors state that “the state-of-the-art models on short context cannot do long-context tasks.” It is unclear if they are referring to which models they are referring to here, as several models such as Mamba and HyenaDNA, are capable of handling very long context lengths.\n3.\tOn lines 63-65, it is stated that most GFMs are restricted to ternary alphabets. This is incorrect as it ignores tokenization, which is a key area of development in GFMs.\n4.\tOn line 75, the authors motivate their use of SeqPack due to the whole genome containing long “junk” regions. I would be wary of resorting to such a description as non-coding regions of the DNA have been shown to contain information, such as regulatory elements and useful motifs. In fact, this might suggest that such an approach renders Omni-DNA to be less useful for tasks that deal with non-coding regions of genomic sequences.\n\nAdditionally, the writing of the paper has several issues which should be addressed.\n\n1.\tWhile I appreciate the inclusion of the ablation studies in Section 2.2, the introduction and subsequent discussion of those studies comes across as abrupt, instead of being used to motivate the design choices made in Omni-DNA.\n2.\tSection 3.3.1 is difficult to follow.\n3.\tWhile it is admirable that Omni-DNA is capable of mapping sequences to functions (Section 4.4), this does lend weight to the argument that Omni-DNA is a more capable GFM than existing models. Moreover, the baseline there is an NLP model (GPPT-4o), which one would not expect to do well on this task anyway.\n\nQUESTIONS:\n**Questions:**\n\n1.\tSeqPack compresses a sequence of length K into a much smaller embedding of length L. Does this lead to a loss of resolution, thereby limiting OmniDNA’s sensitivity to very short sequence tasks, such as those related to single nucleotide polymorphisms (SNPs)?\n2.\tIn Table 3, Caduceus-PH seems to perform comparably to Omi-DNA despite having 300x fewer parameters. Could the additional performance shown by Omni-DNA simply be a result of the much larger parameter space it explores?\n3.\tWhat do you mean by “output all tasks” in line 235?\n4.\tWhen conducting multi-task training, does the choice of tasks impact the benefit one could derive from this approach? Intuitively, one would expect more performance gain if related tasks are learned together.\n\n**Suggestions:**\n\nThere are several typos that should be corrected.\n1.\tLine 14: “outperform” -> “outperforms”\n2.\tLine 84: “Omni-DNA are” -> “Omni-DNA is”\n3.\tLine 85: “data are” -> “data is”\n4.\tLine 211: “HynnaDNA”  “HyenaDNA”\n\nAdditionally, I recommend moving the discussion of empirical insights to after the results section. In its current form, there is little connection between the empirical insights and any of the ensuring discussion.\n\nLIMITATIONS:\nThe authors have not addressed the limitations or potential negative societal impact of their work. In isolation, I do not there are clear negative social impacts arising from this work, but if the downstream tasks are of a sensitive nature, concerns regarding malicious use and user privacy naturally follow. This is especially true due to the multi-task paradigm, where private information from the dataset for one task may “leak” into the embeddings used to perform another task.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Omni-DNA, a family of decoder-only transformer models (from 20M to 1.1B parameters) designed to be a comprehensive Genomic Foundation Model (GFM). The authors aim to address four key limitations in existing GFMs: limited architecture exploration, a single-task fine-tuning paradigm, a lack of multi-modal (textual annotation) capabilities, and the inability for a single model to excel at both short- and long-context tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Pros\n- The paper tackles several important challenges in genomic modeling simultaneously, from architecture design to long-context reasoning and multi-modality. The evaluation is extensive, covering 26 tasks from the Nucleotide Transformer and Genomic Benchmarks, in addition to long-context and generation tasks.\n\n- The work makes a concerted effort to move beyond the standard \"pre-train, fine-tune per task\" paradigm. The demonstration of successful multi-task fine-tuning and the introduction of the SEQ2FUNC dataset for sequence-to-text generation are significant steps toward more versatile and useful genomic models.\n\n## Cons\n- The architecture ablation (RoPE vs. ALiBi, RMSNorm vs. LayerNorm) explores known components, and the conclusion that \"ROPE + Bias-Free Layernorm delivers the best trade-off\"  is an empirical finding for this domain, not a fundamental architectural innovation.\n\n- The multi-task fine-tuning approaches (multi-head and instruction-style next-token prediction) are standard techniques in NLP. While their application to genomics is valuable, it is not a novel methodological contribution.\n\n- The core idea of SEQPACK—learning to compress a sequence by selecting important tokens—is conceptually similar to various token pruning, sequence compression, and \"attention sink\" aware methods in NLP literature. The paper does not adequately compare SEQPACK to these related works, making its specific novelty hard to assess.\n\n- The paper presents Omni-DNA as a holistic framework, which makes it difficult to disentangle the source of performance gains. For instance, in the benchmarks in Table 2 and Table 3, is the superior performance due to the optimized backbone, the larger pre-training dataset (300 billion nucleotides ), the specific pre-training recipe, or a combination? The paper lacks clear ablation studies that would isolate the impact of each \"targeted improvement.\"\n\n- The SEQ2FUNC dataset is a cornerstone of the paper's multi-modal claims, but its creation process is concerning. The dataset is generated by prompting an LLM to \"generate refined, natural-language descriptions\" from coarse annotations. The quality control relies on two LLM passes agreeing and rejecting \"low-confidence cases\".\n\nQUESTIONS:\nSee Cons.\n\nLIMITATIONS:\nThe authors briefly mention that their architecture exploration could be costly and that reinforcement learning could be a future direction. However, they do not adequately address the limitations of the current work. The primary limitation is the immense complexity and cost of the entire framework, which makes it very difficult for other researchers to reproduce or build upon. Furthermore, the paper does not discuss the potential for cascading errors: errors in the LLM-generated SEQ2FUNC dataset could lead to a model that learns to generate incorrect functional annotations. The reliance on LLMs for both dataset creation and evaluation is a significant methodological limitation that is not fully acknowledged.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}