Title: Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis
Venue: NeurIPS 2025
Accepted: yes
Keywords: DNA Language Models

Markovian Regularization
Abstract: Transformers have revolutionized nucleotide sequence analysis, yet capturing long‑range dependencies remains challenging. Recent studies show that autoregressive transformers often exhibit Markovian behavior by relying on fixed-length context windows for next-token prediction. However, standard self-attention mechanisms are computationally inefficient for long sequences due to their quadratic complexity and do not explicitly enforce global transition consistency.

We introduce CARMANIA (Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis), a self-supervised pretraining framework that augments next-token (NT) prediction with a transition-matrix (TM) loss. The TM loss aligns predicted token transitions with empirically derived n-gram statistics from each input sequence, encouraging the model to capture higher-order dependencies beyond local context. This integration enables CARMANIA to learn organism-specific sequence structures that reflect both evolutionary constraints and functional organization.

We evaluate  CARMANIA across diverse genomic tasks, including regulatory element prediction, functional gene classification, taxonomic inference, antimicrobial resistance detection, and biosynthetic gene cluster classification. CARMANIA outperforms the previous best long-context model by at least 7\%, matches state-of-the-art on shorter sequences (exceeding prior results on 20/40 tasks while running $\sim$2.5$\times$ faster), and shows particularly strong improvements on enhancer and housekeeping gene classification tasks—including up to a 34\% absolute gain in Matthews correlation coefficient (MCC) for enhancer prediction. The TM loss boosts accuracy in 33 of 40 tasks, especially where local motifs or regulatory patterns drive prediction. This enables more effective modeling of sequence-dependent biological features while maintaining robustness across non-coding and low-signal regions. Code available at https://github.com/EESI/carmania.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a self-supervised pre-training framework, CARMANIA (Context-Aware Regularization and Markov Integration for Attention-Based Nucleotide Analysis), aimed at improving nucleotide sequence modeling, especially for genomic tasks. The approach combines sliding-window attention with a novel transition matrix (TM) loss to capture global dependencies in nucleotide sequences, overcoming the limitations of traditional attention mechanisms in long sequence modeling. By combining Markov transition dynamics, CARMANIA helps the model learn organism-specific sequence structures and improves prediction performance for a variety of genomic tasks, such as regulatory element prediction, antimicrobial resistance detection, and biosynthetic gene cluster classification.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n\n- Originality: This paper introduces CARMANIA, a self-supervised pretraining framework that combines sliding-window attention with a novel Transition Matrix (TM) loss to enhance the modeling of global dependencies in nucleotide sequences.\n- Significance: CARMANIA tackles challenges in genomic sequence analysis, particularly with long sequences, and enhances the model's predictive power for tasks like gene function prediction and detecting antimicrobial resistance.\n- Performance Improvement: Through experiments, the model shows a notable increase in prediction accuracy across various genomic tasks, with a 34% improvement in the Matthews correlation coefficient (MCC) for regulatory element prediction.\n- Efficiency: The method is computationally efficient, reducing both training and inference times for long sequences. This results in faster processing while maintaining improved performance over other models.\n\n### Weaknesses:\n\n- Technical Complexity: The combination of Transition Matrix loss (TM Loss) and sliding-window attention could be difficult for readers who aren't familiar with advanced methods in genomics. More explanation on how these components work together could help clarify the approach.\n- Limited Experimentation: While CARMANIA shows promising results on several benchmark tasks, the paper does not include validation from real-world experiments, which would be important for confirming its practical value in biological research.\n- Model Size: Despite its strong performance, CARMANIA requires more memory and computational resources compared to some other models, especially when working with large-scale genomic data.\n\nQUESTIONS:\n### Questions:\n\n- Transferability of TM Loss: While the paper focuses on genomic data, could the TM loss approach be applied to other types of biological sequences like protein?\n- Handling Rare Genomic Events: Given that genomic data often includes rare events or underrepresented patterns, how well does CARMANIA handle these rare cases, particularly in low-signal regions of the genome?\n\nLIMITATIONS:\n## Limitations:\n\n- Computational Cost: The paper mentions computational overhead, particularly related to the calculation of the Transition Matrix loss, which could become a bottleneck when scaling to large datasets. While the use of sliding-window attention improves efficiency, there may still be limitations on large-scale applications.\n- Lack of Real-World Validation: Although CARMANIA performs well in experiments, it has not been validated in actual biological experiments.\n- Task-Specific Tuning: The paper discusses the β hyperparameter (used to adjust the weight of the TM loss), which works well for general tasks. However, tasks involving rare, biologically significant motifs may require more specific tuning. The paper does not go into detail on how to fine-tune for such cases.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

comments: SUMMARY:
This paper proposes a novel loss function that leverages a transition matrix to better supervise the model. Additionally, it introduces Sliding Window Attention to the transformer architecture, which not only incorporates inductive bias into the model but also theoretically reduces computational complexity. Compared to previous approaches, the proposed method demonstrates performance improvements on certain downstream tasks.

STRENGTHS AND WEAKNESSES:
Strengths:​​ From a biological perspective, the transition-matrix approach is theoretically sound. Additionally, the inductive bias introduced by Sliding Window Attention has been empirically validated to some extent by previous work ConvNova [1].

​Weaknesses:​​ The paper lacks comparison with recent methods like ConvNova [1]. Moreover, the results appear relatively weak compared to existing approaches. I suggest the authors consider further improving their implementation to enhance model performance, potentially by incorporating TM Loss into current methods. It would be more convincing to conduct experiments on established high-performance baselines such as Caduceus or ConvNova - significant performance improvements on these platforms would substantially strengthen the paper's claims.

[1] REVISITING CONVOLUTION ARCHITECTURE IN THE REALM OF DNA FOUNDATION MODELS. ICLR 2025

QUESTIONS:
None

LIMITATIONS:
The experimental results in this paper appear generally weak, failing to surpass state-of-the-art (SoTA) performance on a considerable number of tasks. Additionally, there is no experimental evidence demonstrating that the proposed TM Loss can actually improve the performance of current SoTA methods. The study also lacks comparative experiments with ConvNova.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript presents CARMANIA, a self-supervised pre-training framework for genomic foundation models. \n\nThe core motivation of this work is to address the challenge that standard autoregressive transformers are limited in capturing long-range dependencies in genomics, as their fixed-size context windows cause them to exhibit local Markovian behavior. To overcome this, the paper introduces a novel Transition Matrix (TM) loss as a global regularizer for pre-training. This TM loss forces the model to learn the empirical n-gram statistics of the entire input sequence, pushing it to capture organism-specific structures and dependencies that extend far beyond its immediate context window.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n### Novelty\n\nThe paper's core contribution, the TM loss, is an original and intuitive idea. It elegantly integrates a classic concept from bioinformatics (Markov models and n-gram statistics) as a regularization prior into a modern Transformer architecture. This approach provides an explicit mechanism to guide the model toward learning global sequence properties, directly addressing the well-known difficulty of capturing long-range dependencies in genomics.\n\n## Weaknesses\n\n### Questionable Efficacy of TM Loss on NT Benchmark\n\nWhile the authors are to be commended for conducting extensive experiments, the effectiveness of the TM loss is not consistently demonstrated across all benchmarks. On the 18 Nucleotide Transformer tasks, the performance difference between CARMANIA and the baseline without TM loss is often marginal or even negative.\n\nFor instance, the model with TM loss performs worse on the H3K4me3 histone modification task ($0.565$ vs. $0.586$). While it performs better on Splice Site Donor prediction ($0.973$ vs. $0.921$), many other tasks show negligible differences. The most significant outlier is the H4ac histone acetylation task, where adding the TM loss leads to a dramatic MCC improvement from $0.197$ to $0.606$. While the paper offers a brief explanation, this is not entirely convincing, as other methods in the comparison do not exhibit similar performance volatility on this task. This substantial gain warrants a more detailed analysis to clarify the underlying mechanism and ensure it is not an artifact. Overall, these results from a major benchmark seem to conflict with the paper's broader claim that the TM loss boosts accuracy in 33 of 40 tasks and weaken the argument for its general applicability.\n\n### Clarity and Reporting of Experimental Details\n\nThe paper's clarity suffers from incomplete reporting of experimental details, making some results difficult to interpret fully. Table 1, which presents a key ablation study, is a primary example of this issue.\n\n- **Ambiguous Metrics**: The table fails to specify which datasets or tasks were used to compute the \"F1 Macro\" and \"BLEU\" scores. The reader must search the main text to infer that these likely correspond to an Antimicrobial Resistance (AMR) task and a human genome benchmark, respectively. This context is essential and should be included in the table's caption.\n- **Relative vs. Absolute Cost**: The table reports \"Relative FLOPs,\" which only shows the proportional cost increase of full attention. It does not provide the absolute FLOPs of the proposed method, making it impossible to gauge the actual computational budget.\n\nQUESTIONS:\nThe proposed model is several times larger than the baselines (Caduceus and HyenaDNA), what is the FLOPS of the proposed method?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}