Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis
Authors: Mohammadsaleh Refahi
Mahdi Abavisani
Bahrad A. Sokhansanj
James R. Brown
Gail Rosen
Venue: NeurIPS 2025 poster ()
Keywords: DNA Language Models
Markovian Regularization
Primary area: machine_learning_for_sciences
TL;DR: We propose CARMANIA, a context-aware regularization method for nucleotide sequence analysis that enforces Markovian properties in Transformers, improving long-range dependencies, convergence speed, and out-of-domain generalization.

Abstract: Transformers have revolutionized nucleotide sequence analysis, yet capturing long‑range dependencies remains challenging. Recent studies show that autoregressive transformers often exhibit Markovian behavior by relying on fixed-length context windows for next-token prediction. However, standard self-attention mechanisms are computationally inefficient for long sequences due to their quadratic complexity and do not explicitly enforce global transition consistency.

We introduce CARMANIA (Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis), a self-supervised pretraining framework that augments next-token (NT) prediction with a transition-matrix (TM) loss. The TM loss aligns predicted token transitions with empirically derived n-gram statistics from each input sequence, encouraging the model to capture higher-order dependencies beyond local context. This integration enables CARMANIA to learn organism-specific sequence structures that reflect both evolutionary constraints and functional organization.

We evaluate  CARMANIA across diverse genomic tasks, including regulatory element prediction, functional gene classification, taxonomic inference, antimicrobial resistance detection, and biosynthetic gene cluster classification. CARMANIA outperforms the previous best long-context model by at least 7\%, matches state-of-the-art on shorter sequences (exceeding prior results on 20/40 tasks while running $\sim$2.5$\times$ faster), and shows particularly strong improvements on enhancer and housekeeping gene classification tasks—including up to a 34\% absolute gain in Matthews correlation coefficient (MCC) for enhancer prediction. The TM loss boosts accuracy in 33 of 40 tasks, especially where local motifs or regulatory patterns drive prediction. This enables more effective modeling of sequence-dependent biological features while maintaining robustness across non-coding and low-signal regions. Code available at https://github.com/EESI/carmania.

Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis
Mohammadsaleh Refahi
Mahdi Abavisani Dataminr New York, NY
Bahrad A. Sokhansanj Drexel University Philadelphia, PA
James Brown Drexel University Philadelphia, PA
Drexel University Philadelphia, PA
Gail Rosen
Drexel University Philadelphia, PA
Abstract
Transformers have revolutionized nucleotide sequence analysis, yet capturing long-range dependencies remains challenging. Recent studies show that autore- gressive transformers often exhibit Markovian behavior by relying on fixed-length context windows for next-token prediction. However, standard self-attention mech- anisms are computationally inefficient for long sequences due their quadratic complexity and do not explicitly enforce global transition consistency.
We introduce CARMANIA (Context-Aware Regularization with Markovian In- tegration for Attention-Base Nucleotide Analysis), self-supervised pretraining framework that augments next-token (NT) prediction with a transition-matri. (TM) loss. The TM loss aligns predicted token transitions with empirically derived n- gram statistics from each input sequence, encouraging the model to capture higher- order dependencies beyond local context. This integration enables CARMANIA learn organism-speeife sequence structures that reflect both evolutionary con- straints and functional organization.
We evaluate CARMANIA across diverse genomic tasks, including regulatory ele- ment prediction, functional gene classification, taxonomic inference, antimicrobial resistance detection, and biosynthetic gene cluster classification. CARMANIA outperforms the previous best long-context model by at least 7%, matches state-of- the-art on shorter sequences (exceeding prior results on 20/40 tasks while running ~2.5 faster), and shows particularly strong improvements keeping gene classification tasks masks-including up to a 34% absolute gain in Matthews correlation coefficient (MCC) for enhancer prediction. The TM loss boosts accu- racy in 33 of 40 tasks, especially where local motifs or regulatory patterns drive effective modeling of sequence-dependent biological
enhancer and house-
prediction. This enables features while maintaining robustness across non-coding and low-signal regions. Code available at battps://gitaii com/
Introduction
Deoxyribonucleic acid (DNA), often referred to as the "language life."" encodes the genetic instruc- tions essential for the development, functioning, and reproduction of all known living organisms and many viruses. The sequence of four nucleotide bases-adening (A), thymine (T), cytosine (C), and guanine (G)-form the foundation of genetic information, dictating the synthesis of proteins and the regulation various biological processes. Advancements in high-throughput sequencing technologies have exponentially increased genomic data availability.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite advances in sequencing, analyzing large-scale biological data remains challenging. Tra- ditional
as motif detection, sequence alignment, and Markov models-cappe
short-range dependencies well Delcher et al. [007]] Steinegger and Söding [2017]. Wood et al. [2019]. but often fail to represent complex or long-range patterns, especially in diverse or non- canonical sequences. Recent work has applied NLP techniques to biological sequences Nguyen et al. [024], Cahyawijjya et al. 2022], Schiff et al. [024], Bo et al. [2025), with large language models (LLMs) showing strong ability to capture complex dependencies. However, genomic sequences are often extremely long, making full attention computationally infeasible. To address this, transformer models use mechanisms like sliding window and sparse attention Dong et al. [2025] Zaheer et al.
2020]. which expand the receptive field but may still struggle to capture long- raaggraage dependencies effectively in genomies.
In this work, we enhance transformers by incorporating Markovian priors, guiding them to follow long-range transition dynamics and better capture complex dependencies within the broader context of genomic sequences. Drawing on recent studies that interpret transformers' capabilities through the lens of Markovian processes Hu et al. [2024). Zekri et al. [2024], we propose novel pretraining strategy that explicitly enforces these Markovian properties.
Our approach, named CARMANIA (Context-Aware Regularization with Markovian Integration for Attention- Based Nucleotide Analysis), supplements the standard next-token prediction objective with an auxiliary loss. This loss aligns the model's 's predicted first-order transition matrix with one derived from the full input sequence, encouraging consistency with its empirical token transition patterns. By doing so, CARMANIA captures the probabilistic structure of genomic sequences and improves context-awareness during next-token prediction.
To scale pretraining to long genomic sequences, we replace full attention with sliding-winde attention (SWA), sparse mechanism that limits each token's 's receptive field to fixed-size window of preceding tokens, reducing complexity from O(n²) to (kk) (A: < n) while preserving biologically meaningful local interactions. TM loss complements this by reinforeing short-range dependencies within the local window while promoting global consistency, since the transition matrix is computed over the entire input sequence. Together, these components enable balance between local motif recognition and broader statistical aligmment. In practice, CARMANIA handles effective contexts up to 160 kbp. which makes it, to our knowledge, the longest-context transformer-based genomic language model to date. An overview of our framework is illustrated in Figure
In addition improving context modeling, pretraining approach supports efficient domain adaptation, as conserved transition patterns across species enable generalization with minimal fine- tuning Bergeron al. [2023).
The main contributions of this work are:
A scalable architecture for long-context genomic modeling that integrates sliding-wing wwindoww attention, RoPE-based positional encoding, FlashAttention, and a widened Transformer backbone, enabling efficient representation learning over extended DNA sequences.
A novel self-supervised objective based on Transition Matrix (TM) loss, which regularizes the transformer model by aligning predicted token transitions with empirical n-gram distributions. This Markovian prior enhances memory retention, out-of-distribution generalization, and biological coherence across long genomic contexts.
Comprehensive evaluation across 40 genomic tasks, including regulatory element prediction, functional gene classification, taxonomic inference, and biosynthetic gene cluster(BGC) prediction, where our model achieves consistent improvements- in benchmarks where alternative inductive biases have previously dominated.
Related Work
2.1 Markovian Models in Genomics
Hidden Markov Models (HMMs) are widely used in genomics for modeling the probabilistic structure of biological sequences. They enable gene prediction, in GeneMark Lukashin and Borodovsky

---

Decoder Block
Embaddiing Layar
Next Token Prediction
Trannition Matri Prodectory
RMS
F
Neaked Grouped Querry Attertion
Pastiional Encoding (RoPE)
630
616
63
62
64
63
Transformer
RMS
Feod forward
Figure I: Proposed Pretraining Framework. We extend LLaMA-style decoder with transition matrix module to capture global nucleotide co-occurrence. The model uses diding-window attention with rotary embeddings and local caching, reducing attention complexity from O O(n). The transition matrix complements local attention by preserving long-range dependencies efficiently.
1998]. and other methods for identifying coding regions Delcher et al. [2007]. Profile HMMs are commonly applied to protein family analysis, capturing conserved regions and domain variability Pot ter et al. 2018). HMMs also power microbial classification tools like Clasnip Chuan et al. [2023]. and Aphmm Firtina et al. (2024) improves the speed and efficiency of profile HMM computations.
Their effectiveness depends on model structure, sequence length, and the biological complexity of the task.
2.2 Language Models in Genomies
The first breakthrough in genomic language models came with DNABERT et al.
[2021] which
employed BERT architecture with k-mer tokenization and sequence masking learn DNA sequence representations. The Nucleotide Transformer DaaiomeerDalla-aee Dalla- al. (2024) scaled this concept, training 2.5 billion parameter model on 850 genomes. Other models like HyenaDNA Nguyen al. [024] used long convolutional blocks for efficient long-sequence handling, while BigBird Refahi al. 2025 2023] applied sparse and sliding window attention to capture long-range dependencies. Caduceus Schiff et al. [2024), built Mamba, introduced bi-directional model preserving reverse complement symmetry.
Together, these advances in tokenization, architecture, and representation learning have led to increasingly effective models for genomic analysis Cahyawijaya et al. [2022). Celikkanat al.) 2024
2.3 The Markovian Core of Autoregressive Transformers
Hu al. 2024) aimed establishing an equivalence between autoregressive language models based on transformers and Markov chains. Specifically, findings emphasize the role of statistical induction heads, which adaptively learn in-context patterns, forming a foundation for next-token predictions. This perspective introduces a compelling connection between autoregressive architectures and Markov chains, paving the way a nuanced understanding of in-context learning (ICL) Zekri al. [2024), Edelman et al. [2024). Zekri et al. (2024) demonstrates that an autoregressive model with vocabulary size and a context window of size K can be represented as Markov chain with a state space of size K).
There has been significant research Edelman et al. 2024 Zekri et al. 2024 Ildiz et al. 2024
In an autoregressive model, the probability of predicting the next token xz-1 given the previous K tokens (Zz, Tt-1,...... It-K+1) can be written as:
Pr(ze41 Ft, 21-1.... ..... 3t-K+t K+1).
(I)
3

---

This conditional probability can be interpreted as a transition probability in a K-th order Markoy chain, where each unique K K-length token sequence defines a state. The transition matrix Q E R?xT is defined as:
Qe.e Pr(æ++1 = Za 2t =z,,,
(2)
where and s' index sequences of K tokens. The rows of Q sum to satisfying the Markov property. Thus, the model's behavior approximates a Markov chain with T states, capturing dependencies within the fixed context window. Although self-attention is globally contextual and non-Markovian in theory. limited context in practice causes autoregressive Transformers to approximate Markovian behavior Ildiz et al. [2024].
3 Pretraining Long Sequences With Markovian Knowledge
We adopt causal Transformer inspired by LL L.L.AA al. 2023 where each layer computes hidden representations h from input tokens (#1, Zn) and the previous layer's outputs h(--1,, using multi-head self-attention predict the next token. To scale to long sequences, we replace full self-attention with SWA, where each token attends only to a fixed-size & window of preceding tokens, Tt-k...... It-1, reducing complexity from O(n²) to O(kn). This not only improves efficiency but also reflects the local nature most biological dependencies in genomic sequences. We further improve efficiency with FlashAttention Dao (2023) and enhance positional encoding via Rotary Positional Embeddings (RoPE) Su et al. [024]. enabling better extrapolation to long contexts.
3.1 Unified n-th Order Transition Tensor Formulation
To complement the next-token prediction objective, we introduce a unified probabilistic framework that captures higher-order dependencies through a transition tensor. Let P. E R.V denote the model's predicted probability distribution over the vocabulary V at position f, where [Pr]; P(z, = x<t.)
We define the n-th order transition tensor T(x) E IR V
as:
L-n+lnm R-1
((m) Lr
(3)
B(L +
==0
where B is the batch size, is the sequence length, V is the vocabulary size, and i,,..., in index token values the n-gram.
This formulation approximates the joint distribution over n consecutive model predictions, providing global summary of sequential dynamics. In practice. we focus on the first-order case (n 2), which corresponds to modeling bigram transitions. This strikes a balance between expressiveness and computational efficiency, and directly supports the TM loss introduced in the next section.
3.2 Pretraining Objective
Our pretraining objective combines two complementary components: next-token prediction and transition matrix alignment. The primary objective is the standard autoregressive language modeling loss, which minimizes the negative log-likelihood of predicting each token 2; given its preceding context:
Cxt = Σ log Po(w; Ii,..., xt-1);
where Pe denotes the model's predicted token distribution.
To reinforce global sequence structure, we introduce Transition Matrix (TM) loss that encourages the model to match the empirical token transition dynamics. Let Pij and q, denote the smoothed n-gram distributions of the input and model outputs, respectively. The KL divergence between these distributions defines the TM loss:
LTM Σ
Pij log
Pij K3
(5)

---

Loaa
TH Lowx
3.-
0.4
1400 50030
18000
20230
26093
50033
8.00 10000
19400
20000
34000
12000
Figure 2: Comparison of training losses with and without explicit TM loss enforcement (3 = vs.
3= 0). Left: Both models show similar reductions in next-token prediction loss. Right: Without TM
loss (3 = 0), the model partially learns transition structure- TM loss rises then Lubilizes-indicatingg
implicit alignment with n-gram patterns.
Table 1: Ablation study showing the impact of TM loss and attention type on model performance
dataset. F1 Macro is evaluated on the Antimicrobial Resistance (AMR) classification tasks, while BLEU and Relative FLOPS are computed on the Human Genome-Shov Short dataset using 10,000 fragments from the GRCh38/hg38 reference genome. Relative FLOPs are reported with respect to the Sliding Window attention baseline, which has an absolute compute cost of approximately 16.7 x 10 FLOPs per sequence.
(FI Macro, BLEUU) and compute cost. All models are trained on the Scorpio-Gene-
Model Setting
F1 Maero (f)
BLEU (T)
Relative FLOPs (L)
Transformer (SWA)
0.873 + 0.130
0.73
1.00
TM Loss
0.882 # 0.131
0.77
1.00
+ Full Attention + TM Loss
0.883 + 0.131
0.71
1.58
The full loss combines both terms:
Crul
LNT + BLLT,
(6)
where B is tunable hyperparameter controlling the influence of the TM loss.
4 Experiments
In this section, we evaluate CARMANIA across a range of genomic tasks. We first describe the pretraining datasets, model architecture, and training setup. followed by ablation studies assessing the impact of the TM loss and attention mechanism on convergence, efficiency, and long-range reten- tion. Finally, present comprehensive benchmark results demonstrating the model's adaptability, generalization, and superior performance over existing methods.
4.1 Experimental Setup
Pre-training Datasets
We pre-train our model on three large-scale genomic datasets: (i) GRCh38
G GRCh38 2013 which provides ~3B base pairs with 10 kbp and 160 kbp fragments; (ii) the Basic Genome Dataset Zhu et al. [2022), consisting of ~ 10B base pairs across 4,600+ genomes with 10 kbp fragments; and (iii) the Scorpio Gene-Taxe Dataset Refahi et al. [2025), comprising ~550M base pairs from 2,046 genomes with 4 kbp fragments. Additional details are Appendix A.
Model and Training Details
Architecture: CARMANIA is a LLaMA- causal Transformer tailored for genomic sequence modeling. It uses 16 attention heads (4 key-value), window size of 128, and 5 custom Transformer layers with an embedding size of 1024 and intermediate dimension of 4608. The model uses SiLU activations and contains 83M parameters. We selected this wide architecture based on the comparative results in Supplementary Table [2] which demonstrate its superior performance and efficiency over deeper models in both in-domain and out-of-domain genomic tasks.
Tooenizztion and n-gram Frequency Calculation: We tokenize DNA sequences at the single- nucleotide level (A, T. C, G) to preserve fine-grainee features such as SNPs and avoid disrupting open
5

---

260
Window Size
I1.11
Pegiesity
E31
(33
1.5
0175
125
11c
Figure 3: Left: Effect of window size on model performance. window size of 128 achieves results comparable to full attention. Right:Inference time per sequence for CARMANIA(63M), HyenalNNA(1 16M), and Caduceus- .9M) across varying sequence lengths.
reading frames, as can occur with subword or non-overlapping k-mer methods. Additionally, during tokenization, compute normalized 4 4 first-order transition matrix for each sequence based on bigram frequencies, where each row defines a probability distribution over possible next nucleotides. This sequence-specific matrix acts as self-supervised training signal, serving as ground truth for the transition matrix loss and guiding the model to learn biologically meaningful transition patterns,
Training Parameters, and Hardware Specifies: The model was trained for two epochs using Py PoTrch on an NVIDIA A A00 GPU (80GB). To fit within memory constraints, batch sizes were set based on sequence length: 35 for 4 kbp. 19 for 10 kbp, and 1 for 160 kbp inputs. We used cosine annealing schedule with an initial learning rate of 5e-4. Additional training details, including optimizer settings and gradient clipping, listed in Supplementary Table [3] The training setup yields an efficiency of approximately 7.57 x 10-9 GPU-hours per token.
4.2 Ablation Studies
Effect of Transition Matrix Loss
We study the effect of adding first-order TM loss alongside the next-token (NT) loss by training on the Scorpio-Gens
dataset for 31k steps under two settings: with TM loss (B = 1) and without it (3 = 0). the 3 case, TM loss is computed but not used during training, allowing us to observe the model's natural transition behavior. The left panel of Figure shows NT loss decreases similarly for both models, indicating that adding TM loss does not hinder token-level learning. In contrast, the right panel reveals different TM loss patterns: for the 3 0 model, the TM loss initially increases then decreases as the model partially and passively captures transition structure, while for the 3 = 1 model, the TM loss consistently decreases from the start. However, shown in supplementary results, convolution-bast models like HyenaDNA require explicit TM regularization to exhibit this property.
Finally, as reported in Table T models trained with TM loss achieve higher macro Fl scores on downstream Antimicrobial Resistance (AMR) classification tasks (detailed in Supplementary Table and improved BLEU scores on the human genome henchmark. We also evaluated this effect across all 40 downstream benchmarks and found that adding TMM loss led to improved performance in 33 of them.
Higher- Order Transition Matrices. As shown in Supplementary Tables 10 and I1 applying second- order TM loss led to performance degradation across tasks compared first-order TM or no TM. This drop is likely due to the sparsity of higher-order transitions in biological sequences, which makes the loss signal unstable. In contrast, first-order TM loss consistently improves performance by capturing reliable and biologically meaningful pairwise dependencies.
Effect of Windowed vs. Full Attention
We assess the effect of attention window size by training on a 100M bp subset of the Scorpio- -ene- Taxa dataset. As shown in Figure B larger windows improve BLEU scores and reduce perplexity. with full attention as an upper bound. Notably, window size of 128 performs comparably to full attention, indicating that TM loss enables smaller windows to capture complex dependencies efficiently. Figure 3 also illustrates inference time comparisons across models on input sequences of varying lengths, up 100k bases. Despite having 83M parameters, model runs approximately

---

With TM Losa
Model
Wiboou 7M Looe
MysnatRe
ba
Genomic Position (bp)
Genomic Position (bp)
Figure 4: Heatmap of average sequence similarity across 50 independent 160 kbp genomic segments in 100 bp windows at 2000 bp intervals. Left: CARMANIA consistently maintains high similarity across all regions, demonstrating superior long-range memory, whereas HyenaDNA shows reduced sequence coherence in later segments. Right: Incorporating the TM loss improves memory retention in CARMANIA, yielding higher sequence similarity across extended contexts.
2.5 times faster than Caduceus becomes significantly slower as sequence length increases, while HyenaDNA (1.6M) maintains relatively fast inference but underperforms on long sequences and remains slower than our model. This demonstrates the practical efficiency of our architecture, and also highlights how windowed attention enables faster inference in Transformer-based models.
Table D compares the performance of models trained with full attention and windowed attention. While full attention achieves slightly better performance on downstream tasks, results in lower BLEU score. In addition, evaluate the computational efficiency of windowed attention. As shown in [ full attention requires 58% more computational resources compared to a window size of 128. This demonstrates the trade-off between accuracy and efficiency, windowed attention reduces complexity from (n²) in full attention to O()² making it a more practical and scalable choice for large-scale sequence modeling.
4.3 Long-Range Sequence Retention
We evaluate each model's ability preserve long-range sequence structure by measuring internal consistency across extended human genome regions. Specifically, we analyze 50 independent 160 kbp genomic segments. Within each segment, we extract 100 bp windows using fixed stride of 2000 bp and compare the model -peedccpre sequences against the corresponding regions in the original input sequence using Hamming similarity. This produces a position- wise similarity profile, which we average across all segments to assess how well each model maintains fidelity over increasing genomic distance.
As shown in Figure | HyenaDNA shows a marked drop in sequence similarity at distal positions. This likely reflects the limitations of its fixed-depth recurrence architecture, where information must propagate layer by layer without explicit token-to-token interaction Nguyen et al. [2024), making it harder retain detailed signals over long spans. In contrast, CARMANIA maintains high similarity across the entire sequence due to its sliding-window attention and the TM loss, which encourages consistency in token transitions. The effect of the TM loss is further illustrated in Figure 4 where models trained without it show noticeably shorter retention spans, suggesting its role in supporting full- full-sequence memorization.
4.4 Downstream Task Evaluations
Genomics Benchmark: Supervised Adaptation
To evaluate task-specific adaptation, we fine-tuned our model on diverse genomics classification datasets. The Genomic Benchmarks collection includes tasks such as regulatory element prediction, enhancer detection, and binary species classification Grešová et al. [2033]. For fair comparison, we fine-tuned the human-pretrained CARMANIA model using 5-fold cross-validation and compared its performance to other ofte--of-the-ar models trained on the human genome. As shown in Table 2 CARMANIA achieves top performance out of tasks and matches the best performance others, demonstrating its strong generalization ability in supervised downstream settings.
7

---

Table 2: Top-1 accuracy (f) across 5-fold cross- validaaiii on Genomic Benchmark tasks. Results for CNN, HyenaDNA, Mamba, and both Caduceus variants are taken from the original Caduceus paper [Schiff et al. 2024 Underlined values indicate second-best results.
Tisk
CNN
HyenaDNA A
Mamba
Caduvus-PH
Caduceus-PS
CARMANIA
Baseline w/o TM
Mouse Enhuncers
0.715 0,087
0,780 a 0.025
0.743 a 0.054
0.754 0.074
0.743 0,05K
0.761 :: 0.019
0,748 x: 0.030
Coding Intergenic
0.892
0,902 0,005
0904 0.004
0915 0.003
0.910 0.003
0,935 0.901
0.001
Human Womn
0.964
0967
0.973
0.968 0.002
0.968
0.966 0.001
Human Enhancers Cohn
0.702 = 0.022
0.729 0.014
0,732 # 0.029
0.747 2 0.004
0.245 0.007
0.724 # 0.005
0.699 = 0.00S
Human Enhancer Ensembl
0.744 0.122
0.849 0.006
0.862 4 0.000
0.893 2: 0.008
0.900 =: 0.006
0.916 2 0.002
0.892 1 0.001
Human Requlatory
0.872 0.005
0.869 0.012
0.814 1.2211
0,572 0.011
0.873 L 0.007
0.895 0.002
0.893 1 0.002
Human OCR Enserabl
0,783
0,828 0.006
0.818 0.000
Human NoRTATA Promoteess
0.861 0.009
0.944 0.002
0.933 + 0.007
0.946 0.007
0:945 0.010
0.963 + 0.002
0.961 + 0.003
Nucleotide Transformer Tasks: Supervised Adaptation
We benchmarked our model on 18 diverse genomics classification tasks introduced by Dalla- Torre et al. Dalla-Torne et al. (2024), spanning histone modification prediction, regulatory annotation, and splice site detection. Following their evaluation protocol, we performed 10-fold cross-validation with early stopping, reporting the mean and standard deviation across different random seeds.
As shown in Table 3 CARMANIA achieves top-1 performance on 5 out of 18 tasks and outperforms the average of all baselines by over 3% across metries. Notably, in enhancer classification tasks, CARMANIA achieves up to 34% absolute improvement in MCC over previous models [hhou et al.
Avsec et al. [202T]. Schiff et al. 2024 Nguyen et al. 2024). Dalla-Torre et al. 2024). These results demonstrate the model's 's strong ability to capture regulatory signals from sequence. A notable gain is observed the H4ac histone mark, where adding the TM loss yields over 40.9% relative improvement in MCC. The benefit of the TM loss, however, is not uniform across all genomics tasks. In enhancer prediction, where positive and negative classes differ strongly in their local motif' distributions (KL divergence = 9.28), the TM loss provides no measurable improvement (A = 0.0%). In splice-site donor prediction (KL 0.99), it yields modest 5.2% gain, and in the H4ac histone mark dataset (KL 0.64), substantial 40.9% improvement. These results demonstrate that TM-driven performance gains inversely track inter-class bigram separability. confirming that TM loss is most effective when local sequence features are ambiguous and long-range dependencies become critical. Biologically, this aligns with our understanding that enhancer regions possess clear, unique motifs that NT loss can easily learn, whereas splice sites rely on short, dispersed sequence cues. In contrast, H4 acetylation involves multiple lysine sites (K5, K8, K12, K16) on the H4 N-terminal tail that act cooperatively to regulate chromatin accessibility Dion et al. [2005). Ma et al. [2024]. Instead of strong individual motifs, these sites follow diffuse and combinatorial sequence patterns, making TM's global co-occurrence modeling particularly advantageous.
2023
Table 3: Performance comparison (f) on 10-fold cross-validation across 18 tasks, including histone marks, regulatory elements, and splice site prediction. We report MCC for histone/enhancer tasks, F1 for promoter/splice site, and accuracy for the "All" splice task. Baselines follow Caduceus [Shhff let al. [2024) and Dalla-Torre al. Dalla-Torre et al. [2024). CARMANIA achieves top performance on 5 tasks and remains competitive elsewhere. Underlined values denote second-best results.
ENFORMER 252M
DNAHERE-2 117M
NEA22 500M
HVENADNA 1.6M
Caducces- 1.M
Caduceus-PS 1.9M
CARMANIA RMM
Baseline w/o
8TM
Histume Markers
H3 HSKLLAC
0.71920048
0.785200033
0.78420.047
0.779200057
0.81528.045
0.79900 029
0,782 211023
0.785 g0001
0,28820.077
0.51620:025
0.55120:021
2400250065
0.631:8.02 026
039120212
0.627 +9.021
0.631 200022
HAMES
0,344-0185
0.501=00020
0.62540:013
360300004
0.601.0011 129
0.60920 109
0,632 #1.011
0.620 s0101
0.29180:06]
0.51110.028
0.550+0.021
0.512+0.004
0.523+0.050
0.48810.100 102
0.515 40.017
0.516 200017
HER AMED
0:21100.09
0.53640.040
0.319H0:045
0.45500000 095
0.48760.155 770
0.388x0.10 101
0.502 a0.025
0.500 #01038
H3KIMES
0.158a00072
0.35220.077
0.41020:033
0.56920.056
0,54420.045
0.440:01222
0.565 40.012
0.586 20.011
H3K79ME3
044
0:613+00130
0.67220 0418
0. 0.697-0.1177
0.676430066
01699 411003
0.655 S0L018
HSKUAC
04428600033
0.542500129
0.562+0:140
0.581
0.622s0.130
0.684501448
0.615 +110013
0.688 s0L013
H4
0732600076
0.79640.027
0.79940.025
0 066360 044
0.811
0.78000020
0.77280:010
0 0.76940 019
HAAC
0.273g0.063
0,46320.041
0.49520.032
0.564400038
0.631:9.054
0.525:01240
0.197200011
Regulatory Annotation
ENHANCER
0.05120.108
0,51610.098
0.54820.144
0,17220117
0,516
0.891100066
0.80020.013
0.78800013
ENHANCER TYPES
0.309H0.1 144
0421+00011
0.424H).115 132
033860
0.439H00.05
0.46=-1995
0.724
7344
PROMOTER: ALL
0954200006
0.97
0.96110.000
0
NONTATA
0955a0000
0.97740.005
0.976a0.005
0.55110
01000.011
0.96240.00
TATA
0.96020.02
0.95520021
0.96620.013
0.92200.400
0.95320.016
0.95700.115
0.942400:008
Splice Site Ausotation
ALT.
0.848e0019
0:939-0.009
0.983=0.008
0956+0.011 1
0:940+0.027
0.03++000
0.96740.008
0.96740
ACCEPTOR
0.11-00.28
0.975+00006
00.01=0.0111
0908+0.010
0.95700053
0.336-00077
0.958+00000
0.94+00001
DONOR
0.906a0.027
0.96310.006
0.985:0.022
0.9:930.024
0.94820.025
0.874102289
0.9773600002
0921
8

---

Table 4: Performance comparison on the Scorpio-Gene- Taxa dataset.
Test
Gene ont
Taxa out
Phylum
Class
Order
Family
Gene
Phylum
Class
Order
Family
Gene
MetaBERTa
U.712
0.584
0.426
0.322
0.282
0.640
0.471
0.290
0.204
0.074
HyenaDNA
0.764
0.636
0.447
0.292
0.846
0.449
0.265
0.113
0.058
0.674
Buseline w/o TM
0.860
0.765
0.589
0.417
0,845
0.547
0.401
0.222
0.145
0.608
CARMANIA
0.861
0.768
0.596
0.419
0.909
0.469
0.316
0.159
0.094
0.728
Task-Adaptive Pre- Tre-Training on the Scorpio-Gene-Taan -axa Dataset
We evaluated task-adaptive pre-training on the Srorpio-GeneTTaxee dataset (Refahi et al. [025 J. which includes three evaluation splits: Test (seen genes and taxa), Gene-out (unseen gene labels), and Taxa-out (unseen phyla). All models were trained on the same data and evaluated using FAISS-
based [Douze et al. 2024] embedding similarity to retrieve the best match from the training set.
As shown in T 4 CARMANIA achieves the highest accuracy on both the main test set and the Taxa-out split, while also showing strong generalization on the Gene-out set. In contrast, MetaBERTa performs well on Gene-out but struggles on Taxa-out, suggesting it may rely more on taxonomic memorization than on learning transferable mapping from gene sequence taxonomy. This highlights& a fundamental challenge in this dataset: there is often trade-off between generalizing across gene identities and taxonomic groups. Most models struggle to do both simultaneously unless they explicitly capture the hierarchical structure linking genes to taxa (Refahi et al. 2025)]
To further examine these results, we visualized the learned embeddings using t-SNE for the 10 most frequent genes (Figure 5). CARMANIA forms compact clusters that align well with both gene identity and taxonomic structure, suggesting it captures both levels of organization. HyenaDNA produces well-separated clusters based on gene identity but lacks alignment with taxonomic labels. In contrast, MetaBERTa embeddings show stronger alignment with taxonomy but weaker separation by gene, as also seen in Supplementary Figure
This is further supported by additional results in Supplementary Table 5 where CARMANIA outperforms all other models on AMR gene classification tasks without fine-tuning, highlighting its strong domain adaptation capability.
Domain Adaptation and Generalization
We evaluated several pre-trained genomic models on the AMR dataset without fine-tuning. The dataset, derived from MEGARes [Bonin et al.) 2023 and CARD Jia et al. (2016). includes three classification tasks: Gene Family, Resistance Mechanism, and Drug Class Yoo al. [2024). Labels were assigned based on the closest match (best hit) in embedding space using FAISSDouze et al. 22244] on frozen representations.
As shown in Table 5 CARMANIA achieves the highest F1 Macro scores across all tasks, indicating that it produces more informative embeddings for AMR classification than prior models, including
HyenaDNANguyen et al. [2024). Caduceus Schiff et al. 2024], MetaBERTa Refahi et al. [2025]. and
Nucleotide Transformer: Dalla- Torre al. [2024).
Table 5: FI Macro scores for AMR classification on Gene Family, Resistance Mechanism, and Drug
Class using FAISS-based best-hit embedding retrieval.
Model
GeneFamily
Resist-Mech DrugClass
MetaBERTa (35.2M)
0.650
0.898
0,886
HyenaDNA (1.6M)
0.623
0.840
0,880
Caduceus- PH (1.9M)
0.597
0.823
0.839
Nucleotide (2.5B)
0.611
0.856
0.859
w/o TM (83M)
0.728
0.974
0,931
CARMANIA (83M)
0.733
0.975
0.942
Long Context Task: Biosynthetic Gene Cluster Classification
To evaluate the ability of genomic language models to capture functional patterns across extended DNA regions, we performed a large-scale classification task on biosynthetic gene clusters (BGCs) using the MiBiG database Kautsar et al. 22020] Liu et al. 20022 where each cluster is labeled by its

---

CARMANIA
HyenaDNA
MetaBERTa(BigBrr))
TrA TepB
IpdA
tuT
map
dif
RuSA
TabG
gnaL
IISH
Figure 5: I-SNE visualization of the 10 most common genes in the Scorpio-Gene-Taal dataset. CARMANIA effectively clusters genes while maintaining taxonomic coherence, leading to superior gene-to-taxonomy classification performance.
associated secondary metabolite class. This task was framed as multi-class classification using raw DNA sequences- withhot relying on protein translation or domain-level annotations. Since BGCs vary in length (average 377k bp), we truncated all sequences to 100k bp for model compatibility.
Unlike previous approaches that depend on external tools like Prodigal and HMMER Hannigan et al. 2019). Liu et al. 2002). our method directly operates on nucleotide sequences. We evaluated perfor- mance using 5-fold cross-validation. As shown in Table AAIMANIA, which incorporates TM loss, achieves the highest accuracy (48.4%), outperforming HyenaDNA by over 7% and significantly surpassing other baselines.
Table 6: 5-fold CV accuracy and standard deviation for BGC classification on 100,000 DNA se- quences.
Model
Caduceus-Ph
HyenaDNA
Baseline w/o TM CARMANIA
Accuracy + Std
0.326 = 0.119
0,412 = 0.130
0.410 & 0.240
0.484 + 0.330
Conclusions
In this paper, we introduced CARMANIA, novel pre-trained genomic model designed for long- sequence analysis and nucleotide-level processing. Unlike existing models, CARMANIA effectively captures both local and global sequence dependencies, making it highly suitable for diverse genomic tasks. Our evaluations demonstrate superior performance across multiple benchmarks, including AMR classification, gene-tar association, regulatory element prediction, species classification, and biosynthetic gene cluster prediction.
Limitations and Future Work: While our method improves sequence representation learning, one limitation lies in the fixed scaling parameter 3 used to balance the next-token and TM losses. We set 3 = 1 and observed that the losses naturally aligned in scale, making this choice effective in practice. However, tasks involving rare but biologically important motifs may require more fine-grained dataset-specific tuning. In cases where (X) + H fixed 3 may lead to overfitting on common patterns while missing informative outliers. Future work could explore adaptive scaling strategies that dynamically adjust based on the training loss distribution. Although the TM loss is computationally efficient due to its matrix- formulation, memory usage increases with larger vocabulary sizes and batch dimensions. This can limit scalability, particularly in domains such as chemistry (e.g., SMILES representations) or natural language, where token sets are larger and sequence variability is higher. To address this, future work could explore memory-effictee approximations, sparse matrix operations, or gradient checkpointing to reduce the computational footprint. In addition, we aim to scale up both the model and the diversity of training datasets to improve generalization across a broader range of genomic and functional prediction tasks. Finally, we note that, like other predictive genomic models, this approach may raise concerns around potential misuse in synthetic biology or privacy-sensine contexts, which should be addressed in future deployments.
10

---

References
Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18 (10):1196-122) 1203, 2021.
Lucie A Bergeron, Søren Besenbacher, Jiao Zheng, Panyi Li, Mads Frost Bertelsen, Benoit Quintard, Joseph Hoffmann Zhipeng Li, Judy St. Leger. Changwei Shao, et al. Evolution of the germline mutation rate across vertebrates. Nature, 615(79511:285-295
2023.
Yu Bo, Weian Mao, Yanjun Shao, Weiqiang Bai, Peng Ye, Xinzhu Ma, Junbo Zhao, Hao Chen, and Chunhua Shen. Revisiting convolution architecture the realm of dna foundation models. arXiv preprint arXiv:2502. 18538, 2025.
Nathalie Bonin, Enrique Doster, Hannah Worley, Lee J Pinnell, Jonathan E Bravo, Peter Ferm, Simone Marini, Mattia Prosperi, Noelle Noyes, Paul S Morley, al. Megares and amr++, v3. O: an updated comprehensive database of antimicrobial resistance determinants and an improved software pipeline for classification using high-throughput sequencing. Nucleic acids research, 51 (D1):D744-D752, 2023.
Samuel Cahyawijaya, Tiezheng Yu, Zihan Liu, Tiffany TW, Xiaopu Zhou, Nancy Y Ip, and Pascale Fung. Snp2vec: Scalable self- supervisdd pre-training for genome-wide association study, BioNLP 2022@ ACL 2022. page 140, 2022.
Abdulkadir Celikkanat, Andres Masegosa, and Thomas Nielsen. Revisiting k-mer profile for effective and scalable genome representation learning. Advances in Neural Information Processing Systems, 37:118930- 118952 2024.
Jiacheng Chuan, Huimin Xu, Desmond L Hammill, Lawrence Hale, Wen Chen, and Xiang Li. Clasnip: web-based intraspecies classifier and multi-locus sequence typing for pathogenic microorganisms using fragmented sequences. PeerJ, 11:e14490, 2023.
Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, et al. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, pages 1-11, 2024.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869) 2023.
Arthur L Delcher, Kirsten A Bratke, Edwin C Powers, and Steven L Salzberg. Identifying bacterial genes and endosymbiont dna with glimmer. Bioinformaties, 23(6):673 6769)99,,,99 2007.
Michael F Dion, Steven Altschuler, Lani Wu, and Oliver J Rando. Genomic characterization reveals simple histone h4 acctylation code. Proceedings the National Academy of Sciences, 102(15):5501-5500 2005.
Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, ZIJIA CHEN, Ameya Sunil Mahabaleshwarkar, Shih- Yang Liu, Matthijs Van keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Celine Lin, Jan Kautz, and Pavlo Molchanov. Hymba: hybrid-head architecture for small language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https
//openreview.o fopenreriew.net/foormm//
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanue Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprini 2oXiv:241108288 2024.
Benjamin I. Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The evolution of statistical induction heads: In-context learning markov chains. arXiv preprint arXiv:2402. 1004, 2024.
11

---

Can Firtina, Kamlesh Pillai, Gurpreet S Kalsi, Bharathwaj Suresh, Damla Senol Cali, Jeremie S Kim, Taha Shahroodi, Meryem Banu Cavlak. Joël Lindegger, Mohammed Alser, et al. Aphmm: Accelerating profile hidden markov models for fast and energy-efficient genome analysis. ACM Transactions on Architecture and Code Optimization, 21(1):1-29, 2024.
Ensembl GRCh38, p13 (genome reference consortium human build 38), insdc assembly, 2013.
Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Simeček, and Panagiotis Alexiou. Ge- nomic benchmarks: collection of datasets for genomic sequence classification, BMC Genomic Data, 24(1):25, 2023,
Geoffrey D Hannigan, David Prihoda, Andrej Palicka, Jindrich Soukup, Ondrej Klempir, Lena Rampula, Jindrich Durcak, Michael Wurst, Jakub Kotowski, Dan Chang, et al. deep learning renome-mining strategy for biosynthetic gene cluster prediction. Nucleic acids research, 47(18): e0-e-1110, 2019.
Jiachen Hu, Qinghua Liu, and Chi Jin. On limitation of transformer for learning hmms. arXiv preprint arXiv:2406.0400 2024.
M. Emrullah Ildiz, Yixiao Huang, Yingcong Li, Ankit Singh Rawat, and Samet Oymak. From self-attention to markov models: Unveiling the dynamics generative transformers, 2024. URL abs//402..... 2402. 13512
https::/aaxtis org abs/
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, 37 (15):2112-2120) 2021.
Baofeng Jia, Amogelang R Raphenya, Brian Alcock, Nicholas Waglechner, Peiyao Guo, Kara K Tsang, Briony A Lago, Biren M Dave, Sheldon Pereira, Arjun Sharma, et al. Card 2017: expansion and model-centric curation of the comprehensive antibiotic resistance database. Nucleic acids research, page gkw1004, 2016.
Satria A Kautsar, Kai Blin, Simon Shaw, Jorge C Navarro-Muñoe Barbara R Terlouw, Justin JJ Van Der Hooft, Jeffrey A Van Santen, Vittorio Tracanna, Hernando G Suarez Duran, Victòria Pascal Andreu, et al. Mibig 2.0: repository for biosynthetic gene clusters of known function. Nucleic acids research, 48(D1)) 2020.
Mingyang Liu, Yun Li, and Hongzhe Li. Deep learning to predict the biosynthetic gene clusters in bacterial genomes. Journal of Molecular Biology, 434(15):167597, 2022.
Alexander V Lukashin and Mark Borodovsky. Genemark. hmm: new solutions for gene finding.
Nucleic acids research, 26(4):110--1115, 1998.
Wendong Ma, Xiaofan Ding, Jiajia Xu, and Terence Chuen Wai Poon. Chhm: manually curated
catalogue of human histone modifications revealing hotspot regions and unique distribution patterns. International Journal Biological Sciences, 20(10):3760, 2024.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024.
Simon C Potter, Aurélien Luciani, Sean Eddy, Youngmi Park, Rodrigo Lopez, and Robert Finn. 2018.
Hmmer web server: 2018 update. Nucleic acids research,
Mohammadsaleh Refahi, Bahrad A Sokhansanj, Joshua C Mell, James R Brown, Hyunwoo Yoo, Gavin Hearne, and Gail I. Rosen. Enhancing nucleotide sequence representations in genomic analysis with contrastive optimization. Communications Biology. 8(1):517, 2025.
MS Refahi, BA Sokhansanj. and GL Rosen. Leveraging large language models for metagenomic analysis. 2023 IEFE Signal Processing in Medicine and Biology Symposium (SPMB), pages 1-6. IEEE, 2023.
12

---

Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv:2403.03234, 2024.
arXiv preprint
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, 35(11):1026-102 2017.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing, 568: 127063, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397,, 2023.
Derrick Wood, Jennifer Lu, and Ben Langmead. Improved metagenomic analysis with kraken Genome biology, 1-:1-13, 2019.
Tao Wu and David F Gleich. Retrospective higher-order markov processes for user trails. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1185-1194, 2017.
Hyunwoo Yoo, Bahrad Sokhansanj. James R Brown, and Gail Rosen. Predicting anti-microbial resistance using large language models. arXiv preprint arXiv:2401.00644 2024.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Yang, al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-1111 1729-17297, 2020.
Oussama Zekri, Ambroise Odonnat, Abdelhakim Benechehab, Linus Bleistein, Nicolas Boullé, and levgen Redko. Large language models as markov chains. arXiv preprint arXiv:2410.02724, 2024.
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert-2: Effi- cient foundation model and benchmark for multi-species genome. arXiv preprint arX1v:3306.15066 2023.
Qiyun Zhu, Shi Huang, Antonio Gonzalez, Imran McGrath, Daniel McDonald, Niina Haiminen, George Armstrong, Yoshiki Vázquez- Baeza, Julian Yu, Justin Kuczynski, et al. Phylogeny-awand analysis of metagenome community ecology based on matched reference genomes while bypassing taxonomy. Msystems, 72)xc0016-222, 2022.
13

---

A Supplementary Material
A.1 Effect of TTM Loss on Other Domains:
Protein Modeling
While CARMANIA was primarily developed for genomic sequences, we further examined the gener- ality of the proposed TM loss in other biological domains. In theory, TM loss should benefit any mod- eling task that relies on capturing long-range dependencies. To validate this hypothesis, we adapted the CARMANIA architecture for amino acid sequences and trained on the Scorpio-Gene-Tann Taxa protein dataset. Specifically, we constructed 20 x 20 bigram frequency matrix representing pairwise transition statisties between standard amino acids and incorporated TM loss into the training objective.
As summarized in Table incorporating TM loss consistently accelerated convergence and improved predictive accuracy across all hierarchical levels. These results confirm that TM loss generalizes beyond nucleotide modeling and effectively enhances protein pequenne representations.
Table Performance comparison on the Scorpio-Gene- Taxa dataset using protein sequences.
Test Set
Gene-Heldout
Taxa- Mexa-Helouut
Phylum
Order
Family
Gene
Phylum
Class
Order
Family
Without TM Loss
84.9
70.6
51.3
99.5
19.8
10.9
3.3
98.4
With TM Loss
91.4
85.5
71.0
31.5
99.6
20.5
11.6
38
1.5
98,7
A.2 Effect of TM Loss on Convolution- Architecture: HyenaDNA
We trained HyenaDNA on the Scorpio-Genc Saneii--Gne--Taxee dataset with and without TM loss to investigate its effect model performance. As shown in Figure 6 the model without TM loss demonstrates better convergence, with the training loss decreasing more effectively compared to the model with TM loss. Notably, the model without TM loss appears to struggle with learning the bigram frequency (transition matrix) inherent to the Markovian model. This observation suggests that convolution based models, even in autoregressive manner, may not capture Markovian dependencies effectively as transformer- based models, which leverage attention mechanisms to model Long-range dependencies more efficiently.
Additionally, we evaluated the models' performance on the AMR detection tasks using the macro F1 score across all tasks. The results, Figure/Zindicate that incorporating TM loss improves performance in CARAMNIA; however, the HyenaDNA model does not exhibit a similar performance gain. This suggests that convolution-baset models may inherently struggle to learn embeddings and representations for tasks that rely on Markovian properties, highlighting a potential architectural limitation in such models for sequence modeling tasks.
LESS
1.00
18 Urou TN LOO
Wi WIth Loaa
0.25
1.0
1000
10000
20000
25000
10033
5030
10000
11500
20000
2502e
s26c0
Figure 6: Left: Next-token loss; Right: TM loss for HyenaDNA. The convolution-based model strug- gles to learn the TM effectively, unlike the attention-based model, which better captures Markovian dependencies.
A.3 Effect of TM Loss on Mamba-Based Architecture: Caduceus
Caduceus Schiff et al. 2024 is a bi-directional, reverse-comp creese-commleemett equivariant DNA language model built upon the MambaDNA architecture, which leverages selective ssaee modeling for long-range sequence processing. While Caduceus is pretrained using a masked language model- ing (MLM) objective distinct from our causal next-token prediction (NT) setup we incorporated the proposed TM loss into the MLM objective while keeping all other configurations unchanged. This modification allows us to assess whether TM loss provides additional benefits beyond the CARMANIA backbone.
14

---

90
88-
Average F1 (%)
82
76-
Wymnati aftveut
CARMANIA withoot CARMANIA with
Figure 7: Effect of Transition Matrix Loss on HyenaDNA and CARMANIA Pre-trained on Scorpio-
gene-taxa
For this experiment, we used the largest available pretrained variant Caduceus (approximately 7.7M parameters) and fine-tuned it the Scorpio Gene-Taxa dataset with and without TM loss. Incorporating TM loss led to a notable performance boost, particularly on the gene classification task, yielding + +12% gain on the test and +6% the held-out set. These results suggest that TM loss provides a meaningful signal even in state-space models like Caduceus, although its impact remains more pronounced in transformer--aee
architectures such CARMANIA.
Table 8: Effect of TM Loss on Caduceus Tine-Tuned on the Scorpio-Gent Caapiio-Gnne-Taae -aan Dataset. Perfor- mance (accuracy, %) across hierarchical taxonomic levels. A denotes the relative improvement when adding TM loss.
Test Set
Gene-Heldout
Taxa-Heldont
Model
Phylum
Class
Order
Family
Gene
Phylum
Class
Order
Family
Gene
(TM w/o TM)
+6.2
+7.0
+5.8
+4.0
+12.4
+0.3
+0.1
0.1
0.0
+6.6
1,5
Caduceus TM
46.2
31.9
19.2
11.7
38.6
30.6
13.7
4.0
1.5
22.5
CARMANIA
86.1
76.8
59.6
41.9
90.9
46.9
31.6
15.9
9.4
72.8
A.4 Sensitivity Analysis: Effeet of /
The sensitivity analysis evaluates the impact of the hyperparameter / on model performance. As shown in Table 9 increasing 3 from 0 to 1.0 improves the macro F1 score (87.36% to 88.17%) and BLEU (0.73 0.77), suggesting that the TM loss helps capture Markovian dependencies. However, when 33 is increased to 5.0, performance significantly degrades, indicating that excessive TM loss disrupts the next-token objective. The best performance is achieved with 1.0, demonstrating balanced contribution from both objectives.
Table Comparison of 3 values with Macro, BLEU, and Perplexity scores.
B
Average F1 Macro
BLEU
Perplexity
0
0.873
0.73
3.6
0.5
0.874
0.76
3.6
1.0
0.882
0.77
3.6
5.0
0.836
0.85
3.8
A.5 Higher-Order Transition Matrix
We explored the impact of incorporating TM loss at different Markov orders in CARMANIA, with results shown in Tables TO and [1 Surprisingly, while the first-order TM consistently improves downstream classification performance across tasks, the second-order TM performs worse than both the first-order TM and the baseline without TM.
This performance degradation in the second-order setting can be attributed the increased sparsity of higher-order transition patterns, particularly in biological sequence data. Since second-order
15

---

transitions require co-oceurrence of triplets, many such patterns are infrequent or entirely missing from the training set, leading to unstable or noisy estimations. This sparsity reduces the model's ability to generalize and can hinder the convergence of the transition- objective Wu and Gleich [2017]. On the other hand, first-order transitions strike a balance by enforcing local smoothness and capturing robust pairwise dependencies that are both statistically reliable and biologically meaningful. This results in more stable training and improved generalization across domains.
Table 10: Comparison of F1 Macro scores across three AMR classification tasks with different TM orders. First-order TM improves performance across all tasks, while second-orde TM leads to degradation.
Model Variant
GeneFamily
Resist-Mech DrugClass
Without TM
0.728
0.974
0.931
1st-Order TM
0.733
0.975
0.942
2nd-Order TM 0.701
0.943
0.915
Table 11: Accuracy comparison the Scorpio-Gene- Taxa test set across different taxonomic levels. First-order TM leads to consistent improvements, especially at the gene level. Second-order TM degrades performance across all levels.
Model Variant
Phylum
Class
Order
Family
Gene
Without TM
0.860
0.765
0.589
0.417
0.845
1st-Order TM
0.861
0.768
0.596
0.419
0.909
2nd-Order TM
0.810
0.703
0.520
0.371
0.690
A.6 Wide vs. Deep: Architectural Trade-off's for DNA Modeling
As shown in Table 12, our wide architecture reduces the number of layers while significantly increasing the hidden and intermediate sizes. This results in network that is both more expressive and more efficient.
Most notably, the wide model achieves a substantially higher accuracy on the Scorpio-Gens Serhio-GGee-aae Taxa classification task (71.05% = 18.07 63.87% + 19.16), demonstrating its superior ability to capture biologically meaningful patterns from genomic sequences. Furthermore, the wide model also outperforms the deep model on the AMR prediction task out-of-domain setting-with higher macro Fl score (88.32% + 13.12 vs. 85.22% + 14.66). These consistent gains across both in-domain and out-of-domain datasets highlight the robustness and generalization ability of the wide configuration.
The observed improvements can be attributed to three key factors:
First, wider layers with large hidden and MLP dimensions better capture motif-level and co- occuureeee patterns, which are prevalent in DNA sequences. This capacity is especially critical for tasks such as gene classification and function prediction, where local patterns carry more signal than hicrarchical abstractions.
Second, the reduced depth allows for better GPU utilization and faster convergence. With fewer sequential operations, the wide model completes training in nearly half the time while requiring fewer steps to achieve peak performance.
Third, by concentrating representational power within each layer, the wide network is better equipped to model long-ran mange dependencies and subtle biological variations without needing deep hierarchical stacks. Given these advantages, we adopt the wide model as the backbone for the rest of our study. Its performance and efficiency make it a strong choice for DNA modeling tasks, especially when scaling to large or diverse datasets.
A.7 Model Size Scalability
We evaluate the effect of model scaling on performance and compute cost using the Basic Genome dataset. Table Table [4mmparees small 4M-parameter model trained on 0.5B-token subset with the full 83M-parameter model trained on 10B tokens. Results show substantial gains in both Fl and BLEU scores, demonstrating that increased model capacity and training data size improve representation quality without disproportionately increasing relative FLOPs.
16

---

Table 12:
Comparison of model configurations between deep and wide architectures. The wide model uses fewer layers but larger hidden and intermediate sizes.
Configuration
Deep Model (80M)
Wide Model (83M)
Number of Layers
24
5
Hidden Size
512
1024
Intermediate (MLP) Size
1664
4608
Attention Heads
8
16
Key/Value Heads
4
4
Attention Window
128
128
Activation Function
SiLU
SiLU
Training Steps
109,500
31,000
Epoch Time
6.1 hours
3.2 hours
Task
Performance Comparison
Soorio-GGnn--ie Test (Accuracy%)
63.87 = 19.16
71.05 = 18.07
AMR (FI Macro%)
85.22 = 14.66
88.32 + 13.12
Table 13: Parameter Ranges for Model Training
Parameter
Range Value
Optimizer
Adam 1, 19, 35
Batch Size
Num Epochs
Learning Rate
5e-4
Weight Decay
0.2
Adam Epsilon
le-6
Adam Betas
(0.9, 0.999)
Warmup Steps
400
LR Scheduler Type
Cosine
Max Grad Norm
0.85,
A.8 Impact of Sequence Length Scaling
To evaluate the effect of input sequence length on model performance, we conducted an ablation study using the GRCh38 dataset, comparing models trained on 10 kbp and 160 kbp fragments. As shown in Table [5] increasing sequence length improves both macro Fl and BLEU score, while maintaining the same computational footprint (FLOPs). This suggests that access longer genomic contexts enables the model to capture more distant dependencies, leading to more robust and informative sequence representations.
A.9 Comparison with Other Convolution-Based Models (ConvNova)
To contextualize CARMANIA'S performance with recent convolution- based models, we compared it against ConvNova Bo et al. 2025). recently proposed architecture for long DNA modeling. Following their evaluation protocol, we used five random seeds on the same test benchmarks to ensure consistency. As shown in Table T6, CARMANIA achieves comparable or superior results on several tasks, demonstrating that its attention- based and Markovian design effectively captures long-range dependencies.
A.10 Pre-training Dataset
The pre-training phase utilizes large-scale genomic sequences to establish robust sequence embed- dings, ensuring scalability, diversity, and compatibility with existing models trained on domain- specific datasets. We incorporate multiple datasets to support comprehensive learning and facilitate direct performance comparisons.
The Human Reference Genome (GRCh38 hg38): This dataset comprises approximately billion base pairs GRCh38 2013 From this genome, we extracted two non-overlapping sets of sequences to ensure coverage across diverse genomic regions. One set consists of long-range sequences with 160 kbp fragments, while the other contains shorter sequences of 10 kbp fragments.
17

---

CARMANIA
HyenaDNA
MetaBERTa(BigBind))
Figure 8: I-SNE visualization of the 10 most common phyla in the Scorpio-Gene-Taxa dataset. CARMANIA effectively captures the structure of inter-cluster taxonomy. As shown in Figure 5 the genes exhibit clear global structure, with phylogenetic groups positioned near each other when colorized based on phyla. This demonstrates that our model outperforms even MetaBFFRTa(BigBidd) preserving the taxonomy signal.
Table 14: Impact of model size performance and compute cost.
Model Size Dataset
Macro(AMR) (t) BLEU (Human dataset) (?)
Relative FLOPs (J)
4M purams / 0.5B tokens
0.809 # 0.114
0.41
0.07
params 10B tokens
The Basic Genome Dataset: Introduced by Zhu et [2022] this dataset includes 10 billion base pairs from 4,634 bacterial, archaeal, viral, and eukaryotic genomes. Sequences were extracted as 10 kbp fragments from each genome, ensuring representation across various species and evolutionary lineages.
The Scorpio Gene- Taxa Dataset: Developed by Refahi al. 2025 this dataset spans 580 million base pairs from 2,046 bacterial and archaeal species, encompassing 497 distinct gene types. To maintain consistency, fragments of 4 kbp were extracted, with shorter sequences padded as needed. Table 17 summarizes the key statistics of pre-training datasets.
A.11 Downstream Datasets
We used five distinct datasets for fine-tuning and evaluation, covering a range of genomic classification tasks and label spaces. These include regulatory clement prediction, gene function classification, taxonomic inference, antimicrobial resistance (AMR) detection, and biosynthetic gene cluster (BGC) prediction. Table 18 provides detailed statistics for each dataset.
The Antimicrobial Resistance classification dataset used in our evaluation is based on the benchmark introduced by Yoo et al. Yoo et al. (2024). It integrates sequences from two major sources: MEGARes Bonin et al. [2023]. a manually curated database of antimicrobial resistance genes annotated by gene family and resistance mechanism; and CARD Jia et al. [2016), which provides detailed molecular annotations of AMR genes, mechanisms, and associated drugs.
AMR:
BGC Dataset. To evaluate functional classification over extended DNA regions, we used the biosynthetic gene cluster (BGC) classification dataset derived from the MiBiG database Kautsar et al. 2020 Liu et al. 22022. where each cluster is labeled with a secondary metabolite class. Since BGCs vary in length (average 377k bp), we truncated all sequences to 100k bp for model compatibility.
database/Kautsar
Nucleotide Transformer Tasks. We evaluated our model on 18 genomic classification tasks introduced by Dalla-Torre et al. Dalla-allaooooo Torre et al. [2024). covering histone mark prediction, regulatory element annotation, and splice site detection. These tasks span wide range of sequence-based regulatory functions, offering comprehensive benchmark for assessing model generalization.
18

---

Table 15:
Effect of sequence length on performance. Longer input improves both Fl and BLEU without increasing FLOPs.
Training Setup
F1 Macro (f)
BLEU (1)
Relative FLOPs (L)
Trained on 10k Sequences
0.765 = 0. 0.120
0.82
1.00
Trained on 160k Sequences
0.800 0.138
0.84
1.00
Table 16: Genomics Benchmark Results. Top-1 accuracy (1) is reported for pretrained HyenaDNA, Caduceus-Phi ConvNova, and CARMANIA, alongside the CNN baseline. The best result is in bold, and the second-best is underlined.
Task
CNN
HyenaDNA
Caduceus-Ph
ConvNova
CARMANIA
Enhancers
Mouse Enhancers
0.730 0.032
0.779 0.013
0.754 0.074
0.784 a 0.009
0.795 a: 0.002
Human Enhancers (Cohn)
0.702 0.021
0.718 0.008
0.747 0.004
0.743 0.005
0.725 0.002
Human Enhancers (Ensembl)
0.744 + 0.122
0,832 + 0.006
0.893 0.008
0.900 0.004
0.895 + 0.002
Species Classificttion
Coding vs. Intergenic
0.892 0.008
0.904 0.008
0.915 0.003
0.943 0.001
0,930 + 0.001 0,970 0.005
Human Worm
0.942 0.002
0,961 0.002
0,973 0.001
0.967 0.002
Regulatory Elementy
Human Regulatory
0.872 #: 0.005
0,862 # 0.004
0.872 + 0.011
0.873 = 0.002
0.894 + 0.002
Non- TATA Promoters
0.009
0.005
0.946
0.965
Human OCR (Ensembl)
0.698 0.013
0,744 0.019
0.828 0.006
0.793 0.004
0.778 0.002
We evaluated supervised adaptation on the Genomic Benchmarks collection Grešová et al.] 2023 which includes range of classification tasks such as regulatory element prediction, enhancer detection, and binary species classification.
Genomics Benchmark Tasks.
The Scorpio Gene-Taxe Dataset:
Developed by Refahi et al. [2025). this dataset spans 580 million base pairs from 2,046 bacterial and archaeal species, encompassing 497 distinct gene types. To main- tain consistency. fragments of 4 kbp were extracted, with shorter sequences padded as needed. The dataset features Test, Gene Out, and Taxa Out splits, carefully designed assess both memorization and generalization- by holding out specific genes or entire taxonomic groups (e.g., phyla) while preserving hierarchical relevance. This enables robust evaluation gene-level and taxonomy-level generalization.
B Confirming the Transition Matrix Implementation
The first-order transition probabilities for the next token given the current token defined
P(w++1 ws) = [P(A] ws), P(T wi), P(C wx), P(G) ws)]
Similarly, for the token after next:
P(ws42 wi+1) = [P(A wi41), P(T w;41), P(C) w;+1), P(G | w:+1)]
In our implementation, the bigram transition matrix for position i is computed as:
T, = P(w.1 wx) P(w++2 wi+1)"
Expanding this matrix explicitly:
P(A A
w:)
P(A
W++1)
P(A
P(T
W(+1
P(A
wx)
P(C
wi+1)
P(A
w.)P(C
wa)
P(G
w
T:
P(T
P(A
w.+1)
P(T
P(T
w/.
w4+1
P(T
m) P(C
wi-1)
P(T
P(G
wi+1
P(C
P(A
W++1)
P(C
W++1
P(C
P(C
w(11
P(C
P
W4+1
P(G
P(A
W++1)
P(G
P.
w+11
P(G
PCC
w(+1)
P(G
PG
W4+1
Summing over the entire sequence length N, we obtain the predicted (but not yet normalized) bigram matrix:
19

---

Table 17: Statistics of the Pre-training datasets.
Dataset
Number Samples
Number Tokens
Length range (bp)
Human Genome-long GRCh38
2013
19,029
3
160K
Human Genome-shorf GKC
GKC
2013
303,921
3B
10k
Basic Genome Zhu el 40ZZ
1,010,237
10B
10k
Scorpio-Gene- Serto--Gnee-aaxa Taxa Kefahi et
2025
547,523
580M
114-13,227 (4k for training)
Table 18: Statistics of the fine-tuning datasets.
Dotaset
Number of Tasks
Classes
Length Kange (hp)
Genomics Benchmark
Binary (one mulii-class)
70-4,776
Scorpio Gene- Taxa Reran caal 6043
10
Multi-ciass (497-1.929)
114-13.227
Antimicrabial Resistance Picueetir 10a et al M024
3
Multi-clats
21
Nucleatide
Dalli-
Binory (one
Biosynthetic Gene Clester Predictio Kauitar al 2020
201-8M (avg 377k; Truncated to 100%)
N
T Pred
T:
To confirm equivalence with the bigram frequency matrix, consider the specific cell (T - A):
T Predd T + A) P(T P;) w141)
If replace probabilities with actual sequence counts, where P(w; T) if wz T and otherwise, and normalize each row to represent the transition probabilities, we get:
Σ I(w; T) I[w,+1 = A) ExE(AT.C..) E I(w; = T) I[ws+1 = x)
Prea (T + A)
This aligns with the actual bigram frequency:
count + A)
Pactual T - A)
(xT(AA,,C count(T + x)
Both matrices are row-normalized, ensuring the sum of each row is 1:
Σ
T Pred +2
Σ
Pactual (T + x) =
(E(A.T.C.C;
,E(A.T.C.G)
Thus, when probabilities are replaced with actual sequence values, the predicted transition matrix T Pred is equivalent to the actual bigram frequency matrix Pactual, confirming that our method is inherently inspired by bigram frequency calculations while extending them with learnable model probabilities.
20

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We carefully reviewed the abstract and introduction, and they do not contain any claims that are not justified in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the paper, we discuss both the strengths and limitations of our approach in the Experimental Results and Conclusion sections, particularly regarding model generalizability and range retention boundaries.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: We clearly describe the assumptions behind our appraoch and provide a full derivation of the first-order transition tensor and loss in Section 3.1 and 3.2. Due to space constraints, extended derivations and higher-order analysis are included in the supplementary material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We use publicly available datasets and describe all model architectures, hyper- parameters, and training procedures in detail. The code will be released upon acceptance to further support reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
22

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All datasets used are publicly available and cited appropriately. We will release the code and detailed instructions for reproducing the main experimental results in the supplementary material and a public repository upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
public/guides ottp///eaaaammmiiiooooooiiiiiiie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. http/uullcggggucce http///ooeSuuiiissseeeeeee asioonooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper provides detailed descriptions of data splits, optimizer settings, training schedules, and hyperparameters in Section 4 and Appendix A. These include the batch size, learning rate, number of epochs, and evaluation metrics used across all benchmarks.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For datasets with official splits, we follow the provided train/test partitions with- out additional resampling. For other datasets, we adopt evaluation protocol from prior work, including 5-fold 10-fold cross-validation where applicable. Additionally, we report
23

---

average performance across tasks when appropriate to summarize model generalization, as detailed in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the GPU type, memory configuration, and training time estimates in Section 4.1 and Appendix A. These include hardware specifications such as A100 GPUs and per-task training durations to support reproducibility. Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurips: http://hwwarpppscccppppccttccccc ctw.h//eeeiiiiciieeeeeeeeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We briefty discuss the potential applications of our method in genomic discov- ery and functional annotation, which could support advancements in health, biotechnology. and microbial ecology. We also acknowledge possible risks related to dual-use, privacy (in human genomics), or misuse of predictive models, as noted in the Broader Impact section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The datasets and models used in this work are not considered high risk for misuse; they are based on publicly available genomic data and not involve sensitive personal or dual-use information.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
25

---

Answer: [Yes]
Justification: All external datasets and models used in this work are properly cited, and rely exclusively publicly released assets with appropriate licenses, noted in the references and Appendix A.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This work does not introduce any new datasets, models, software assets beyond those already publicly available and cited.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
26

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Large language models not used part of the methodology experimental components of this research. Any use of LLMs was limited to minor editing and formatting support and does not impact the scientific contributions.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanstandard components.
Please refer to our LLM policy (https /ttp:::///uriipp cc/Confeerences/2225 /LLM) for what should or should not be described.
27


Reviews:
[{"review_id": "oJ02pEqyeJ", "reviewer": "Reviewer_6bow", "summary": "This manuscript presents CARMANIA, a self-supervised pre-training framework for genomic foundation models. \n\nThe core motivation of this work is to address the challenge that standard autoregressive transformers are limited in capturing long-range dependencies in genomics, as their fixed-size context windows cause them to exhibit local Markovian behavior. To overcome this, the paper introduces a novel Transition Matrix (TM) loss as a global regularizer for pre-training. This TM loss forces the model to learn the empirical n-gram statistics of the entire input sequence, pushing it to capture organism-specific structures and dependencies that extend far beyond its immediate context window.", "questions": "The proposed model is several times larger than the baselines (Caduceus and HyenaDNA), what is the FLOPS of the proposed method?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n### Novelty\n\nThe paper's core contribution, the TM loss, is an original and intuitive idea. It elegantly integrates a classic concept from bioinformatics (Markov models and n-gram statistics) as a regularization prior into a modern Transformer architecture. This approach provides an explicit mechanism to guide the model toward learning global sequence properties, directly addressing the well-known difficulty of capturing long-range dependencies in genomics.\n\n## Weaknesses\n\n### Questionable Efficacy of TM Loss on NT Benchmark\n\nWhile the authors are to be commended for conducting extensive experiments, the effectiveness of the TM loss is not consistently demonstrated across all benchmarks. On the 18 Nucleotide Transformer tasks, the performance difference between CARMANIA and the baseline without TM loss is often marginal or even negative.\n\nFor instance, the model with TM loss performs worse on the H3K4me3 histone modification task ($0.565$ vs. $0.586$). While it performs better on Splice Site Donor prediction ($0.973$ vs. $0.921$), many other tasks show negligible differences. The most significant outlier is the H4ac histone acetylation task, where adding the TM loss leads to a dramatic MCC improvement from $0.197$ to $0.606$. While the paper offers a brief explanation, this is not entirely convincing, as other methods in the comparison do not exhibit similar performance volatility on this task. This substantial gain warrants a more detailed analysis to clarify the underlying mechanism and ensure it is not an artifact. Overall, these results from a major benchmark seem to conflict with the paper's broader claim that the TM loss boosts accuracy in 33 of 40 tasks and weaken the argument for its general applicability.\n\n### Clarity and Reporting of Experimental Details\n\nThe paper's clarity suffers from incomplete reporting of experimental details, making some results difficult to interpret fully. Table 1, which presents a key ablation study, is a primary example of this issue.\n\n- **Ambiguous Metrics**: The table fails to specify which datasets or tasks were used to compute the \"F1 Macro\" and \"BLEU\" scores. The reader must search the main text to infer that these likely correspond to an Antimicrobial Resistance (AMR) task and a human genome benchmark, respectively. This context is essential and should be included in the table's caption.\n- **Relative vs. Absolute Cost**: The table reports \"Relative FLOPs,\" which only shows the proportional cost increase of full attention. It does not provide the absolute FLOPs of the proposed method, making it impossible to gauge the actual computational budget.", "was_revised": true, "final_justification": "> Why 3 initial rating? \n\nThe sliding-window attention has been widely used in various sparse attention implementations, while the effectiveness of TM loss is not clearly validated in the experiments. \n\n> Why 4 final rating? \n\nThe additional  KL divergence experiments have successfully addressed my concerns in the effectiveness of the TM loss. \n\n> Why not a higher score? \n\nThe limited contribution of sliding-window attention is major concern. \nThe results, while promising, does not make a significant difference. \n\n> Why not a lower score? \n\nThe TM loss can be an important asset to the community.", "rebuttal": "## Reviewer 6bow\n\nWe appreciate Reviewer *6bow*’s alignment with the other reviewers in recognizing the paper’s originality and novelty, and their thoughtful articulation of how the TM loss bridges classic bioinformatics concepts with modern transformers to address long-range dependencies in genomics.\n\n### Variable Impact of TM Loss Across Datasets\n\nThe benefit of the TM loss isn’t uniform across all genomics tasks. Sometimes standard next‑token (NT) training captures everything the model needs, and other times it falls short. NT loss alone suffices when the classes already differ strongly in their local bigram vocabulary, because the transformer’s fixed context window can learn those distinctive motifs directly. TM loss becomes necessary when classes look nearly identical at the bigram level, since it injects a global co‑occurrence signal that NT can’t provide.\n\nTo confirm this, we computed the KL divergence between class‑conditional bigram histograms for Enhancers, Splice‑Site Donors, and H4ac datasets, where the impact of TM loss is different. As shown below, we find high KL for Enhancers (9.28) but much lower values for Splice‑Site Donors (0.99) and H4ac (0.64), and we see that TM-driven performance gains track precisely with lower KL.\n\n**Table: TM loss gain vs. bigram KL divergence**  \nTasks with lower KL benefit more from TM, reflecting its strength in capturing global patterns when local signals are weak.\n\n| Task              | Class Conditional Bigrams' Mean KL (Pos vs. Neg) | Δ (CARMANIA - Baseline) % |\n|-------------------|--------------------------------------------------|-----------------------------|\n| Enhancer          |               9.28                                             | 0.0%                        |\n| Splice Site Donor |           0.99                                             | +5.2% ↑                     |\n| H4ac              |                 0.64                                             | +40.9% ↑                    |\n\nBiologically, this matches our understanding that enhancer regions have clear, unique motifs that NT loss can easily pick up, while splice sites rely on short, scattered patterns. In contrast, H4 acetylation involves several sites on the H4 tail (K5, K8, K12, K16) that act in combination to mark open chromatin. Instead of strong individual motifs, these sites are linked to subtle and spread-out sequence patterns [1], which makes TM’s global view especially helpful.\n\nIn short, NT loss excels when “what comes next” is already class‑skewed, while TM loss becomes crucial when “how local pieces team up across the sequence” carries the real information.\n\nAlso, as input sequences get longer, fixed-size context models capture less of the overall pattern, making it harder for next-token loss to learn useful signals. Therefore, the longer the sequence, the more important the TM loss becomes for capturing global patterns that local windows miss.This trend is reflected in our results: we observe substantial improvements on long-sequence tasks such as **biosynthetic gene cluster classification (100k)** in *Table 4* and **gene class prediction (4k)** in *Table 5*. We also highlight this in *Table 13* of the Supplementary Material, where increasing the input context length leads to higher accuracy on the same AMR task.\n\nNote that in the paper we aimed for thorough validation across diverse datasets. While TM loss provides varying gains, it consistently performs on par with SOTA. In practice, users can assess its utility by testing whether class-conditional n‑gram distributions differ significantly, or by using biological priors to judge if the signal is driven by a single strong motif (high KL) or multiple redundant ones.\n\nWe will add these insights to the final version of the paper.\n\n> [1] Dion et al. *Genomic characterization reveals a simple histone H4 acetylation code.* Proceedings of the National Academy of Sciences.\n\n---\n\n### Clarity in Table 1\n\nWe updated the table caption to clearly indicate the corresponding tasks and datasets used for each metric.\n\n**Updated Table 1 Caption:**  \n>Ablation study showing the impact of TM loss and attention type on model performance (F1 Macro, BLEU) and compute cost. All models are trained on the *Scorpio Gene-Taxa* dataset. F1 Macro is evaluated on the *Antimicrobial Resistance (AMR)* classification tasks, while BLEU and Relative FLOPs are computed on the *Human Genome-Short* dataset using 10,000 fragments from the GRCh38/hg38 reference genome.\n\n---\n\n### Relative vs. Absolute Computational Cost\n\nWe calculated the absolute FLOPs and will include them in the paper along with the relative FLOPs.\n\n**Table: Absolute and Relative FLOPs per Sequence for Attention Types (Length = 10,000)**\n\n| Attention Type         | FLOPs                          | Relative FLOPs |\n|------------------------|--------------------------------|----------------|\n| Full Attention         | 26.5 × 10¹²                    | 1.58           |\n| Sliding Window         | 16.7 × 10¹²                    | 1.00           |"}, {"review_id": "9z3ERGxq7j", "reviewer": "Reviewer_Db4V", "summary": "This paper proposes a novel loss function that leverages a transition matrix to better supervise the model. Additionally, it introduces Sliding Window Attention to the transformer architecture, which not only incorporates inductive bias into the model but also theoretically reduces computational complexity. Compared to previous approaches, the proposed method demonstrates performance improvements on certain downstream tasks.", "questions": "None", "limitations": "The experimental results in this paper appear generally weak, failing to surpass state-of-the-art (SoTA) performance on a considerable number of tasks. Additionally, there is no experimental evidence demonstrating that the proposed TM Loss can actually improve the performance of current SoTA methods. The study also lacks comparative experiments with ConvNova.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths:​​ From a biological perspective, the transition-matrix approach is theoretically sound. Additionally, the inductive bias introduced by Sliding Window Attention has been empirically validated to some extent by previous work ConvNova [1].\n\n​Weaknesses:​​ The paper lacks comparison with recent methods like ConvNova [1]. Moreover, the results appear relatively weak compared to existing approaches. I suggest the authors consider further improving their implementation to enhance model performance, potentially by incorporating TM Loss into current methods. It would be more convincing to conduct experiments on established high-performance baselines such as Caduceus or ConvNova - significant performance improvements on these platforms would substantially strengthen the paper's claims.\n\n[1] REVISITING CONVOLUTION ARCHITECTURE IN THE REALM OF DNA FOUNDATION MODELS. ICLR 2025", "was_revised": true, "final_justification": "extensive experiment solved my concern.", "rebuttal": "## Reviewer Db4V\n\nWe appreciate Reviewer *Db4V*’s alignment with the other reviewers in recognizing the paper’s originality and novelty, as well as their acknowledgment that TM loss can be applied to any high-performing model and potentially improve performance beyond the specific backbone we introduced.\n\n---\n\n**Comparison with ConvNova**\n\nConvNova is a promising and very recent model. At the time of our rebuttal, pretrained weights for ConvNova remain unreleased. To enable a direct comparison, we followed their evaluation setup (5 random seeds on a fixed test set) and benchmarked CARMANIA accordingly. As shown in the table below, CARMANIA performs on par with ConvNova. It outperforms ConvNova on three tasks, ranks second-best on several others, and overall demonstrates competitive performance.\n\nWe will include this comparison in the final version of the paper.\n\n**Table: Genomic Benchmark Results**  \nTop-1 accuracy (↑) is reported for pretrained HyenaDNA, Caduceus-Ph, ConvNova, and the CNN baseline. The best result is in **bold**, second-best is _underlined_.\n\n| Task                         | CNN            | HyenaDNA       | Caduceus-Ph     | ConvNova         | CARMANIA         |\n|------------------------------|----------------|----------------|------------------|------------------|------------------|\n| **Enhancers**               |                |                |                  |                  |                  |\n| Mouse Enhancers             | 0.730 ± 0.032  | 0.779 ± 0.013  | 0.754 ± 0.074    | _0.784 ± 0.009_  | **0.795 ± 0.002** |\n| Human Enhancers Cohn        | 0.702 ± 0.021  | 0.718 ± 0.008  | **0.747 ± 0.004**| _0.743 ± 0.005_  | 0.725 ± 0.002     |\n| Human Enhancers Ensembl     | 0.744 ± 0.122  | 0.832 ± 0.006  | 0.893 ± 0.008    | **0.900 ± 0.004**| _0.895 ± 0.002_   |\n| **Species Classification**  |                |                |                  |                  |                  |\n| Coding vs. Intergenic       | 0.892 ± 0.008  | 0.904 ± 0.008  | 0.915 ± 0.003    | **0.943 ± 0.001**| _0.930 ± 0.001_   |\n| Human vs. Worm              | 0.942 ± 0.002  | 0.961 ± 0.002  | **0.973 ± 0.001**| 0.967 ± 0.002     | _0.970 ± 0.005_   |\n| **Regulatory Elements**     |                |                |                  |                  |                  |\n| Human Regulatory            | 0.872 ± 0.005  | 0.862 ± 0.004  | 0.872 ± 0.011    | _0.873 ± 0.002_  | **0.894 ± 0.002** |\n| Human Non-TATA Promoters    | 0.861 ± 0.009  | 0.887 ± 0.005  | 0.946 ± 0.007    | _0.951 ± 0.003_  | **0.965 ± 0.002** |\n| Human OCR Ensembl           | 0.698 ± 0.013  | 0.744 ± 0.019  | **0.828 ± 0.006**| _0.793 ± 0.004_  | 0.778 ± 0.002     |\n\n---\n\n**Performance**\n\nIn our experiments, we aimed for thorough and transparent validation across diverse datasets. While our method often performs on par with state-of-the-art approaches, it yields substantial gains on several challenging tasks (particularly in long-sequence settings and functionally meaningful categories). These include regulatory element prediction (e.g., enhancer and enhancer type) and functional classification tasks (e.g., biosynthetic gene cluster classification, gene class prediction, gene family classification).\n\nThe model’s strength arises from several architectural choices: the use of a LLaMA-based backbone, support for long-context modeling, the incorporation of TM Loss, and a relatively wide (but shallow) architecture. These design elements enable the model to effectively capture global dependencies, long-range sequence interactions, class-conditional n-gram divergences, and the co-occurrence patterns of biologically relevant motifs.\n\nAs shown in Table **1**, **CARMANIA** achieves substantial performance gains on 11 particularly challenging tasks, with improvements ranging from +1.8% to +33.2%. An additional 9 tasks show moderate improvements of up to 1.5%. For the remaining tasks, **CARMANIA** performs comparably to the state-of-the-art (SOTA), noting that no single model consistently outperforms others across all tasks. These consistent and meaningful gains demonstrate the robustness and generalizability of our approach.\n\nAs the reviewer suggested, TM loss can be applied and could benefit beyond the proposed architecture to evolving new architectures such as the recently proposed ConNova model.\n\n\n**Table: CARMANIA vs. Second-Best Performance Across Tasks**\n\n| Task                     | CARMANIA (%) | Second Best Method | Second Best (%) | Δ (%) |\n|--------------------------|--------------|---------------------|------------------|------------|\n| ENHANCER                | 88.0         | NT-V2               | 54.8            | +33.2      |\n| ENHANCER Type           | 72.4         | Caduceus-PS         | 43.9            | +28.5      |\n| BGC                     | 48.4         | HyenaDNA            | 41.2            | +7.2       |\n| GeneFamily (AMR)             | 73.3         | MetaBERTa           | 65.0            | +8.3       |\n| Resistance-Mechanism (AMR)     | 97.5         | MetaBERTa           | 89.8            | +7.7       |\n| Gene-Class  (Scorpio-gene-taxa)            | 90.9         | HyenaDNA            | 84.6            | +6.3       |\n| Drug-Class  (AMR)             | 94.2         | MetaBERTa           | 88.6            | +5.6       |\n| Gene-Class(held-out) (Scorpio-gene-taxa)  | 72.8         | HyenaDNA            | 67.4            | +5.4       |\n| Human Regulatory        | 89.5         | Caduceus-PS         | 87.3            | +2.2       |\n| Coding vs. Intergenic   | 93.5         | Caduceus-PH         | 91.5            | +2.0       |\n| Human NonTATA Promoters | 96.3         | Caduceus-PS         | 94.5            | +1.8       |\n\n---"}, {"review_id": "1sfajBdLU5", "reviewer": "Reviewer_eyPb", "summary": "This paper proposed a self-supervised pre-training framework, CARMANIA (Context-Aware Regularization and Markov Integration for Attention-Based Nucleotide Analysis), aimed at improving nucleotide sequence modeling, especially for genomic tasks. The approach combines sliding-window attention with a novel transition matrix (TM) loss to capture global dependencies in nucleotide sequences, overcoming the limitations of traditional attention mechanisms in long sequence modeling. By combining Markov transition dynamics, CARMANIA helps the model learn organism-specific sequence structures and improves prediction performance for a variety of genomic tasks, such as regulatory element prediction, antimicrobial resistance detection, and biosynthetic gene cluster classification.", "questions": "### Questions:\n\n- Transferability of TM Loss: While the paper focuses on genomic data, could the TM loss approach be applied to other types of biological sequences like protein?\n- Handling Rare Genomic Events: Given that genomic data often includes rare events or underrepresented patterns, how well does CARMANIA handle these rare cases, particularly in low-signal regions of the genome?", "limitations": "## Limitations:\n\n- Computational Cost: The paper mentions computational overhead, particularly related to the calculation of the Transition Matrix loss, which could become a bottleneck when scaling to large datasets. While the use of sliding-window attention improves efficiency, there may still be limitations on large-scale applications.\n- Lack of Real-World Validation: Although CARMANIA performs well in experiments, it has not been validated in actual biological experiments.\n- Task-Specific Tuning: The paper discusses the β hyperparameter (used to adjust the weight of the TM loss), which works well for general tasks. However, tasks involving rare, biologically significant motifs may require more specific tuning. The paper does not go into detail on how to fine-tune for such cases.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n- Originality: This paper introduces CARMANIA, a self-supervised pretraining framework that combines sliding-window attention with a novel Transition Matrix (TM) loss to enhance the modeling of global dependencies in nucleotide sequences.\n- Significance: CARMANIA tackles challenges in genomic sequence analysis, particularly with long sequences, and enhances the model's predictive power for tasks like gene function prediction and detecting antimicrobial resistance.\n- Performance Improvement: Through experiments, the model shows a notable increase in prediction accuracy across various genomic tasks, with a 34% improvement in the Matthews correlation coefficient (MCC) for regulatory element prediction.\n- Efficiency: The method is computationally efficient, reducing both training and inference times for long sequences. This results in faster processing while maintaining improved performance over other models.\n\n### Weaknesses:\n\n- Technical Complexity: The combination of Transition Matrix loss (TM Loss) and sliding-window attention could be difficult for readers who aren't familiar with advanced methods in genomics. More explanation on how these components work together could help clarify the approach.\n- Limited Experimentation: While CARMANIA shows promising results on several benchmark tasks, the paper does not include validation from real-world experiments, which would be important for confirming its practical value in biological research.\n- Model Size: Despite its strong performance, CARMANIA requires more memory and computational resources compared to some other models, especially when working with large-scale genomic data.", "was_revised": true, "final_justification": "I am not an expert in the area of Nucleotide Analysis. The method and experiments seem solid and useful.", "rebuttal": "## Reviewer eyPb\n\nWe appreciate Reviewer *eyPb*’s recognition of our method’s gains in prediction accuracy, impact on long genomic sequences, and computational efficiency, alongside their agreement with other reviewers on its novelty and originality.\n\n---\n\n**Clarifying TM Loss and SWA Integration**\n\nThe transition matrix (TM) loss and sliding window attention (SWA) introduce two enhancements to the standard transformer objective of predicting the next token. TM loss aligns the model’s predicted first-order transition probabilities with those computed from the full sequence, ensuring that its outputs reflect the genome’s global token-transition statistics. SWA limits each token’s receptive field to a fixed window of its predecessors, reducing attention complexity from $O(n^2)$ to $O(kn)$ and focusing the model on biologically meaningful local patterns. In short, SWA reduces computation by attending only to local tokens, while TM loss keeps it consistent with global statistics.\n\nWe revised the introduction in the final version to clarify this.\n\n---\n\n**Real-World Experimentation**\n\nWe agree that validating models in real-world biological contexts is essential. Accordingly, all our evaluations are based on biologically curated, real-world datasets. CARMANIA was fine-tuned and tested on five diverse datasets spanning over 40 tasks—including enhancer prediction, splice site detection, antimicrobial resistance classification, gene–taxa mapping, and biosynthetic gene cluster annotation.\n\nThese tasks were selected to capture practical genomic challenges, such as rare regulatory events, long-context dependencies, and taxonomic diversity. All datasets come from trusted, expert-curated sources (e.g., MiBiG, MEGARes, CARD, NCBI Genome [1–5]) and were carefully split to support meaningful generalization.\n\nAdditional details are provided in Supplementary Material A.9, and we will more clearly highlight the real-world nature of these datasets in the revised manuscript.\n\n**References:**\n\n1. Terlouw et al. *MIBiG 3.0: a community-driven effort to annotate experimentally validated biosynthetic gene clusters.* Nucleic Acids Research  \n2. Lakin et al. *MEGARes: an antimicrobial resistance database for high-throughput sequencing.* Nucleic Acids Research  \n3. Alcock et al. *CARD 2020: antibiotic resistome surveillance with the comprehensive antibiotic resistance database.* Nucleic Acids Research  \n4. Tho Hoan Pham et al. *Qualitatively predicting acetylation and methylation areas in DNA sequences.* Genome Informatics  \n5. Grešová et al. *Genomic benchmarks: a collection of datasets for genomic sequence classification.* BMC Genomic Data\n\n---\n\n**Model Size**\n\nWhile transformer-based models generally require more memory than convolutional architectures, **CARMANIA** was specifically designed to address this limitation. It retains the expressive power and modeling flexibility of transformers while incorporating architectural efficiencies that significantly reduce computational overhead.\n\nAs demonstrated in our experiments, **CARMANIA** achieves inference speeds within 2.5× of state-space models—even on very long genomic sequences. To the best of our knowledge, it is the longest-context transformer-based genomic language model to date, capable of processing sequences exceeding 160,000 base pairs.\n\nCompared to earlier transformer models such as DNABERT and Nucleotide Transformer—which support shorter contexts (~1,000 bp) and exceed 2.5 billion parameter **CARMANIA** provides a more scalable and practical solution for large-scale genomic analysis.\n\n---\n\n**Rare Genomic Events**\n\nCARMANIA handles rare events and low-signal regions robustly. In benchmarks for histone-modification prediction, enhancer classification, and splice-site detection—where signals are sparse and motifs weak CARMANIA outperforms existing models by learning subtle patterns across the genome.\n\n---\n\n**Enhancing Protein Modeling With TM Loss**\n\nIn theory, TM loss should benefit any task that gains from longer‐range context. To confirm this, we tested its effectiveness on protein sequences. We adapted the CARMANIA architecture for amino acid data and trained it on the Scorpio-Gene-Taxa dataset. We constructed a 20×20 bigram frequency matrix capturing all pairwise standard amino-acid statistics, and we integrated TM loss into the training objective.\n\nAs shown below, incorporating TM loss consistently accelerated convergence and improved predictive accuracy, confirming its transferability to protein‐sequence modeling.\n\n**Table: Performance Comparison on the Scorpio-Gene-Taxa Dataset Using Protein Sequences**\n\n|                      | **Test Set**: Phylum | Class | Order | Family | Gene | **Gene Held-Out**: Phylum | Class | Order | Family | **Taxa Held-Out**: Gene |\n|----------------------|----------------------|-------|-------|--------|------|---------------------------|--------|--------|--------|--------------------------|\n| Without TM Loss      | 91.1                 | 84.9  | 70.6  | 51.3   | 99.5 | 19.8                      | 10.9   | 3.3    | 1.1    | 98.4                     |\n| With TM Loss         | **91.4**             | **85.5**| **71.0** | **51.5** | **99.6** | **20.5**                | **11.6** | **3.8** | **1.5** | **98.7**                 |\n\n---\n\n**TM Loss Overhead**\n\nThe TM loss overhead remains negligible in practice. For example, in our experiments, training on a dataset of 580 million tokens with an A100 GPU (batch size 35, sequence length 4000) showed only a slight increase of **~3%** in training time per epoch (3h 28m vs. 3h 38m).\n\n---\n\n**Finetuning $\\beta$**\n\nIt’s true that fine-tuning $\\beta$ in CARMANIA could improve performance for specific tasks. We opted for a fixed $\\beta$ to ensure fairness in general-purpose pretraining. Notably, even without tuning, our model performs strongly on challenging tasks such as histone marks, enhancers, and promoters."}]

Author rebuttal:

Meta-review: This paper introduces CARMANIA (Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis), a self-supervised pretraining framework that augments next-token (NT) prediction with a transition-matrix (TM) loss. Reviewers recognize the contribution of TM loss. Reviewers proposed questions including experiments, comparison, and clarity pre rebuttal. Most concerns are resolved during rebuttal and reviewers agreed on a borderline accept score. AC recommends acceptance.
Decision: Accept (poster)