JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model
Authors: Qihao Duan
Bingding Huang
Zhenqiao Song
Irina Lehmann
Lei Gu
Roland Eils
Benjamin Wild
Venue: NeurIPS 2025 poster ()
Keywords: genomics
foundation model
hybrid architecture
learning efficiency
Primary area: machine_learning_for_sciences
TL;DR: JanusDNA, the first bidirectional DNA foundation model built upon a novel pretraining paradigm, integrating the optimization efficiency of autoregressive modeling with the bidirectional comprehension capability of masked modeling.

Abstract: Large language models (LLMs) have revolutionized natural language processing and are increasingly applied to other sequential data types, including genetic sequences. However, adapting LLMs to genetics presents significant challenges. Capturing complex genomic interactions requires modeling long-range global dependencies within DNA sequences, where interactions often span over 10,000 base pairs, even within a single gene. This poses substantial computational demands under conventional model architectures and training paradigms. Additionally, traditional LLM training approaches are suboptimal for DNA sequences: autoregressive training, while efficient for training, only supports unidirectional sequence understanding. However, DNA is inherently bidirectional. For instance, bidirectional promoters regulate gene expression in both directions and govern approximately 11% of human gene expression. Masked language models (MLMs) enable bidirectional understanding. However, they are inefficient since only masked tokens contribute to loss calculations at each training step. To address these limitations, we introduce JanusDNA, the first bidirectional DNA foundation model built upon a novel pretraining paradigm, integrating the optimization efficiency of autoregressive modeling with the bidirectional comprehension capability of masked modeling. JanusDNA's architecture leverages a Mamba-Attention Mixture-of-Experts (MoE) design, combining the global, high-resolution context awareness of attention mechanisms with the efficient sequential representation learning capabilities of Mamba. The MoE layers further enhance the model's capacity through sparse parameter scaling, while maintaining manageable computational costs. Notably, JanusDNA can process up to 1 million base pairs at single-nucleotide resolution on a single 80GB GPU using its hybrid architecture. Extensive experiments and ablation studies demonstrate that JanusDNA achieves new state-of-the-art performance on three genomic representation benchmarks. Remarkably, JanusDNA surpasses models with 250x more activated parameters, underscoring its efficiency and effectiveness. Code available at https://anonymous.4open.science/r/JanusDNA/.

JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model
Qihao Duan 145,, Bingding Huang", Zhenqiao Song'.
Irina Lehmann', Lei Gu Roland Ei1. Benjamin Wild"
Berlin Institute of Health, Charité Universitätsmedizin Berlin
Colleee of Big Data and Internet, Shenzhen Technology University
2I Language Technologies Institute, Carnegie Mellon University
Epigeneticc Laboratory, Max Planck Institute for Heart and Lung Research Department of Mathematics and Computer Science, Freie Universität Berlin 'Health Data Science Unit, Heidelberg University Hospital and BioQuant 'Intelligent Medicine Institute, Fudan University
*Corresponding authors
{qihao. duan, irina. lehmann, roland. eils, Denjamin. wild)@bih-charrde huangbingding@szzu. edu. cn zhenqias@andrew cmu. edu lei. ler--gummpppi- wwtpppiiib.mmmd....... mpg.de
Abstract
Large language models (LLMs) have revolutionized natural language processing and are increasingly applied to other sequential data types, including genetic se- quences. However, adapting LLMs to geneties presents significant challenges. Capturing complex genomic interactions requires modeling long-range global de- pendencies within DNA sequences, where interactions often span over 10,000 base pairs, even within a single gene. This poses substantial computational demands under conventional model architectures and training paradigms. Additionally, traditional LLM training approaches are suboptimal for DNA sequences: autore- gressive training, while efficient for training, only supports unidirectional sequence understanding. However, DNA is inherently bidirectional. For instance, bidirec- tional promoters regulate gene expression both directions and govern approxi- mately 11 % of human gene expression. Masked language models (MLMs) enable bidirectional understanding. However, they are inefficient since only masked tokens contribute loss calculations each training step. To address these limitations, we introduce JanusDNA, the first bidirectional DNA foundation model built upon novel pretraining paradigm, integrating the optimization effi ciency of autoregressive modeling with the bidirectional comprehension capability of masked modeling. JanusDNA's architecture leverages Mamba-Attention Mixture-of-Experts (MoE) design, combining the global, high-resolution context awareness of attention mechanisms with the efficient sequential representation learning capabilities of Mamba. The MoE layers further enhance the model's 's capacity through sparse parameter scaling, while maintaining manageable com- putational costs. Notably, JanusDNA can process to 1 million base pairs single-nucleotide resolution on a single 80GB GPU using its hybrid ar- chitecture. Extensive experiments and ablation studies demonstrate that Janus- DNA achieves new state-of of-te-of-thh-art art performance on three genomic representation benchmarks. Remarkably, JanusDNA surpasses models with 250x more acti- vated parameters, underscoring its efficiency and effectiveness. Code available btttss//ggihhie com Saaem/Qiaao--uaae DaniaaoDDDaa///auDDAA JanusDNA
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Modeling the language of DNA is crucial for investigating its biological function, offering potential advancements in understanding genotype-phenotype associations, disease diagnosis, and new drug development I. Large language models (LLMs) have demonstrated remarkable success in large- scale nonlinear representation learning for natural language processing (NLP) tasks, such as text generation, translation, and summarization 2 3 4]. This success has inspired researchers to explore LLM applications in other domains [5] 6) including bioinformatics [7]. However, applying general LLMs directly to DNA sequence data is challenging. DNA sequences are represented as series of nucleotides, lacking clear semantic meaning and posing challenges for interpretation by L.LMs. Additionally, non-coding regions in DNA can be remotely related to coding regions both upstream and downstream, necessitating models capable of processing long-range dependencies while maintaining bidirectional understanding. These factors make it challenging to apply LLMs directly to DNA sequence data without significant modifications or adaptations [8]. Some models have been developed specifically for DNA sequence representation [9] 10 I1]. However, these models still face some limitations.
Current limitations (1) Limited Sequence Length and Low Resolution: Capturing complex genomic interactions requires modeling long-range dependencies within DNA sequences, where interactions can span over 10,000 base pairs even within a single gene [1]. This necessitates a model that effectively handle long-range dependencies and relationships within the sequence. However, many current models solely rely on global attention mechanisms inspired by their superior success in natural language applications [13 14). yet they often struggle to effectively process long genomic sequences and uncover meaningful long-range interactions [I]] K-mer tokenization is However, this method introduces a trade-off' between sequence length and resolution [15]. potentially leading to the loss of crucial information, especially cases where single nucleotide polymorphisms (SNPs) are essential for understanding gene function. (2) Unidirectional Understanding: Many genomic processes are influenced by bidirectional interactions, with essential regulatory elements located both upstream and downstream of key genomic regions. For example, bidirectional promoters initiate transcription in both orientations [66 17.. Additionally, intergenic enhancers transcribed predominantly bidirectionally often function as weak promoters in both directions. Conversely, for elements with unidirectional transcription (both enhancers and promoters), transcription direction typically correlates with the orientation in which the element functions Accurately modeling these bidirectional interactions is crucial for understanding genomic function and making precise predictions, imposing significant demands on model architecture and training strategies [8. However, some existing decoder-only models based on State Space Models (SSMs) [I5 19 20 are primarily unidirectional or limited in their capacity effectively understand bidirectional context, thus constraining their ability to comprchensively capture these complex interactions. (3) Low Training Efficiency: Long-range modeling and bidirectional understanding of DNA sequences both require substantial computational resources and memory, especially for those requiring attention for a more global representation. Therefore, training efficiency significantly impacts the model's s performance, especially under limited computational resources. Most bidirectional models are trained using masked training paradigms, such as BERT (21), which utilize only a small fraction of tokens (typically I 15%) for loss calculation at each step. This limitation can hinder the model's ability to learn effectively from the entire sequence within limited training steps, thereby requiring more training epochs to adequately cover the full training data.
frequently used to expand the context window of DNA sequence models
promoter in vivo [18.
Additionally. the masking process itself introduces extra computational overhead. In contrast, autoregressive (next-token prediction) training is more efficient, as nearly all tokens contribute to the loss at each training step, allowing the model learn more effectively within fixed number of steps as sequence length increases 222. However, it's important to note that autoregressive models are inherently unidirectional, limiting their ability incorporate bidirectional context.
In response to the aforementioned issues, we introduce JanusDNAF! the first bidirectional DNA foundation model built upon a novel pretraining paradigm. Our architecture employs two principal strategies: (1) Hybrid Architecture: To achieve powerful global understanding while maintaining
Janus, the ancient Roman god of transitions and duality, is symbolized by two faces gazing in opposite directions.

---

computational efficiency for long contexts, we integrate the strengths of state-space models (SSMs) 23 and Mfxture-of-Expeee (MoE) designs [24 25| into attention mechanisms (26], enabling the model to effectively capture long-range global dependencies and complex interactions within DNA sequences. (2) Ridirectional Efficient Training: While preserving the bidirectional understanding of DNA sequences typically achieved through masked training, we significantly improve learning efficiency by computing the loss over all tokens in each training, same as in autoregressive modeling. Notably, JanusDNA is capable of processing up to 1 million base pairs at resolution with global atfention on single 80GB GPU, making suitable for large-scale under- standing in genomic research. We evaluated JanusDNA on 35 diverse genomic tasks showcase its superior global understanding as well as long-range representation ability.
In summary, our contributions are as follows:
We propose JanusDNA, novel bidirectional DNA foundation model capable of capturing global long-range dependencies and interactions at single-nucleotide resolution.
We introduce efficient Hybrid processing ultra-long genomic sequences within practical computational budgets.
architecture designed for
We present Janus Modeling, novel and efficient pretraining paradigm that effectively combines the strengths autoregressive and masked modeling. facilitating effective global bidirectional learning.
We demonstrate state-of-the-art performance across diverse genomic benchmarks, outper- forming significantly larger models. In particular, JanusDNA significantly surpasses the expert model Enformer eQTL prediction tasks, despite having far fewer parameters.
2 Preliminary and Related Work
2.1 Large Language Model Pretraining Paradigms
Autoregressive Language Modeling (ALM) is generative pretraining paradigm in which the model predicts the next token in sequence given all previous tokens. Trained on large corpora, the model learns statistical properties to generate coherent text. The training objective
LCLM Σ log P( 22...., 21-1),
(I)
where T is the sequence length, and 24 denotes the token at position t. The model generates text by sampling from the learned probability distribution over the vocabulary at each time step 15 19 20 227. Each token contributes to overall loss, and the model minimizes the average loss across all tokens. However, to maximize generative performance (28), ALM is unidirectional, causing a limited ability to model bidirectional contexts, which crucial for DNA sequence understanding
29
Masked Language Modeling (MLM) is non-causal pretraining paradigm where the model predicts masked tokens in a sequence using surrounding context. The training objective is:
LMEM Σ log P(za/1gj Tja............................ Tix):
where N is the total number of tokens, 2; is the masked token, and Ty .., Ije are the un- masked tokens. This approach enables the model to learn bidirectional representations, capturing dependencies in both directions [I 28 29
10 30 31. However, MLMs mostly follow the BERT-style training paradigm, which masks fixed percentage of tokens in the input sequence, e.g., 15% 8 9 [O]. This can lead to inefficiencies, as only a small fraction of the data is used for loss computation during each iteration. In contrast, autoregressive training paradigms take advantage of nearly the entire data, significantly improving training efficiency and overall performance.
A detailed review of related work DNA language models is provided in Appendix A
3

---

(A) DNA
(C) JanuaDNA Model
(D) Mixture of Experts
Strend Right Left
Staadd A Right
MOV. brbddees
00000
Stand
DOGK
PT
200
Marto-Thii Laper
Strand B
00000
Double-stranded Modeling
Steand A AAAL AALLERRRS
Straed
Siceeti bat: Laye!
MICE
VICE
Cwwbdd
(E) Bidirectional Fusion Mask
Nambo
Marba
Erteeid Baciward Enoiediis 07
AT
XT X2 mf
Ougus
Feature Fuston
Brand
Enbeddinn
head
Head
Head
Madi-teak Theett
Befrectoon
Figure 1: The JanusDNA Architecture for Bidirectional DNA Modeling, JanusDNA employs a hierarchical bidirectional strategy to comprehensively model DNA sequences. (A) DNA, with its inherent double-stranded nature. (B) The model processes both the forward and reverse complement strands independently in parallel to capture complete biological context, with their embeddings subsequently combined for downstream tasks. (C) The core JanusDNA model architecture processes single input strand using parallel left-to-right and right-to-left pathways. Each pathway consists of Mamba-FFF and Mamba-Mon MoEE layers for effective and efficient sequential encoding. (D) The MoE architecture enhances model capacity and specialization by dynamically and sparsely routing inputs to a subset of expert networks, enabling efficient computation and improved representation learning. (E) The Bidirectional Global Fusion mechanism, utilizing specific attention mask, integrates the forward and backward representations from (C) to ensure that each nucleotide's ' embedding is informed by its complete sequence context.
JanusDNA
We propose Janus modeling, an efficient bidirectional training method with global attention, and JanusDNA, powerful hybrid DNA foundation model.
As illustrated Figure [. JanusDNA processes bidirectional DNA sequences from both left-to-right and right-to-left using two independent stacks of Mamba and Mixture-of- (MoE) layers. These stacks generate forward and backward representations independently, ensuring no information leakage. The two representations are then fused create unified representation that encapsulates bidirectional information. Each token position is predicted based on all upstream and downstream tokens. The following sections detail the efficient bidirectional training method with global attention Janus Modeling, and the hybrid Mixture-of- (MoF) architecture JanusDNA.
3.1 Bidirectional Efficient Training
conventional pretraining paradigms face trade-off: Masked Language Models (MLMs) offer bidirectional understanding but suffer from low training efficiency due to sparse loss signals, especially for those requiring global attention, while Autoregressive Models are efficient in training but inherently unidirectional. To overcome this, we introduce Janus modeling, a novel pretraining objective designed to achieve efficient, fully bidirectional sequence understanding with global attention, as illustrated in Figure 2
As discussed in Section 2
The core idea of Janus modeling is to predict every token in sequence of length T using its complete bidirectional context, i.e., all tokens preceding Ia (01,...,,I.-) and all tokens succeeding Ie (Iz+ The training objective is therefore:
T
Eridirectional
log
I, Ittly...,,y

---

(A) Autoregressive Modeling
(B) Masked Modeling
(C) Janus Modeling
Figure 2: Modeling Interpretation. Janus modeling treats each token target for loss calculation, enabling higher training efficiency compared to masked modeling by full sequence learning while keeping bidirectional context understanding.
This objective ensures that every token contributes to the loss, maximizing training efficiency, while simultaneously demanding bidirectional comprehension.
To realize this objective, Janus modeling employs a two-stage process: independent context encoding followed by global fusion mechanism:
The input sequence (I) (1...., IT) is processed by two parallel and independent stacks of layers (detailed in Section
Independent Context Encoding:
A forward pass processes the sequence from left-to-right, generating sequence of hidden Each hidden state H, is function of the input tokens (Z),...; and primarily captures information about the left context of xt.
states Rivd = (Hf .I...... Hf
H ForwardEncode((z).....
A backward pass processes the sequence from right-to-left, generating a sequence of hidden states Rowd S H .... H Each hidden state H is function of the input tokens (IT, Tt) and primarily captures information about the right context of Iz.
H BackwardEEnodderrryy.....
(5)
These two sets of representations, Riwd and Rowd. are generated independently, ensuring no premature information leakage between past and future contexts before the explicit fusion step. The entire model is trained end-to-end using Lnidirectional from Equation
To compute P(z, (1 .1...,, 2t- Ze+1..., Tr) for each x; as per Equation the left-context information captured in Riwa and the right-context information captured in Rewd must be integrated. This is organically achieved via global attention mechanism, specifically implemented with FlexAttention (26) for efficiency. The representations from both passes, R = H are concatenated form combined input sequence
Bidirectional Global Fusion:
(Iff ...., Hff and Read (H for the attention layer: Rimpur H
H. H ..... H This Pamput sequence has length of 27. The core of the fusion lies in a carefully designed attention mask, MMj. which dictates how tokens in Rappur
attend each other. This mask ensures that the prediction for xa is based only on H. for l < t and H. for j t, preventing information leakage. The mask, also illustrated in Figure [E), is defined as:
9idx > kujdx
kviax T and lstx < T, kuia 2 T and lax 2 T,
Oidx 5 kwjaxs
Mij
(6)
kvijx 2 T + + 2,
if kviax 2 T and Lidx < T,
Qiax 2 kwiix
T + 2, if kviax < T and Tats > T.
Here, hat and kwia are the 0-indexed indices of the query and key-value pairs within the 27 le-ghh Rimput. respectively. T is the original sequence length. The mask Mij is a binary matrix where allowed
5

---

attentions are and disallowed are 0 (or -o0 after softmax). The first two cases handle causal attention within the Ridd and Rowd segments, respectively. The third and fourth cases manage the cross- attention between Riad and Rowd segments, precisely controlling information flow to maintain the integrity of bidirectional prediction without information leakage relative to the token being predicted.
The output of this attention mechanism provides a fused representation H, for each token Tt, which is then used make the final prediction following repositioning step, where the repre- sentations of the same token are summed, except for the first and last tokens, due to their repre- sentations containing information from only one
Masked vs. Janus training
240
Accuracy
0.25
direction, while optimizing Lbiiirrccionall
2-4
Janus 128
This Janus modeling approach, conceptually depicted in (Cg)re2(()), enables each token to be a learning target informed by its full bidirec- tional context, thereby enhancing training effi- ciency compared to traditional MLMs (Figure 2 (B)) and overcoming the limitations of unidirec- tional models (Figure (A)). (A)).
Janus 32
Masked
Masked 61
Training Steps
Figure 3: Superior Learning Efficiency of Janus Modeling. Comparison of last-token prediction ac- curacy between Janus modeling and conventional masked modeling over 10k training steps. Janus modeling consistently achieves higher accuracy for the same model architecture and training duration, demonstrating
Empirical Validation of Training Efficiency To empirically assess the learning efficiency of Janus modeling against conventional bidirec- tional approaches, masked modeling, we con- ducted a comparative experiment focused on last-token prediction. This task was chosen as it allows direct comparison: both standard masked language model and our Janus modeling approach can predict the final token ør given the
enhanced efficiency learning from sequence data. The number in the legend indicate hidden dimention.
preceding context æ ZT-1, ensuring a fair basis for evaluation.
Masked
Strand
Last Token
Strand
Strand Left Right
Strand Right Left
Strand Left Right
Right Left
Mamba
Mamba
Mamba
Mamba
Full global attention
Janus global attention
Figure 4: Janus and Masked Modeling Efficiency Validation. Both models are pre-trained from scratch using identical hyperparameter settings, with the only difference being the masking strategy applied in the final fusion attention layer. Last-token prediction is used to enable fair comparison of learning efficiency between the two models.
For this demonstration, we configured two model variants as Figure 4 Janus Model: A single-layer bidirectional Mamba architecture cquipped with a FlexAttention layer [26] for bidirectional fusion, utilizing the Janus-specile attention mask Mij (Equation 6 This model predicts 2, based on the fused bidirectional context of æi..... It-1, T2+1, IT as per the Janus modeling objective. Masked Language Model: As baseline, we construct comparable single-layer Mamba architecture followed by the same FlexAttention layer without an attention mask, trained using the conventional masked language modeling (MLM) objective, where fraction of tokens (typically 15%) contribute to the loss.
Both models were trained on the human reference genome (HG38) 32 for 10,000 steps. Each training sample had context length 131,075 tokens, processed with batch size of 1. We evaluated performance across three hidden dimensions: 32, 64, and 128, keeping other hyperparameters

---

consistent. Pre-training for Janus models, benefited from their sparse attention masks, takes around 27 minutes per 1,000 steps, nearly twice as fast as masked models.
For evaluation, both the masked and Janus models are given the full input sequence with the last token 27 masked. This evaluation is conducted
test set containing 1,920 sequences.
The results, presented in Figure clearly demonstrate that models trained with the Janus modeling method significantly outperformed those trained with the masked modeling approach in prediction accuracy, given the same number of training steps. This finding substantiates that Janus modeling is more effective leveraging the DNA sequences for learning, thereby achieving superior training efficiency while maintaining robust bidirectional understanding.
3.2 Hybrid cixture-offExperts (MoE) Model
After introducing the bidirectional efficient training method, we developed bidirectional backbone model enhance sequence representation. Leveraging the efficient bidirectional fusion method, we propose JanusDNA, hybrid model that integrates the strengths of Mamba, Attention, and MoE.
Architecture As illustrated in Fig. JanusDNA incorporates three primary components for se- quence representation: Mamba, MoF, and Attention. Mamba, a State Space Model (SSM) [23]. efficiently encodes input sequences using high-dimensional parameters. Compared to traditional Transformer architectures, Mamba is more memory- and computationally efficient, making it particu- larly suitable for processing long DNA sequences.
The Mixture-of-Experts (MoE) architecture provides a scalable approach to significantly increase model capacity without proportionally increasing computational costs during training and inference 33 In Jan,sDNA, MoE layers replace the feedforward network (FFN) layers in the Mamba blocks specific ratio following (24, achieving balance between performance and efficiency. To ensure balanced utilization of the experts, auxiliary loss is introduced (3), encouraging the model to distribute input data evenly across all experts. This auxiliary loss computed based on the router logits, which represent the probabilities of selecting each expert for a given input, as shown in Eq. 7 Given experts indexed by i = N and batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P:
Lotal = a N
fo
(7)
where fi is the fraction of tokens dispatched expert i,
fi
T
Σ
Hargmax p(x) = =i},
(8)
ZEB
and P; is the fraction of the router probability allocated to expert i,
P.
T Σ (z))
(9)
IEB
Here, (x) represents the probability of routing token x to expert i, while T is the total number of tokens in the batch B. This auxiliary loss encourages balanced utilization of all experts, improving overall model performance.
Bidirectional sequences are processed through independent stacks of Mamba and MoE layers, each designed to enhance the model's representational capacity. The Mamba layers efficiently capture local contextual dependencies within the sequence, leveraging their memory-efficiee state-space modeling, while the MoE layers provide sparse scaling to enhance the model's representational capacity without incurring proportional computational overhead.
The forward and backward representations generated by these layers are fused using Attentton 26). an optimized attention mechanism that supports sparse masks with reduced memory consump- tion. This fusion enables the model to integrate both local and long-range global information streams, resulting in comprehensive bidirectional representation for improved performance.
7

---

Reverse Complement In the double-helix DNA structure, each strand contains semantically equivalent information, with the reverse complement (RC) of strand oriented in the opposite direction and its bases complemented relative to the forward strand (A paired with TT, and C paired with G) g. However, recognizing both the forward and RC versions of non-palindromic motifs, such as GATA and TA7C, poses a significant challenge, as it is akin to learning two distinct motifs [35]. To address this, we adopt post-hoc reverse complement representation strategy [36]. Specifically, the DNA sequence and its reverse complement are processed in parallel using the identical model. The resulting representation vectors are then pooled to form a unified, enriched representation as shown in Figure I (B). This approach enables the model to effectively learn from both the original and reverse complement sequences, improving its ability to capture intricate patterns and relationships within DNA sequences, thereby enhancing performance across various tasks. We further conduct ablation experiments on reverse complement design in Appendix B.1.3
4 Experiments
4.1 Pre-training on Human Reference Genome
To ensure fair comparison with prior work, we pre-train our model only the human reference genome (HG38 (32) following the training setup described in [8] Specifically, we adopt single nucleotide-level tokenization to capture high-resolution input sequences and avoid overlooking critical DNA information that may be lost when using k-mer tokenization, commonly used in attention- models [5]. Additionally, single nucleotide-level tokenization is employed to facilitate downstream research on Single Nucleotide Polymorphisms (SNPs). Please note that performance on downstream tasks depends on the model architecture, as well as the composition and diversity of the pretraining data [II 19 20 [0]. Here, we specifically focus only on the architecture, and thus use only the human reference genome for pretraining ensure fair comparison.
4.2 Downstream Tasks
We evaluate our model on three different benchmarks: Genomic Benchmark 377 Nucleotide Transformer Benchmark [IT], and DNALONGBENCH [38]. We follow all benchmark settings of Genomic Benchmark and Nucleotide Transformer Benchmark as described in [8]. Accordingly, we adopt the reported results from [8] as our reference. Considering the practical computational cost of sparse MoËs, adjust the model's hidden size match slightly reduce the number of activated parameters compared to the baseline [ ensuring a fair comparison. As we introduce the model with middle attention layer in the ablation experiments (Section B.I.I]. present results for models both with and without mid-attention on the Genomic Benchmark and Nucleotide Transformer Benchmark.
4.2.1 Genomic Benchmark
Table 1: Genomic Benchmarks. Top-1 accuracy (t) across 5-fold cross-validation (CV) for a supervised CNN baseline, pretrained HyenaDNA, Caduceus models, ConyNova and JanusDNA models. Best values per task are bolded, second best are underlined. Error bars indicate the difference between the maximum and minimum values across 5 random seeds used for CV.
JANUSDNA MLP w/ MID-ATIS w/o MID-ATTN (426K)
JANUSDNA MLP
MODELS
CNN
HYENADNA (436k)
CADUCEUS PH
CADUCEUS PS
CoNA (386K)
ACTIVATED FARAM
(264K)
(470k)
(470k)
(431K)
MOUSE ENHANCERS
0.715=0.087 0. 0.780+0.025
0.75440.074 O. 7934:00056 0.78440.009
0.770=0.
0.769+20.029
CODING VS. INTERGENOMIC
0.892=0.008 0.90410.005
0.915:0.003
0.910=0.000 003 0.943=0.00
0.912 003
0.911100001
HUMAN VS. WORM
0.942=0.002 0.96440..02 0.973=2.001
0.968=0.002 0.967+0.002
00971
0.971
HUMAN ENHANCERS COIN
0.702=0.021 0. 7.7299000014 0. 7..77700.00 004 0.745=0.007
0.743:10.005
0.741 005
0.742:10.066
HUMAN ENHANCER ENSEMBL
0.893+0.005
0.900=00006 0.900=00000
0.897 004
0.89940.004
HUMAN REGULATORY
0.872=0.0050.300 0.809=00012 0.872 0.873=0.007
0.873+0.002
0.877
0.868-20.008
HUMAN OCR ENSEMBL
0.698_0.013 013-00.0130.78340.oorr 0. 0.828=0.006 0.818=0.006
0.793+600004
0.822=0.000
0.82449..00
HUMAN NONTATA PROMOTERS
1661=00.099.9/411..00
0.926+0.007 0.915=00.010 0.95110.003
0.957 0.95-110.010
We start with the Genomic Benchmark, which a collection of regulatory element classification
tasks with sequence lengths mostly ranging from 200 to 500, and one up 4,776. We take the hidden
8

---

state embedding of the final layer and apply a pooling layer on sequences to obtain a fixed-length representation. We then upply a linear layer map the representation the number of classes for each task. We perform 5-fold cross-validation for each task using the same seed as 8 As shown in T our model achieves state-of-the-arr performance on 3 out of 8 tasks, outperforming the previous best model, while the remaining tasks are close to the best model. Please note that this benchmark is already quite saturated, as shown in Table [ and we do not expect improvements in pretraining to meaningfully improve benchmark performance further.
Table 2: Nucleotide Transformer Tasks. Performance (1) across 10-fold CV for Enformer, DNABERT-
2, Nucleotide Transformer v2, HyenaDNN, Caduceus- ConvNova, and JanusDNAmip- Metrics vary by task: MCC for histone markers and enhancer annotation, F1-score for promoter annotation and splice site acceptor/donor, and accuracy for splice site "all". Best values per task are bolded, second best are italicized. Given the disparity in model size, we also underline the best value among models with fewer than 2M activated parameters. Error bars indicate the difference between the maximum and minimum values across 10 random seeds used for CV.
> 100M ACTIVATED PARAM. MODELS
2M ACTIVATED PARAM. MODELS JANUSDNA MLP W/ MIDATIN (2.00IM)
JANUSDNA MLP W/O MIDATIN
ENFORMER
DNABERT-2 (117M)
NT-v2 (500M)
HYENADNA CADUCEUS-PII (1.6M)
CONVNOVA (L.7M)
(252M)
(1.9M)
(2.009M)
Histone Markers
H3
0.71910.048 0.78600.8800000 0.78410.00
0.77910.037
0.81520.048
0.81210.017
1.883510000 009 0.83110.00 023
H3K14AC
0.288 0.516++0.02
0.551++0.02
0.612#0.065
0.631
0.621+00009
0.729
0.718440.022
НЗК36ME3
0.344::0.05 0.591110002
0.62511.013
0.613:100041
0.601
0.661100010 O. 0.702:0.015
0.609:10.028
Н3К4МEI
0.291 0.511110.026 0.550110.021
0.512:0.024 0.523:0.039
0.531110.028
0.615:1..03 0.616
НЗK4ME2
0.211 069 0. 0.336: 040 0.31910.045
0.455:200095
0.487=0.
0.485:10.032 O. 0.58920.00 023 0. 0.686 019
НЗК4МMЗ
0.158 0.22000.00000 0.352 0.110:10.00
0.51910.056
0.54410.045
0.5661100277
68688810 026 0.675110.014
H3K79MB3
0.496 042 0.00 0.626+0.026
0.672:0.048
0.697=0.077
0.700=1.007
0.747 0. 0.743
НЗКЯАC
0.420 063 0. 5.2::.00200 0.50210.040
0.68110.061
0.62210.08 030
0.05810.011
0. 0.67331000 0.04 0.6611:0.02
H4
0.73220.076 0.796110.227 0.799
0.76310.044
0.811=0.022
0.80810.008
0.81220.011
0.813=0.013
H4AC
0.273:000083
0. 0.63300001 0.49520.032
0.564:10.036
0.621
0.636:10.011
0.69822.013 013 0.705=000025
Regulatory Annotation
ENHANCER 1.451=0.100/ 0.516 0.548
0.517
0.546=0.073
0.586=u.Das
0.559 042
0.542++0.04
ENBANCER 0.30910.1133 0.123:00.051 0.12110.182
0.38610.118
0.439::0005
0.50011.018
0.50310.00
0.492.. 0.09
PROMOTER: ALL 0.954 001 0.971+00.006 0.976=u.oe 00G
0.960+00.ous
0.970=u.004
0.967+t0.oo
0.970=u..02 002
0.970++o.uu
NONTATA 0.955 0100000000000 0.972:10.005 0.976:00.om 005
0.959:10.006
0.96910.011 011
0.908:10.003
0.97110 004
0.971 003
TATA 0.960=0.023 0..
0.966-+0.013
0.944+0.040
0.953#0.016
0.969=0.003
0.958=0.007
0.960-+-0000
Splice Site Amnotation
ALL
0.84820.019 0.939- 0.983=0.000
0.95620.011
0.940=0.027
0.965:20004
0.967=0.005
0.943+0.020
ACCEPTOR
0.914 0. 0.975++00000 0. 0.981
0.958
0.937
0.971+00003
0.957=u.012
0.961+0.000
DONOR
0.906=0.027 0.963 0. 0.922
0.949:10002 0.94820.025
0.965.+0.033 0.94820.008 0.935+000116
4.2.2 Nuclcotide Transformer Tasks
Next, we evaluate model on the Nucleotide Transformer tasks, which include 18 datasets covering histone marker prediction, regulatory annotation prediction, and splice site annotation prediction. Following the evaluation metrics outlined in [Л] we perform 10-fold cross-validation for each task, adhering to the same experimental settings as g].
As shown in Table our model achieves state-of tfite--o-thh--rr performance on 12 out of 18 tasks, surpassing previous models, including those with 250 times more activated parameters. While the promoter and splice site annotation tasks exhibit slightly weaker performance compared to the best larger model, this underscores the potential importance of training data scale and diversity for these specific tasks. For clarity, we present only the results of Caduceus-PH in the table due to space constraints, as Caduceus-PP performs slightly worse than Caduceus-PPP
4.2.3 DNA Long Range Benchmark
To further assess our model's ability to capture long-range dependencies in DNA sequences, we evaluate on the expression Quantitative Trait Loci (eQTL. prediction task from DNALONGBENCH
388 with the sequence length of 450,000. The eQTL task measures whether a nucleotide variant can influence the expression of target gene based on the sequences of the gene and its surrounding regions.
9

---

Table 3: DNALongBench eQTL Tasks. The AUROC for expert model Enformer, Caduceus-PPi and JanusDNA, The best results are bolded.
JANUSDNA
JANUSDNA W/o MID-ATTN
MODELS
EXPERT MODEL CADUCEUS-PIL
MLP
ACTIVATED
(252M)
(7.7M)
(7.662M)
MID-ATNN
(7.745M)
ARTERY TIBIAL
0.741
0.690
0.803
0.852
ADIPOSE SUBCUTANBOUS
0.736
0.759
0.7/1
0. 0.769
CULTURED FIBROBLASTS
0.802
MUSCLE SKELETAL
0.621
0.789
0.803
0.864
TIBIAL
0.683
SKIN NOT SUN EXPOSED SUPRAPUBIC
0.710
0.812
0.875
0,903
SKIN SUN EXPOSED LOWER L.R
0.700
0.692
0.706
0.846
THYROID
0.612
0.703
0.752
0.793
WHOLE BLOOD
0.689
0.769
0.794
0.821
Due to limited computational resources, compare model against the state-of-the-art DNA language model, Caduceus-PH [8], which is also trained with the same data scale for fair comparison, and the expert model for this task, Enformer [3)] The results for Enformer are taken directly from DNALONGBENCH [38] . For Caduceus-PPt we entirely fine-tune the official HuggingFace-relesse released weights, which are pretrained on sequences of length 131k. Our model is pretrained and entirely fine-tuned using the same setup and sequence length as Caduceus-PP to ensure a fair comparison. As shown in Table JanusDNA achieves the best performance 8 out of 9 datasets, significantly outperforming the expert model and Caduceus- despite using fewer computational parameters.
5 Conclusion
Summary In this work, we introduced a novel global modeling paradigm for bidirectional DNA sequence representation, combining the bidirectional capability masked language modeling with the speed and optimization benefits of autoregressive approaches. We proposed JanusDNA, Mamba MoE-based DNA foundation model with global attention that enhances genomic sequence understanding while maintaining low memory complexity, supporting the processing of up to I million base pairs (1 Mbp) on a single 80GB GPU. Experimental results demonstrate that JanusDNA outperforms HyenaDNA, Caduceus, and other Transformer baadsdd models across range of benchmark tasks and the expert model on eQTL tasks. By leveraging global attention mechanisms and efficient long-range sequence processing, JanusDNA offers powerful framework for advancing research on long-range genomic interactions.
Limitations and Future Work Although JanusDNA demonstrates high learning efficiency on a fixed data scale, current training is restricted to the human reference genome for fair architectural comparison. Expanding the corpus to include human genomic variants (e.g.. 1000 Genomes Project) and non-human species (e.g., primates) could further boost modeling capacity and biological insight. JanusDNA also lacks integration of multimodal data such as epigenetic states (e.g., chromatin accessibility, histone modifications) and single-cell transcriptomic profiles, which vital for resolving cell-type-specific regulation and predicting chromatin- intraatii-iffueeeed phenotypes. Future work will incorporate these modalities and explore functional roles of key genomic features (c.g.. CTCF- mediated chromatin loops, enhancer RNAs, non-coding risk variants). Experimental validation (e.g.. CRISPR, organoids) will prioritize therapeutic targets, while clinical collaborations will evaluate JanusDNA's utility in personalized diagnostics and drug discovery. These efforts aim to evolve JanusDNA into unified framework linking genome structure, epigenetic regulation, and disease mechanisms.
6
Acknowledgments
The authors acknowledge the Scientific Computing of the IT Division at the Charité Univer- sitätsmedizin Berlin and the Science Computing at Shenzhen Technology University for providing computational resources that have contributed to the research results reported in this paper. B.W. and R.F. acknowledge support by the Collaborative Research Center (SFB 1470) funded by the German Research Council (DFG).
10

---

References
[1] Gonzalo Benegas, Chengzhong Ye, Carlos Albors, Jianan Canal Li, and Yun Song. Genomic language models: opportunities and challenges. Trends in Genetics, 2025.
[2] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 194.277,,2. 2024.
[3]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han. Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[5] Yuxiao Hu, Qian Li, Dongxiao Zhang, Jinyue Yan, and Yuntian Chen. Context-alignment: Activating and enhancing Ilm capabilities in time series. arXiv preprint arXiv:2501.03747, 2025.
[6] Yanhao Jia, Ji Xie, S Jivaganesh, Hao Li, Xu Wu, and Mengmi Zhang. Seeing sound, hearing sight: Uncovering modality bias and conflict ai models in sound localization. arXiv preprint 1.Kv:25005./2117 2025.
[7] Wei Liu, Jun Li, Yitao Tang, Yining Zhao, Chaozhong Liu, Meiyi Song, Zhenlin Ju, Shwetha V Kumar, Yiling Lu, Rehan Akbani, et al. Drbioright 2.0: an llm-powered bioinformatics chatbot for large-scale cancer functional proteomics analysis. Nature communications, 16(1):2256, 2025.
[8] Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling, 2024.
[9] Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, 37(15):2112-2120 2021.
[10] Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert- 2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306. 15006, 2023.
[11] Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo de Almeida, Hassan Sirelkhatim, et al. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, 22(2):287-297, 2025.
[12] Sarah Nemsick and Anders S Hansen. Molecular models of bidirectional promoter regulation. Current opinion in structural biology, 87:102865, 2024.
[13] Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. arXiv preprint 2rX1v:2402.01002. 2024.
[14] Georgios Pantazopoulos, Malvina Nikandrou, Alessandro Suglia, Oliver Lemon, and Arash Eshghi. Shaking up vims: Comparing transformers and structured state space models for vision & language modeling. arXiv preprint arXiv:2409.05392 2024.
[15] Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow Callum Birch- Sykes. Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems,
2023.
[16] Daniel Schulz, Bjoern Schwalb, Anja Kiesel, Carlo Baejen, Phillipp Torkler, Julien Gagneur, Johannes Soeding, and Patrick Cramer. Transcriptome surveillance by selective termination of noncoding ma synthesis. Cell, 1575)) 2013.

---

[17] Wu Wei, Vicent Pelechano, Aino I Järvelin, and Lars M Steinmetz. Functional consequences of bidirectional promoters. Trends in Genetics, 27(7):267-211 2011.
[18] Olga Mikhaylichenko, Vladyslav Bondarenko, Dermot Harnett, Ignacio E Schor, Matilda Males, Rebecca R Viales, and Eileen EM Furlong. The degree of enhancer or promoter activity is reflected by the levels and directionality of erna transcription. Genes & development, 32(1):42- 57, 2018.
[19] Eric Nguyen, Michael Poli, Matthew G Durrant, Brian Kang, Dhruva Katrekar, David B Li, Liam Bartic, Armin W Thomas, Samuel H King, Garyk Brixi, al. Sequence modeling and design from molecular to genome scale with evo. Science, 386(6723))ead09333 2024.
[20] Garyk Brixi, Matthew G Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A Gonzalez, Samuel King, David B Li, Aditi T Merchant, et al. Genome modeling and design across all domains of life with evo 2. BioRxiv, pages 2025-02, 2025.
[21] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04000. 2018
[22] Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05661 2024.
[23] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
[24] Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwarte et al. Jamba: A hybrid transformer- manssormer-mamba mamba language model. arXiv preprint arXiv:2403.19887 2024.
[25] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specializa- tion mixture-of-experts language models. arXiv preprint arXiv:2401.006 100066, 2024.
[26] PyTorch Team. Flexattention: A new way to simplify and enhance attention mechanisms. wwwp://wwwww...o/.ttttt.t..o.oa.
2024. Accessed: 2025-05-07.
[27] Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. segpt: toward building foundation model for single-cell multi-omics using generative ai. Nature Methods, 21 14800 2024.
[28] Xingyi Cheng, Bo Chen, Pan Li, Jing Gong, Jie Tang, and Le Song. Training compute-optimal protein language models. biorxiv. 2024.
[29] Micaela E Consens, Cameron Dufault, Michael Wainberg, Duncan Forster, Mehran Karimzadeh, Hani Goodarzi, Fabian J Theis, Alan Moses, and Bo Wang. To transformers and beyond: large language models for the genome. arXiv preprint arXiv:2311.07621, 2023.
[30] Christina V Theodoris, Ling Xiao, Anant Chopra, Mark D Chaffin, Zeina R Al Sayed, Matthew C Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng. Shirley Liu, et al. Transfer learning enables predictions in network biology. Nature, 61867965)616--24, 2023.
[31] Gonzalo Benegas, Sanjit Singh Batra, and Yun S Song. Dna language models are powerful predictors of genome-wide variant effects. Proceedings of the National Academy of Sciences, 120(44):e311 1219120, 2023.
[32] Genome Reference Consortium et al. Genome reference consortium human build 37 (grch37). Database (GenBank or RefSeq), 2009.
[33] Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, page 1-20, 2025.
[34] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23(1), January 2022.
12

---

[35] Hannah Zhou, Avanti Shrikumar, and Anshul Kundaje. Towards a better understanding of complemett equivariance for deep learning models in genomics. Machine Learning in Computational Biology, pages 1-33. PMLR, 2022.
[36] Babak Alipanahi, Andrew Delong, Matthew T Weirauch, and Brendan J Frey. Predicting the sequence specificities of dna-and ma- be-dinndig proteins by deep learning. Nature biotechnology, 33(8):831-853 2015.
[37] Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Šimeček, and Panagiotis Alexiou. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24(1):25, 2023.
[38] Wenduo Cheng, Zhenqiao Song, Yang Zhang, Shike Wang, Danqing Wang, Muyu Yang, Lei Li, and Jian Ma. Dnalongbench: A benchmark suite for long-range dna prediction tasks. bioRxiv, pages 2025 01, 2025.
[39] Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska- Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10):1196-1200) 2021.
[40] Wenduo Cheng, Zhenqiao Song, Yang Zhang, Shike Wang, Danqing Wang. Muyu Yang. Lei Li, and Jian Ma. Dnalongbench: A benchmark suite for long-range dna prediction tasks.
[41] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference Machine Learning, pages 28043-28078 PMLR, 2023.
[42] Badri Narayana Patro and Vijay Srinivas Agneeswaran. Mamba-360: Survey of state space models as transformer alternative for long sequence modelling: Methods, applications, and challenges. arXiv preprint 2rXiv:2404.16/11 2024.
[43] Mingqian Ma, Guoqing Liu, Chuan Cao, Pan Deng, Tri Dao, Albert Gu, Peiran Jin, Zhao Yang, Yingce Xia, Renqian Luo, al. Hybridna: A hybrid transformer-mambal long-range dna language model. arXiv preprint arXiv:2502.10800 2025.
[44] Tri Dao. Flashattentiinn-2 2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869/ 2023.
[45] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69%0 2014.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims are summarized in Section 3 Also Section and Appendix B.2 for more experimental evidence.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include the limitations of our work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

We detail the assumption and proof of theoretical result on training efficiency in Section
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We use publicly-accese accessible human reference genome [22 and benchmarks
337 [T 40). We explain our setting in Section 4 and Appendix B.2 We upload the codes
and instructions to recover the results. Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We use publicly-accessibee dataset HG38 32 We upload the codes and instructions to recover the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
Details are summarized in Section and Appendix B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We show the standard emror in most training curves. We calculate standard error across 10 random seeds for nucleotide transform tasks, and 5 random seeds for genomic benchmark in Section We random seeds for ablation in Appendix l But no error bars for BenCh [40] due to limited computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have the Appendix B.3 on this.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp////eearrpeecc////////hhaaa
Question:
Answer: [Yes]
Justification: conform.
We have read and understood the code ethics and have done our best to
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work focuses biological information understanding for the academic use. This work is not related to any private or personal data, and there are no explicit negative social impacts.
Guidelines:
17

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We do not foresee any high risk for misuse of this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we credited them in appropriate ways.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release our code base with included readme files.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [No]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did not involve LLMs as any important, original or non-standard compo-
nents.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

A Related Work
A.1 DNA Language Models
Attention-based Models Attention-based DNA language models, such as DNABERT 2 DNABERT2 [0], and Nucleotide Transformer MT. have demonstrated significant success in model- ing DNA sequences. These models employ k-mer encoders to group consecutive nucleotide base pairs into single tokens, enabling efficient sequence representation. However, their global attention mecha- nisms restrict scalability to sequences of approximately 12,000 base pairs (bps), limiting their ability to capture long-rand range dependencies. Furthermore, the use of k-mer tokenizers reduces modeling resolution, posing challenges for tasks like Single Nuclcotide Polymorphism (SNP) analysis.
State Space Models State Space Models (SSMs) have been applied to genomic tasks, with Hye- naDNA [15] utilizing the Hyena operator [4] to process sequences up million base pairs (bps). Despite this capability, its unidirectional design limits the model's ability to capture bidirectional genomic contexts 22 To overcome this, Caduceus [ introduces a bidirectional Mamba archi- tecture, which aggregates information from both upstream and downstream sequences, enhancing genomic context comprehension. However, basedd models often face challenges in memory recall tasks when compared to transformer-basee approaches [42]. Additionally, their masking-basee training paradigm less efficient, as uses only 15% of the data for loss computation per iteration. In contrast, autoregressive training paradigms take advantage of nearly 99% of the data, significantly improving training efficiency and overall performance. Latest SSM-based DNA models, such as Evo [19] and Evo2 (20). introduce stripedHyena and StripedHyena2. showing better scaling rate and improved throughput compared to HyenaDNA. However, to gain better generative performance, they still rely on the autoregressive training paradigm, facing the same limitation HyenaDNA for bidirectional understanding.
Hybrid Models Hybrid models incorporate multiple encoding mechanisms such as convolutional neural networks (CNNs), Mamba, and Attention to leverage the complementary strengths of each architecture. Enformer (39] combines CNNs with Transformers, enabling the model to capture long-range genomic interactions spanning up 100 kilobases. HybriDNA 43 integrates Mamba and Transformer components, extending its receptive field to 131 kilobases.
B
Experimental Details
B.1 Ablation Experiments
B.1.1 JanusDNA Hybrid Architecture
To determine the optimal hybrid architecture for DNA sequence modeling that effectively balances local and global attention, we perform ablation experiments various configurations of the uni- directional encoder (i.e., modules preceding the bidirectional fusion layer). Referring to [24], we also explore the value of the additional mid-attention layer. Specifically, we evaluate the following configurations: 1) mamba and FFN blocks only, 2) mamba and FFN blocks with mid- iid-attention, 3) mamba and FFN blocks with MoE, and 4) mamba and FFN blocks with both mid-attention and MoE. The ratio of MoE to replace FFNs is set to 0.5. In models with mid-attention, the two independent bidirectional Mamba blocks at the 4th layer are replaced with two independent causal Attention blocks implemented using Attentoon
The models are pre-trained sequences of lengths 1024 and 13 131072, with batch sizes of 128 and 1, respectively, using a single GPU. All other hyperparameters and training settings are consistent with the pre-training setup described earlier. To ensure a consistent number of activated parameters across different models, we adjust the model configurations as Table 4 The training perplexity results for these configurations are shown in Figure
The training perplexity results reveal that the model with mamba, FFN, and mid-attention exhibits higher perplexity compared to the model with only mamba and FFN, while the model with mamba, FFN, and MoE achieves lower perplexity. This suggests that mid-attention may not enhance training efficiency, whereas MoE contributes positively. Notably, the model combining mamba, FFN, mid-
21

---

Table 4: Hyperparameter settings for JanusDNA ablation experiments.
JANUSDNA (ALL WITH MAMBA)
FFN
MIDATTN+FEN
MOE+FFN
MIDATIN-MOF+FEN
LAYERS
WIDTH
148
148
128
128
ACTIVATED PARAMS (M)
5.973
5.973
6.084
6.080
TOTAL PARAMS (M)
5.973
5.973
28. 18.104
28.100
GLOBAL STEPS
10K
10K
10K
10K
EXPERT NUMBER OF MOE
HEAD NUMBER OF ATTENTION
4
MULTIPLE NUMBER OF FFN WIDTH
4
4
OPTIMIZER
ADAMW
OPTIMIZER MOMENTUM
B1, B2 0.9, 0.95 -e-
LEARNING RATE
LR SCHEDULER
COSINE DECAY
WEIGHT DECAY (MODEL)
0.1
Loss Curves for Sequence Length 1024
Loss Curves for Sequence Length 131k
1.07
1.08
FAN
FFN
1.00
Niskton FFN
1.27
MisAttn FFN
FFN Midhttn MoE.
Midkitn MoE
1.06
1.03
1.1
1.02
1:01
5000
6000
7000
8000
9000
10000
5000
6000
7000
8000
9000
10000
Training Steps
Training Steps
Figure 5: Training perplexity of Mid-attention and MoE ablation models on 1024-length and 131k- length sequences.
attention, and MoE achieves the lowest perplexity, indicating that mid-attention and MoE can synergistically improve training performance.
To further investigate the role of mid-attention, we conduct additional ablation experiments comparing models with and without mid-attention (all incorporating MoE due to demonstrated benefits) on the Nucleotide Transformer Benchmark with 3-fold cross-validation since Genomic Benchmark is too saturated to show apparent differences. We pretrain and fine-tune models with hidden dimensions of 32, 72, and 128. The results, summarized in Figure
indicate that models with mid-attention perform better at hidden dimension 32, However, as the hidden dimension increases, models without mid-attention outperform those with mid- mid-attention at dimension of 72. dimension of 128, the performance of models with and without mid- attennino becomes comparable. These findings suggest that mid-attention provides diminishing benefits as the hidden dimension grows larger, eventually becoming negligible at higher dimensions.
Given that larger hidden dimensions are generally preferred for large-scale DNA sequence modeling, and considering that mid-attention introduces additional computational overhead, prefer to models with mamba and FFN blocks without mid-attention for downstream tasks. Nonetheless, we include the experiments the model with mid-attention in our formal evaluations on Genomic Bench- mark and Nucleotide Transformer Benchmark to ensure completeness and provide comprehensive analysis.
B.1.2 Reverse Complement (RC)
DNA follows the complementary base-pate lise-pairing principle, meaning each DNA strand has reverse complement strand with equivalent genetic information. However, despite this theoretical equivalence, many biologically important motifs (e.g., transcription factor binding sites) are non- poli-dddorooicc Rec- ognizing both the forward and RC versions of such motifs (for instance, motifs like GATA and TATC) is challenging, as it effectively requires the model to learn two distinct representations. Therefore, explicitly integrating RC information allows the model more robustly and comprehensively capture DNA sequence patterns. Nonetheless, the utility of RC depends heavily on the specific biological
22

---

Accuracy JanusDNA with and without Mid-Attn Dimensions Nucleotide Transformer
10
07
04
H3K79ma3
HBK36me3
H3K-4me3
N3K4me2
H3K44ml1
H4ac
H3K14ac
HDKoac
Hd
H3
splice sites donars
spiice sites acceptors splice silte
enharcers, types enhanceess
promoter no tara promater rata
prometer all
Figure 6: Mid attention ablation results on Nucleotide Transformer.
context. For instance, genomic elements such as splice sites are strictly defined by single DNA strand, making RC inclusion potentially suboptimal or misleading.
Regarding computational overhead, RC is introduced only during inference by averaging the embed- ding representations of a strand and its RC prior to decoding. Thus, the additional computational cost is minimal, primarily involving an extra forward pass through the model for the RC strand.
To quantitatively assess the impact of incorporating RC, we conducted ablation experiments using the NT benchmark. We fine-tuned JanusDNA models under consistent experimental conditions (learning rate 1e-3, batch size 256), with and without RC during prediction. The results clearly indicate that models utilizing RC generally outperform those without RC across most tasks. Notably, the exception to this pattern was observed in splice site prediction tasks, where RC inclusion led to inferior performance, consistent with the biological reality that splice sites are inherently strand-specific.
Table 5: Performance of JanusDNA with and without middle attention, with and without Re- verse Complement on Nucleotide Transformer Benchmark. Top-1 accuracy (1) across 5-fold cross- validation (CV) for different model variants. Best values per task within each group (left two columns are JanusDNA without middle attention, right two columns are NanusDNA with middle attention) are bolded. Error bars indicate the standard deviation across 5 random seeds used for CV.
JANUSDNA
TASKS
MID-ATTN
w/O MID-ATTN
w/o RC
w/ RC
W/O RC
W/ RC
H3
0.789 028
0.828210.00 020
0.796=2.002
0.830
H3K14AC
0.089.10.00 029 0. 0.729910.02 022
0.602:10.015 0.700:10.05
33336ME3
0.661440.021 0.701 +000022
0.658 66.06-
0.688+0.00
113K4M11
0.574:60002 026 0. 2..609:00000 022
0.555500.000
0.605 028
H3K4ME2
0.546:1 5.888811.000 .20
0.53210-020
0.581 024
H3K4MMB
0.640:10.00 003 0. G81-10.0000 016
0.6255:0.00 0.015 0.675.100 014
H3 13333
0.723-40.00
0. 0.743-40 +..-9000009
НЗКЯАC
0.038:1:000 023 0. 673 014
0.63 100016 O. 658 020
H4
0.781.10.00 020 O. 1.810=00.00 022
0.775100.019 O. 0.813:1 ::00:.0.000 00.011
H4AC
0.6533+0.023 :..099
0.629 00007 0.684 1684+0..02
ENHANCERS
0.38243.035 0. 0.396 033
0.379
0.397 0065
ENHANCERSTYPES
0.475:10.00 0.488110.00 066
0.49010:046 0.492:10. 096
PROMOTERALL
0.961110.00 0001:00000 0. 0.969 0.969110.0G 002
0.96210.000 003 0. 0.9701100000 002
PROMOTERNOLATA
0.96140.004
0.961 000000 0.970++0.000
PROMOTERTNIA
0.916:10.012 0. 9.955-10..0 010
00.1710.010 0. 0.953:10.03
SPLICESITESALL
0.961 000.00 0.948$+0000
0.946=0.00 0.92240.015
APLICESITESACCEPTORSS
O 920--000000
0.906+0.001
0.93 0.904
SPLICESIITEDDOOOS
0.910520.00 0.8931n.006
0.921 009 0.87420.011
B.1.3 Janus and Casual Modeling
We conducted an ablation comparing Masked and Janus modeling using the same architecture as in Figure order to further demonstrate the value of bidirectional modeling, we here conduct the ablation experiments between Janus modeling and Casual modeling.
23

---

Table 6: Comparasion of Janus models and Casual models on Last-Token prediction.
Janus
Casual
Mize(M)
0.056
0.207
0.795
0.061
0.209
0.814
step(k)),im
32
64
128
40
76
152
1
0.428
0.439
0.445
0.421
0.426
0.434
2
0.445
0,451
0,460
0.454
0.441
0,453
3
0.450
0.462
0.472
0.443
0.447
0.471
4
0.451
0.466
0.473
0.443
0.451
0.468
5
0.453
0.470
0.479
0.454
0.457
0.478
6
0.455
0.473
0.482
0.451
0.453
0.481
7
0.458
0.474
0.484
0.449
0.457
0.482
8
0.459
0.477
0.488
0.459
0.465
0.479
9
0.461
0.479
0.490
0,453
0.467
0.479
10
0.461
0.480
0.491
0.459
0.465
0.484
Table 7: Comparasion of Janus models and Casual models on Middle-Token prediction.
Janus
Casual
size(M)
0.056
0.207
0.795
0.061
0.209
0.814
step(k)),din dim
32
64
128
40
76
152
0.428
0.436
0.440
0.384
0.400
0.408
2
0.438
0.454
0.465
0.429
0.425
0.443
3
0.437
0.449
0.467
0.436
0.446
0.454
4
0.443
0.465
0.471
0.434
0.451
0.454
5
0.444
0.477
0.477
0.434
0.449
0,451
6
0.456
0.474
0.482
0.428
0.442
0.446
7
0.458
0.471
0.477
0.438
0.457
0.461
8
0.453
0.478
0.478
0.434
0.455
0.461
9
0.458
0.481
0.485
0.447
0.458
0.465
10
0.456
0.482
0.488
0.443
0.456
0.465
Conducting direct ablation with same architecture between Casual modeling and Janus modeling challenging due to inherent architectural differences. The core strength Janus modeling lies in its bidirectional context understanding, requiring bidirectional architecture. Casual modeling, by definition, is strictly unidirectional, and adapting it to bidirectional architecture would conflict with its fundamental principles.
However, we can still maintaint the most faireness by keeping models with same parameters. We constructed Casual models with layer of single mamba encoder and FlashAttention22 layer with causal attention mask, ensuring comparable model sizes to the Janus models. Following same pre-training settings, compared the two kinds of models in last-token prediction and middle-token prediction. we expected similar performance for the prediction of the last token (where there is no bidirectional information, both models have exactly the same information predict the last token), whereas we expected Janus models to outperform Casual models for the prediction of a token the center of the sequence (where Janus benefits from the bidirectional context).
The results are presented table 0 and table We confirm that the Janus model consistently outperforms the Casual models for tokens in the center of the sequence across all evaluated model sizes. To our surprise, Janus models also slightly outperform the Casual models even on the last token prediction task, suggesting Janus models indeed learn richer DNA representations through bidirectional training.
B.1.4 Multilayer Perceptron feature enhancer
The features fused by attention can be further enhanced through the addition of Multi-Layer Perceptron (MLP). To verify this, we conduct ablation experiments and name the JanusDNA models equipped with an MI.P as JanusDNA-ML.P. distinguishing them from the original JanusDNA models without the MLP attached after the feature fusion attention module. We perform the ablation

---

experiments on both the Nucleotide Transformer Benchmark and the DNALONGBENCH Benchmark, as shown in TableBBand T The inclusion of the Meads to notable performance improvements across both benchmarks.
Table 8: Performance of JanusDNA with and without MLP on Nucleotide Transformer Tasks. Performance (1) across 10-fold CV for janusDNA and JanusDNA mlp variants. Metrics vary by task: MCC for histone markers and enhancer annotation, Fi-score for promoter annotation and splice site acceptor/donor, and accuracy for splice site "all". Best values per task are bolded, second best are italicized. Since all models are approximately fewer than 2M activated parameters, we underline the best value(s) among them. Error bars indicate the difference between the maximum and minimum values across 10 random seeds used for CV.
JANUSDNA
JANUSDNA
JANISDNA JANUSDNA MID-ATTN (1980M)
W/O MIDAUIN
(1.88MM)
W/ MIDATTN (2.001M)
W/o MIDSTIN (2.009M)
Histone Markers
113
0.821#0.021
0.824#0.012 0. 0.835=0.009
0..01
H3K14AC
0.665:0.034
0.685:10.016 0. 0.022 0.71810.026
H3K36ME3
0.65810.022
0.670:10012 0. 70210.011 0.699110025
H3K4ME1
0.56310.041
0.57110.018 0.615:10.06 035 0.616-0.018
H3K4ME2
0.509 006
0.548 022 0.589++.022 0.58610.019
H3K4ME3
0,605:10.00 0300 0.629:0.022 0.688 1.006 0.675100.01
HЗK79ME3
0.716:20.017
0.72740.023
0.747 0. 0.743+000009
НЗК9AC
0.64110.024
0.639110019
0.67310.011 0.00110.027
H4
0.809=0.021
0.816=0 0.812 011 0.813++00013
H4АC
0.637:10.060 060
0.65340.034
0.698 013 0.705 023
Regulatory Annotation
ENHANCER
0.564+0.022
0.535+0.036 0.55940.042 0.54244.004
ENHANCER TYPES 0.112:10.049 0.470:10.025 0.50310.038
0.192110.096
PROMOTER: ALL 0.969++0.002 0.971=0.061
0.970=+0.00. 002
0.970++0.00x
NONTATA 0.97110.009
0.97110:002
0.97110
0.971110000
TATA 0.956#000010
0.9088
0.958#000007
0.960 0.000
Splice Site Anmotation
ALL
0.963=1.002 0.960110000
0.967=100005 0.943+0.020
ACCEPTOR
0.94910.020 0.93911.022
0.90720.012 0.961 009
DONOR 0.947=0.015
0.936+000014 0.94840.00s 0.935
B.2
Formal Experiment
B.2.1
Pre-training
Architecture configuration We utilize Mamba, FFN, MoE blocks the primary building blocks for unidirectional representation, which are then followed by a bidirectional fusion layer. There are 8 layers in each unidirectional encoder and each layer consists of one Mamba block and one FFN block. The MoE block is to replace the FFN block at a certain ratio, which is set to 0.5 in our experiments. The number of experts set to 16 and the dimension of FFN set to 4 times the hidden dimension. The bidirectional fusion layer is achieved by a FlexAttention layer (26] with 4 attention heads.
Meanwhile, we also implement version of the model with mid-attention, which replaces the Mamba block at the fifth layer with a mid-attention layer. The mid-attention layer is implemented with FlexAttention with 4 attention heads. The attention mask is set to half triangle allow the model to only attend to the tokens ahead of the current token to keep causality.
Pre-training setup To ensure fair comparison with prior work, we pre-train our model on the human reference genome (HG38 (32) following the training setup described in (8). We use cross- entropy loss for pre- pre-training. The model is trained with learning rate of 8 x 10 10-3, maintaining constant token count of 220 tokens per batch. Two sequence lengths are used: 1024 and 131072, with corresponding batch sizes of 128 and I, respectively, across 8 GPUs. Optimization is performed using Adam W (45) with a weight decay of 0.1, $1 = 0.9, and 82 = 0.95. A cosine learning rate scheduler is applied, incorporating warmup phase for 10% of the training steps. The learning rate starts at 1 x 10 and peaks at 1 x 10-4. The coefficient for the MoE auxiliary loss is set to 0.2. The gradient clipping threshold is set 1.0.
For the 1024-length model, training is conducted for 10,000 steps, while the 131072-length model trained for 50,000 steps.
25

---

We pre-trained three scales of the model: 32, 72, and 144 hidden dimensions to keep the same activated parameters for fair comparison with baseline models on different benchmarks. The hidden dimensions of 32 and 72 are used for the 1024-length model, while the 144 hidden dimension is used for the 131072-length model. The feedforward dimension set to times the hidden dimension, and the number of attention heads is set to 8. The coefficient for the MoE auxiliary loss is set to 0.2, the number experts set to 16. All hyperparameter settings are listed as Table 9
Table 9: Hyperparameter settings for JanusDNA pretraining (select models).
JANUSDNA
W/ MIDATIN
w/ MIDATTN
LAYERS
8
8
8
8
8
WIDTH
32
72
32
72
144
ACTIVATED PARAMS (M)
0.422
1.980
0.428
1.989
7.664
TOTAL PARAMS (M)
1.798
8.947
1.804
8.956
35.533
MAX SEQ. LEN.
1024
1024
1024
1024
131072
BATCH SIZE
1024
1024
1024
1024
8
GLOBAL STEPS
10K
10K
10K
10K
50K
EXPERT NUMBER MOE
16
16
16
16
HEAD NUMBER OF ATTENTION
4
MULTIPLE NUMBER OF FEN WIDTH
4
4
4
4
4
COEFFICIENT OF AUXILIARY MoE LOSS
0.2
0.2
0.2
0.2
0.2
RUNTIME(H800 WITH EVALUATION EVERY 2K STEPS) 3H3M
3H7M
3H2M
3H8M
9H17M
OPTIMIZER
ADAMW
OPTIMIZER MOMENTUM
81, 82 0.9, 0.95
LEARNING RATE
8e -3
LR SCHEDULER
COSINE DECAY 0.1
WEIGHT DECAY (MODEL)
B.2.2 Benchmarks
Genomic Benchmark For the Genomic Benchmark tasks, we follow the experimental setup of B and report their results for comparison. To ensure fair evaluation, fine-tune 52-dimensional models to match the activated parameter count the baselines.
We apply 5-fold cross-validation, splitting the training set into 90/10 train/validation splits and using early stopping based on validation performance with seeds of 21, 3, 5). Models are fine-tuned for 10 epochs with a batch size of 256.
For learning rate selection, we perform hyperparameter tuning over 1 10-3, 10 as and report the best- best-performing configuration across cross cross- validation,,, summarized in Table I0, We use crossyentropy loss for fine-tuning. For JanusDNA, the coefficient for the MoE auxiliary loss set to 0.2. Table 10: JanusDNA models with and without mid-attention hyperparameter selection for learning rate genomic benchmarks for top-1 accuracy averaged over 5-fold cross-validation.
DATASET
w/ MIDATTN w/o MIDATTN
MOUSE ENHANCERS
1e-3
2e 2e-3 3
CODING INTERGENOMIC
le
le
HUMAN VS. WORM
le
le -3
HUMAN ENHANCERS COHN
1e-3
1e
HUMAN ENHANCER ENSEMRI.
1e-3
1e
HUMAN REGULATORY
le-3
le-
HUMAN OCR ENSEMBL
-e-
2c
HUMAN NONTATA PROMOTERS
1e
1e
Nucleotide Transformer Tasks For the Nucleotide Transformer tasks, adopt the experimental setup from 8 and report their results for comparison. To ensure a fair comparison, we fine-tune 72-dimensional models match the activated parameter count of the bascline models.
26

---

We use 10-fold cross-validation with seeds of {1, 2, 3, 5, 7,8,9, 10), splitting the dataset into 90/10 train/validation subsets and applying early stopping based on validation performance. Each model is fine-tuned for 20 epochs.
We cross-entropy loss for fine-tuning. We conduct hyperparameter tuning over the following search space, the same as [ learning rates in x 10 2 x 10 and batch sizes in 128, 256, 512. For each task, we report the best-performing configuration across cross-validation, summarized in Table TT
For JanusDNA, the coefficient for the MoE auxiliary loss is set to 0.2.
Table I1: JA Hyperparameter Selection for Nucleotide Transformer Tasks. Model with Mid-Attention and Model without Mid-Attention fine-tuning hyperparameters chosen based on best performance averaged over 10-fold cross-validation.
JANUSDNA
w/ MIDATTN
w/O MIDATTN
LR
BATCH SIZE
LR
BATCH SIZE
H3
le
256
2e
128
Н3К14AC
1e-3
256
le 3
256
НЗК3бMЕ3
le
256
2e
512
НЗK4MB1
le
256
le
256
HISTONE MARKERS
Н3K4MB2
le 3
256
le
256
Н3K4МB3
le
256
le
256
H3K79ME3
le 1e
256
le
256
НЗК9АC
le
256
le
128
H4
le
256
1e-3
Н4ΔC
le
256
le
256
ENHANCERS
le
512
le
512
ENHANCERS TYPES
256
2e
256
REGULATORY
PROMOTER ALL
le
512
le
128
ANNOTATION
PROMOTER NO TATA
1e-3
256
1e-3
PROMOTER TATA
2e
256
1e
128
SPLICE SITES ACCEPTORS
2e-3
128
2e
128
SPLICE SITE ANNOTATION
SPLICE SITES ALL
2e
128
2e-3
SPLICE SITES DONORS
2e-3
128
2e-3
128
DNALONGBENCH For the DNALONGBENCH eQTL tasks, we compare JanusDNA with both the expert model Enformer and the state-of-the-an architecture Caduceus-PH. We ob- tain the pre-trained weights of Caduceus- from Hugging Face: https:://uugiiggfass co/ kuleshov- group/ caduceus- aeb_seqlen- 1k_ d_modell- 256 1a_layer-44 4-1r--e-13
For all models, we extract the hidden state embeddings from the final layer and apply a pooling layer to obtain fixed-length representation for each input sequence. A linear classification head is then used map these representations the target number of classes for each cell type.
To ensure comparability, we fine-tune the JanusDNA model with 44--iimessional embeddings, matching the activated parameter count of Caduceus- PH. Fine-tuning is conducted for epochs using a learning rate of 4 x and batch size of 8. We use cross-cntron cross-entropy loss for fine-tuning, and for JanusDNA, the coefficient for the MoE auxiliary loss is set 0.02. Training is distributed across eight 80GB GPUs, with each GPU processing one batch. All models are fine-tuned and evaluated using float32 precision to ensure stability and faimess in comparison. Due to limited computational resources, we conduct single run per sub-dataset using the same set of hyperparameters across all experiments.
Transcription Factor Prediction (Mouse) We conducted experiments on non-human Genome Understanding Evaluation (GUE) tasks [10) focusing on transcription factor binding site prediction in mouse genomes to evaluate its generalization capability.
27

---

We followed DNABERT2's & experimental setup, performing standard fine-tuning with a batch size of 32. Unlike DNABERT2's 1000-epoch fine-tuning at a learning rate of 3e-5, we fine-tuned the JanusDNA model for only 10 epochs at learning rate of 1e-3.
Results, shown in Table T2 it is very interesting can perform similarly to DNABERT2 on these tasks, despite being pre-trained only on the human reference genome. In future work, we plan explore further how more diverse pre-training data affects the model performance.
that JanusDNA, with fewer active parameters,
Table 12: Transcription Factor Prediction (Mouse). Performance (1) across various models. Metrics are MCC for different categories (0-4). Best values per category are bolded. Given the disparity in model size, also underline the best value among models with fewer than 100M activated parameters.
TRANSCRIPTION FACTOR PREDICTION (MOUSE)
MODEL
ACTIVATED FARAMS
0
2
100M activated Param. Models
DNABERT-2 (PRE-TRAINC) ON GUE)
117M
0.6412
0.863
0.813
0.735
0,508
DNABERT-2 PRE-TRAINED
WT-50OM-HUMAA
480M
0.310
0.750
0.617
0.292
0.293
NT-500M-10000
0.393
0.755
0.647
0.331
0.340
NT-2500M. 10000
2537M
0.483
0.800
0.701
0.423
0.434
NT-2500M-MUUL MULTT
2537M
0.633
0.838
0.715
0.694
0.471
100M activated Param. Models
DNABERT (3-MER)
86M
0.423
0.791
0.699
0.554
0.420
DNABERT (4-MER)
86M
0.494
0.800
0.726
0.518
DNABERT (S-MER)
87M
0.425
0.793
0.622
0.4999
0.403
(6-MER)
JANUSDNA-72DIG
2.009M
0.619
0.850
0.875
0.843
0.502
B.3 Details of resources used
We use 80GB NVIDIA H100, A100, A800 GPUs for pre-training and fine-tuning.
28


Reviews:
[{"review_id": "gI4vBTcNNp", "reviewer": "Reviewer_hQUx", "summary": "JanusDNA introduces the first **bidirectional DNA foundation model** that overcomes key limitations in genomic sequence modeling: (1) inability to capture long-range dependencies (>10k bp) at single-nucleotide resolution, (2) unidirectional bias in existing models, and (3) low training efficiency of masked language modeling (MLM). Its core innovations include:\n\n- **Hybrid Mamba-Attention-MoE architecture** for efficient million-bp-scale modeling.\n- **Janus Modeling**, a novel pre-training paradigm combining autoregressive training efficiency with bidirectional comprehension.\n- **Reverse-complement integration** to handle bidirectional genomic motifs.\nEvaluated on 35 genomic tasks, it achieves SOTA on 12/18 Nucleotide Transformer tasks and 8/9 eQTL tasks despite using **7.66M activated parameters** (vs. 250× larger baselines).", "questions": "According to Figure 1B, the model processes both strands of the DNA double helix through left-to-right and right-to-left modeling. Given that DNA follows the complementary base-pairing principle (where one strand theoretically determines the sequence of its complement), simultaneous modeling of both strands appears redundant. Could you clarify the advantage and biological significance behind this design choice, or provide some ablation experiments?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### **Strengths & Contributions**\n\n1. **Unprecedented sequence length & resolution**:\n    - Processes **1 million bp** at single-nucleotide resolution on single 80GB GPU, demonstrating good efficiency.\n2. **Architectural innovation**:\n    - The Mamba-Attention-MoE design integrates local/global context modeling with sparse expert scaling, outperforming SSM/Transformer hybrids.\n3. **Biological relevance**:\n    - Explicit modeling of **reverse-complement strands** captures non-palindromic motifs, which is critical for gene regulation.\n4. **Empirical rigor**:\n    - Extensive ablation studies validate design choices, and SOTA results on eQTL prediction surpass specialist models like Enformer.\n        \n        ---\n        \n\n### **Limitations & Weaknesses**\n\n1. **Narrow data scope**:\n    - Trained *only* on the human reference genome (HG38) (Page 7, §4.1), limiting cross-species generalizability. And Lacks epigenetic data (e.g., chromatin states), noted as future work.\n2. **Limited improvement in Genomic Benchmark**\n    1. The model demonstrates suboptimal performance on the Genomic Benchmark, failing to surpass Caduceus baseline models in most tasks. While the authors claim this benchmark is 'saturated', recent work ConvNova [1] achieved significant performance improvements on these same tasks using CNN-based models with fewer parameters, outperforming multiple Mamba and Transformer-based models. The authors should provide comparative analysis against relevant models.\n\n[1] Revisiting Convolution Architecture in the Realm of DNA Foundation Models, https://openreview.net/forum?id=B07dLVWLyD", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer hQUx,\n\nWe are very grateful for your detailed review and recognition of our work's contributions. We have carefully considered your comments and respond as follows:\n\n**W1: Narrow data scope.** We fully agree that the data scope of this work is limited to the human reference genome to achieve the fairest possible comparison with existing methods since the primary contribution of this work is the model architecture and pre-training paradigm itself as you acknowledged. We truly appreciate that. We fully agree that, for many DNA language model applications, diverse pre-training data is crucial and we are actively working on extending the data scope to genomes of additional species, more individual human genomes, and incorporating more modalities such as epigenetic information and different sequencing data types to make the JanusDNA architecture more generalizable and robust. Thank you very much for your suggestion.\n\n**W2: Limited improvement in Genomic Benchmark.** We greatly appreciate your recommendation of this reference paper. We have carefully reviewed it and acknowledge its powerful and novel approach and now include the ConvNova performance in our results tables.\n\nAdditionally, we made further improvements to our architecture and reran all experiments. Specifically, we noticed that the model benefits from additional MLP layers after the final attention layer that fuses the forward and backward information as an example shows below.\n\n```bash\nSequential(\n      (0): Linear(in_features=32, out_features=64, bias=True)\n      (1): SiLU()\n      (2): Linear(in_features=64, out_features=32, bias=True)\n      (3): SiLU()\n    )\n```\n\nThis simple modification does not significantly change the model design or size, but most performance metrics improved considerably. We updated the manuscript with these new results for the cam-ready version. \n\nWe attached the updated results for your reference, where `JanusDNA_mlp` is the latest model: \n\n### Genomic Benchmark\n\nFollowing the settings from Caduceus, we fine-tuned the models with a batch size of 256, obtaining the new results shown in columns 5 and 6, surpassing ConvNova on 4 out of 8 tasks. Since ConvNova did not specify a batch size, we performed additional parameter searches across batch sizes of 128, 256, and 512. The results are shown in columns 7 and 8, surpassing ConvNova on 7 out of 8 tasks. (5-fold cross-validation).\n\n| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| batch |  | 256 | 256 | 256 | 256 | best among 128, 256, 512 | best among 128, 256, 512 |\n|  | ConvNova | JanusDNA | JanusDNA | JanusDNA_mlp | JanusDNA_mlp | JanusDNA_mlp | JanusDNA_mlp |\n|  |  | w/ Mid-Attn | w/o Mid-Attn | w/ Mid-Attn | w/o Mid-Attn | w/ Mid-Attn | w/o Mid-Attn |\n| Size | 386K | 422k | 427k | 426k | 431k | 426k | 431k |\n| MouseEnhancers | 0.784±0.009 | 0.752±0.041 | 0.750±0.031 | 0.770±0.048 | 0.769±0.029 | 0.798±0.029 | **0.802±0.050** |\n| HumanEnhancersCohn | 0.743±0.005 | 0.745±0.004 | 0.740±0.002 | 0.741±0.005 | 0.742±0.006 | **0.743±0.004** | **0.743±0.003** |\n| HumanEnhancerEnsembl | 0.900±0.004 | 0.899±0.003 | 0.900±0.005 | 0.897±0.004 | 0.899±0.004 | 0.901±0.003 | **0.904±0.001** |\n| Coding vs. Intergenomic | **0.943±0.001** | 0.913±0.001 | 0.914±0.002 | 0.912±0.003 | 0.911±0.001 | 0.912±0.003 | 0.913±0.002 |\n| Human vs. Worm | 0.967±0.002 | 0.972±0.001 | 0.971±0.001 | 0.971±0.001 | 0.971±0.001 | 0.971±0.001 | **0.971±0.001** |\n| HumanRegulatory | 0.873±0.002 | 0.877±0.002 | 0.870±0.006 | 0.877±0.005 | 0.868±0.008 | **0.877±0.003** | 0.869±0.002 |\n| HumanNonTATAPromoters | 0.951±0.003 | 0.949±0.006 | 0.945±0.006 | 0.957±0.004 | 0.954±0.010 | **0.957±0.004** | 0.954±0.008 |\n| HumanOCR_Ensembl | 0.793±0.004 | 0.825±0.002 | 0.825±0.003 | 0.822±0.003 | 0.824±0.001 | 0.823±0.003 | **0.824±0.003** |\n\n### Nucleotide Transformer (NT)\n\nPerformance on NT also improved significantly, as shown in columns 6 and 7. Columns 4 and 5 represent performance without the two additional simple MLP layers. (Columns 8 and 9 are for addressing Q1 to save space about ablations on reverse complement design.) (10-fold cross-validation)\n\n| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | Caduceus-PH | Convnova | JanusDNA | JanusDNA | JanusDNA_mlp | JanusDNA_mlp | JanusDNA_mlp | JanusDNA_mlp |\n|  |  |  | w/ midattn | w/o midattn | w/ midattn; w/ rc | w/o midattn; w/ rc | w/ attn; w/o rc; 256, 1e-3 | w/ attn; w/ rc; 256, 1e-3 |\n| size(M) | 1.9 | 1.7 | 1.980 | 1.988 | 2.001 | 2.009 | 2.001  | 2.001 |\n| H3 | 0.815±0.048 | 0.812±0.017 | 0.821±0.021 | 0.824±0.012 | **0.835±0.009** | 0.831±0.023 | 0.789±0.028 | **0.828±0.020** |\n| H3K14ac | 0.631±0.026 | 0.644±0.009 | 0.665 ± 0.034 | 0.685±0.016 | **0.729±0.022** | 0.718±0.026 | 0.689±0.029 | **0.729±0.022** |\n| H3K36me3 | 0.601±0.129 | 0.661±0.019 | 0.658 ± 0.024 | 0.670±0.012 | **0.702±0.015** | 0.699±0.025 | 0.661±0.021 | **0.701±0.022** |\n| H3K4me1 | 0 .523±0.039 | 0.554±0.023 | 0.563 ± 0.041 | 0.571±0.018 | 0.615±0.035 | **0.616±0.018** | 0.574±0.025 | **0.609±0.022** |\n| H3K4me2 | 0.487±0.170 | 0.485±0.032 | 0.509 ± 0.056 | 0.548±0.022 | **0.589±0.023** | 0.586±0.019 | 0.546±0.026 | **0.588±0.020** |\n| H3K4me3 | 0.544 ±0.045 | 0.566±0.027 | 0.605 ± 0.030 | 0.629±0.022 | **0.688±0.026** | 0.675±0.014 | 0.640±0.013 | **0.681±0.016** |\n| H3K79me3 | 0.697±0.077 | 0.700±0.007 | 0.716 ± 0.017 | 0.727±0.023  | **0.747±0.013** | 0.743±0.009 | 0.723±0.025 | **0.747±0.013** |\n| H3K9ac | 0.622±0.030 | 0.658±0.011 | 0.641 ± 0.024 | 0.639±0.019 | **0.673±0.014** | 0.661±0.027 | 0.638±0.023 | **0.673±0.014** |\n| H4 | 0.811±0.022  | 0.808±0.008 | 0.809 ± 0.021 | **0.816±0.008** | 0.812±0.011 | 0.813±0.013 | 0.781±0.020 | **0.810±0.022** |\n| H4ac | 0.621±0.054 | 0.636±0.011 | 0.637±0.060 | 0.653±0.034 | 0.698±0.013 | **0.705±0.023** | 0.653±0.023 | **0.696±0.019** |\n| enhancers | 0 .546 ±0.073 | **0.586±0.038** | 0.564 ± 0.022 | 0.535±0.036 | 0.559±0.042 | 0.542±0.044 | 0.382±0.035 | **0.396±0.033** |\n| EnhancersTypes | 0.439±0.054 | 0.500±0.018 | 0.462±0.049 | 0.470±0.025 | 0.503±0.038 | 0.492±0.096 | 0.475±0.053 | **0.488±0.066** |\n| PromoterAll | 0.970±0.004 | 0.967±0.001 | 0.969±0.002 | **0.971±0.002** | 0.970±0.002 | 0.970±0.003 | 0.964±0.003 | **0.969±0.002** |\n| PromoterNoTata | 0.969±0.011 | 0.968±0.003 | 0.971±0.003 | **0.971±0.002** | 0.971±0.004 | 0.971±0.003 | 0.961±0.004 | **0.969±0.004** |\n| PromoterTata | 0.953±0.016 | **0.969±0.003** | 0.956±0.010 | 0.958±0.008 | 0.958±0.007 | 0.960±0.008 | 0.946±0.012 | **0.954±0.010** |\n| SpliceSitesAll | 0.940±0.027 | 0.965±0.004 | 0.963±0.022 | 0.960±0.009 | **0.967±0.005** | 0.943±0.020 | **0.961±0.003** | 0.948±0.008 |\n| SpliceSitesAcceptors | 0.937±0.033 | **0.971±0.003** | 0.949±0.020 | 0.939±0.022 | 0.957±0.012 | 0.961±0.009 | **0.928±0.010** | 0.906±0.016 |\n| SpliceSitesDonors | 0.948±0.025 | **0.965±0.003** | 0.947±0.015 | 0.936±0.014 | 0.948±0.008 | 0.935±0.016 | **0.915±0.008** | 0.893±0.006 |\n\n### DNALONGBENCH\n\nPerformance also improved significantly after adding the two simple MLP layers.\n\n|  | Caduceus-PH | JanusDNA; w/o midattn | JanusDNA_mlp; w/o midattn|\n| --- | --- | --- | --- |\n| size | 7.7M | 7.662 M | 7.745 M |\n| AT | 0.690 | 0.802 | **0.851** |\n| AS | 0.759 | 0.740 | **0.768** |\n| CCF | 0.689 | 0.770 | **0.801** |\n| MS | 0.789 | 0.803 | **0.864** |\n| NT | 0.841 | 0.877 | **0.913** |\n| SNSES | 0.812 | 0.874 | **0.903** |\n| SSELL | 0.691 | 0.706 | **0.845** |\n| Thyroid | 0.703 | 0.752 | **0.792** |\n| WB | 0.768 | 0.794 | **0.821** |\n\n**Q1: Value of reverse complement design and ablation?**\n\nThank you very much for raising this important point. We fully agree that clarifying the value of the reverse complement (RC) design is crucial.\n\nAs you correctly pointed out, DNA follows the complementary base-pairing principle, meaning each DNA strand has a reverse complement strand with equivalent genetic information. However, despite this theoretical equivalence, many biologically important motifs (e.g., transcription factor binding sites) are non-palindromic. Recognizing both the forward and RC versions of such motifs (for instance, motifs like GATA and TATC) is challenging, as it effectively requires the model to learn two distinct representations. Therefore, explicitly integrating RC information allows the model to more robustly and comprehensively capture DNA sequence patterns.\n\nNonetheless, the usefulness of RC information can depend heavily on the specific biological context. For instance, certain genomic elements such as splice sites are defined strictly by a single DNA strand (the template strand), making the inclusion of RC information potentially suboptimal or even misleading.\n\nTo quantitatively assess the impact of incorporating RC information, we have conducted additional ablation experiments using the NT benchmark. We fine-tuned all models under consistent experimental conditions (learning rate of 1e-3, batch size of 256) across 10-fold cross-validation, with the only difference being with or without of RC information during prediction.\n\nThe results (presented in the last two columns of the above Table on NT, following the main experiments) clearly indicate that models with RC information generally outperform those without RC across most tasks. Notably, the exception to this pattern was observed in splice site tasks, where RC inclusion led to inferior performance, reflecting the biological reality that splice sites are inherently strand-specific.\n\nBased on your insightful suggestion, we will clarify and expand our discussion on this topic in Section 3.2 (Reverse Complement) and include this new experiment in the final manuscript. We truly appreciate your constructive feedback, which has enabled us to further strengthen our work."}, {"review_id": "8u4wmIIpgg", "reviewer": "Reviewer_yT7i", "summary": "The paper proposes a new genomic foundation model (GFM), called JanusDNA, that seeks to improve upon existing GFMs by incorporating the bidirectional nature of DNA in the model architecture, and improves the training efficiency via Janus modelling.  In short, each sequence is processed by two independent stacks of Mamba feed-forward network and Mixture-of-Experts (MOE) layers, in the forward and reverse direction before being combined through a bidirectional fusion mask. The training objective seeks to predict every token, as opposed to just the masked tokens in Masked Language Modeling (MLM), which improves the training efficiency of JanusDNA. Evaluation on multiple genomic benchmarks demonstrates that JanusDNA supersedes existing models on a variety of tasks.", "questions": "**Questions:**\n\n1.\tWhy is Janus pre-training faster than masked models (per 1000 steps)? On first looks, there appear to be more operations per training step in Janus modeling.\n2.\tLines 200-204 makes a distinction between masked models and the Janus model. What is the difference between the T-th token being masked for masked models and it being omitted for the Janus model? Is this distinction meaningful?\n3.\tHow is the JanusDNA model fine-tuned (line 317)?\n\n**Suggestions:**\n\n1.\tSSM is first referred to in line 61 but the full form of this term is not stated until later. I recommend defining the acronym here.\n2.\tWhen the hybrid MoE is introduced in Section 3.2, it is not evident how this helps JanusDNA deal with longer sequences.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The core idea presented in the paper is sound and draws its motivation from biological domain knowledge. In particular, JanusDNA accounts for (a) the reverse complementarity of the DNA double-strand structure and (b) the bidirectional nature of context in DNA sequences.\n\nThe paper is also largely clear in its presentation of its ideas with a natural flow and good use of visualizations to aid the reader In understanding the various components of the proposed model. The use of formal mathematical notations also helps with the clarity of the presented ideas. \n\nI would also like to laud the authors on including error bounds for the results in Tables 1 and 2. \n\nWhile I do not believe there are significant weaknesses in this work, there are several questions that ought to be answered. Most of these have been delineated In a subsequent section but there are a couple of questions that I would like to raise here.\n\n1.\tIn Eq, 6, how did the authors arrive at the current formulation of the cross-attention in the bidirectional fusion mask? The text refers to the need “to maintain the integrity of bidirectional prediction without future peeking” but it is not clear to me what this means.\n2.\tWhen discussing the training efficiency towards the end of Section 3.1, It is unclear what the metric for computational complexity is. There is reference to time taken per 1000 steps in the text, but Figure 3 plots the number of training steps. A well-established measure of complexity, clock time or FLOPs, would be more convincing here.", "was_revised": true, "final_justification": "The additional clarifications and results have significantly strengthened the paper. I have raised my score accordingly to reflect this.", "rebuttal": "Dear Reviewer yT7i,\n\nWe sincerely appreciate your detailed review and recognition of our work. Below, we address your thoughtful and constructive suggestions.\n\n**Q1: Detailed explanation of Eq. 6** \n\nThank you very much for highlighting this point; we are happy to clarify.\n\nFirst, the unidirectional representation follows an autoregressive structure, where, for example, the representation at position 0 predicts the token at position 1, the representation at position 1 predicts the token at position 2, and so forth.\n\nThus, it is understandable that the masks in the self-attention regions (top-left and bottom-right in Figure 1(E)) allow attention to themselves, as reflected by the first and second lines in Eq. 6.\n\nSecond, the cross-attention parts (top-right and bottom-left) indicate fusion of the two unidirectional representations. Similar to the forward autoregressive representation, the reverse representation uses the representation at position T to predict token T-1, position T-1 predicts token T-2, etc. Therefore, to predict a middle token (excluding tokens at positions 0 and T) using these bidirectional features, we account for an offset of \"+2\". For instance, when predicting token 5 (with indexing starting from 0), the forward representation at position 4 and the backward representation at position 6 are utilized. However, if here is \"+1\", for instance, when predicting token 5 (with indexing starting from 0), the forward representation at position 4 and the backward representation at position 5 are utilized, but the backward representation at position 5 includes the original information of token 5, causing information leakage, and training loss will decrease to 0 quickly correspondingly.\n\nWe arrived at the formulation illustrated in Figure 1(E), which we have now improved to clearly indicate that the diagonal in the top-left and bottom-right region should also be unmasked. We also removed the confusing phrase “future peeking” in the manuscript and now clarify that we want to prevent information leakage. Thank you again for bringing this to our attention.\n\n**Q2: Metrics of computational complexity (refine figure)**\n\nThank you for suggesting a more common complexity metric. We now additionally present the training efficiency based on the clock time required for model training along with corresponding performance on the last-token prediction task.\n\nListing these details in a table may not be intuitive; therefore, in camera ready version of the paper, we update Figure 3 by plotting accuracy against training time to clarify this comparison. Thank you for this valuable recommendation.\n\n| Accuracy on Last Token Prediction |  |  |  |  |  |  |  |  |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | mask-32dim |  | janus-32dim |  | mask-64dim |  | janus-64dim |  | mask-128dim |  | janus-128dim |\n| Training time(mins) | 0.056 M | Training time(mins) | 0.056 M | Training time(mins) | 0.207 M | Training time(mins) | 0.207 M | Training time(mins) | 0.795 M | Training time(mins) | 0.795 M |\n| 60 | 0.404929 | 25.50 | 0.427821 | 60.5 | 0.4102496 | 25.50 | 0.439397 | 60 | 0.425551 | 25.50 | 0.444796 |\n| 120 | 0.434058 | 51.00 | 0.444575 | 121 | 0.4411103 | 51.00 | 0.451205 | 120 | 0.447582 | 51.00 | 0.459588 |\n| 180 | 0.441789 | 76.50 | 0.449629 | 181.5 | 0.4478824 | 76.50 | 0.461554 | 180 | 0.450395 | 76.50 | 0.471902 |\n| 240 | 0.443333 | 102.00 | 0.450901 | 242 | 0.4508481 | 102.00 | 0.466316 | 240 | 0.453748 | 102.00 | 0.473491 |\n| 300 | 0.444255 | 127.50 | 0.452616 | 302.5 | 0.4521624 | 127.50 | 0.470291 | 300 | 0.456034 | 127.50 | 0.479219 |\n| 360 | 0.447273 | 153.00 | 0.454893 | 363 | 0.4541665 | 153.00 | 0.472556 | 360 | 0.459998 | 153.00 | 0.481693 |\n| 420 | 0.449191 | 178.50 | 0.457761 | 423.5 | 0.4563688 | 178.50 | 0.474091 | 420 | 0.465365 | 178.50 | 0.484446 |\n| 480 | 0.450238 | 204.00 | 0.459449 | 484 | 0.4587279 | 204.00 | 0.477486 | 480 | 0.469885 | 204.00 | 0.48797 |\n| 540 | 0.451589 | 229.50 | 0.460683 | 544.5 | 0.4609852 | 229.50 | 0.478885 | 540 | 0.473288 | 229.50 | 0.490162 |\n| 600 | 0.452361 | 255.00 | 0.461319 | 605 | 0.461945 | 255.00 | 0.47961 | 600 | 0.474498 | 255.00 | 0.491077 |\n\n**Q3: Why is Janus pre-training faster than masked models (per 1000 steps)? On first looks, there appear to be more operations per training step in Janus modeling.**\n\nFor this ablation experiment, to keep the comparison as fair as possible, we used the same architectures for both the Janus and the masked language model, except masks for the final attention layer. \n\nThe Janus model consists of one layer with two parallel unidirectional encoders followed by a FlexAttention layer that incorporates a specially designed sparse attention mask for Janus-type feature fusion. As depicted in Figure 1(E) (which we revised to make the masking and prediction indices more intuitive to understand), the two unidirectional features are concatenated into a tensor of shape (batch, sequence_length × 2, dim).\n\nCompared to the Janus model, the architecture of the masked model is similar, where there is one layer of two separated parallel unidirectional encoders (same as Janus model) followed by a FlexAttention, but with no attention mask. This is because for masked training, partial tokens have been masked in the dataset loading process, causing the full attention requirement for bidirectional feature fusion. Meanwhile, to keep the comparison as fair as possible, the two unidirectional features from the layer of unidirectional encoders will also be concatenated as a tensor with the shape of (batch, sequence_length * 2, dim). \n\nTherefore, the speed difference primarily arises from the attention stage, where the Janus model employs sparse attention, while the masked model uses dense, full attention. Additionally, the masking pre-processing for the masked model incurs some computational overhead. Consequently, the Janus model achieves faster performance than the masked model when the architectures are identical.\n\nImportantly, an alternative implementation of the masked model would concatenate features along the feature axis rather than the sequence axis, resulting in final attention over a tensor with shape (batch, sequence_length, dim * 2). This approach would make the computational costs of Janus and masked pretraining nearly equivalent. However, we opted against this implementation in our ablation study because concatenating along the feature axis would alter the masked model's capacity, compromising the comparability between the two approaches. We have now clarified this point in the description of the ablation experiment.\n\n**Q4: Lines 200-204 makes a distinction between masked models and the Janus model. What is the difference between the T-th token being masked for masked models and it being omitted for the Janus model? Is this distinction meaningful?**\n\nThank you for closely examining this point. There is indeed no distinction; all models receive identical inputs, where the last token is masked for prediction. We will revise this description in future versions to eliminate confusion. We appreciate your careful observation.\n\n**Q5: How is the JanusDNA model fine-tuned (line 317)?**\n\nFor the DNALONGBENCH tasks, all models, including Caduceus and JanusDNA, were fine-tuned entirely (full model fine-tuning) using a learning rate of 4e-4, a batch size of 8, and one batch per GPU over three epochs. We specified these hyper-parameters explicitly in Appendix D.2, lines 579–583, and will ensure this is clearly stated in future revisions. \n\nThank you for pointing out this ambiguity.\n\n**S1: SSM is first referred to in line 61 but the full form of this term is not stated until later. I recommend defining the acronym here.**\n\nThank you for identifying this oversight. We will define the acronym \"SSM\" upon its first mention in line 61 in the camera ready version of the manuscript.\n\n**S2: When the hybrid MoE is introduced in Section 3.2, it is not evident how this helps JanusDNA deal with longer sequences.**\n\nWe appreciate your insightful comment. Our intention was not to claim explicitly that MoE handles longer sequences more effectively. Rather, MoE models achieve superior performance compared to dense models with an equivalent number of inference parameters. This can be observed in Figure 4, where MoE models consistently demonstrate improved perplexity compared to dense models, across both shorter (1024) and longer (131072) sequence lengths.\n\nMeanwhile, for many downstream tasks of DNA language models, we need to run inference on the entire genome (3 billion nucleotide pairs), potentially for many individuals. Therefore inference performance is crucial, requiring better  performance while less running time, and MoE helps with that. We will clarify this point in the discussion in the camera ready version of the manuscript.\n\nThank you once again for your valuable feedback. Your suggestions and comments will significantly enhance the quality of our manuscript, and we are committed to making the necessary improvements to meet the high standards of the NeurIPS community."}, {"review_id": "4ggto1PDRL", "reviewer": "Reviewer_tdwT", "summary": "This paper introduces JanusDNA, a novel bidirectional DNA foundation model that addresses key limitations in current genomic language models. The authors propose a new pretraining paradigm called \"Janus modeling\" that combines the training efficiency of autoregressive models with the bidirectional understanding capabilities of masked language models. The model architecture employs a hybrid Mamba-Attention-Mixture-of-Experts (MoE) design, enabling it to process sequences up to 1 million base pairs at single-nucleotide resolution on a single GPU. The model is evaluated on three genomic benchmarks, achieving state-of-the-art performance on many tasks while using significantly fewer parameters than competing models.", "questions": "1- Can the authors provide a more detailed explanation of why Janus modeling is nearly twice as fast as masked modeling? This seems counterintuitive given the bidirectional processing. Did you ensure both models used the same architecture for fair comparison? \n\n2- Could you provide an ablation study comparing MLM, CLM, and Janus modeling objectives using the same model architecture? This would help isolate the contribution of your proposed training paradigm.\n\n3- The paper mentions processing both forward and reverse complement strands. What is the computational overhead of this approach, and have you ablated its contribution to performance?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1- The Janus modeling approach is innovative, allowing each token to be predicted using its full bidirectional context while maintaining high training efficiency. This addresses a fundamental trade-off in language model pretraining.\n\n2- The hybrid architecture combining Mamba, Attention, and MoE is novel.\n\n3- JanusDNA achieves state-of-the-art performance on multiple benchmarks.\n\nWeaknesses:\n1- There is an inconsistency in the training efficiency claims (lines 198-199). The paper states that Janus models train \"nearly twice as fast as masked models,\" but this seems counterintuitive given the bidirectional processing and attention mechanisms involved. The explanation for this efficiency gain is insufficient.\n\n2- The model is only pretrained on the human reference genome (HG38) for fair comparison with baselines. This limits the model's potential and generalizability compared to models trained on more diverse genomic data.\n\n3-  The paper would benefit from a direct comparison of different pretraining objectives (MLM, CLM, and Janus modeling) using the same architecture to isolate the contribution of the training paradigm.", "was_revised": true, "final_justification": "I have raised my score to recommend Accept. This change is a direct result of the authors' outstanding rebuttal, which convincingly addressed my primary reservations. My concerns about the model's training efficiency were clarified with a detailed architectural explanation. More importantly, the authors conducted two new, substantial ablation studies during the rebuttal period. These experiments provide the strong empirical evidence that was previously missing, first by directly comparing the Janus pre-training paradigm against a causal language model, and second by quantitatively assessing the impact of the reverse complement strategy. These additions have transformed the paper from a promising concept into a well-supported contribution.\n\nThe single remaining limitation, the pre-training on only the human reference genome, is acceptable in this context. The authors rightly justified this as a necessary choice for fair comparison with baselines in a paper focused on methodological novelty, and they have acknowledged it as an area for future work. The weight of the successfully resolved issues, backed by new data, far outweighs this reasonable scope limitation.", "rebuttal": "Dear reviewer tdwT,\n\nWe greatly value your time and effort in providing this constructive review and recognition of our work. We carefully considered your suggestions and propose the following response for your further review:\n\n**W1&Q1: Training speed (Janus model vs masked model)**\n\nThanks very much for pointing this out. For the Janus model, the architecture is one layer of two separated parallel unidirectional encoders followed by a FlexAttention with a specifically designed attention mask for Janus-type feature fusion. As shown in Figure 1(E) (which needs to be modified a bit that the diagonal of the left top part should be unmasked), two unidirectional features are concatenated shaped (batch, sequence_length * 2, dim) , and the attention mask of Janus fusion layer is sparse due to its design. \n\nCompared to the Janus model, the architecture of the masked model is similar, where there is one layer of two separated parallel unidirectional encoders (same as Janus model) followed by a FlexAttention, but with no attention mask. This is because for masked training, partial tokens have been masked in the dataset loading process, causing the full attention requirement for bidirectional feature fusion. Also, to keep the comparison fair, the two unidirectional features from the layer of unidirectional encoders will also be concatenated shaped (batch, sequence_length * 2, dim). \n\nThus, the speed difference primarily arises from the attention stage, where the Janus model employs sparse attention, while the masked model uses dense, full attention. Additionally, the masking pre-processing for the masked model incurs some computational overhead. Consequently, the Janus model achieves faster performance than the masked model when the architectures are identical.\n\nImportantly, an alternative implementation of the masked model would concatenate features along the feature axis rather than the sequence axis, resulting in final attention over a tensor with shape (batch, sequence_length, dim * 2). This approach would make the computational costs of Janus and masked pre-training nearly equivalent. However, we opted against this implementation in our ablation study as concatenating along the feature axis would alter the masked model's capacity, compromising the comparability between the two approaches. We have now clarified this point in the description of the ablation experiment.\n\n**W2: limited pre-training data**\n\nWe fully agree that the data scope of this work is limited to the human reference genome to achieve the fairest possible comparison with existing methods since the primary contribution of this work is the model architecture and pre-training paradigm itself as you acknowledged. We truly appreciate that. We fully agree that, for many DNA language model applications, diverse pre-training data is crucial and we are actively working on extending the data scope to genomes of additional species, more individual human genomes, and incorporating modalities such as epigenetic information and different sequencing data types to make the JanusDNA architecture more generalizable and robust. Thanks very much for your suggestion.\n\n**W3&Q2: MLM, CLM, Janus modeling ablation upon the same model architecture?**\n\nThank you for this insightful suggestion. We conducted an ablation comparing MLM and Janus modeling using the same architecture as in Figure 3. However, conducting a direct ablation between CLM and Janus/MLM is challenging due to inherent architectural differences. The core strength of Janus modeling lies in its bidirectional context understanding, requiring a bidirectional architecture. CLM, by definition, is strictly unidirectional, and adapting it to a bidirectional architecture would conflict with its fundamental principles.\n\nNevertheless, we acknowledge the value in exploring this further. We ran a new ablation experiment to compare the performance of a unidirectional CLM and a bidirectional Janus model. In this comparison, we expected similar performance for the prediction of the last token (where there is no bidirectional information, so both models have exactly the same information to predict the last token), whereas we expected Janus to outperform CLM for the prediction of a token in the center of the sequence (where Janus benefits from the bidirectional context). To maintain fairness, causal models were constructed with one layer of a single mamba encoder and a FlashAttention2 layer with a causal attention mask, ensuring comparable model sizes to the Janus models.\n\nThe results of this new experiment are presented in the table below and will be included in the camera ready version of the manuscript. We confirm that the Janus model consistently outperforms the CLM for tokens in the center of the sequence across all evaluated model sizes. To our surprise, Janus also slightly outperforms the CLM even on the last token prediction task, suggesting Janus models indeed learn richer DNA representations through bidirectional training. We will clarify this point further in the manuscript.\n\n|  | last token |  |  |  |  |  | middle token |  |  |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | janus |  |  | clm |  |  | janus |  |  | clm |  |  |\n| size(M) | 0.056 | 0.207 | 0.795 | 0.061 | 0.209 | 0.814 | 0.056 | 0.207 | 0.795 | 0.061 | 0.209 | 0.814 |\n| step(k)\\dim | 32 | 64 | 128 | 40 | 76 | 152 | 32 | 64 | 128 | 40 | 76 | 152 |\n| 1 | 0.428 | 0.439 | 0.445 | 0.421 | 0.426 | 0.434 | 0.428 | 0.436 | 0.440 | 0.384 | 0.400 | 0.408 |\n| 2 | 0.445 | 0.451 | 0.460 | 0.454 | 0.441 | 0.453 | 0.438 | 0.454 | 0.465 | 0.429 | 0.425 | 0.443 |\n| 3 | 0.450 | 0.462 | 0.472 | 0.443 | 0.447 | 0.471 | 0.437 | 0.449 | 0.467 | 0.436 | 0.446 | 0.454 |\n| 4 | 0.451 | 0.466 | 0.473 | 0.443 | 0.451 | 0.468 | 0.443 | 0.465 | 0.471 | 0.434 | 0.451 | 0.454 |\n| 5 | 0.453 | 0.470 | 0.479 | 0.454 | 0.457 | 0.478 | 0.444 | 0.477 | 0.477 | 0.434 | 0.449 | 0.451 |\n| 6 | 0.455 | 0.473 | 0.482 | 0.451 | 0.453 | 0.481 | 0.456 | 0.474 | 0.482 | 0.428 | 0.442 | 0.446 |\n| 7 | 0.458 | 0.474 | 0.484 | 0.449 | 0.457 | 0.482 | 0.458 | 0.471 | 0.477 | 0.438 | 0.457 | 0.461 |\n| 8 | 0.459 | 0.477 | 0.488 | 0.459 | 0.465 | 0.479 | 0.453 | 0.478 | 0.478 | 0.434 | 0.455 | 0.461 |\n| 9 | 0.461 | 0.479 | 0.490 | 0.453 | 0.467 | 0.479 | 0.458 | 0.481 | 0.485 | 0.447 | 0.458 | 0.465 |\n| 10 | 0.461 | 0.480 | 0.491 | 0.459 | 0.465 | 0.484 | 0.456 | 0.482 | 0.488 | 0.443 | 0.456 | 0.465 |\n\n**Q3: Computational overhead of RC and ablation on its value?**\n\nThank you very much for raising this key point. We fully agree that clarifying the computational overhead and performance impact of RC is essential.\n\nIncluding RC enables the model to more robustly and comprehensively capture DNA sequence patterns, especially for biologically important motifs (e.g., transcription factor binding sites) that are non-palindromic. Recognizing both forward and RC versions of such motifs (for instance, motifs like GATA and TATC) is challenging, as it effectively requires the model to learn two distinct representations. Nonetheless, the utility of RC depends heavily on the specific biological context. For instance, genomic elements such as splice sites are strictly defined by a single DNA strand, making RC inclusion potentially suboptimal or misleading.\n\nRegarding computational overhead, RC is introduced only during inference by averaging the embedding representations of a strand and its RC prior to decoding. Thus, the additional computational cost is minimal, primarily involving an extra forward pass through the model for the RC strand.\n\nTo quantitatively assess the impact of incorporating RC, we conducted ablation experiments using the NT benchmark. We fine-tuned models under consistent experimental conditions (learning rate 1e-3, batch size 256), with and without RC during prediction. The results clearly indicate that models utilizing RC generally outperform those without RC across most tasks. Notably, the exception to this pattern was observed in splice site prediction tasks, where RC inclusion led to inferior performance, consistent with the biological reality that splice sites are inherently strand-specific.\n\nBased on your insightful feedback, we will clarify this discussion further in Section 3.2 and include the ablation experiment results explicitly in the final manuscript. We sincerely appreciate your constructive comments, which have helped strengthen our paper.\n\n|  | w/ attn; w/o rc | w/ attn; w/ rc | w/o attn; w/o rc | w/o attn; w/ rc |\n| --- | --- | --- | --- | --- |\n| H3 | 0.789±0.028 | **0.828±0.020** | 0.795±0.026 | **0.830±0.015** |\n| H3K14ac | 0.689±0.029 | **0.729±0.022** | 0.662±0.015 | **0.700±0.015** |\n| H3K36me3 | 0.661±0.021 | **0.701±0.022** | 0.658±0.016 | **0.688±0.012** |\n| H3K4me1 | 0.574±0.025 | **0.609±0.022** | 0.555±0.030 | **0.605±0.028** |\n| H3K4me2 | 0.546±0.026 | **0.588±0.020** | 0.532±0.020 | **0.581±0.024** |\n| H3K4me3 | 0.640±0.013 | **0.681±0.016** | 0.625±0.015 | **0.675±0.014** |\n| H3K79me3 | 0.723±0.025 | **0.747±0.013** | 0.710±0.020 | **0.743±0.009** |\n| H3K9ac | 0.638±0.023 | **0.673±0.014** | 0.631±0.016 | **0.658±0.020** |\n| H4 | 0.781±0.020 | **0.810±0.022** | 0.775±0.019 | **0.813±0.011** |\n| H4ac | 0.653±0.023 | **0.696±0.019** | 0.629±0.017 | **0.684±0.020** |\n| enhancers | 0.382±0.035 | **0.396±0.033** | 0.379±0.041 | **0.397±0.065** |\n| EnhancersTypes | 0.475±0.053 | **0.488±0.066** | 0.490±0.046 | **0.492±0.096** |\n| PromoterAll | 0.964±0.003 | **0.969±0.002** | 0.962±0.003 | **0.970±0.002** |\n| PromoterNoTata | 0.961±0.004 | **0.969±0.004** | 0.961±0.004 | **0.970±0.005** |\n| PromoterTata | 0.946±0.012 | **0.954±0.010** | 0.947±0.010 | **0.953±0.019** |\n| SpliceSitesAll | **0.961±0.003** | 0.948±0.008 | **0.946±0.007** | 0.922±0.019 |\n| SpliceSitesAcceptors | **0.928±0.010** | 0.906±0.016 | **0.932±0.014** | 0.904±0.008 |\n| SpliceSitesDonors | **0.915±0.008** | 0.893±0.006 | **0.921±0.009** | 0.874±0.011 |"}, {"review_id": "OCdJztJflx", "reviewer": "Reviewer_BnYd", "summary": "This paper proposes JanusDNA, a hybrid SSM-transformer MoE architecture applied to genomics foundation models. JanusDNA achieves competitive or SOTA performance across a wide array of tasks such as the genomics benchmark, NT benchmark, and DNA long range benchmark. A distinction of JanusDNA from previous approaches is that it leverages the forward and reverse reads of genomics sequencing to create a new type of inductive biases, which are shown by the authors to be more efficient than uni-directional autoregression and MLM.", "questions": "NA", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### __Strengths__\n\n- Genomics foundation model is an important topic that has received a lot of attention. Having an additional model that contribute to this space is a plus.\n- Unlike natural language, genomics sequences sourced in the wild may not attain a canonical order (i.e. they may be read from left to right or right to left). Thus, bidirectional attention certainly has a better inductive bias for genomics modeling, but as numerous research in scaling language models have shown, MLM does not scale well compared to autoregressive models. I believe the two-stream design for forward & backward reads is appropriate in this context.\n- Orthogonal to the design of inductive bias, the authors have explored an array of best practices in scalable foundation model variants, most notably, the MoE architecture and hybrid SSM-attention models. While these improvements are not novel, to this reviewer's knowledge this paper is the first in investigating their application in genomics modeling.\n- Janus-DNA achieves competitive results across a wide array of genomics modeling tasks.\n\n### __Weaknesses__\n\n- While I agree that the choice of dual-stream autoregression is appropriate, it'd be good to include a discussion about this point to contextualize this choice _for genomics modeling_. \n- As the authors have discussed, Janus-DNA is only pretrained on Human Reference Genome and may lack the knowledge needed for modeling other species, but it'd be good to see some performance on these tasks (such as EPM and Mouse from GUE) to understand the performance gap.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer BnYd,\n\nWe sincerely express our gratitude for your recognition of our work and your insightful, detailed feedback. We have carefully formulated the following responses:\n\n**W1: While I agree that the choice of dual-stream autoregression is appropriate, it'd be good to include a discussion about this point to contextualize this choice *for genomics modeling*.**\n\nThank you very much for raising this important point. We are very thankful for the help in making this statement more clear. \n\nAutoregressive modeling relies on unidirectional representations, where understanding and predicting a current pattern depends on preceding ones. This approach is extensively applied in Natural Language Processing (NLP), as the logical sequence of human language allows subsequent elements to be inferred from prior context. Such modeling naturally aligns with language generation tasks. However, DNA, as a complex biological \"language,\" exhibits distinct patterns significantly different from human languages in regulating phenotypes. Unlike human languages, genomic processes frequently involve bidirectional interactions, with critical regulatory elements located both upstream and downstream of key genomic regions.\n\nFor example, bidirectional promoters initiate transcription in both orientations [1, 2]. Additionally, intergenic enhancers transcribed predominantly bidirectionally often function as weak promoters in both directions. Conversely, for elements with unidirectional transcription (both enhancers and promoters), transcription direction typically correlates with the orientation in which the element functions as a promoter in vivo [3].\n\nThus, bidirectional modeling is essential for comprehensively understanding genomic processes.\n\nIn addition to dual-stream autoregression, masked modeling is a classic method for bidirectional sequence modeling. However, due to its masking logic, only a subset of tokens contribute to model optimization, as discussed in our \"Empirical Validation of Training Efficiency\" section (Section 3.1).\n\nTherefore, Janus modeling, the dual-stream autoregressive method proposed in our work, facilitates effective bidirectional DNA modeling while maintaining high learning efficiency, making it particularly suitable for genomic modeling.\n\nWe briefly addressed this topic in Section 1 (lines 55–60). Incorporating the above discussion will further enhance clarity and biological relevance. We will include this expanded content in the camera-ready version, as you suggested. Thank you again for highlighting this important point.\n\n[1] Wei, Wu, et al. \"Functional consequences of bidirectional promoters.\" Trends in Genetics 27.7 (2011): 267-276.\n\n[2]Schulz, Daniel, et al. \"Transcriptome surveillance by selective termination of noncoding RNA synthesis.\" Cell 155.5 (2013): 1075-1087.\n\n[3]Mikhaylichenko, Olga, et al. \"The degree of enhancer or promoter activity is reflected by the levels and directionality of eRNA transcription.\" Genes & development 32.1 (2018): 42-57.\n\n**W2: As the authors have discussed, Janus-DNA is only pretrained on Human Reference Genome and may lack the knowledge needed for modeling other species, but it'd be good to see some performance on these tasks (such as EPM and Mouse from GUE) to understand the performance gap.**\n\nThank you very much for your interest in exploring further possibilities for our work; we greatly appreciate it. Currently, we are integrating genomic data from additional species, variants, and incorporating modalities, such as epigenetic states (e.g., chromatin accessibility, histone modifications) and single-cell transcriptomic profiles. These additions are vital for understanding cell-type-specific regulation and predicting chromatin-influenced phenotypes within the JanusDNA framework.\n\nWe fully agree that evaluating models pre-trained on the human reference genome across other species to determine performance gaps will help explore genomic similarities between species and contribute to future biological research. We will pursue this direction in our future work. We express our sincere gratitude for your insightful suggestions and the valuable research ideas proposed.\n\n\nThank you once again for your valuable feedback. Your comments significantly enhance the quality of our manuscript, and we are committed to making the necessary improvements to meet the high standards of the NeurIPS community."}]

Author rebuttal:

Meta-review: All reviewers recommended acceptance, citing the paper’s technical soundness, biological relevance, and strong empirical results. Concerns about training efficiency and generalization were well addressed during the rebuttal phase. The additional ablation studies and expanded discussion on biological motivations further strengthened the submission.
Decision: Accept (poster)