Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA
Authors: Lifeng Qiao
Peng Ye
Yuchen Ren
Weiqiang Bai
chaoqi liang
Xinzhu Ma
Nanqing Dong
Wanli Ouyang
Venue: NeurIPS 2024 poster (2024)
Keywords: genomics
tokenization
foundation models
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: Existing DNA tokenization methods borrowed from NLP are poorly suited for DNA's unique properties. We introduce MxDNA, a framework that allows the model to learn an effective DNA tokenization strategy, showing robust performance on various tasks.

Abstract: Foundation models have made significant strides in understanding the genomic language of DNA sequences. However, previous models typically adopt the tokenization methods designed for natural language, which are unsuitable for DNA sequences due to their unique characteristics. In addition, the optimal approach to tokenize DNA remains largely under-explored, and may not be intuitively understood by humans even if discovered. To address these challenges, we introduce MxDNA, a novel framework where the model autonomously learns an effective DNA tokenization strategy through gradient decent. MxDNA employs a sparse Mixture of Convolution Experts coupled with a deformable convolution to model the tokenization process, with the discontinuous, overlapping, and ambiguous nature of meaningful genomic segments explicitly considered. On Nucleotide Transformer Benchmarks and Genomic Benchmarks, MxDNA demonstrates superior performance to existing methods with less pretraining data and time, highlighting its effectiveness. Finally, we show that MxDNA learns unique tokenization strategy distinct to those of previous methods and captures genomic functionalities at a token level during self-supervised pretraining. Our MxDNA aims to provide a new perspective on DNA tokenization, potentially offering broad applications in various domains and yielding profound insights. Code is available at https://github.com/qiaoqiaoLF/MxDNA.

Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA
Lifeng Qiao Peng Ye 1.3f, Yuchen Ren 1.4 Weiqiang Bai', Chaoqi Liang',
Xinzhu Ma Nanqing Dong', Wanli Ouyang
Shanghai Artificial Intelligence Laboratory, SShanghai Jiao Tong University The Chinese University of Hong Kong. "The University of Sydney yepeng@pjlab.. waw.la......... .rg.cn
Abstract
Foundation models have made significant strides in understanding the genomic language of DNA sequences. However, previous models typically adopt the tok- enization methods designed for natural language, which are unsuitable for DNA sequences due to their unique characteristics. In addition, the optimal approach tokenize DNA remains largely under-explored, and may not be intuitively understood by humans even discovered. To address these challenges, we in- troduce MxDNA, novel framework where the model autonomously learns effective DNA tokenization strategy through gradient decent. MxDNA employs sparse Mixture of Convolution Experts coupled with deformable convolu- tion to model the tokenization process, with the discontinuous, overlapping, and ambiguous nature of meaningful genomic segments explicitly considered. On Nucleotide Transformer Benchmarks and Genomic Benchmarks, MxDNA demon- strates superior performance to existing methods with less pretraining data and time, highlighting its effectiveness. Finally, we show that MxDNA learns unique tokenization strategy distinct those of previous methods and captures genomic functionalities at token level during self-supervised pretraining. Our MxDNA aims to provide new perspective on DNA tokenization, potentially offering broad applications in various domains and yielding profound insights. Code is available at htttechuu..oomgggeeeeeeeeeeeeeoe http:/iiaqiioo///MM
Introduction
Foundation models natural language processing (NLP) have achieved remarkable success, trans- forming how machines understand and generate human language 2123 Inspired by this success, researchers are now exploring the application foundation models to decode the complex "language" of genomic sequences, aiming to potentially revolutionize our understanding of genomics 5 6 7
Tokenization, critical initial step in NLP models, leverages human knowledge of natural language structures such as grammar and punctuation segment text into meaningful units. However, DNA sequences present a distinct challenge: they lack natural delimiters and their "grammar" is not not readily understood by humans. These challenges make the tokenization process of DNA sequences not straightforward.
Various tokenization methods have been employed by existing foundation models to analyse DNA sequences 7 4 6
(. For example, single nucleotide tokenization 7 treats each nucleotide as an segments the DNA blocks of k consecutive nucleotides, and Byte-Pair
individual token, K-mer A 6
Work done during intersshi Shanghai Artificial Intelligence Laboratory. Corresponniig Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Simple Heuristics
Simple Heuristics
Information -oth-eeeeiiiic
Ideal G enomic Tokenization
Natural
tolentistion
We proe Barticattion
toleentzation
Language
Meeningful:
ATCGGTCAGCTAGCTNGC
Single Character
Character N-gram
Byte Pair Encoding ng
D Disconlinoou:
ATC
TCAGC
TAGC
A Appicaaion
Direct
Nye
DNABERT Nuclaotide Transformer
DNA BERT2
TAGC
Ovedapping:
ATCGTTCAGTAGCTAGC
Single Nucleotide
K-mer
Byte Pair Encoding
Genomic Language
ARTECTAACTIACCAGE
AECOCTCACCTBEEAGE
GACTAACAAG
Ambiguous:
ATCGSTCAGCTAGCTAGG
ATCGGTCAGCTAGCTAGC
The progression from basic tokeniza- tion methods to more sophisticated techniques, with the direct but unsuitable applications from natural language to genomic language. Right: the ideal tokenization properties for Benomics--Meaninge Discontinuous, Overlapping, and Ambiguous-onn outlined in (8) which our MxDNA aims to achieve.
Figure I: Evolution of tokenization and Ideal Properties. Left:
Encoding (BPE) iteratively merges the most frequent pairs of existing tokens. All of these methods are borrowed directly from NLP as depicted in Fig. [ each with its own inherent limitations. Single nueleotide tokenization, while offering high resolution for input, leads an extremely large number of tokens, significantly increasing the complexity of the model. K-mer comes in two forms: overlapping and non-overlapping. Overlapping K-mer, despite its attempt to capture more contextual information, does not offer substantial benefits over single nucleotide approaches and can suffer from information leakage [.. overlappag K-mer greatly reduces tokenized sequence length but can disrupt potentially meaningful unit by splitting it into separate K-mers. BPE, adopted from NLP, attempts to optimize vocabulary size but often results in suboptimal segmentation that may not correspond to meaningful units [10
Unlike natural languages. where linguistically meaningful units such as words and sentences are almost standardized and well understood, the optimal approach to tokenize DNA remains under- explored due to the complex and varied nature of genomics. In NLP, common tokenization strategies have been validated by human knowledge, but such understanding does not extend to DNA. Con- sequently, rather than relying manually crafted tokenization rules, may be better to trust a neural network to learn and determine the most effective tokenization strategy for genomic sequences. Addi- tionally, recent research suggests that biologically meaningful protein tokens can be discontinuous, overlapping, and may require mapping to several tokenization possibilities [8] 12 13,, properties that are likely applicable to DNA sequences due to the genetic central dogma 14. To handle these complexities, can further equip model with capabilities manage discontinuities, overlaps,
and the ambiguities of genomic sequences explicitly.
Building on the analysis above, we introduce MxDNA ("Mx" draws from Mixture of Experts (15), a novel framework designed autonomously learn an effective DNA tokenization strategy solely through gradient decent. The core of the framework starts with a sparse Mixture of Convolution Experts that identifies and embeds basic units within DNA sequences. Unlike conventional Mix- ture of Experts models, which focus on scaling up the model while maintaining computational efficiency 115 16 [7], the experts MxDNA are uniquely designed to capture DNA basic units of varied lengths. Following this, a deformable convolution [18 [19] assembles these basic units into final tokens. Throughout the process, MxDNA is explicitly equipped to manage the inherent disconti- nuities, overlaps and ambiguities in genomic sequences, enabling it to handle complex biological characteristics it encounters. Furthermore, we incorporate a cross-attention mechanism to align the output resolution with the original input during pretraining on a masked language modeling task M
The proposed MxDNA demonstrates strong performance on both the Nucleotide Transformer Bench- marks [6] and the Genomic Benchmarks [20]. Despite only being pretrained on human reference genome, it still outperforms or matches previous models
6 5 some pretrained on multi-
species data and for much longer duration, achieving state-of-the-ar average performance and the best on 15 of the 26 individual tasks. Finally, by visualizing the learnt tokenization process, we illustrate that MxDNA learns unique tokenization strategy distinct to those of previous methods

---

and captures genomic functionalities at token level during self-supprvised pretraining, potentially offering novel biological insights. Our contributions can be summarized as follows:
Learnt Tokenization: We highlight the unsuitability of current DNA tokenization methods directly borrowed from NLP. Based on the belief that humans may not know the best tokenization approach but a model could potentially discover it, we propose a novel approach where the model autonomously learns an effective tokenization strategy.
Architectural Design: We introduce sparse Mixture of Convolution Experts coupled with deformable convolution to dynamically learn tokenization, specifically designed to manage the inherent discontinuities overlaps and ambiguities in genomic sequences. Additionally, we leverage cross attention align input and output sequence length to enable self self-supervised pretraining.
MxDNA demonstrates robust quantitative performance with less pre- training data compared to some existing models, achieving state-of-the-er average per- formance on both Nucleotide Transformer Benchmarks and Genomic Benchmarks. Fur- thermore, visual analysis of the tokenization behaviour and the token embedding space highlights the unique strategy and capability to capture genomic functionalities at token level of MxDNA, potentially offering new biological insights.
Empirical Results:
2 Background and Related Work
2.1 Tokenization Methods
Tokenization is a fundamental step in both natural language processing (NLP) and DNA sequence modelling, transforming complex texts or DNA sequences into manageable tokens. In NLP, whites- pace tokenization uses spaces and punctuation as delimiters but faces out-of-vocabulary issues. Similarly, in both fields, character (or single nucleotide in DNA) tokenization provides high res- olution but can lead to computational inefficiency [2] 22 23 N-gram in NLP and K-mer in
DNA analysis both use contiguous sequences of N (K) items from given inputs 24 25 6 26 4 but can disrupt meaningful units due to their fixed-length nature (non-overlapping) or lead to potential information redundancy across both domains to reduce vocabulary size by merging frequent pairs of existing tokens [ However, it might not adequately encode more complex patterns and are unreliable for finding linguistically sound tokens [1] These rule-based methods show limitations in different aspects, and our study aim to develop learning-based tokenization method without these limitations.
leakage (overlapping) [5] 9.. Byte-Pair Encoding (BPE) is employed
2.2
DNA Foundation Models
Recent advancements in DNA modeling have leveraged foundation models to decode the complex language of genomes. DNA BERTERT 44 pioneers the use of a BERT-like pretrained model for genomic sequence analysis, enhancing the understanding of nucleotide relationships via attention mechanisms. Nucleotide Transformer 66] offers comprehensive analysis of foundation models pretrained DNA sequences, with model sizes reaching up to 2.5 billion parameters and pretraining data drawn from the 1000G [8 human genomes and 850 various species. DNABERT2 (5) introduces an enhanced genome foundation model, utilizing an efficient BPE tokenizer and techniques to address input length constraints, resulting in reduced time and memory consumption while improving performance.
HyenaDNA 7 introduces a genomic foundation model capable of handling context with million tokens single nucleotide resolution, enabling the first exploration in-context learning in genomics. DNAGPT 266 extends the traditional GPT model by integrating tasks such as binary classification of DNA sequence order and numerical regression for predicting guanine-cytosine content, alongside developing a comprehensive token language. Caduceus (33 designs an architecture that leverages the long-range Mamba (29 block to support bi-directionality and reverse complementarity equivariance, addressing specific challenges in genomic analysis. Following VQVAE (0). VQDNA [31] employs a convolutional encoder alongside a nector-uanttied codebook to model tokenization, sharing a similar motivation with us yet ultimately adopting distinct solutions. Each work offers unique insights and innovations to the filed. Our research specifically concentrate on the tokenization methods for DNA, hopefully providing unique contributions to the filed.
3

---

3
Method
Transformer Blocks
ATCOGтCл NCceTTCoCCTCCTTC CCTACCTA
Tokenization
ABc TokGc
Trans former Blocks
RBc TOAGZ
Cross Attention
LM Head
MaskedLM M
Pretraining
moce
mac
Task Head
Downstream
Tokens
Tokens
Finetuning
Nucleotides
Nucleotides
Basic Units R Recognition (Sec. 3.2.1) (A daptive and A mbiguous Property)
Basic Units ssembly (Sec. 3.2.2)
(Overlapping and Discontiouous Property)
Gating Weights
Convolution
Expert
Offset Estimation
Convolution
Basic Units Scoring
Non-Max Suppression
Expert
AFC
GG
ATE
Deformable Convolution
Convolution
Expert
U[1-E, 1+s) Jitter
Modulation Estimation
Convolution
Expert
Figure 2: Our proposed MxDNA. (Top) Overall pipeline of the MxDNA model: Black arrows indicate pretraining data flow, and red arrows indicate finetuning data flow. The learnt tokenization module tokenizes single nucleotide input into learnt tokens. (Bottom) Illustration of the learnt tokenization module: Meaningful basic units are recognized with linearly scoring layer and non- maximum suppression, embedded through convolution experts (Sec. (Sec.3..2II) and assembled into final tokens by deformable convolution. (Sec. (3.22) This process ensures meaningful, discontinuous, overlapping, and ambiguous tokenization, addressing the unique properties of genomic data.
3.1 Motivation
The concept of "correct" tokenization in genomic sequences analysis remains undefined due the complex nature of genomics. Unlike natural languages, where linguistically meaningful units are well-understood, biological units in genomics are not limited to contiguous nucleotide or amino acid sequences. Instead, they often encompass discontinuous, overlapping, and ambiguous segments crucial for understanding biological functions [8] Current DNA modeling practices directly borrow tokenization methods from natural language processing (NLP), such as single nucleotide tokenization, K-mer and Byte-Pair Encoding (BPE). These fixed, predefined approaches. though useful, often fail to capture the unique properties of DNA sequences, which lack explicit delimiters and consist of biologically meaningful units that defy simple segmentation.
Recognizing these challenges, MxDNA was developed based on the belief that although an optimal tokenization schema for genomic sequences is yet to be discovered, with the desired tokenization properties-sue such as handling discontinuities, overlaps, and ambiguities, and allow learn and adapt its tokenization strategy all by itself.
explicitly equip our model
3.2 Learnt Tokenization Module
This section introduces our learned tokenization module, which central to our approach. The module first identifies meaningful basic units within the input sequence, which are then assembled into tokens with discontinuous, overlapping, and ambiguous properties. Implementation details are in Appx. A.2

---

3.2.1 Basic Units Recognition
Basic Units Scoring Initially, MxDNA identifies the basic units that serve the building blocks for tokens. It estimates the probability of the existence of various sized basic units centred at each nucleotide position by linear gating mechanism commonly used in Mixture of Experts models. Following this, one-dimensional non-maximum suppression is applied to eliminate redundant proposals and select the most significant basic units.
Specifically, given the input nucleotide sequence X E RRixd, where I the sequence length and d is the hidden dimension, X is first linearly scored to produce S E R where represents the number of experts. Training-time multiplicative jitter noise [16] is applied to introduce ambiguity, while ensuring deterministic inference. The jitter noise is applied by multiplying the scores with random factor sampled uniformly between [1 0.01, + 0.01], resulting in slight perturbations to the probability distribution used for tokenization.
Modified non-maximum suppression then applied to S, where Siy indicates the presence probability of a basic unit of length I.j centered at position i, and I. No is predefined set of lengths. The results are tracked using an expert mask M N', where each M; is natural integer indicating the presence of basic unit's 's center of length M, at position i.
Basic Units Embedding After identifying the basic units, the nucleotides within each unit are aggregated to form embeddings. Convolution kernels of corresponding sizes are applied to the center of each basic unit to capture local features. The initial scoring and gating into specific convolution experts is similar to the Mixture of Experts paradigm, with each expert being a convolutional unit focusing on a specific segment rather than a single nucleotide.
Specifically, a basic unit at position i of length L,- = M, is processed by the convolution expert Ej with kernel size Lj, and weighted by softmax (S.)), aggregating the nucleotides within the unit. This transforms the original input e [xd into an array basic units U [g where k is the number of basic units:
Eg (2
| ++1++144411 softmax (St) where T,p = M2 1
M2 > 0
U,
(I)
M₁
Then, the unwanted entries (i]M, = 0} of U are removed to keep the basic units U E RRkxd only, where & is the number of basic units.
3.2.2 Basic Units Assembly
Distal Relation Estimation Building upon the identified basic units, the more complex genomic patterns that extend beyond simple segmentation are modelled by a one-dimensional deformable convolution. This technique uniquely accommodates the modeling of complex local geometric transformations, adaptively adjusting to the input sequence. The linkages between the distal basic units are modeled by the offsets and modulation factors of each basic unit.
Following 18 19]. offsets AP E R and modulation factors AM E R are computed based on the basic units U model the distal relationships among them. This strategy ensures the combination of basic units is discontinuous, and reuses units across tokens achieve the overlapping property.
Final Tokens Embedding Using the computed offsets and modulation factors, deformable con- volution is applied embed basic units into final tokens. The embedding process for each position incorporates deformations of the convolution kernel specified by the offsets, with the results modu- lated by the modulation factors.
Specifically, a one-dimensional deformable convolution with kerel size f is applied to embed these basic units into the final learnt tokens T 4xd.
T; =
Σ
Wp Up Am
(2)
pet pE(-14 +1 (1[11+1.....
[+]
5

---

For fractional location p' = - +++p + Ap, bilinear interpolation is implemented as:
Up =
Σ
max (0, 1 - lp' -ql) Uq
(3)
DE ()...)
3.3 Overall Pipeline
The framework begins with single nucleotide input represented Xiput R This single nucleotide resolution imput allows for fine-grained analysis of genomics from the beginning.
Initially, imput is processed through several transformer encoder blocks designed to extract global relationships within the sequence, producing X Rixd This sets the stage for effective tokenization. Following this, the learnt tokenization module transforms the nucleotide sequence X into a more manageable form T E R improving the efficiency and focus of subsequent layers. The tokenized output T is then passed through another series of transformer encoder blocks to further refine the token representation to Toupput E R enhancing the model's ability to encode deeper genomic information.
For the mask language modeling pretraining stage, the enriched nucleotide level representation X serves the query, with the refined tokenized output Tutpuu acting as both the key and value. This setup maps the output resolution to single nucleotides, essential for reconstructing masked tokens. During the finetuning stage, the [CLS] token of Toutput used for classification by convention.
Experiments
In this section, we first introduce the implementation and pretraining settings of MxDNA. Then, evaluate MxDNA against other foundation models on Genomic Benchmarks (20] and Nucleotide Transformer Benchmarks [6]. Next, we present ablation studies the effect of different tokeniza- tion methods and different components of MxDNA. Finally, we conduct simple analysis on the tokenization behaviors of MxDNA. Experiment settings and results are detailed in Appx. A.4
4.1 Model Implementation Pretraining
Our MxDNA is built on the architecture Nucleotide Transformer v2 100M model with 512 hidden units and 22 layers, totaling approximately 100M parameters. Specifically, the model's learnt tokenization module includes 10 convolution experts with kernel sizes ranging from to 10, along with deformable convolution block with a kernel size of three. We integrate this module by replacing the fifth transformer block, aiming to avoid introducing additional computations.
MxDNA is pretrained the whole Human Reference Genome 32 masked language modeling task [I with 15% of the nucleotides randomly masked. An auxiliary balancing loss with a weight of 0.01 is used prevent degradation towards a single expert, following (16). The model undergoes training for 500k steps for main performance comparisons and 100k steps for ablations.
4.2 Downstream Evaluation
We primarily follow the evaluation settings of HyenaDNA 77.. performing evaluation on Nucleotide Transformer Benchmarks and Genomic Benchmarks. To ensure fair comparison, we fully finetune all the BERT-like DNA foundation models including Nucleotide Transformer v2 (6) DÑABERT 4 DNABERT2 (5), MxDNA under same hyperparameter settings. For HyenaDNA, we utilize the hyperparameters recommended by and we report the average performance with sample standard deviations.
All experiments are repeated with three random seeds,
4.2.1 Genomic Benchmarks
First, we begin our evaluation on the Genomic Benchmarks 20 which consists of eight regulatory element classification tasks. For this benchmark, all BERT-like models are finetuned for 10 epochs with Top-1 accuracy reported for each dataset.
The Nucleotide Transformer and DNABERT2 pretrained much larger datasets than other models.

---

Table 1: Genomic Benchmarks. Average performance across three random seeds for Nucleotide Transformer v2 100M, DNABERT, DNABERT2, HyenaDNA and MxDNA with sample standard deviations. We highlight the best values in bold type and underline the second best.
Dataset
NTv2
DNABERT
DNABERT2
HycnaDNA
MxDNA
Average
88.
+ 0.00
87.50
= 2013
88.29
=019
87.17 #
89.13 0.13
Mouse Enhancers
83,94
0.41
81.54
0.86
81.34
0.84
80.99
0.72
80.57
0.97
Coding vs Intergenomic
94.50
0.06
93.13
+ 0.05
94.94
0.34
90.74
0.111
95.28
0.08
Human vs Worm
96,88
96,98
0.07
97.57
0.04
96.53
0.04
97.64
0,01
Human Enhancer Cohn
74.33
74.54
75.93
73.36
74.67
Human Enhancer Ensembl
92.05 0.38
92.18 + 0.14
92.31
= 0.13
88.12 0.17
93.13
0.35
Human Regulatory
93.79 + 0.12
88,16 + 0.09
87.9 0.54
93.08 0.22
94.11
0.08
Human OCR Ensembl
78.51 0.55
81.40
80.94
79.15 2.34
81.05
Human TATA Promoters
91.05
+
92.06 + 95.34 0.17
95.39
0.26
96.56 0.29
Table 2: Nucleotide Transformer Benchmarks. Average performance across three random seeds for Nucleotide Transformer v2 100M, DNABERT, DNABERT2, HyenaDNA and MxDNA with sample standard deviations. We highlight the best values in bold type and underline the second best.
Dataset
NTv2
DNABERT
DNABERT2
HyenaDNA
MxDNA
Average
70.70 2012
68.61
2016
76.66
200
75.96
0.00
78.14 2012
Histone Markers Avg.
55.22 1 0.21
51.67 1.017
65.89
0.48
65.24 = 0.26
68.14
H3
78.22
77.41
= 65.13
82.31
80.86
82.78
14ck
51.76 + 0.99
46.51 +1.53
= 1.10
65.96 = 0.26
68.27 10.19
H3K36me3
59.18 0.53
50.98
0.75
68.19
0.82
64.31 0,33
67.05
1.05
13344e1
51.87 1.00
43.83
56.09
0.48
55.04
1.07
56.15
H3K4me2
29.63 1.68
32.38
49.25
2.08
49.96
0,90
55.59
1.08
H3K4mc3
38.76 1 1.30
31.49
1. 3.40
57.90
1 0.92
60.92
0.72
63.68
1: 0.34
H3K79me3
60,98 1.06
60.48
0.50
71.94
0.44
70.97
0.77
74.29
0.08
H3K9ac
54.57 6.71
52.55
64.35
62.57
64.78
H4
79.19 1. 0.49
79.60
1. 0.55
80.71
+: 0.43
78.73
0.66
81.18
0.25
H4ac
48.02 1,40
41.53
= 0.244
63.03
081
63.06
0.62
67.65
0.39
Regulatory Annotation Avg.
84.86 = 0.01
84.83
= 0.29
86.16
0.1
84.87
= 0.20
86.16
0.07
Enhancer
78.16 0.29
79.13
0.56
79.40
: 0.29
78.81
=: 0.54
79.90
=: 0.39
Enhancer Types
58.14 14 + 1.02
54.73
+ 1.24
59.84
: 0.45
58.36 = 0.58
60.50
0.59
Promoter All
96.23 0.61
97.05
97.37
0.44
96.19 0.14
97.16
Promoter Non-TATA
96.69
97.02
97.65
96,43
97.24
Promoter TATA
95.10 + 0.24
96.22
0.35
96,55
0.24
94.55 = 0.38
96.01 0.09
Splice Site Annotation Avg.
98.71 :: 0,08
98.02
96.69 = 0.00
96.82 = 0.111
98.09 20009
All
98.36 0.04
97.83
95.68
96.78 =
98.14
Accpetor Donor
98.69
+ 0.14
97.81 0.28
97.71 = 0.1
96.52 = 0.19
98.01 = 0.13
99.09 0:05
98.43 0.05
96.67
0.17
97.16 0.16
98.10 0.13
As shown in Table [ MxDNA achieves the best performance on 5 out of 8 datasets and ranks the top-2 on 7 out 8 datasets. On average, MxDNA shows an improvement 0.84 points compared to the second-best model, DNABERT2. These results demonstrate MxDNA's robustness and effectiveness regulatory element classification.
4.2.2 Nucleotide Transformer Benchmarks
Next, we evaluate MxDNA on the Nucleotide Transformer Benchmarks [6], which includes 18 datasets across three task types: histone marker prediction, regulatory annotation prediction, and splice site annotation prediction. For this benchmark, the BERT-like models are finetuned for 20 epochs. Following [23] we use the Matthews Correlation Coefficient (MCC) for histone markers tasks, Fl score for regulatory and splice site annotation tasks, except accuracy for splice site all task.

---

Table 3: Average results on Nucleotide Transformer Benchmarks and Genomic Benchmarks with
different tokenization methods. We highlight the best values bold type, underline the second best.
Method
NT Benchmarks
Genomic Benchmarks
Single Nucleotide (1-mer)
75.07 026
88.56 1.00
Overlapping k-mer (6-mer)
74.35 035
88,55 0.07
Non-overlapping k-mer (6-mer)
67.65 + 0.17
86.83 0.06
Byte-pair Encoding (4096 tokens)
74.96
87.30
MxDNA (Learnt Tokenization)
77.52 0.18
88.89 0.05
Table 4: Average results on Nucleotide Transformer Benchmarks and Genomic Benchmarks with components added successively. We highlight the best values in bold type, underline the second best.
Method
NT Benchmarks
Genomic Benchmarks
Single Nucleotide Baseline
75.07
88.56
+ Mixture of Convolution Experts
77.00 = 005
88.72 0007
Deformable Convolution
77.35 0.12
88.86 0.18
+ Jitter Noise (MxDNA)
77.52 0.18
88.89 0.05
As shown in Table 2 MxDNA achieves the best performance on 10 out of 18 datasets and ranks in the top-2 on 16 out of 18 datasets. On average, MxDNA shows an improvement of 1,48 points compared the second- best model, DNABERT2. Notably, MxDNA significantly outperforms all other models in the histone markers tasks while maintaining competitive performance in regulatory annotation and splice site annotation tasks.
4.3 Ablation Studies
Different Tokenization Methods: We compare various tokenization methods by pretraining mod- els for 100k steps using the same backbone but different tokenization methods. The results in Table 3 show that our learnt tokenization significantly outperforms traditional methods such as non-overlapping K-mer, BPE, overlapping K-mer, and single nucleotide tokenization. Among the rule-based methods, single nucleotide tokenization performs best, possibly because it doesn't 't incor- porate human biases and focuses solely on the raw data, though may make difficult capture higher-level semantics. Conversely, non-overlapping K-mer might disrupt meaningful units, BPE might fail to segment DNA sequences meaningfully, and overlapping K-mer could suffer from information leakage.
Different Components: We assess the impact of individual components by incrementally integrat- ing each into the baseline model and pretraining them for 100k steps. Starting with a baseline of single nucleotide tokenization, we sequentially add the Mixture of Convolution Experts, the deformable convolution and jitter noise, resulting in the proposed MxDNA. The results in Thole substantial performance gains from the Mixture of Convolution Experts alone, demonstrating the effectiveness of the idea which allows the model to learn tokenization autonomously rather than depending predefined tokenization. There are also noticeable performance improvements contributed by the deformable convolution and jitter noise, showing the effectiveness of explicitly equipping the model with capabilities to handle discontinuities, overlaps and ambiguities.
4.4 Analysis
We conduct analysis of the tokenization behaviors of MxDNA against previous methods on both sample and dataset level, and present a output embedding analysis at a token level. Notably, MxDNA exhibits unique tokenization strategy distinct from prior methods and is able to inherently capture and differentiate genomic functionalities at token level during self-supervised pretraining, potentially providing new biological insights. Visualization details are in Appx A.6
We first visualize the tokenization of a DNA sequence. For MxDNA, two individual forward passes with identical input and model yield slightly different results during training. It is
Sample Level
8

---

Imput DNA Sequezce CCC.MGAAAAGCCTTTTAGCCCCGAAAAAAA
Adagatie: Televiinattonn MADIA
Previous Rule-bed Methoos
Forward Pass
Basic Units Recugrition
Porward Pass
Buscc Vaits
Single Nudieoide
Recaquilca
Overlagpngg
GAGMGGGGMEGGAGGEE BATAIAAAAAAGAAAAAAAAAAAAAAAAAAAA
Basic Unis soambly
Basit Dniss Assemall
Not-ovedlapinn K.
GECTTTGGSTTAGGGAAMIII AGCAGCCTOTGAGG
Bybe Pair Exooding
ABSTAGCGGAATIGAGT
ABCCTG1GAGCTAG
GOCTGTGAGO AGCOAAA
Figure 3: Tokenization results of MxDNA over two individual forward passes (left) compared to those traditional rule-based methods (right). block of the same colour refers single token.
Token Length Distribution of BPE
Proportion of Tokeens
0.10
0.05
0.80
10
Token Length
Token Length Distribution of MxDNA
0.86
Propertian of Tokens
0,10
0.08
0.06
0.80
Token Length
Histone Marker
Enhancer
Promoter
Splice Site
Figure 4: Distribution of token lengths for BPE (top) and MxDNA (bottom) across different down- stream datasets, illustrating the distinct strategy of MxDNA for handling DNA tokenization. For the sake of simplicity, regard the basic units tokens for MxDNA.
worth mentioning that there are usually a small number of differences between the two results, and we display the region where the tokenization outcomes are different to show the ambiguous property for illustrative purposes. For previous rule-based methods, tokenization is static and performed only once. As depicted Fig. 3] our learnt tokenization method tokenize the DNA sequence in way distinctly different from previous rule-hased method. Moreover, the discontinuous, overlapping and ambiguous tokenization results validate our design choices to manage these properties.
Dataset Level To gain more insights, we measure the distribution of token lengths across different downstream datasets for both MxDNA and BPE. For simplicity, we regard the basic units as tokens for MxDNA. BPE and MxDNA shows very distinct distribution of token lengths. As shown in Fig. BPE tends to produce bell-shaped distribution, inherently biased by its requency-based merging nule. Conversely, MxDNA's distribution closer to a uniform distribution with preferences for specific lengths, reflecting its adaptive, task-oriented segmentation capabilities. Moreover, the

---

MxDNA
DNABERI
DNABERTZ
HyenaDNA-tiny
HyenaDNA-large
NTv2-100M
Histone Marker
Enhancer
Promoter
Spiice Site
Figure 5: t-SNE visualization of the output embeddings at token level across different functional sequences different models, demonstrating MxDNA's unique capability to inherently capture and differentiate genomic functionalities at token level.
variability in token distribution across datasets might suggest that DNA sequences of different functions might possess distinct patterns and meaningful units.
Token Embedding Analysis Next, we use t-SNE to visualize the pretrained output tokens in sequences with different genomic functions of different foundation models. As is shown in Fig. 4.4 without any finetuning, the token embedding distributions of MxDNA are different across sequences with different functions: the tokens of Histone Marker, Promoter and Splice Site form unique clusters. While for all other foundation models, their tokens do not form clear clusters as MxDNA does. This shows MxDNA's superior capability inherently capture and differentiate genomic functionalities at token level, suggesting its robustness and specificity in representing biological sequences even before any supervised finetuning applied.
Conclusion
Summary We present MxDNA, framework developed to autonomously learn effective DNA tokenization strategies solely through gradient descent. MxDNA demonstrates strong performance against existing models and tokenization methods across 26 diverse genomic tasks in Nucleotide Transformer Benchmarks and Genomic Benchmarks with no additional cost. We also perform an analysis of the tokenization mechanism and the token embedding space of MxDNA, showing its distinct tokenization strategy against previous methods and unique capability to capture genomic functionalities at token level.
Limitations & Future Works While MxDNA demonstrates strong quantitative performance on various downstream tasks, direct biological validation of the model's tokenization decision remains limited. Furthermore, the evaluation on long range tasks is lacking due to quadratic cost of self- attention, although the learnt tokenization is expected to help reduce sequence length effectively Future research will focus on refining MxDNA's design to learn a better and more interpretable tokenization strategy, and testing its applicability to broader genomic analyses especially on more long range tasks.
and can be combined with sub-quadratic architectures 29 33
10

---

Acknowledgments and Disclosure of Funding
This work is funded in part by Shanghai Artificial Intelligence Laboratory and supported by the Beijing Super Cloud Computing Center serve platform.
References
[1] Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 181000008 2018.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, ct al. Language models are few-shot learners. Advances in neural information processing systems, 13:187771901, 2020.
[3] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ann Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[4] Yanrong Ji. Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language genome. Bioinformatics, 37(15)2112-2120, 2021.
[5] Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2.066. /5006, 2023.
[6] Hugo Dalla- Torre, Liam Gonzalez, Javier Mendoza-Revine Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop. Bernardo P. de Almeida, Hassan Sirelkha- tim, Guillaume Richard, Marcin Skwark, Karim Beguir, Marie Lopez, and Thomas Pierrot. The nucleotide transformer: Building and evaluating robust foundation models for human genomies. bioRxiv, pages 2023-01, 2023.
[7] Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Woroww Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024.
[8] Mai Ha Vu, Rahmad Akhar, Philippe A Rohert, Bartlomicj Swiatczak, Geir Kjetil Sandve, Victor Greiff, and Dag Trygve Truslew Haug. Linguistically inspired roadmap for building biologically reliable protein language models. Nature Machine Intelligence, 5(5):485-496 2023.
[9] Chaoqi Liang. Weigiang Bai, Lifeng Qiao, Yuchen Ren, Jianle Sun, Peng Ye, Hongliang Yan, Xinzhu Ma. Wangmeng Zuo, and Wanli Ouyang. Rethinking the bert-like pretraining for dna sequences. arXiv preprint arXiv:2310.0766 2023.
[10] Kaj Bostrom and Greg Durrett. Byte pair encoding suboptimal for language model pretraining. arXiv preprint or ari:::2004.03722 2020.
[11] Valentin Hofmann, Janet B Pierrehumbert, and Hinrich Schütze. Superbizarre not superb: Derivational morphology improves bert's interpretation of complex words. arXiv preprint 10Xi:2/011000400 2021.
[12] Philippe A Robert, Rahmad Akbar, Robert Frank, Milena Pavlović, Michael Widrich, Igor Snapkov, Andrei Slabodkin, Maria Chernigovskaya, Lonneke Scheffer, Eva Smorodina, et al. Unconstrained generation of synthetic antibody-antigen structures to guide machine learning methodology for antibody specificity prediction. Nature Computational Science, 2122)8555-655 2022.
[13] Rahmad Akbar, Philippe A Robert, Milena Pavlovié. Jeliazko R Jeliazkov, Igor Snapkov. Andrei Slabodkin, Cédric Weber, Lonneke Scheffer, Enkelejda Miho, Ingrid Hobæk Haff, et al. A compact vocabulary of paratope-cpitope interactions enables predictability antibody- binding. Cell Reports, 34(1)) 1), 2021.
[14] Francis Crick. Central dogma of molecular biology. Nature, 227 563, 1970.
[15] Noam Shazcer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-cxperts layer. arXiv preprint arXiv: 170110063.638 2017.
[16] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1335 39, 2022.
11

---

[17] Albert Q Jiang. Alexandre Sublayrolles, Antoine Roux. Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint urXiv:240I.04000 04088, 2024.
[18] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEFE international conference on computer vision, pages 764-773, 2017.
[19] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9308-9316, 2019.
[20] Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Simeček, and Panagiotis Alexiou. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24(1):25, 2023.
[21] Yoon Kim, Yacine Jernite, David Sontag, and Alexander Rush. Character-aware neural language models. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[22] Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character- language modeling with deeper self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3159-3166, 2019.
[23] Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling, arXiv preprint arXiv:2403.03:: 2024.
[24] Peter F Brown, Vincent J Della Pietra, Peter V Desouza, Jennifer Lai, and Robert L Mercer. Class-based n-gram models of natural language. Computational linguistics, 18(4):467-48)) 1992.
[25] Paul McNamee and James Maytield. Character n-gram tokenization for european language text retrieval. Information retrieval, 7:73-99 90- 2004.
[26] Daoan Zhang, Weitong Zhang, Bing He, Jianguo Zhang, Chenehen Qin, and Jianhua Yao. Dnagpt: A generalized pretrained tool for multiple dna sequence analysis tasks. bioRxiv, pages 2023- 07, 2023.
[27] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv: 1588 07909, 2015.
[28] Marta Byrska-Bishoy Uday S Evani, Xuefang Zhao, Anna O Basile, Haley J Abel, Allison A Regier, André Corvelo, Wayne E Clarke, Rajeeva Musumuri, Kshithija Nagulapalli, et al. High-coverage whole-genome sequencing of the expanded 1000 genomes project cohort including 602 trios. Cell, 185(18):3426335 3440. 2022.
[29] Albert Gu and Tri Dao. Mamba: Lincar-tima sequence modeling with selective state spaces. arXiv preprint arXiv:2312. 00752, 2023.
[30] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation lcarning. Advances in neural information processing systems, 30, 2017.
[31] Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, Cheng Tan, Jiangbin Zheng, Yufei Huang, and Stan Z. Li. Vadna: Unleashing the power of vector quantization for multi-species genomic sequence modeling, arXiv preprint 10011200. 2024.
[32] Valerie A Schneider. Tina Graves-Lindsay, Kerstin Howe, Nathan Bouk, Hsiu-Chuan Chen, Paul A Kitts, Terence D Murphy, Kim D Pruitt, Françoise Thibaud- Thibaud-Nissen, Derek Albracht, al. Evaluation of grch38 and novo haploid genome assemblies demonstrates the enduring quality of the reference assembly. Genome research, (7)5884 864, 2017.
[33] Michacl Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hicrarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pages 28043-28078. PMLR, 2023.
[34] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703, PMLR, 2020.
[35] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 2xXiv22010.19299 2020.
12

---

[36] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot lext-to-image generation. In International conference on machine learning. pages 8821-8831, Pmlr, 2021.
[37] Hangbo Bao, Li Dong. Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint htt:::200..022.222 2021.
[38] Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 6077-6086, 2018.
[39] Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott. Multimodal pretraining unmasked: meta-analysis and a unified framework of vision-and-lane and--anagdd berts. Transactions of the Association for Computational Linguistics, 9:978-999 2021.
[40] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks, Advances in neural information processing systems, 28, 2015.
[41] Delong Chen, Samuel Cahyawijaya, Jianfeng Liu, Baoyuan Wang, and Pascale Fung. Subohject-level image tokenization. arXiv preprint arXiv:2402. 14327, 2024.
[42] Young Kyung Kim, Matías Di Martino, and Guillermo Sapiro. Vision transformers with natural language semantics. arXiv preprint urXiv:2402.1786. 2024.
[43] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[44] Nithin Chalapathi, Yiheng Du, and Aditi Krishnapriyan. Scaling physics-informed hard constraints with mixture-of-exppets. arXiv preprint arXiv:2402. 13412, 2024.
[45] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et Conformer: Convolution-augment transformer for speech recognition. arXiv preprint arXiv:2005.08100 2020.
[46] Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 22-31, 2021.
[47] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. Advances in neural information processing systems, 34:30392-30400 304000, 2021.
[48] Ziga Avsec Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinsk Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, (8(10):11961100) 2021.
[49] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiigg 568:127063. 2024.
[50] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv: 2020.
[51] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2022.
[52] Tri Dao. Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 163599 2022.
[53] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08859) 2023.
[54] Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary Vito,o, Elias Ellison, Will Feng, Jiong Gong. Michael Gschwind, Brian Hirsh, Sherlock Huang. Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang. Yinghai Lu. CK Luk. Bert Maher, Yunjie Pan. Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang,
13

---

William Wen, Shunting Zhang. Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume (ASPLOS 24). ACM, April 2024.
[55]
William Falcon The PyTorch Lightning team. PyTorch Lightning. March 2019.
[56]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Davison, Sam Shleifer, Patrick von Platen, Clara Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: Sfate-of the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Octoher 2020. Association for Computational Linguistics.
[57] Wenzel Jakob, Jason Rhinclander, and Dean Moldovan, pybind11 scamless operability between c++1l and python, 2017. https://githb..com/pp howp.///oooooooooonnnnnin.......
[58] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, Dubourg, J. Vanderplas, A. Passos, Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-lcam: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2888 2011.
[59] Charles Harris, Jarrod Millman, Stéfan van Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Aray programming with NumPy. Nature, 585(7825):357-362, September 2020.
[60] J. D. Hunter. Matplotlib: 2d graphics environment. Computing in Science Engineering, 9(3):90-95, 2007.
[61] Michael L. Waskom, seaborn: statistical data visualization. Journal Open Source Software, 6(60):3021, 2021.
14

---

A Appendix supplemental material
A.1 Extended Related Works
A.1.1 Image Tokenization
Tokenization computer vision (CV) attempts to transform images into formats that can be effi- ciently processed by machine learning models especially transformers. In line with Character-level tokenization, (34] directly using raw pixels as input units. The Vision Transformer (ViT) [35] splits
an image into patches of identical sizes and treats these patches tokens in NLP, demonstrating
remarkable performance on standard image recognition tasks. Inspired by [30]. [36 37] utilized
an image tokenizer learnt by discrete variational autoencoder (dVAE) map pixels into discrete
tokens according to a visual codebook. Meanwhile, researchers also utilize detection or segmentation
features for visual representations. For instance, 38 39
used pretrained Faster R-CNN model 40
to extract region features. Recently, [1
22
exploit Segment Anything Model (SAM)
4 to
construct sub-word tokenization and semantic tokenization respectively.
A.1.2 Mixture of Experts
(Sparse) MoE is first designed to improve the capacity of neural networks while maintaining total computations. [15] uses MoE as a general purpose neural network component and realizes sparse gating, demonstrating its use as a practical way to massively increase model capacity. By replacing FNN with Mixture of Experts, [16 17 successfully combine sparse MoE and Transformers, achieving superior capabilities with less computational cost. Previous methods generally replace layer of the neural networks with multiple, sparsely activated identical alternatives, governed by gating mechanism. Recently, [4] explicitly adds interpretability to each expert by letting each expert solve the constraint over smaller decomposed domains through differentiable optimization.
A.1.3 Deformable Convolution
Deformable convolution explicitly equips the model with ability to adapt to the geometric varia- tions of different objects [18. 19 Unlike the attention mechanism, which focuses on capturing long-range relationships, deformable convolution locally samples feature maps using learnt offsets and modulation factors. By modeling complex geometric transformations effectively, deformable convolution networks achieve significant performance improvements in various tasks, including image classification, object detection and semantic segmentation.
A.1.4 Combination of Convolution and Transformer
The integration of convolutional layers with transformer architectures has emerged as a powerful approach across various domains, effectively combining the strengths of both techniques. Con- former 45 applies this hybrid design to audio processing, enhancing the capture of local and global dependencies in audio signals. In computer vision, the CvT [46] introduces convolutions into trans- formers to improve efficiency and representational power, while Early Convolution [47) in Vision Transformer incorporates convolutional layers early in the architecture enhance input feature representations. Extending to genomic data, Enformer [48] applies a similar approach as in 47 to model complex dependencies in DNA sequences and reduce the computational cost, showcasing the potential of hybrid architectures to handle highly specialized data types like genomic sequences.
A.2 Method Details
For the convolution expert, we adapt design principles from 45). Our 1-D convolution expert starts with a pointwise convolution Win paired with a Gated Linear Unit (GLU), followed by a 1-D grouped convolution W. Waj. Subsequent to the grouped convolution, Layer Normalization (LayerNorm) and a Swish activation layer Wout (Swish()) are applied. The grouped convolution here has number of groups equal to the factor of hidden size closest kernel size, and number of output channel equal to number of input channels. This ensures that each convolution expert has similar parameter counts in spite of different kernel sizes.
Eg (X) Wont (Swish (LayerNorm(Wgr, GLU (Win (x)xxx))
(4)
15

---

Term
Description
Number of nucleotides
Dimension of hidden states
Number of experts
Number basic units
Kernel size of the deformable convolution
Indices of nucleotides or tokens
Indices of experts
X E Rixd
Input nucleotide sequence
S E Rix
Confidence scores of basic units existence
L E N
Kernel sizes of convolution experts
M E N
Mask indicating the existence of basic units Convolution experts
Ej E R4,xd 7 Rd
U Raxd
Basic units
AP E REXS
Offsets ofthe deformable convolution
AM E R*xs
Modulation factors of the deformable convolution
TE k
Final tokens
Table 5: Glossary of terms used in describing the method.
A.2.1 Non-Maximum Suppression
The pseudocode in describes the selection process for optimal basic units based on scores, ensuring no overlaps, and using kernel sizes to guide the selection.
The input consists of: positions (all nucleotide positions), kernel sizes (all kernel sizes), scores (scores for each (position, kernel size) pair) for the possibility of basic unit of given size existing at given position. The output is mask indicating selected basic units with their corresponding kernel sizes.
Algorithm Detailed Mon-Maximum Suppression for Basic Unit Placement
1: procedure NMS(positions P [1, 2, ... 1]. kernel sizes L N scores S R ( and j E [1.2..... n).
2: Sort all (P,, Lj) pairs by S; in descending order, where E
3: Initialize an output array with zeros M E No.
4: for each (Pt, I3) pair in the sorted pairs do
5: Calculate the start and end of the region at P. with width I.j.
6: if the region is not overlapped with any region in M then Mp. E N + L3 8: end if end for
7:
9:
return M.
11:
end procedure
A.2.2 Sparse Mixture of Convolution Experts
The pseudocode in 2 outlines the selective activation of convolutions positions determined by Non-Maximum Suppression, using corresponding kernel sizes.
The input consists of: input (embeddings of nucleotides). positions (at nucleotide level) of selected basic units with their corresponding kernel sizes. The output is the embeddings of the selected basic units.
A.2.3 Deformable Convolution
The pseudocode in details how deformable convolution dynamically adjusts based on input features by modifying its parameters for each input segment.
The input consists of: input (embeddings of selected basic units). The output the embeddings of the final tokens.
16

---

Algorithm 2 Detailed Sparse Convolution
1: procedure SPARSE CONVOLUTION/input X E Rixd selected positions with kernel sizes
M TENN N
2: k 4- number of non-zero elements in M.
3: Initialize output array with zeros U E Rexd
4: Initialize counter ent. = 0.
5:
for each i in ( do
6:
if M, * 0 then
7:
f- cnt 1.
8:
Extract the segment of X centered at with width M,.
9:
Uent Rd + Apply the convolution expert of kernel size M, to the segment.
10: end if
end for
12: return U.
end procedure
Algorithm 3 Detailed Deformable Convolution
1:
procedure DEFORMABLE CONVOLUTION(inqU U Rxdd
2: Initialize an output array with zeros E Rexd for each i in (1, ke] do
3:
4:
Calculate offsets AP; R based on Uz.
5:
Calculate modulation factors A,, E R based on U.- Extract the deformed segment of U centred i according to AP; Weight the segment by A.M. 8: T; Rdd Compute the segment's dot product with the convolution kernel of size f.
6:
7:
9:
end for return T. end procedure
11:
A.3 Back Propagation
The tokenization module is learnt solely through gradient descent. In this section, we will focus the gradients with respect to the Basic Units Scoring Block and Distal Relation Estimation Block.
A.3.1 Sparse Mixture of Convolution Experts
Recall the forward process in Eq.
I
U. Fiy X [ My +1+++1 [1 softmax (Si), 1 (M, 0), where L, M2
( M, > 0
6 Sts
1(, > 0) =
softtaxxS.),, = Σx S
0, M, < 0
The gradient w.r.t the score Sy is follows:
aU, E, (x 1 matt | [ 11) 1(M, > 0)
@softmax(Sa)
where L, = M,
aS
0Sa
(6)
@softmax (S.),
softmax (S.)n (1 softmax(S.),)
k =
05
-oftmax ssfftax(S;) k +
A.3.2 Deformable Convolution
Recall the forward process in Eq. 2
17

---

=
E
Wp Up+ Am
(7)
f - - 1
Up =
max (0, 1 Ip' -y) Uq
(8)
The gradients w.r. the offset Ap and the modulation Am are as follows:
OT: aAp
max (0, 1 - li + + Ap al) Un (9)
Σ
" Am-
Ap
pEf-121 +1... 1111
0T;
Σ
Up Uitp+Ap 0Am
0Am
E
up . -U+pt+A (10)
0Am
pe p
[5]]
p +1........ [+]]
A.4
Experiment Setting Details
A.4.1
Settings
Model Implementation MxDNA is built on the Nucleotide Transformer V2 architecture which incorporates several architectural improvements recognized in the NLP community, such rotary positional encodings [4). SwishGLU MI.P (50), and the exclusion of linear bias terms (3] (T). Consistent with Nuclcotide Transformer V2 100M. MxDNA has 512 hidden units, an expansion factor of 4, 16 attention heads, and 22 layers, totaling approximately 100M parameters. Specifically, the model's learnt tokenization module includes 10 convolution experts with kernel sizes ranging from 1 to 10, along with deformable convolution block with a kernel size of three. We integrate this module by replacing the fifth transformer block, aiming to avoid introducing additional computations. We utilize FlashAttention
52 53
for efficient attention calculations.
Pretraining Following 4. MxDNA is pretrained on the whole Human Reference Genome
using Masked Language Modeling. We removed all sequences gaps and unannotated regions and extracted 70 to 510-nt-long sequences as training data. We mask 15% of the tokens, with 80% replaced by a special [MASK] token, 10% replaced with random vocabulary token, and 10% left unchanged. All masking happens at the initial input stage(single nucleotide, 6mer tokens, bpe tokens). For model using single nucleotide tokenization, non-overlapping 6mer and BPE, the masking is performed randomly and mask out 15% of total tokens except of special tokens. For model using overlapping 6mer, we follow the strategy used in [4] with contiguous k-length spans of certain k-mers are masked, totalling around 1.5% of the tokens. An auxiliary balancing loss with a weight of 0.01 is used to prevent degradation towards single expert, following [16]. The model is trained with a learning rate lc-4 and a batch size of 512. We employ the AdamW optimizer with B1 = 0.9, 82 = 0.98, 1e - 6, weight decay of 0.01, and cosine annealing learning rate scheduler with linear warm-up over the first 10% of steps. The model undergoes training for 500k steps for main performance comparisons and 100k steps for ablations.
Downstream We download the data from http:: httpps://huggingfaace http:://wgggggggaaee.o.eoeeeeeea nucleotide, transformer benchmark for Nucleotide Transformer Benchmarks and https:// github. com/ML MMMML-BB www--iioooo---eee CC.foo/CCCE/////.ii...cc. in Nucleotide Transformer Benchmarks, the BERT-like models are finetuned using PEFT (parameter efficient finetuning) without providing the exact hyperparameters. Believing that fully fine-tuning these models will better leverage their capabilities and provide fairer comparison we decide to proceed with full finetuning for all models. We keep the original data splits in 6 20 We do not perform cross validation models and we decide to follow the practice of HyenaDNA (7) instead. Additionally, we repeat all experiments under three random seeds, report the average results with sample standard deviations.
genomic hww.nii__eehhhhrkeeeeee for Genomic Benchmarks. Moreover,
( does since it will be too computationally expensive for BERT-like
the BERT-like models are fully finetuned with a batch size of 32 and a learning rate of 3e-5. We employ the AdamW optimizer with 31 0.9, B2 0.999, 1e 8, and weight decay of 0.01.
18

---

Models are trained for 10 epochs on Genomic Benchmarks and 20 epochs on Nucleotide Transformer Benchmarks, with the learning rate linearly warmed up over the first epoch and then decaying to zero during the remaining epochs. For the Mouse Enhancers dataset (sequence lengths with mean 2381, std = 984.4, max = 4707), we truncate the sequence to a maximum length of 4096, which is considered acceptable. For DNABERT, which can not handle sequences of length over 512, we truncate the sequence to a maximum length 512.
For HyenaDNA, we fully finetune the pretrained model from https://huggingttee co/ Longgaafart/hyenndaa--iiaaa 1k wewleseelle-eee -256 using the hyperparameteess provided by [ in docker image wwwpanannnanaaaaanae
latest dna- nt6 latest for Nucleotide Transformer Benchmarks, and https://huggingfaace co/ Loo/LongSaarrrr aatiSSafarr/yenannaa. tiny- sk-seelenn with modified hyperpa- rameters recommended by [23] for Genomic Benchmarks. Their research suggests that training with sequence lengths 2 to times the length of sequences used in downstream tasks typically yields the best performance. Thus, the tiny models are the best choice for most of the downstream tasks in Nucleotide Transformer Benchmarks and Genomic Benchmarks since most of tasks have sequence length of around few hundreds and the tiny model are pretrained with 1000 length sequence. Notice that although our reproduced results a bit lower than the results reported by the authors of HyenaDNA, the performance of MxDNA is still better than originally reported results most of the tasks.
A.4.2 Metries
This section defines the metrics used evaluate the performance models on various genomic tasks. On Nucleotide Transformer Benchmarks, We used the Matthews Correlation Coefficient (MCC) for histone marker tasks, scores for regulatory and splice site annotation tasks, except accuracy for splice site all task. Top-1 1 Accuracy is used for all tasks in Genomics Benchmarks.
Matthews Correlation Coefficient (MCC) The Matthews Correlation Coefficient a robust statistical rate which takes into account true and false positives and negatives and is generally regarded as a balanced measure that can be used even if the classes are very different sizes.
TP x TN - FP x FN VTP+ (P)(TP FN) +N) + FP)(TN where TP. TN, FP, and FN are the numbers of true positives, true negatives, false positives, and false negatives, respectively.
MCC
FN)
F1 Score We use the macro-averaged Fl score, which is computed using the arithmetic mean of all the per-class FI scores. The (per-class) F1 score is the harmonic mean of precision and recall and is particularly useful when the costs of false positives and false negatives are high.
precision recall
(per-class) F1 2
precision recall
where precision = TP+FP and recall
TP
TP
TP4FN
Accuracy Accuracy is the proportion of true results (both true positives and true negatives) among the total number of cases examined.
TP+TM
Accuracy = TP +N FF+FN
A.5 Ablation Results Details
A.5.1 Different Tokenization Methods
Detailed results on each datasets with different tokenization methods are presented in Table Table "1-mer" stands for single nucleotide. "Ovlp 6-mer" stands for overlapping 6-mer. "Non- Ovlp" stands for non-overlapping 6-mer. "BPE" stands for Byte-pair Encoding with vocabulary
19

---

Table 6: Genomic Benchmarks. Different tokenization methods.
Dataset
1-mer
Ovlp 6-mer
Non-Ovlp 6-mer
BPE
MxDNA
Average
88.56
88.55 our
86.83
0.0
87.30 0.66
88.89
Mouse Enhancers
77.56 0.94
78.81 0.s
82.43
0.0
80.44 144
80.44
0.63
Coding vs Intergenomic
95.05
2.13
94.97
006
92.73 0:00
92.25 2.11
94.78 0.04
Human vs Worm
97.52 004
97.14 0.11
96,35 0.04
96.59 0.01
97.27
Human Enhancer Cohn
73.70
73.07 0.56
72.72 0.19
72.92 ::
73.98
Human Enhancer Ensembl
92.79 = 000M
92.98 um
91.79 0.23
92.38 1: u.s
92.73 =: 0:08
Human Regulatory
94.03 = 0,02
94.21 0.88
93.73 0.08
89.70 0.10
94.10 = 0.12
Human OCR Ensembl
80.84
0.50
81.36 1.0
75.64 0.64
77.87 4: 0.4
80.62 = 0.42
Human NoTTATA Promoters
97.00 0.05
95.84 0.09
89.24 0.24
96.22 0.17
97.22 0.23
Table 7: Nucleotide Transformer Benchmarks, Different tokenization methods.
Dataset
1-mer
Ovlp 6-mer
Non-Ovlp 6-mer
BPE
MxDNA
Average
75.07
74.35 0.3
67.65 2011
74.96 016
77.52 2018
Histone Markers Avg.
63.13 a 033
61.88 0.06
50.36 1::8
64.58 0.3
67.29 0.23
H3
80.92 u85
80.94
0.64
74.77 0.33
80.26 t.15
82.14
= 0.7.6
H3K14ac
62.00 1.79
62.17 #0,00
46.33
65,91 0.72
68.29 #065
H3K36me3
62.59 +
63.26
50.49
63.83
65.46
H3K4mel
51.66+ 0.57
50.11 = 3.63
41.26
54.33 + 0.53
54.97 = 1.50
H3K4me2
49.51 1: 0.9
35,59 5.72
29.99
49,48 0.24
55.30
H3K4mc3
54,14
54.15
30.83
58.10
63.82
H3K79mc3
70.36. 158
71.30
0.04
59,99
70.89 0.94
73.74 0.0.8
H3K9ac
60.63 1 275
64.70
1 0.32
50.24
1 1.31
62.22 0.54
63.15
1. 0.26
H4
80.23 1 0.79
80.09 0.59
78.27
79,29 0,441
80.89
0.23
H4ac
59.25 =
56.49
41,42
61.45
65.14
Regulatory Annotation Avg.
85.38
84.95 2.12
84.
84.62
85.70
Enhancer
78.93 0.70
76.65 0,80
78.48
77.17 0.50
79.73
Enhancer Types
58.90
59.12
56.50
59.57
59.79
Promoter All
96.92 # 0.8
96.94 0.12
95.76 =: 0,10
95.77 0.09
96.87
+: 0,10
Promoter Non-TATA
97.04 + u.5
96.84 o.0
95.87 +: 0.08
95.83 0.16
96.81 +: 0.13
Promoter TATA
95.09
95.21 2018
94.02 0.11
94.77 0.33
95.32 0.17
Splice Site Annotation Avg.
97.70 0.19
98.25 0.06
97.85 0:08
93.46 0.02
98.00 1013
All
98.07 u.77
98.18 0.11
97.91 = 0.15
93.25 D41
98.05 0.13
Accpetor
97.61 0.44
98.18
97.65
94.40 0.34
97.68
Donor
97.42
98.39 0.16
98.00
92.72
98.28
Table 8: Genomic Benchmarks. Different components.
Dataset
I-mer
MoCE
Def Conv
Noise (MxDNA)
Average
88.56 0.002
88.72
88.86 1:1
88.89
Mouse Enhancers
77.56 0.94
79.66 0.79
80.36
80.44 0.63
Coding Intergenomic
95.05 +
94.64
94.57
94.78
Human vs Worm
97.52 + ou04
97.20 + c.a
97.31 0.0
97.27 0.07
Human Enhancer Cohn
73.70 1:033
74.78 0.52
73.75
73.98 0.40
Human Enhancer Ensembl
92.79
92.59 0.19
92.78
92.73
Human Regulatory
94.03 1 0.02
93.93
0.04
94.15
1:111
94.10
0.12
Human OCR Ensembl
80.84 1 0.50
80.27
1 0.34
81.14
1 0.8
80.62
0.42
Human Non't TATA Promoters
97.00 o.s
96.92
0.32
96.81
0.6
97.22 0.23
20

---

Table 9: Nucleotide Transformer Benchmarks. Different components.
Dataset
1-mer
+ MoCCE
+ Def Conv
+ Noise (MxDNA)
Average
75.07 022
77.00 + 0.0s
77.35 + 0.12
77.52 +018
Histone Markers Avg.
63.13 0.3
66.58 = 2.11
67.02 + 0.10
67.29=axx
H3
80,92 0.85
81.18 +
81.65 0.82
82.14
H3K14ac
62.00 1.79
67.68
* 0.44
66.12 + 0.46
68.29 #0.65
H3K36me3
62.59 1.50
66.31
+ 0.46
65.26 + 1.33
65.46 1.74
H3K4me1
51 31.66+t 0.57
53.18
56,14 + 104
54.97 1,50
H3K4me2
49. 99.51 0.5
53.86 #:
54.13 + 1:22
55.30 =
H3K4mc3
54. 14 1 0.95
63.82 1: 1,72
61.42 +: 1.58
63.82 0.92
H3K79mc3
70,36 1.58
72.72
1: 0.71
72.88 1 0.75
73.74 1 0.78
H3K9ac
60.63 2778
63.78
64,95
1:77
63.15
H4
80.23 0,079
79.96
0.42 64.09
81.47
1. 0.7
80.89 0.23
H4ac
59.25 1.36
0.73
66.21
1. 0.41
65.14 1.23
Regulatory Annotation Avg.
85.38 0.15
85.60
= 0.32
85.71 + 0.29
85.70 0021
Enhancer
78.93 =0000 0.70
79.07 =
78.91 + 0.39
79.73 =
Enhancer Types
58,90 i 0.72
60.07
1. 0,82
60.28 1 1:20
59.79 1 0.52
Promoter All
96,92 0.08
96.92
96,80 1: 00s
96.87
1: 0.10
Promoter Non-TATA
97.04
0.05
96.71
96.96 0.11
96.81
Promoter TATA
95.09 0.34
95.25
+: 0.54
95.69 +: 025
95.32 = 0.17
Splice Site Annotation Avg.
97.70 0.1
97.39 =
97.86 + 0.11
98.00 10
All
98.07 07
98.20
98.13 009
98.05 2019
Accpetor Donor
97.61 0.44
97.08 + 0.13
97.76 + 0.21
97.68 = 0.20
97.42 0.58
96.90 0.52
97.69 + 0.50
98.28 0.07
size of 4096 borrowed from DNABERT2. All models are trained for 100k steps with same backbone but different tokenization methods.
A.5.2 Different Components
Detailed results on each datasets with components added from the single nucleotide tokenization baseline are presented in Table 8 and Table "1-mer" stands for the baseline. "+ MoCE" stands for adding the sparse Mixture of Convolution Experts."> "+ Def Conv" stands for adding the deformable convolution block. "+ Noise" stands for adding the multiplicative jitter noise. These components are added successively, finally equivalent to MxDNA. All models are trained for 100k steps with same backbone and components added sequentially.
A.6
Visualization Details
The BPE tokenizer used here is directly borrowed from DNABERT2 with a vocabulary of size 4096. The visualization methods for traditional tokenization methods straightforward. Below are the visualization details for MxDNA.
Sample Level: For MxDNA, we perform a forward process of the model using the sequence as input, extracting the Mask of basic units existence M. Offsees AP and Modulation factors AM of the deformable convolution. First, we colour the recognized basic units based on M. Then, we determine the distal relations using A and A.M Specifically, a distal relation is considered to be visualized only if the the product of the corresponding modulation weight and the bilinearly interpolated offset weight exceeds one. Eventually, final learnt token is made up of group of related basic units and coloured by the colour of the central basic unit.
For dataset level, we first finetune the MxDNA model on downstream datasets and use the refined models to generate the Mask of basic units existence M for all samples the dataset. We then calculate the proportion of the lengths of each recognized basic units as indicated by M. Specifically, we finetune MxDNA on H3, Enhancer, Promoter All and Splice Site All in Nucleotide Transformer Benchmarks for Histone Marker, Enhancer, Promoter and Splice Site respectively.
Dataset Level:
21

---

Table 10: Comparison of various models based on their computational complexity.
Model
Flops (G)
Macs (G)
Parameters (M)
Number of Tokens
DNABERT2
24.80
12.39
117.07
104.2
NTv2 100M
16.63
8.31
97.89
86
DNABERT
99.48
49.70
89.20
507
HyenaDNA tiny d256
1.67
0.832
1.64
511
HyenaDNA tiny
0,441
0.219
0,436
511
MxDNA
35.94
17.93
100.09
512 + 101.6
Learnt Tokenization Module
0.914
0.446
11.69
512
101.6
Single Nucleotide Baseline
94.85
47.38
92.95
512
Table 11: Assets used this work
Asset
License
GRCh38 32
CC BY 4.0
Genomic Benchmarks 20
Apache-2.0
Nucleotide Transformer
6
CC BY-NC-SA 4.0
DNABERT 4
Apache-2.0
DNABERT2 5
Apache-2.0
HyenaDNN 7
Apache-2.0
FlashAttention 52
53
BSD-3-Clause
Pytorch [54]
BSD-3-Clause
Pytorch Lightning
655
Apache-2.0
Huggingface 56
Apache-2.0
Pybind11 557
BSD-3-Claue
Scikit-Learn 558
ASD-3-Clause
Numpy 559
BSD-3-Clause
Matplotlib 60
Matplotlib License
Seaborn 61
Apache-2.0
For the token embedding analysís, we utilize various pretrained models to embed sequences with different functions and analyse the output embeddings at the token level. Initially, perform principal component analysis to reduce the dimensionality to one hundred, which facilitates the visualization process. Subsequently, we employ t-SNE to visualize these token embeddings in two-dimensional space. Specifically, we the H3, Enhancer, Promoter All, and Splice Site All sequences from the test set of the Nucleotide Transformer Benchmarks to represent Histone Marker, Enhancer, Promoter, and Splice Site, respectively.
Token Embedding Analysis:
A.7 Computational Resources
train and evaluate the models on NVIDIA RTX 3090 and NVIDIA 100 GPUs. The pretraining of MxDNA takes around days for 500k steps using 4 A100 GPUs. Finetuing MxDNA on all the downstream tasks takes approximately 1.5 Days using 100 GPU. This true for other BERT-like foundation models with around 100M parameters.
The detailed computational costs of the models (averaged across 5 samples sequence length of 510) are outlined in Table I0 The integration of a mixture of convolution experts and the deformable convolution introduces an increased computational overhead initially due to the O log(!)) time complexity of the learned tokenization mechanism (where / represents the number of nucleotides). This complexity mitigated by the substantial reduction in sequence length after tokenization, which decreases the number of tokens processed by subsequent transformer layers.
A.8 Assets
The assets used in this work along with their licenses including data, pretrained weights, benchmarks. libraries, and software are presented in Table
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As pointed out in Sec. I We propose method where the model autonomously learns to tokenize DNA and achieves great performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
The results can The paper is generally empirical, not theoretical.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The detailed experiment settings are provided in Appx. A.4 be reproduced following these settings.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: Our code will be make public in the github repository once ready. We use publicly available data for experiments illustrated in Appx. A.4
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source Benchmarks).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssicnpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The detailed experiment settings are provided in Appx. A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We finetune all the models under three random seeds and report the average results with sample standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experiments compute resources are given in Appx. A.7
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We review and follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper focuses on tokenization techniques of DNA foundation models. While there maybe misuse of DNA foundation models, our learnt tokenization method will neither increase nor decrease the risk of the potential misuse.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets summarized in Appx. A.8 and cited throughout the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets for now since it is not ready.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "aDZgF11Qs0", "reviewer": "Reviewer_JPfb", "summary": "The paper introduces MxDNA, a novel framework designed to autonomously learn effective DNA tokenization strategies through gradient descent. Unlike traditional methods borrowed from natural language processing, MxDNA employs a sparse Mixture of Convolution Experts and deformable convolution to address the discontinuous, overlapping, and ambiguous nature of meaningful genomic segments. MxDNA demonstrates superior performance on Nucleotide Transformer Benchmarks and Genomic Benchmarks with less pretraining data and time compared to existing models.", "questions": "1. What are the computational trade-offs of using a mixture of convolution experts and deformable convolution?", "limitations": "The authors have addressed the technical limitations well but could enhance their discussion on broader societal impacts.\n\n- Acknowledgement: The authors have acknowledged the limitations of their work, such as the lack of biological validation and challenges in handling long-range dependencies due to the quadratic cost of self-attention.\n\n- Proposals for Future Work: They propose future research directions to address these limitations, such as integrating sub-quadratic attention mechanisms and hybrid models.\n\n- Discussion: The paper does not explicitly address potential negative societal impacts, such as data privacy concerns or ethical considerations in genomic research.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "### Originality:\n- Introduces MxDNA, a novel framework for adaptive DNA sequence tokenization, leveraging a mixture of convolution experts and deformable convolution, which is a significant departure from traditional NLP-based tokenization methods.\n\n### Quality:\n- Demonstrates superior performance on Nucleotide Transformer Benchmarks and Genomic Benchmarks with less pretraining data and time, highlighting the robustness and effectiveness of MxDNA.\n- Provides thorough empirical evaluation with comprehensive benchmarks, showcasing state-of-the-art performance.\n\n### Clarity:\n- The paper is well-organized, with clear explanations of the novel methods and their implementation.\n- Includes visual aids and diagrams to help elucidate the complex processes involved in the MxDNA framework.\n\n### Significance:\n- Offers a new perspective on DNA tokenization, potentially leading to broader applications in genomics and new biological insights.\n- The learned tokenization strategy distinct from existing methods could pave the way for future advancements in genomic sequence modeling.", "weaknesses": "### 1. Evaluation on Long-Range Tasks\n- Propose Alternatives: Discuss integrating sub-quadratic attention mechanisms like Linformer or Longformer to reduce computational costs and allow for long-range genomic analysis.\n- Hybrid Models: Suggest hybrid architectures combining local and global attention to balance computational efficiency and capture of long-range dependencies.\n\n### 2. Analysis of Tokenization Behavior\n- Detailed Token Analysis: Perform in-depth analysis by clustering tokens based on genomic functions and locations, using visualization tools like t-SNE.\n- Functional Correlation: Correlate tokens with known genomic features and discuss the implications of their alignment or misalignment.\nVisualization and Interpretability: Include visualizations of token distributions and introduce interpretable metrics to evaluate token significance.\n\n### 3. Clarity in Methodological Innovations\n- Enhanced Diagrams and Pseudocode: Use detailed diagrams and pseudocode to clarify the operation of mixture of convolution experts and deformable convolution.\n- Algorithm Descriptions: Expand the appendix to include detailed, step-by-step algorithm descriptions.\n- Examples and Glossary: Provide concrete examples of the tokenization process and include a glossary of terms to aid understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback. We have acknowledged the points raised W1/W2/W3 and discussed these in the limitations section of our paper. \n\n\n**W1: Evaluation on Long-Range Tasks**\n\n- We evaluate our model with pre-trained sequence length of 510 on the long range task proposed by DeepSEA [r1] with Sequence length of 1000 (Which is used by HyenaDNA [r2] and Nucleotide Transformer [r3]). The results in AUROC are as follows:\n\n| Model     | TF   | DHS  | HM   | Avg.     | Pretraining Data|\n| --------- | ---- | ---- | ---- | -------- |---|\n| DeepSEA   | 95.8 | 92.3 | 85.6 | 91.2     | NA |\n| HyenaDNA  | 96.4 | **93.0** | 86.3 | **91.9**| Human Reference Genome|\n| DNABERT [r4]   | 95.2 | 91.9 | 82.3 | 89.8     | Human Reference Genome|\n| DNABERT2 [r5]  | 96.2 | 92.6 | 86.3 | 91.7     | Multi-species |\n| NTv2 100M | 96.4 | 92.7 | **86.6** | **91.9**  | Multi-species |\n| MxDNA     | **96.5** | 92.9 | 86.3 | **91.9** | Human Reference Genome|\n\nIt achieves comparable performance to HyenaDNA and Nucleotide Transformer v2 100M, and outperforms DNABERT, DNABERT2 and DeepSEA. Notably, DNABERT2 and Nucleotide Transformer v2 100M are pre-trained on multi-species data, while MxDNA is pre-trained on human reference genome only. Also, HyenaDNA is aimed at long-range tasks, and it is expected to perform better on this task.\n\n- Our research mainly focuses on learnt tokenization. Since the pretraining of foundation models are costly and the time limit of rebuttal, we will explore the combination of sub-quadratic attention mechanism and hybrid architectures in the future.\n\n**W2: Analysis of Tokenization Behavior:**\n- We use t-SNE to cluster tokens based on genomic functions as in the PDF. We use four types of data in the Nucleotide Transformer Benchmarks as input (\"H3\" for Histone Marker, \"enhancers\" for Enhancer, \"promoter_all\" for Promoter and \"splice_sites_all\" for Splice Site), and analyse the output embeddings of different pretrained models at token level. As is shown in the figure, without any finetuning, the token embedding distribution of MxDNA is different across sequences with different functions : the tokens of Histone Marker, Promoter and Splice Site form unique clusters. While for all other foundation models, their tokens do not form clear clusters as MxDNA does.\n- Though it will be good if the tokens correlate with known genomic features such as motifs, we follow the motifs discovery pipeline of DNABERT but the motifs do not match with existing motifs. This mismatch may imply that the model has learnt its unique way to interpret genomic sequences from the way biological experiments do.\n- For token distribution analysis, there is a token lengths distribution visualization in our paper which shows different patterns in different downstream tasks. Additionally, we use t-SNE to visualize the token distribution in the embedding space and show that the clusters formed by MxDNA is clearer than other models. As for interpretable metrics, we may consider Silhouette Coefficient or other metrics to evaluate the quality of the clusters in the future.\n    \n    \n**W3: Clarity in Methodological Innovations:**\n- We acknowledge the need for enhanced clarity in our methodological innovations. We will try to improve the clarity of our main paper and appendix. The modified method section with high-level motivation and description added is presented in the global rebuttal. A glossary of terms is given below:\n\n| Term                                            | Description                        |\n| ----------------------------------------------- | ---------------------------------- |\n| $l$                                             | Number of nucleotides              |\n| $d$                                             | Dimension of hidden states         |\n| $n$                                             | Number of experts                  |\n| $k$                                             | Number of basic units              |\n| $f$                                             | Deformable convolution kernel size |\n| $i$ | Indices of nucleotides or tokens  |\n| $j$ | Indices of experts |\n| $\\mathbf{X} \\in \\mathbb{R}^{l \\times d}$        | Input nucleotide sequence          |\n| $\\mathbf{S} \\in \\mathbb{R}^{l \\times n}$        | Expert confidence scores           |\n| $\\mathbf{L} \\in \\mathbb{N}^{n}$                 | Expert kernel sizes                |\n| $\\mathbf{M} \\in \\mathbb{N}^{l}$                 | Basic units existence mask         |\n| $\\mathbf{E} \\in \\{\\text{Conv1D}\\}^{n}$          | Expert convolution kernels         |\n| $\\mathbf{U} \\in \\mathbb{R}^{k \\times d}$        | Basic units                        |\n| $\\Delta \\mathbf{P} \\in \\mathbb{R}^{k \\times f}$ | Deformable convolution offsets     |\n| $\\Delta \\mathbf{M} \\in \\mathbb{R}^{k \\times f}$ | Deformable convolution modulations |\n| $\\mathbf{T} \\in \\mathbb{R}^{k \\times d}$        | Final Tokens                       |\n\n\n\n**Q1: What are the computational trade-offs of using a mixture of convolution experts and deformable convolution?:**\n\n- The integration of a mixture of convolution experts and deformable convolution introduces an increased computational overhead initially due to the O(n log(n)) complexity of the learned tokenization mechanism (where n represents the number of nucleotides). This complexity is mitigated by the substantial reduction in sequence length after tokenization, which decreases the number of tokens processed by subsequent transformer layers. \n- Additionally, our implementation in C++ using Pybind11 leverages batch-wise parallelism, which provides a significant speed advantage over typical Python implementations. This reduction in sequence length leads to diminished overall computational demands, as transformer computations generally scale quadratically with the number of tokens. \n- The detailed computational costs of the models are outlined as follows (Average Across 5 samples of sequence length of 510):", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TWuCDH1Bu5", "reviewer": "Reviewer_nT57", "summary": "The paper introduces MxDNA, a novel framework for adaptive DNA sequence tokenization. Unlike traditional tokenization methods borrowed from NLP, MxDNA uses a sparse Mixture of Convolution Experts coupled with deformable convolution to autonomously learn an effective tokenization strategy through gradient descent. This method explicitly considers the discontinuous, overlapping, and ambiguous nature of meaningful genomic segments. The paper demonstrates superior performance on Nucleotide Transformer Benchmarks and Genomic Benchmarks, highlighting MxDNA's effectiveness with less pretraining data and time.", "questions": "- Can the authors provide more intuitive explanations and visualizations for the sparse Mixture of Convolution Experts and the deformable convolution? Is there some real biological cases matches the learned pattern?\n\n- Could MxDNA used as a tokenization method to other sequence model to enhance their performance?", "limitations": "The authors have acknowledged the limitations, such as the lack of direct biological validation of the model's tokenization decisions and the challenges with long-range tasks due to the quadratic cost of self-attention. The paper would benefit from a more detailed discussion of these limitations and potential strategies to address them in future work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: The approach of learning tokenization through gradient descent rather than relying on predefined rules is innovative and well-suited to the complexities of genomic data.\n- Clarity: The paper is mostly clear and well-organized, with a logical flow from motivation to method to results.", "weaknesses": "- Important baseline: VQDNA (Li et al, ICML 2024) seems a high related work with MxDNA, which should be discussed in the paper.\n\n- Theoretical Justifications: The theoretical underpinnings of why the learned tokenization method performs better are not fully explored. More rigorous proofs and explanations would strengthen the paper. Will be Better with some biological cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback.\n\n**W1: Discussion of VQDNA**:\n- We didn't include VQDNA[r1] in our initial submission because it was published very close to the NeurIPS deadline.\n- We share a similar motivation with VQDNA. following VQVAE, VQDNA employs a convolutional encoder with a vector-quantized codebook to model tokenization, but our MxDNA uses a sparse mixture of convolution experts with deformable convolution to model tokenization. The VQDNA is pretrained on multi-species data while our MxDNA is only pretrained on Human Reference Genome. The settings used for benchmarking is also different, but both of the two methods outperforms the similar baseline DNABERT2 (BPE) on Histone Markers Prediction (Epigenetic Marks Prediction).\n- we will add a detailed comparison with VQDNA in the final version of our paper. \n\n**W2: Theoretical Justifications**:\n- The motivation of MxDNA is based on the fact that human do not understand the DNA language well, but the model may do a better job than human. Providing concrete theoretical justifications for such an intuition-based approach is challenging. \n- We use t-SNE to cluster tokens based on genomic functions as in the PDF. We use four types of data in the Nucleotide Transformer Benchmarks as input (\"H3\" for Histone Marker, \"enhancers\" for Enhancer, \"promoter_all\" for Promoter and \"splice_sites_all\" for Splice Site), and analyse the output embeddings of different pretrained models at token level. As is shown in the figure, without any finetuning, the token embedding distribution of MxDNA is different across sequences with different functions: the tokens of Histone Marker, Promoter and Splice Site form unique clusters. While for all other foundation models, their tokens do not form clear clusters as MxDNA does.\n- There is also a token lengths distribution visualization in our paper which shows different patterns in different downstream tasks. It is also very different from token length distribution of K-mer and BPE.\n\n\n**Q1: Intuitive Explanations and Biological Cases**:\n\n- Detection of Basic Units: The design of the basic units recognition is similar to the process of object detection in computer vision, where the model learns to detect objects in an image. A detection model proposes several bounding boxes, eliminates duplicate detections ands select the most relevant bounding boxes by non-maximum suppression. The case of MxDNA is similar, with the data being 1D instead of 2D. Then, each bounding box is embedded through a convolution kernel of corresponding kernel size, giving the embedding of the basic units. \n- Deformable Convolution for Adaptation: Following the initial detection, deformable convolution is utilized to address the dynamic and irregular patterns found in genomic data. Unlike standard convolutions with fixed geometries, deformable convolution adjusts its receptive fields dynamically. This flexibility is critical for accurately modeling discontinuous and overlapping genomic features.\n- Some biological analyses including token embedding distribution and token length distribution are dicussed in W2.\n\n\n\n**Q2: Extension to Other Sequence Models**:\n\n- Given the architectural similarities with existing genomics models, primarily BERT-like frameworks with minor modifications, MxDNA's tokenization approach is likely to enhance the performance of other genomic sequence models. \n- Additionally, we have extended our methodology to RNA sequences, utilizing the training procedure from the recently introduced Beacon [r2] framework. The superior performance on downstream datasets underscores MxDNA's  potential to significantly improve performance across different types of biological sequences:\n\n| Model       | Isoform Accuracy (R^2 %) | Mean Ribosome Loading (R^2 %) |\n| ----------- | ------------------------ | ----------------------------- |\n| Beacon-B512 | 72.00                    | 72.35                         |\n| MxDNA       | **81.30**                    | **81.21**                         |\n\n\n**L1: more detailed discussion of limitations and potential strategies**:\n\n- Direct biological validation: Our method start from the fact that human do not understand the DNA language well, but the model may do a better job than human. For the design of specific methods, we are inspired by discontinuous, overlapping and ambiguous properties proposed by [r3]. However, we only validate our results on two benchmarks empirically and direct biological validation is lacking. In the future, we may use the regulatory activity data in [r4] to perform clustering or correlation analysis on learnt tokens and biological traits.\n- Long-range validation: As presented in the table below, our methods itself will reduce the sequence length and the total computations. It is the quadratic self-attention that prevents us from evaluating on more long range tasks. In the future, we will consider similar strategies used by [r5]: hybrid architecture that are generally cheap in computation while maintaining strong long-range interaction ability.\n\n| Model                          | Flops (G) | Macs (G) | Parameters (M) | Number of Tokens |\n| ------------------------------ | --------- | -------- | -------------- | ---------------- |\n| MxDNA                          | 35.94     | 17.93    | 100.09         | 512 -> 101.6     |\n| Learnt Tokenization Module     | 0.914     | 0.446    | 11.69          | 512 -> 101.6     |\n| Single Nucleotide Baseline     | 94.85     | 47.38    | 92.95          | 512              |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nLulO5nwjn", "reviewer": "Reviewer_RCJz", "summary": "DNA language models currently use standard tokenization schemes from NLP that might be unsuitable for modelling DNA sequences. This paper proposes a tokenization scheme called MxDNA that is specifically designed for DNA language modelling. MxDNA presents a learnable tokenization scheme that uses a mixture of convolutional experts and deformable convolution to learn task-aware tokens using gradient descent. This scheme can handle the variable and often ambiguous length of meaningful DNA elements which can also overlap each other. The authors demonstrate that a modified Nucleotide Transformer with the MxDNA tokenization outperforms existing models on most benchmarks from Genome Benchmarks and the ones from Nucleotide Transformer.", "questions": "1. During pretraining, what was the length of the sequences being modelled? I could not find this in the main paper or the appendix. \n2. When you mask out 15% of the tokens for masked language modelling, is this masking performed at the nucleotide level or after the tokens have been generated using MxDNA? If it's after the tokens have been generated using MxDNA, how do you prevent the initial transformer layers (before MxDNA, that process nucleotide level sequences) from not attending to the masked tokens?\n3. In the appendix, it is mentioned that the tiny HyenaDNA model was used for benchmarking. One of the central contributions of HyenaDNA was to increase the sequence length during modelling, so how does MxDNA compare to the larger HyenaDNA models that were trained with longer sequences?\n4. In lines 244-246, it is mentioned that two forward passes with the same sequence and model can yield different results. Why is this happening? Shouldn't we expect the forward pass to be deterministic?", "limitations": "The authors have identified the limitations of their approach and I do not foresee any negative societal impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Originality: The main novel contribution of this paper is the MxDNA tokenization method that enables the creation of learnable DNA tokens. Related works have been cited, including those on recent DNA language models.\n\n- Quality: The benchmarking results in the paper are generally thorough - the authors benchmark MxDNA against existing state-of-the-art DNA language models on established tasks and show that MxDNA generally outperforms existing models.\n\n- Clarity: The need for a learnable tokenization scheme is well-motivated but in general, I believe that the clarity of this paper needs to be significantly improved for acceptance. I have listed my concerns in the next section.\n\n- Significance: MxDNA makes a case for its usage in DNA language models based on its performance. Since DNA language modelling is becoming an increasingly popular research area in computational genomics, this work could be useful to the community if the clarity of this paper is improved.", "weaknesses": "My main concerns with this paper are related to the clarity of the information presented. The lack of clarity in method descriptions did not allow me to fully comprehend it and I do not think one could reproduce the authors' results using these descriptions.\n\nParts of the paper that need to be improved are listed below:\n- Although works related to DNA language modelling are cited, there isn't a clear motivation for using the specific modules of MxDNA. From the experimental results, we see that these modules work well when combined but a reader would want to know how the authors arrived at this formulation.\n- The main description of the tokenization module in Section 3.2 is very dense. Although I am familiar with language models, genomics, and deep learning more generally, I was not able to understand how this core module works. The appendix contains implementation details but these too are very dense. I believe that the paper will greatly benefit from a clear description of the module that first motivates the techniques being employed before providing a high-level description of each of these techniques. Then, more details about how these modules are used in MxDNA can be presented.\n- Section 4.1 on how the pretraining is performed omits many essential details. For example, is the full human genome used for training? Are repeat regions removed? What is the length of sequences being modelled and how is the masking performed (this could be addressed by a clearer Section 3.2)? I could not find any details beyond hyperparameters in the appendix.\n- The \"Sample Level\" part of Section 4.4 (including Figure 3) is confusing. Why is it desirable for two forward passes with the same sequence and model to yield different results? What is the source of this stochasticity? Shouldn't we expect the forward pass to be deterministic?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback.\n\n**W1: Related Works and Motivation:**\n\n- We appreciate your feedback emphasizing the need for clearer motivations behind the modules used in MxDNA. Our approach is fundamentally inspired by the desired properties for genomic tokenization—Meaningful, Discontinuous, Overlapping, and Ambiguous—as outlined in the literature [r1]. These characteristics guide our development of a learnable tokenization method tailored to meet these specific genomic needs.\n- Our initial step in addressing these requirements involves aggregating neighboring nucleotides into meaningful basic units. Initially, we consider using strided convolutions because they excel at capturing local features within small windows. However, the fixed kernel size and stride of standard strided convolutions limit their adaptability, which is crucial for learning a dynamic tokenization strategy. To overcome this, we explore using a variety of convolution kernel sizes applied adaptively across different sequence positions, drawing from the Mixture of Experts (MoE) framework. In our adaptation, we replace traditional mlp experts with convolution experts that have varying kernel sizes, allowing us to capture basic units of different lengths effectively.\n- To select the most significant tokens from these convolutions, we employ a one dimensional non-maximum suppression technique on the gating logits, which helps refine the selection of basic units. \n- Following the aggregation of basic units, we seek a method to handle more complex genomic patterns that go beyond simple segmentation. This leads us to integrate Deformable Convolution [5, 6], known for its capability to model complex local geometric transformations. The one-dimensional adaptation of deformable convolution is particularly well-suited for genomic sequences, enabling the model to address discontinuous properties by predicting offsets that link distal basic units and to handle overlapping properties by reusing basic units across different tokens. \n- This comprehensive design allows MxDNA to effectively capture and represent the complex structural dynamics of genomic data.\n   \n**W2: Tokenization Module Clarity**\n- We acknowledge that the description of the tokenization module in Section 3.2 is dense and could be clearer. This module is crucial as it learns to tokenize the sequence end-to-end together with the rest of the model. The modified method section with high-level motivation and description added is presented in the global rebuttal. We will modified the appendix in the final version for clarity and put the detailed implementation in the code as well.\n\n**W3: Pretraining Details:**\n- We largely follow the pre-training procedure of DNABERT. We use the full human genome for pretraining.\n-  We removed all sequences gaps and unannotated regions and extracted 70 to 510-nt-long sequences as training data. We do not remove the repeated regions. \n- All masking happens at the initial input stage(single nucleotide, 6mer tokens, bpe tokens). For model using single nucleotide tokenization, non-overlapping 6mer and BPE, the masking is performed randomly and mask out 15% of total tokens except of special tokens. For model using overlapping 6mer, we follow the strategy used in DNABERT, with contiguous k-length spans of certain k-mers are masked, totalling ~15% of the tokens.\n\n**W4: Stochasticity in Sample Level:**\n- The stochasticity described in Section 4.4 and Figure 3 arises from the jitter noise added. Following Switch Transformer [r2], we only add stochasticity in training but not in inference, thus making the inference deterministic. \n- Furthermore, this stochastic element is minor with multiplicative noise sampled uniformly between 1-0.01 and 1+0.01, adding a little jitter to the probability distribution when determining the tokenization. It only add slight variability during training and will not affect the tokenization results significantly.\n\n\n**Q1: Sequence Length During Pretraining:**\n- The lengths of sequences used in pretraining varied from 70 to 510 nucleotides, largely following the protocol used in DNABERT [r3] (5 to 510-nt-long in DNABERT). This range was chosen to adequately represent the diversity of genomic data while ensuring efficient processing. \n\n**Q2: Masking Strategy**\n- Masking during the pretraining phase is implemented at the nucleotide level before any tokenization by MxDNA. This approach prevents potential information leakage by ensuring that the initial transformer layers do not have access to masked tokens. \n- Cross-attention mechanisms are used to align the reduced token count from MxDNA with the original sequence length, allowing the model to perform masked language modelling effectively using the learnt tokens.\n\n**Q3: Comparison with HyenaDNA:**\n- We selected the tiny HyenaDNA [r4] model for benchmarking based on the recommendations of the HyenaDNA authors, who advocate for the use of tiny models and perform extensive hyperparameter search on each downstream datasets.\n- Their research suggests that training with sequence lengths 2 to 4 times the length of sequences used in downstream tasks typically yields the best performance. Thus, the tiny models are the best choice for most of the downstream tasks in Nucleotide Transformer Benchmarks and Genomic Benchmarks since most of tasks have sequence length of around a few hundreds and the tiny model are pre-trained with 1000 length sequence. \n- We provide the results of the largest HyenaDNA model on Nucleotide Transformer Benchmarks as follows (Without extensive hyperparameter search: Epochs: 100, Batch Size: 256, Learning Rate: 6e-4, Scheduler: Cosine Decay):", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xcSL8gReCv", "reviewer": "Reviewer_eT4M", "summary": "The paper proposes an alternative method for tokenization in genomic foundation models. Existing approaches adopt methods from natural language processing (NLP) and apply those to tokenize genomic sequences. While these tokenization methods have been validated by human knowledge, there is no basis for their use as is for tokenization in genomics.\n\nThe proposed approach, MxDNA, attempts to learn a tokenization strategy with three properties – (a) discontinuity (b) overlapping tokens (c) ambiguity. These are considered to be inherent to genomic sequences, and consequently, an appropriate tokenization strategy should reflect these properties. The tokenization module comprises basic unit recognition and assembly, and forms one block of the entire encoder pipeline. It is sandwiched between transformer encoder blocks to learn global relationships, and transformer encoder blocks to refine the learned token embedings.\n\nThe authors close out the paper by comparing the performance of MxDNA against existing foundation models, including DNABERT, DNABERT-2, Nucleotide Transformer and HyenaDNA, on the Genomic and Nucleotide Transformer benchmarks. The authors have also conducted ablation studies on the various tokenization schemes in the MxDNA framework, and the components of the tokenization scheme used in MxDNA.", "questions": "Questions:\n\n1.\tAre the “discontinuous” and “overlapping” properties not somewhat contradictory, i.e., discontinuous tokens would not overlap? Perhaps, a better way of stating this is that the sequence tokens need not align with the genomic sequence.\n2.\tThe standard deviations in the results are based on three experimental runs. Are these values statistically meaningful? This is important when trying to make substantive conclusions from the included results. In its current form, several data points in the paper fall within the 1-SD error bars of the top 2 results.\n3.\tBuilding upon Question 2, the standard deviations for MxDNA appear to be far lower than any of the competing foundation models. Why Is this so? If anything, the noise in the tokenization of MxDNA should increase the variance in its performance, thereby lead to higher values of standard deviation.\n4.\tDo any of the tasks in either benchmark require nucleotide-level resolution? This may unfairly bias the overall results towards models that perform tokenization at the single-nucleotide level, instead of k-mers or BPE.\n\nSuggestions:\n\n1.\tFrom the description of the pipeline in Section 3.3, it is unclear if the tokenization scheme is learned separately. My understanding is that this too is learned during the pretraining of MxDNA, but this is not apparent from the description.\n2.\tIn the ablation study of Table 3, only k-mers of length 6 are used. It is possible, and in fact likely, that different values of k are appropriate for different tasks. Hence, a different value o f k might lead to a higher value on the benchmarks than single nucleotide tokenization.\n3.\tGrammatical errors:\n\n            a. Line 65: “only pretrained”  “only being pretrained”\n            b. Line 141: “Ambiguous property”  “ “Ambiguity property”\n            c. Line 611: “tokenizer used here are”  “tokenizer used here is”", "limitations": "The authors highlight the limitations of MxDNA as follows:\n\n1.\tThey make the claim that a better method for tokenization may help in the discovery of biologically meaningful units, but state that the evaluation of the tokens learned by MxDNA has not been biologically validated. \n2.\tThey also state that the range of MxDNA is limited due to its use of quadratic self-attention.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The key innovation in MxDNA is the learned tokenization scheme, the motivation for which is clear and logical – there is no reason to expect that NLP tokenization schemes should work well for genomic sequences. Moreover, the properties of the MxDNA tokenization method appear to exhibit seemingly desirable properties. Consequently, it is easy to understand the rationale behind the design choices made by the authors in this paper.\n\nIn addition to this, the authors have used a varied selection of recent and commonly-used genomic foundational models in their experiments for the purposes of benchmarking. Pretraining on data that is used to train some of these other models, and performing the experiments on existing benchmarking data, also lends greater credence to the results in the paper.", "weaknesses": "The paper does not read well as a stand-alone manuscript. Many details are omitted from the main paper. The included descriptions of the components of MxDNA are also difficult to comprehend. This applies to the pseudocodes in the appendix too; Algorithms 2 and 3 use many undefined/poorly defined function calls and hence, do not measurably aid a reader in understanding the operations they are describing. Instead, the authors should look to write the pseudocode such that key aspects of the operations are apparent as the reader can always look to their released code for a full implementation. More generally, an effort should be made such that the main paper is self-contained such that the reader can understand the overview of the MxDNA architecture and its building bocks. Details can be relegated to the with the appendix being used to elaborate on details of the various steps. \n\nI am also unconvinced by the use of an ambiguous tokenization strategy, introduced via jitter noise in MxDNA. While I can understand the need for tokenization to depend on the context, this should map to a fixed subsequence to map to different tokens in different contexts. On the other hand, a fixed sequence and context should map to tokens in a fixed manner. \n\nLastly, given the compute demands of foundation models, the absence of any discussion on computational complexity (training time, FLOPs, etc.) is a little worrying. For instance, the authors of DNABERT-2 show that large models may have a similar number of FLOPs to their smaller counterparts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback.\n\n**W1: Clarity in Method Description:**\n\n- We appreciate your feedback regarding the need for a more self-contained and more understandable manuscript. The modified method section with high-level motivation and description added is presented in the global rebuttal. We will modify the appendix in the final version for clarity and put the detailed implementation in the code as well.\n\n**W2: Tokenization Strategy and Jitter Noise:**\n\n- Following Switch Transformer [r1], we only add stochasticity in training but not in inference, thus making the inference deterministic. \n- Furthermore, this stochastic element is minor with multiplicative noise sampled uniformly between 1-0.01 and 1+0.01, adding a little jitter to the probability distribution when determining the tokenization. It only adds slight variability during training and will not affect the tokenization results significantly. \n- The jitter noise here may serve as a kind of data augmentation as well. As reported in the Switch Transformer paper, the noise in the tokenization can be beneficial to the model. \n- During the training stage, as a sequence based model, the model does not have access to the context information directly, the jitter noise may make the model more robust when transfer to other context and not overfit to the sequence only.\n   \n**W3: Computational Complexity Discussion:** \n\n- The integration of a mixture of convolution experts and deformable convolution introduces an increased computational overhead initially due to the O(n log(n)) complexity of the learned tokenization mechanism (where n represents the number of nucleotides). This complexity is mitigated by the substantial reduction in sequence length after tokenization, which decreases the number of tokens processed by subsequent transformer layers with quadratic costs.\n- The detailed computational costs of the models are outlined as follows (Average Across 5 samples of sequence length of 510):\n\n\n| Model                          | Flops (G) | Macs (G) | Parameters (M) | Number of Tokens |\n| ------------------------------ | --------- | -------- | -------------- | ---------------- |\n| DNABERT2 [r2]                       | 24.80     | 12.39    | 117.07         | 104.2            |\n| Nucleotide Transformer v2 100M [r3] | 16.63     | 8.31     | 97.89          | 86               |\n| DNABERT [r4]                       | 99.48     | 49.70    | 89.20          | 507              |\n| HyenaDNA tiny d256 [r5]            | 1.67      | 0.832    | 1.64           | 511              |\n| HyenaDNA tiny                  | 0.441     | 0.219    | 0.436          | 511              |\n| MxDNA                          | 35.94     | 17.93    | 100.09         | 512 -> 101.6     |\n| Learnt Tokenization Module     | 0.914     | 0.446    | 11.69          | 512 -> 101.6     |\n| Single Nucleotide Baseline     | 94.85     |  47.38   |    92.95       |     512          |\n\n\n\n**Q1: Discontinuous and Overlapping Tokens:**\n\n- The discontinuous properties means that the tokens may not be continuous in the original nucleotide sequence, but they can overlap with each other in token-level. This is exactly that sequence tokens need not align with the genomic sequence. We state it in this way because we want to keep the original statement in [r6]. We will refine our explanation to avoid confusion and add your statement in the final paper.\n\n\n**Q2: Statistical Significance of Experimental Runs:**\n\n- It is important in machine learning to draw a conclusion based on the average results of multiple runs to reduce the effect of randomness. Also, we believe it is better to provide standard deviations based on multiple runs to give a sense of the variance in the results, though we acknowledge that three runs may not be sufficient to provide statistically meaningful values. \n- However, the computational cost of training these models multiple times is currently prohibitive. Indeed, there are some data points in the paper that fall within the 1-SD error bars of the top 2 results, but we believe that the overall trends of average results are still clear.\n\n**Q3: Low Variance in MxDNA's Performance:**\n- We were initially surprised by the lower standard deviations observed in MxDNA's performance compared to other models, but we decide to report the results as they are. In fact, as mentioned in line 246, adding noise only make a few difference to the tokenization results during training. Also, as reported table 11 of Switch Transformer, the noise in the tokenization can be beneficial to the model. \n- Furthermore, following Switch Transformer, we only add jitter noise in training stage and during inference, the model is deterministic with no noise added. This kind of data augmentation may contribute to a lower standard deviations.\n\n**Q4: Nucleotide-Level Resolution:**\n\n- The tasks are all classification tasks at sequence level, and are not enforced to require nucleotide-level resolution. \n- Indeed, different tasks may benefit from different tokenization strategy such as single nucleotide, K-mer or BPE. We believe it is a feature of different tokenization methods rather than a unfair bias. For example, although single nucleotide tokenization may enjoy single nucleotide resolution, it will lead to much more computations compared to non-overlapping K-mer or BPE.\n- Consequently, it is kind of a advantage of MxDNA since it can capture different level of information (single nucleotide resolution, token level, sequence level) in the same model without increasing much computations. This adaptive feature makes MxDNA performs better in various downstream tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## **General Description:**

 Thanks for the valuable feedback provided by all reviewers. We appreciate all the reviewers JPfb (R1), nT57 (R2), RCJz (R3) and eT4M (R4) for approving our contributions: (1) innovative method (R1, R2, R3, R4), (2) thorough experiments (R1, R3, R4). Besides, the concerns are mainly concentrated on (1) presentation clarity (R3, R4), (2) more experiments and biological analysis (R1, R2). Under the NeurIPS policy, we will follow reviewers’ suggestions to refine method section of the paper at our discretion. 

## **Additional Experiments:** 

In the responses, we show additional experimental results and analysis including:

1. Computational cost evaluation including flops, macs, parameters and tokens (R1: Q1, R4: W3) (Table r1)
2. Biological analysis via t-SNE clustering of tokens based on genomic functions (R1: W2, R2: W2) (Figure r1)
3. Long-range evaluation on chromatin activity prediction (R1: W1) (Table r2)
4. Evaluation of RNA sequence modeling ability (R2: Q2) (Table r3)
5. Evaluation of larger HyenaDNA model (R3: Q3) (Table r4)
6. Evaluation of different values of K in K-mer tokenization (R4: S2) (Table r5)

The results of t-SNE clustering are in the PDF file and the additional results tables are in both the PDF file and in the response.

##  **Method Section Revision:**

 For the clarity problem of the method section, we modify the two subsections the method section of the main paper, adding higher level description as follows:

### **Basic Units Recognition**

**Basic Units Scoring** Initially, MxDNA identifies the basic units as building blocks of our tokens. Analogous to bounding boxes proposal in object detection, MxDNA estimates the probability of the existence of various sized basic units at each nucleotide position. This is achieved by a linear gating mechanism commonly employed in Mixture of Experts models. Following this, one-dimensional non-maximum suppression is applied to eliminate redundant proposals and select the most significant basic units. 

Specifically, given the input nucleotide sequence $X \in \mathbb{R}^{l \times d}$, where $l$ is the sequence length and $d$ is the hidden dimension, we first linearly score $X$ to produce $S \in \mathbb{R}^{l \times n}$, where $n$ represents the number of experts. This scoring incorporates multiplicative jitter noise, introducing the **Ambiguous** property. We then apply a modified non-maximum suppression to $S$, where $S_{ij}$ indicates the presence score of a basic unit of length $L_j$ centered at position $i$, and $L \in \mathbb{N} ^ n$ is a predefined set of lengths. The results are tracked using an expert mask $M \in \mathbb{N}^l$, where each $M_i$ is a natural integer indicating the presence of a basic unit's center of length $M_i$ at position $i$.

**Basic Units Embedding** Subsequently, after estimating the existence of the basic units, we aggregate the nucleotides within each unit to form their embeddings. Giving the advantage of capturing local features, convolution kernels of corresponding sizes are applied at the center of each basic unit. The initial scoring, followed by gating to specific convolution experts, is similar to the Mixture of Experts paradigm, though each expert here is a convolutional expert focusing on a specific segment rather than a single nucleotide.

Specifically, a basic unit at position $i$ of length $L_j = M_i$ is processed by the convolution expert $E_j$ with kernel size $L_j$, and weighted by $\text{softmax}(S_i)_j$, thus aggregating the nucleotides within the unit. This procedure transforms the original input $X \in \mathbb{R}^{l \times d}$ into an array of basic units $U \in \mathbb{R}^{l \times d}$:

**Equation 1** in original paper.

To achieve sparse activation of convolution experts efficiently, we extract the nucleotides within basic units of the same length, apply convolution operation with stride equal to kernel size and place the output with reduced length back to the corresponding center positions. Then, the unwanted entries $\{ i | M_i = 0 \}$ of are removed to keep the basic units $U \in \mathbb{R}^{k \times d}$ only, where $k$ is the number of basic units.


### **Basic Units Assembly**

**Distal Relation Estimation** Building upon the identified basic units, we address the more complex genomic patterns that extend beyond simple segmentation through the application of one-dimensional deformable convolution. This technique uniquely accommodates the modeling of complex local geometric transformations, adaptively adjusting to the input sequence. The linkages between the distal basic units are initially modeled by the offsets and modulation factors of each basic unit. 

Specifically, following Deformable Convolution, we compute offsets $\Delta P \in \mathbb{R}^{k \times f}$ and modulation factors $\Delta M \in \mathbb{R}^{k \times f}$ based on the basic units $U$ to model the distal relationships among them. This strategy ensures that the combination of distal basic units addresses the **Discontinuous** property and the reuse of basic units across different tokens meets the **Overlapping** property.

**Final Tokens Embedding**  Utilizing the calculated offsets and modulations, we apply deformable convolution to embed the basic units into final tokens accordingly. The embedding process for each position incorporates deformations of the convolution kernel specified by the offsets, with the resulted modulated by the modulation factors.

Specifically, we apply an one-dimensional deformable convolution with kernel size $f$ to embed these basic units into the final learnt tokens $T \in \mathbb{R}^{k \times d}$. The token embedding for each position $i$ is formulated as:

**Equation 2** in original paper.

For a fractional location $p' = i+p+\Delta p$, bilinear interpolation is as follows:

**Equation 3** in original paper.

Meta-review: The work introduces a novel framework for learning DNA sequence tokenization. There is a clear consensus around the novelty of the work and its potential impact to the field of DNA-language model. The authors have also adequately addressed concerns related to the presentation and the inclusion of additional benchmarks, meeting the reviewers' expectations.
Decision: Accept (poster)