HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
Authors: Eric Nguyen
Michael Poli
Marjan Faizi
Armin W Thomas
Michael Wornow
Callum Birch-Sykes
Stefano Massaroli
Aman Patel
Clayton M. Rabideau
Yoshua Bengio
Stefano Ermon
Christopher Re
Stephen Baccus
Venue: NeurIPS 2023 spotlight (2023)
Keywords: genomics
hyena
foundation models
large language models
transformers
TL;DR: HyenaDNA is a genomic foundation model trained on the human genome using a context length of 450k tokens.

Abstract: Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution (i.e. DNA "characters") where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena’s new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level – an up to 500x increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics for simple adaptation to novel tasks without updating pretrained model weights. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data.1 On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points. Code at https://github.com/HazyResearch/hyena-dna.

HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
Eric Nguyen"', Michael Poli*. Marjan F
Armin W. Thomas Callum Birch Sykes², Michael Wornow Aman Patel Clayton Rabideau Stefano Massaroli Yoshua Bengio*, Stefano Ermon Stephen A. Baccus Christopher Rel.f
Abstract
Genomic (DNA) sequences encode an enormous amount of information for gene regulation, protein synthesis, and numerous other cellular properties. Similar to natural language models, researchers have proposed foundation models in ge- nomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer baassdeeer--aeeed genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), signifi- cantly limiting the modeling long-range interactions in DNA. In addition, these methods rely tokenizers fixed k-mers aggregate meaningful DNA units,
losing single nucleotide resolution (i.e. DNA "characters") where subtle genetic variations can completely alter protein function via single nucleotide polymor- phisms (SNPs). Recently, Hyena, a large language model based on implicit con- volutions was shown match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena's new long-range capa- bilities, we present HyenaDNA, genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level an up to 500x increase over previous dense attention- based models. HNe scales sub-quadratically in sequence length (training up 160x faster than Transformer). uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables including the first use in-context learning in genomics for simple adaptation to novel tasks without updating pretrained model weights. On a long-range species classification task, HyenaDNA is able to effectively solve the challenge by increasing the con- text length IM without downsampling. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the datasets using model with orders of magnitude less parameters and pretraining data.² On the GenomicBsnchmarks, Hyena DNA surpasses SotA on of 8 datasets average by +10 accuracy points, and by much as +20 accuracy points hancer identification. Code available at https://github..c wtwp:ithuu..oo/HHae. www.///aaaeeeseeeaayyaaeeee dna.
(SotA) on 12 of 18
'Equal contribution. i Equal senior authorship. 'ttanford University. HHavvrrd University. $SynTensor. "Mila and Université de Montréal. *On benchmarks from Nucleotide Transformer, Hye uses model with 1500x fewer parameters (2.5B vs 1.6M) and 3200x less pretraining data (3202 vs human reference genome).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

DNA Sequence Length Warm-up
HyenaDNA Pretraining
Adaptation to New Tasks
ACCTAGGT
Regulatory element type
x xoooooOS 2
MLP
Training o))
* Frocen HyenaDNA
xOOO L
Hyena Operator
DOO L
X
ACSTACGT
Sequence length warm-up for stability & faster training
ACGTAGO
Tuneable Soft Prompt Tokens
Sequence
(Single Nucleatide Tokens)
Figure 1.1: HyenaDNA recipe for long-range foundation models in genomics. The Nye architecture a simple stack of Hyena operators [37] trained using next token prediction. (See Fig. 1.3 for block diagram of architecture). We introduce new sequence length scheduling technique to stabilize training. and provide a method to leverage the longer context length to adapt to novel tasks without standard fine-tuning by filling the context window with learnable soft prompt tokens.
Introduction
Understanding and learning from DNA sequences has long been goal of biologists and deep learn- ing researchers, as its "language" encodes instructions essential for all living things [16]. The map- ping from DNA instructions, genotypes, to observable function and traits, phenotypes, remains on- going research effort. Towards this goal, researchers have proposed using foundation models (FMs) in genomics to learn generalizable features from unstructured whole genome data that can then be fine-tuned for number of tasks including predicting the location and function of genes, identifying regulatory elements, and analyzing the evolution of species [25, 10, 20, 3, 53, 58]. In contrast to protein sequences, which have had successes in protein language models [29, 31, 32, 17, 5, 41, 14], DNA sequences are orders of magnitudes longer (e.g. the human genome is 3.2B nucleotides) with long-range dependencies and interactions that span over 100k+ nucleotides coming the long-range limitations of current generation models could help drive the next wave of innovations in Al powwered drug discovery and therapeutics, and enable genomic FMs to understand and learn in-context whole patient genomes in personalized way.
length [1]. Over-
Limitations of current models Previous genomic FM approaches have relied on attention-based Transformers [25, 10, 53, 58],
PPL vs Context on the Human Genome
3.3
but face number of challenges unique to DNA sequences. The attention mechanism scales quadratically in sequence length, with
Layer Transformer
3.2
Layer HyenaDNA
Perplexity
Layer HyenaDNA
current genomic FMs pretraining on only 512
3.1
Layer HyenaDNA
4,096 tokens context [25, 58, 10, 55],
<0.001% % of the human genome. Also preva- lent is the reliance DNA "words", and tokenizers to aggregate meaningful DNA units. However, single nu- cleotide alterations represent physical analogs where, for example, single nucleotide poly- morphisms (SNPs) and mutations can have a profound impact on biological properties in- cluding regulatory activity [33]. In contrast, natural language semantics can often be con-
fixed k-mers, akin
2.9
200k
400k
600k
800k
1M
Sequence Length
Figure 1.2: Pretraining the human reference genome using longer sequences leads to better perplexity (improved prediction of next token).
served when single character or word changes occur over very long contexts. Therefore, having both long-range context and single nucleotide resolution simultaneously is critical, and remains particular challenge in genomics.

---

Hyena Order-N
HyenaDNA Block
+ Conv
Hyena
Add
Long Convolution
Windoow
MLP
xN
Elem-wise Gate
Dense+Conv
Add
Long Convolution
Sine Act
Dense
Hyena
Elem-wise Gate
Dense Conv
Operator
Dense+ Conv
Figure 1.3: HyenaDNA block architecture. Hyena operator is composed of long convolutions and element-wise gate layers. The gates are fed projections of the input using dense layers and short convolutions. The long convolutions are parameterized implicitly via an MLP that produces the convolutional filters. The convolution itself is evaluated using a Fast Fourier Transform convolution with time complexity (L logg L).
Toward longer context models Recently, Hyena [37]. large language model based on implicit convolutions, was shown to match attention in quality while reducing computational time complex- ity, thereby allowing a longer context to be processed. Hyena uses a parameter-effiient global con- volutional filter along with data-controlled gating mechanism, which enables context-specific operation over every token. Indeed, Hyena showed that for simple associative recall tasks using synthetic data, shallow 2 layer model could effectively process context lengths at 13 tokens. We hypothesize that Hyena's core operations can unlock the potential to capture both the long-range and single nucleotide resolution of real genomic sequences attention-baser approaches. To test this, we explore two questions: (i.) Can a convolutional long-context model be used effectively at sin- gle nucleotide resolution? (ii.) What new capabilities could long-context genomic foundations models enable?
HyenaDNA The result of our investigation is HyenaDNA, genomic FM pretrained on the human reference genome context lengths up million tokens single nucleotide resolution an up to 500x increase over existing genomic FMs using dense-attention. HyenaDNA scales sub- quadratically sequence length (training up to 160x faster than attention sequence length 1M), uses single nucleotide tokens, and has global receptive field at each layer. Our contributions include a "full-stack" recipe for building genomic FMs, including architecture design, a warm-up schedule to speed up training on ultralong sequences, and an efficient downstream adaptation procedure based on soft prompting and in-context learning.
Full-stack genomics modeling We start with a decoder-only Hyena architecture pretrained us- ing next nucleotide (token) prediction. We forego standard aggregating tokenizers, using a single- character tokenizer and a minimal DNA vocabulary of 4 nucleotides (plus special tokens). Training stability becomes issue ultralong sequences (200k+). To overcome this issue, introduce sequence length warm-up scheduler that gradually increases sequence length in stages. At sequence length 450k, training time is reduced by 40%, while boosting accuracy by 7.5 accuracy points species classification task. Furthermore, we design downstream adaptation procedures to leverage longer context windows, simpler and more flexible alternatives standard fine-tuning in ge- nomics. This includes novel soft prompt technique where learnable tokens (up to 32k) are injected directly into the input sequence itself, enabling competitive downstream results without the need to update pretrained model.
Genomic downstream tasks We apply our pretrained HyenaDNA models to 29 diverse down- stream genomic tasks to showease its long-range ability as well as fine-grain resolution. fine-tuned benchmarks from the Nucleotide Transformer [10], HyenaDNA achieves state-of-the-

---

art (SotA) on 12 of 18 datasets while using model with orders of magnitude less parameters and pretraining data (see Tab. 4.2). On the GenomicBenchmarks [23], HyenaDNA supasses SotA on 7 of 8 datasets on average by +10 accuracy points, and by as much as +20 accuracy points on en- hancer function identification. On a novel species classification task, HyenaDNA effectively solves the challenge by increasing the context length to million tokens. In a challenging chromatin profile experiment, a 919-way multi-task, HyenaDNA performs competitively against a larger SotA sparse- attention BigBird Transformer [55]. Finally, we analyze the learned embeddings of pretrained HyenaDNA model by clustering sequences by biotype (gene or transcription type) and compare the results with existing genomic FMs,, showing that HyenaDNA can serve effective universal featurizer in genomics.
Preliminaries and Related Work
2.1 Transformers and Attention
Powering many recent foundation models is the attention mechanism. Given length-L sequence R'XD, (single- isingle-headed) layer of scaled self-attention [2, 51] is map from R.xD to R which performs the following operations:
A(x) (w JT,, y y= Ax))
(2.1)
where D is the embedding dimension, Wa. W. DXD are learnable linear maps and indicated row-wise softmax (and optional scaling). Attention computes all pair-wise comparison for every token, and scales as ((L2) in sequence length. This allows a global context high resolution, but limits the size of the context on current hardware.
Previous methods to reduce the quadratic cost of attention have used specialized methods to ap- proximate full dense attention [18]. In sparse attention, elements attend only to a subset of all other positions. Alternatively, linear attention methods construct approximations A(u) that can be eval- uated in subquadratic time. Both of these classes of methods, however, trade lower time complexity (allowing longer sequences) for loss in expressivity.
2.2 Long Context Strategies in Genomics
To achieve longer context, genomic models have relied on two strategies: i. tokenization and ii dilation and downsampling. Tokenization is necessary step in masked language modeling (MLM) with bidirectional Transformer architectures (BERT) [13], a common model in genomies. These tokenizers use fixed k-mers (short overlapping sequences of length k) or frequency-based byte pair encoding (BPE), that attempt to aggregate DNA into meaningful units [25, 55J. Consequently, these aggregation techniques create large new vocabularies (compared to the natural vocabulary of 4 nucleotides) that are less generalizable [49]. The second strategy uses dilated convolutions and downsampling, both of which essentially average skip elements between weights [18]. A canonical example is the Enformer, which uses dilation and downsampling to reach context lengths of 100k nucleotides to predict gene expression tracks [1]. Common across tokenization, dilation, and downsampling is the sacrifice of single nucleotide resolution to reach longer context.
2.3 Large Convolutional Models
A discrete convolution between input & of length L and (learnable) filter la is given by:
= (h * x)2
e or equivalently
= Tx.
(2.2)
where T E RL is the Toeplitz matrix corresponding to the convolution. Historically, convolutions have played an important role in deep learning and more broadly signal processing. More recently, it has been shown that by stacking & long convolution layers, where & is parametrized through a function 7/o i.e. = (L), one can achieve state-of-ther thettat performance on a variety of benchmarks involving long sequences, for example the Long Range Arena (LRA) [48, 24, 47, 19]. Different "ye have been proposed in the literature: state-space models [24, 19], and implicit parametrizations via
neural fields [45, 44, 37]. On language, the H-family of implicit convolution language models,

---

H3 and Hyena, [12, 37] used long convolutions and gating to match Transformer performance in log, L) time, notably lower than the O(L2) of attention-based models.
HyenaDNA takes inspiration from these approaches, showing that attention-free, long-context causal models can achieve high performance downstream genomic tasks. These extended long-range capabilities enable us to explore new paradigms in genomics, such as in-context learning to easily adapt new tasks without updating pretrained models.
3 HyenaDNA Long-Range Genomic Foundation Models
In this section, we introduce the HyenaDNA approach to long-range genomic sequence modeling. We start with a description of the model architecture, then discuss sequence length warm-up and soft prompting techniques for downstream adaptation.
3.1 The HyenaDNA Model
The HyenaDNA model a decoder-only, sequence- sesuuence-t--eqqueee -o-ueuuence architecture defined by a stack of blocks consisting of Hyena operator [37], followed by ferd-forward neural network (see Fig. 1.3).
Given input E RR (I. denotes sequence length), a Hyena² operator can be defined as:
L2
W 22
Txa
02, v) - H(21, T2) H(21, 22) D.,, Dr,
21
W m:
T
(3.1)
W
D
Ta
D
where 22, V are projections of the input, and Ta L is the Toeplitz matrix constructed from a learnable long convolution filter produced as the output a neural network, (Tx);s = hı=j. The convolution filter values themselves are obtained through a small neural network To taking as input the time (position) index and optionally positional encodings, /2 (a))). which enable the operator to process very long sequences without growing linearly in the number of parameters. Further, the matrices Da, Dæz E RLxx are constructed with 21, 22 the diagonals, and evaluated as clement- wise gating. The projections are obtained by applying a dense linear layer and short convolution to the input sequence, shown in Figure 3.1.
Figure 3.1: bination of long convolutions and data- controlled gating D, and can be a drop-in re- placement for attention.
The Hyena operator is com-
Proposition 3.1. Hyena operator can be evaluated in OOLLogg L) time.
Efficient evaluation crucial on settings involving extremely long sequences such as genomics. In the general case where the embedding dimension D > and x E RLLD the linear projections Way, W W, E DxD are right multiplied to x. and D independent Hyena operators are then applied to each dimension.
3.2 Training Long Sequence Models
The subquadratic cost of HyenaDNA in sequence length allows the model to process ultralong sequences directly at the single nucleotide level without the need for frequency-based aggregation tokenizers. This enables fine-grain resolution for both short and long sequences, critical for detecting single nucleotide polymorphisms or mutations and modeling long-range dependencies in gene expression.
Tokenization
We use the natural DNA vocabulary and refer to each nucleotide as a token. The tokens include "A". "G". "C", "T", and "N" (a non-specific nucleotide) and special character tokens for padding, separation, and unknown characters. Tokens are mapped to embedding dimension D.
Sequence length warm-up for ultralong sequences Directly training on long sequences can af- feet training stability as the variance in gradient increases [28]. Training on shorter sequences ini- tially (followed by longer sequences) was used by [38] to train small scale Transformers and reduce training time, while [28] used sequence length warm-up address stability on up 2k tokens.
We discuss D and order Hyena operators for simplicity.

---

For ultralong sequences (200k+), we develop new warm-uu schedule that gradually increases the se- quence length in stages to improve both stability and decrease training time.
Test Acc (%) vs. Training Time
100
80
Our sequence length schedule starts at L1 = 64, then doubles the window each stage while keep- ing the global batch size constant. By doing so, iterations at each consecutive stage will include more tokens, ensuring the scheduler can also act as a form of batch size warm-up. In Fig. 3.2, we observe sequence length scheduling to be partic- ularly important at sequence lengths greater than 450k, where at this length training time is reduced by 40% and improving ultimate accuracy by 7.5% points for species classification task described later in section 4.4.3.
60
40
Narmwarmup
Warmup
20
200
400
Time (min)
Sequence length warm-up duces the training time of NyenaDNA at se- quence length 450k by 40% and boosts accu- racy by 7.5 points on species classification.
Figure 3.2:
3.3 Downstream Adaptation
Tuneable prompting for long-context models
Prompts have been traditionally used to guide the output of FM [30] by prepending additional context to an input. Expanding on this approach, soft tuneable prompting was introduced to inject learnable tokens (as weights) into the input directly [27] as an alternative to model fine-tuning.
With an extended context length (L), we're able to explore new paradigms in adapting FMs after pretraining. Given downstream task with prompts Tp E RT and corresponding labels Up. we prepend N < L T trainable parameters 0 of dimension D after the embedding step:
I f foncat/embed(r.)) 0), x E (X
(3.2)
The resulting sequences I are then processed by the model, and 0 is optimized on a loss function involving the input sequence's 's label My- Crucially, soft prompting requires utilization of small subset of prompt and label pairs to optimize 0.
During soft prompting, HyenaDNA only optimizes the parameters of the prompt in the input quence while keeping all other model parameters fixed. Soft prompting thereby provides a flexible and computationally efficient approach to adapting genomic FMs new downstream tasks.
4 Experiments
In 4.1, we start with pretraining HyenaDNA on the human reference genome [22]. We then evaluate HyenaDNA on existing short-range (<5k nucleotides) downstream benchmarks in 4.2 assess the performance of single nucleotide resolution. In 4.3, we explore what new capabilities emerge with longer range genomic modeling in the form of in-context learning. Finally, push the limits of ultralong context performance in 4.4.
4.1 Pretraining on the Human Genome
We pretrain HYenaDNA on the human reference genome [22] using next nucleotide (token) predic- tion. Starting with a stack of decoder-only Transformer blocks, we swap attention for the Hyena operator, and compare against a bascline Transformer (GPT) with Flash Attention [11]. We add gradient checkpointing to DNA decrease the memory footprint by 3x on longer sequences (> 160k). We then scale DNA along dimensions of model depth (2 to 8 layers), width (128 to 256 dimensions), and sequence length (1024 to IM). At sequence length 1M, HyenaDNA is 160x faster than its Transformer counterpart shown in Fig. 4.1.
As shown in Fig. 1.2, we observe that as context length increases, perplexity improves during pretraining. However, this improvement comes at the expense of more training time and tokens. For models too shallow to effectively process longer context, perplexity can begin to degrade (increase), observing inflection points with longer sequences. In this way, increasing context can serve

---

novel regularization dimension. For genomic pretraining, we provide the following guidelines. 1. In optimizing for faster training time, shorter context enable lower perplexity be reached faster. 2. In optimizing for best overall perplexity, longer context allows for lower perplexity at the cost of training on more tokens. See I for experiment details.
4.2 Single Nucleotide Resolution
Our first downstream tasks short-range genomic sequences (<5k) aimed at evaluat- ing single nucleotide resolution performance on sequence-level classification using stan- dard fine-tuning.
Runtime vs Context (log-scale)
10
Transformer
104
millisee (log)
HyenaDNA
103
10a
GenomicBenchmarks We start with the newly released GenomieBenchmarkss [23], which is comprised of 8 regulatory element classification datasets with sequence lengths of 200-500, and one up to 4,776. The original baseline model uses a short-range CNN. We fine-tune the pretrained Transformer (GPT) and HyenaDNA from 4.1, both having sin- gle nucleotide resolution, DNABERT model [25]. NNA sets a new SotA 7 of datasets and by up to 20% points on the human enhancer identification task, shown in Tab. 4.1. See A.2 for additional experiment details and ablations.
10a
160x faster
11
10a
104
10a
109
Sequence Length (log)
Figure 4.1: Runtime (forward & backward
pass) for Transformer & HyenaDNA: 2 layers, width=128, gradient checkpoint, batch size=1, A100 80GB. At 1M tokens HyenaDNA is 160x faster than Transformer.
well as the
Nucleotide Transformer Next, we benchmark against 18 datasets from the Nucleotide Trans- former (NT) [10]. which includes predicting reg- ulatory elements for
Table 4.1: GenomicBenchmarks Top-1 accuracy (%) for pretrained HyenaDNA, DNABERT and Transformer (GPT from 4.1), and the previous SotA baseline CNN (scratch).
DATASET
CNN
DNABERT
GPT HYENADNA
Mouse Enhancers Coding vs Intergenomic
69.0
66.9
80.1
85.1
87.6
92.5
88.8
91.3
enhancers, promoters, epigenetic marks, and splice sites from DNA se- quences of length 200-600 nucleotides. We compare against NT base models, which were pretrained using masked language modeling (BERT) and then fine-tuned. The NT models ranged from 500M to 2.5B parameters, and pretrained on up to 3202 genomes. All NT models use 6-mer sequences of 1000 tokens long. For HyenaDNA, we attach a linear decoder head and fine-tune a pretrained model, surpassing SotA on 12 of 18 datasets using model with orders of magnitude less parameters and pretraining data, shown in Tab. 4.2. See A.2 for additional experiment details and ablations.
Human Worm Human Enhancers Cohn
93.0
96.5
95.6
96.6
69.5
74.0
70.5
74.2
Human Enhancers Ensembl
68.9
85.7
83.5
89.2
Human Regulatory
93.3
88.1
91.5
93,8
Human Nontata Promoters
84.6
85.6
87.7
96.6
Human OCR Ensemhl
68.0
75.1
73.0
80.9
4.3 In-context Learning for Genomic Sequences
Compared natural language FMs, which have shown strong success with in-context learning. HyenaDNA's vocabulary is very small. DNA sequences are also less diverse in structure, e.g. there's no concept of labels descriptions that follow DNA sequence. This makes challenging to per- form "pure" in-context learning (relying only on inference), since new concepts such as classifica- tion labels would require new symbols. To overcome this limitation and explore the potential for in-context learning in genomics, we make use of two variants of in-context learning: soft prompting and instruction fine-tuning. Each involve brief tuning phase introduce the concept classifica- tion using only the existing vocabulary.

---

Procedure In both variants, we use the GenomicBenchmarks in 4.2, and HyenaDNA model pretrained on se- quence length 160k from 4.1.
Table 4.2: Nucleotide Transformer (NT) Benchmarks
The Matthews correlation coefficient (MCC) is used as the performance metric for the enhancer and epigenetic marks dataset, and the Fl-score is used for the promoter and splice site dataset.
In the first experiment, we evaluate soft prompting approach by prepend- ing a sequence of soft tuneable to- kens (2 to 32k) directly in the input sequences. We include brief tuning phase (< 20 epochs), updating the soft tokens only. to provide Nye with the ability to indicate the target classes. To denote classes, we repur- pose HyensDNA's fixed vocabulary: for binary classification, for example, we indicate the two classes with the letters "A" and "N".
MODEL PARAMS GENOMES
NT
NT
NT
HyA
500M
2.5B
2.5B
1.6M
3,202
850
1
Enhancer
53.5
59.3
58.0
62.6
Enhancer types
48.5
50.0
47.4
55.7
H3
73.7
77.6
81.4
81.7
13K4mel
35.8
44.5
55.9
57.1
H3K4me2
28.1
30.0
32.6
53.9
H3K4me3
26.3
28.1
42.1
61.2
H3K9ac
46.2
50.8
57.5
65.1
14K1
37.7
47.1
55.0
66.3
H3K36mc3
46,7
53.3
63.2
65.3
In the second experiment, we evalu- ate few-shot learning approach to in-context learning [6] by prepending, consecutively, (2 to 32) demonstra- tions of each class and its sequence into the prompt. As before, we en- code class labels by the use of indi- vidual letters of DNA's existing vocabulary. additionally perform brief instruction-tuning period [52] for each dataset to familiarize HyenaDNA with this task structure by tuning the pretrained model on small subset of the dataset.
H3K79me3
57.7
59.2
64.2
71.6
H4
76.2
78.9
82.2
79.6
H4ac
34.4
42.3
50.1
63.7
Promoter all
95.4
96.6
97.4
96.5
Promoter non-TATA
95.6
96.9
97.7
96.6
Promoter TATA
94.8
95.8
96.4
96.7
Splice acceptor
96.5
98.5
99.0
96.6
Splice donor
97.2
98.2
98.4
97.3
Splice all
97.2
97.8
98.3
97.9
Ocr-Ensembl
Nontata--rromoters
Enhhancers-Conn
Enhancers-Ensembl
Ensembi-Regulatory
50
25
2
2
2
24
211
2 202-22222200 201
20
20 213
2 2' 2 2
2 2
201
#Tuneable Tokens
*Tuneable Tokens
#Tuneable Tokens
#Tuneable Tokens
*Tuneable Tokens
Figure 4.2: Filling long-context with soft tuneable tokens. HyenaDNA is able to learn new tasks in-context when adding sequence of tuneable tokens the input sequences. Longer sequences of tuneable tokens lead to better performance.
Results In Fig. 4.2, HyenaDNA's performance on novel tasks improves more tuneable tokens are added into input sequences, and saturates close to baseline performance (Tab. 4.1: with the exception of the Human Regulatory dataset). By contrast, we find that increasing k- demonstra- tions to the input does not necessarily improve performance. A higher number tuning samples is needed before k-shot demonstrations start to boost accuracy shown in Tab. A.I. See A.3 for experiment details.
4.4 Ultralong-Range Genomics
our final experimental section, we focus on pushing the limits of using long context effectively in genomics. In 4.4.1, tackle challenging 919 binary multi-task against sparse- attentioo baseline. In 4.4.2 we analyze the learned embeddings HyenaDNA and its use in clustering long sequences by functional annotation, and in 4.4.3 showease novel ultralong-range species classification task.
4.4.1 Chromatin Profile Prediction

---

The prediction of chromatin profiles and epigenetic markers from DNA sequences an important and challenging task to quan- tify the functional effects of non-coding variants. These variants include single nu- cleotide changes DNA that can affect the downstream expression of genes [56]. The DeepSEA dataset [57] is compiled from 919 chromatin features including transerip- tion factor (TF) binding profiles, DNase t-yypessensitive sites (DHS) and histone mark (HM) profiles. For a given sequence, the task is jointly predict 919 labels cor- responding to the chromatin profile (similar to peak detection) of a central region of the sequence, indicating the presence of such functional effects. The input also includes flanking regions that pro- vide broader contextual information needed to incorporate long-range interactions. We fine-tune our pretrained Hyena DNA models from 4.1 and perform competitively against DeepSea CNN and the SotA sparse attention BigBird [55] baselines using 5-30x fewer parameters. See A.4 4 for experiment details.
Table 4.3: Chromatin profile prediction Median AUROC computed over three categories: Tran- scription factor binding profiles (TF), DNase
hypersensitive sites (DHS) and histone marks (HM).
AUROC DHS
MODEL
PARAMS
LEN
TF
HM
DeepSEA BigBird
40
95.8
92.3
85.6
110 M
8k
96.1
92.1
88.7
1k
96.4
93.0
86.3
HyenaDNA
3.5 M
8k
95.5
91.7
89.3
4.4.2 Biotype Embeddings
Sequence embeddings, colored by biotype
DNABERT
Nucleotide Transformer
HyenaDNA
Protein Coding
IncRNA
Provessed Pseudogene
Unprocessed Pseudogene
saRNA
miRNA
TEC
snoRNA
MiscRNA
Figure 4.3: Embedding visualisation. t-SNE of the embeddings generated by DNABERT, Nu-
cleotide Transformer and HyenaDNA coloured by Ensembl biotype annotations.
Next, we analyze the pretrained embeddings from HyenaDNA and compare them with DNABERT [25] and the Nucleotide Transformer [10]. We encode sequences human genes corresponding to
different biological function annotations obtained from the Ensembl dataset known as biotypes [9]. In cases where the length of the input exceeds the context window the encoder, the sequence is chunked (by the max length of the encoder) and averaged.
We fit the embeddings using an XGBoost [7] classi- fier on the 10 most frequent biotypes, and apply t- SNE [50] for visualization. As shown in 4.3, dis- tinct clusterings emerge visually. while quantitatively. HyenaDNA produces the highest Fl score in biotype classification (with much smaller model), indicating that during pretraining, DNA learns informative features related to biological function.
Table 4.4: Embedding quality Weighted FI classification score on 10 biotypes.
MODEL
PARAMS
LEN
Fl
DNABERT NT
110 M
512
64.6
500 M
6k
66.5
HyenaDNA
M
160k
72.0

---

4.4.3 Species Classification
The majority of the genome is conserved across species humans and non-human primates, for example, have <10% sequence divergence [43], making them difficult to discriminate. This allows us to to design ultralong-range sequence modeling task to test whether model can determine random genetic sequence. To train, we randomly sample DNA sequences
the source species from 5 different species, and fine-tune pretrained HyenaDNA and Transformer models from 4.1 to predict the species label. We observe in Tab. 4.5 that both models struggle on shorter sequences of length 1024, but performance improves with longer contexts the distinct mutational profile of each species becomes more evident. NA effectively solves the task by using a context length of 450k million, where Transformer cannot due infeasible training time limitations. See A.6 for experiment details.
Conclusion
Summary We presented HyenaDNA, a genomic foun- dation model pretrained the human reference genome with context lengths up to 1 million tokens at single nu- previ-
Table 4.5: Species classification Top- accuracy (%) for 5-way classifi- cation (human, lemur, mouse, pig, hippo). The X symbol indicates in- feasible training time.
cleotide resolution an up to 500x increase ous genomic FMs using dense-attention. HYenaDNA is able to learn generalizable features that then be fine- tuned for tasks including identifying regulatory elements and on a 919-way chromatin profile prediction task. We also explored the first use of in-context learning in ge- nomics to enable simpler adaptation downstream tasks without any updates to pretrained weights.
MODEL
LEN
AcC
Transformer
1k
55.4
HyenaDNA 1k
61.1
Transformer
32k
88.9
HyenaDNA
32k
93.4
Limitations and Future Work While demonstrating competitive results and introducing novel capabilities, it worth noting that HyenaDNA was pretrained on only one human reference genome. Incorporating genomes of multiple humans and species could increase generalizabil- ity in learned features and reduce bias. Furthermore, our current focus in this study was exclusively on DNA se- quences. Extending biological or chemical sequences, such as proteins and drug molecules, has the potential to unlock multi-modal capabilities similar those observed in natural language and vision FMs [39, 40, 54].
Transformer
250k
x 97.9
HyenaDNA
250k
Transformer
450k
x 99.4
HyenaDNA
450k
Transformer
IM
HyenaDNA
99.5
framework to incorporate other
With respect to model size, HyenaDNA is significantly smaller than previous genomic FMs and was pretrained using up to 8 Nvidia A100 (80GB) GPUs. We expect increasing model size, and compute, may lead to additional long-range capabilities. Notably, with model parallelism, becomes feasible to extend the context length by orders of magnitude beyond this current work, and leave that open to future research.
Furthermore, beyond discriminative applications, the use of long context models in generative tasks unlocks exciting prospects for the design of synthetic regulatory elements, genes and protein com- plexes. In conclusion, the continued advancements of long-range sequence models with single nu- cleotide resolution hold great promise in driving innovation in genomic research and unraveling the complexities of biological systems.
Acknowledgments
We would like to thank Guatam Machiraju, Elliott Epstein, Archis Joglekar, Jared Dunnmon, Nazim Bouatta and Anshul Kundaje for helpful discussion and feedback on earlier drafts, and Together for providing the compute used to train models in this paper. We gratefully acknowledge the sup- port of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCFI763315 (Beyond Spar- sity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); US DEVCOM ARL under No. INF-1112-- (Interactive Human-Al Teaminn) ONR under No. N000141712266 (Unifying Weak Supervision); ONR 4-0014-20-1-2480 Understanding and Applying Von-Fuclidean Geom-
10

---

etry in Machine Learning: N000142012275 (NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), Department of Defense (DoD) through the National De- fense Science and Engineering Graduate Fellowship (NDSEG) Program, and members of the Stan- ford DAWN project: Facebook, Google, and VMWare. This work is supported by NSF (1651565), AFOSR (FA95501910024), ARO (W911NF-21-1-012)) hub, and Sloan Fellowship. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
ONR, DOE (DE-SC0022222), CZ Bio-
References
[1] Ž. Avsec, V. Agarwal, D. Visentin, J. R. Ledsam, A. sael, J. Jumper, P. Kohli, and D. R. Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10): 1111-199---,,, 2021.
K. R. Taylor, As-
[2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 1409.040474 2014.
[3] G. Benegas, S. S. Batra, and Y. S. Song. DNA language models are powerful zero-shot predic- tors of non-coding variant effects. bioRxiv, pages 2022-08, 2022.
[4] R. Bommasani, D. A. Hudson, Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bemsttiin J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint 1rXiv:2108.07255: 2021.
[5] N. Brandes, D. Ofer, Y. Peleg, N. Rappoport, and M. Linial. ProteinBERT: a universal deep- learning model of protein sequence and function. Bioinformatics, 38(8):2102-2110 2022.
[6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in nearul information processing systems, 33:1877-1901, 2020.
[7] T. Chen and C. Guestrin. Xghoost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[8] D. M. Church, V. A. Schneider, T. Graves, K. Auger, F. Cunningham, N. Bouk, H.-C. Chen, R. Agarwala, W. M. McLaren, G. R. Ritchie, et al. Modernizing reference genome assemblies. PLoS biology,
2011.
[9] Cunningham, E. Allen, J. Allen, J. Alvarez-Jarreta, M. Amode, M. Armean, O. Dustine-Orimolee A. G. Azov, I. Barnes, R. Bennett, et al. Ensembl 2022. Nucleic acids research, 50(D1):D988 -D95,, 2022.
[10] H. Dalla-Torre, L. Gonzalez, J. Mendoza-Revilla, N. L. Carranza, A. H. Grzywaczewski, F. Oteri, C. Dallago, E. Trop, H. Sirelkhatim, G. Richard, M. Skwark, K. Beguir, M. Lopez, and T. Pierrot. The Nucleotide Transformer: Building and evaluating robust foundation models for human genomics. bioRxiv, 2023.
[11] Dao, Fu, S. Ermon, A. Rudra, and C. Ré. FlashAttention: Fast and memory exact attention with 10-awareness. In Advances in Neural Information Processing Systems, 2022.
[12] T. Dao, D. Y. Fu, K. K. Saab, A. W. Thomas, A. Rudra, and C. Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14003 2022.
[13] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv: 18X.v:881000480: 2018
11

---

[14] A. Elnaggar, M. Heinzinger, C. Dallago, G. Rehawi, Y. Wang, L. Jones, T. Gibbs, T. Feher, C. Angerer, M. Steinegger, et al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE transactions on patlern analysis and machine intelligence, 44 (10):7112-717, 2021.
[15] ENCODE Project Consortium. An integrated encyclopedia of dna elements in the human genome. Nature, 489(7414):57, 2012.
[16] ENCODE Project Consortium. Expanded encyclopacdias of DNA elements in the human and mouse genomes. Nature, 583:699-710, 2020.
[17] N. Ferruz, S. Schmidt, and B. Höcker. ProtGPT2 is deep unsupervised language model for protein design. Nature communications, 13(1):4348, 2022.
[18] Q. Fournier, G. M. Caron, and D. Aloise. A practical survey faster and lighter transformers. ACM Computing Surveys, 2021.
[19] D. Y. Fu, E. L. Epstein, R. Nguyen, A. W. Thomas, M. Zhang, T. Dao, A. Rudra, and C. Ré. Simple hardware-efficient long convolutions for sequence modeling. arXiv preprint arXiv:2302.06646, 2023.
[20] D. Gankin, A. Karollus, M. Grosshauser, K. Klemon, J. Hingerl, and J. Gagneur. Species- aware DNA language modeling. bioRxiv, pages 2023-01, 2023.
[21] Q Geng, R. Yang, and L. Zhang. A deep learning framework for enhancer prediction using word embedding and sequence generation. Biophysical Chemistry, 286:106822, 2022.
[22] Genome Reference Consortium. Genome reference consortium human build 38 (grch38). National Center for Biotechnology Information, 2013. URL httpt:h//ww.w..... ncbi. nlm. nih.gov http///asemmlyc/ccccc 000001405 26/.
[23] K. Gresova, Martinek, D. Cechak, P. Simecek, and P. Alexiou. Genomic Benchmarks: A collection of datasets for genomic sequence classification. bioRxiv, 2022.
[24] A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.0033 2021.
[25] Y. Ji. Z. Zhou, H. Liu, and R. V. Davuluri. DNABERT: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome. Bioinformatics, 37 (15):2112-2100) 2021.
[26] J. Kent, C. W. Sugnet, T. S. Furey, K. M. Roskin, T. H. Pringle, A. M. Zahler, and D. Haus- sler. The human genome browser at ucse. Genome research, 12(6))996- 1006, 2002.
[27] B. Lester, R. AI-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning, arXiv preprint S.iv:2104.08642 2021.
[28] C. Li, M. Zhang, and Y. He. The stability-efficieney dilemma: Investigating sequence length warmup for training GPT models. In Advances in Neural Information Processing Systems, 2022.
[29] Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, A. dos Santos Costa, M. Fazel-Zarandi, Sereu, Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022.
[30] Liu, W. Yuan, J. Fu, Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-33 2023.
[31] A. Madani, B. Krause, E. R. Greene, S. Subramanian, B. P. Mohr, J. M. Holton, J. L. Olmos Jr, C. Xiong, Z. Z. Sun, R. Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, pages 1-8, 2023,
12

---

[32] J. Meier, R. Rao, R. Verkuil, J. Liu, T. Sercu, and A. Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural Information Processing Systems, 34:29287-29303 2021.
[33] Nasser, D. T. Bergman, C. Fulco, Guckelberger, B. R. Doughty, T. A. Patwardhan, T. R. Jones, T. H. Nguyen, J. C. Ulirsch, F. Lekschas, K. Mualim, H. M. Natri, E. M. Weeks, G. Munson, M. Kane, H. Y. Kang, A. Cui, J. Ray, T. M. Eisenhaure,
L., Collins, K. Dey, H. Plister, A. L. Price, C. B. Epstein, A. Kundaje, R. J. Xavier, M. J. Daly, H. Huang, H. K. Finucane, N. Hacohen, E. S. Lander, and J. M. Engreitz. Genome-wide enhancer maps link risk variants to disease genes, Nature, 593:238-22 2433, 2021.
[34] M. Oubounyt, Z. Louadi, H. Tayara, and K. T. Chong. DeePromoter: Robust promoter predic- tor using deep learning, Frontiers in Genetics, 10, 2019.
[35] T. H. Pham, D. H. Tran, T. B. H. Ho, K. Satou, and G. Valiente. Qualitatively predicting acetylation and methylation areas in DNA sequences. Genome Informatics, 16(2):3-11, 2005.
[36] D. K. Pokholok, C. T. Harbison, S. Levine, F. Lewitter, D. K. Gifford, and R. A. Young. Genome-wide map of nucleosome acetylation and methylation in yeast. Cell, 122(4):517-527, 2005.
[37] M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and C. Ré. Hyena Hierarchy: Towards larger convolutional language models. arXiv preprint 23XIv:2302.10866, 2023.
[38] O. Press, N. A. Smith, and M. Lewis. Shortformer: Better language modeling using shorter inputs. arXiv preprint arXiv:2012.15832. 2020.
[39] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[40] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[41] R. Rao, J. Meier, T. Sercu, S. Ovchinnikov, and A. Rives. Transformer protein language models are unsupervised structure learners. Biorxiv, pages 2020-12, 12, 2020.
[42] Roadmap Epigenomics Consortium. Integrative analysis of 111 reference human epigenomes. Nature, 518(7539):317-330, 2015.
[43] J. Rogers and R. A. Gibbs. Comparative primate genomics: emerging patterns of genome content and dynamics. Nature Reviews Genetics, 15(5):347-359, 2014.
[44] D. W. Romero, R.-J. Bruintjes, J. M. Tomezak, E. J. Bekkers, M. Hoogendoorn, and J. C. van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. arXiv preprint arXiv:2110.08059) 2021.
[45] D. W. Romero, A. Kuzina, E. Bekkers, J. M. Tomezak, and M. Hoogendoom. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102.02611,222
[46] N. Scalzitti, A. Kress, Orhand, T. Weber, L. Moulinier, A. Jeannin- Jeamnin-Girardon, O. Collet, Pierre anf Poch, and J. D. Thompson. Spliceator: multi-species splice site prediction using convolutional neural networks. BMC Bioinformatics, 22(1):1-26, 2021.
[47] J. T. Smith, A. Warrington, and S. W. Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208. 04933, 2022.
[48] Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. Long range arena: benchmark for efficient transformers. arXiv preprint uxXiv:2011.04006 2020.
13

---

[49] Y. Tay. V. Q. Tran, S. Ruder, J. Gupta, H. W. Chung, D. Bahri, Z. Qin, S. Baumgartner, C. Yu, and D. Metzler. Charformer. Fast character transformers via gradient- based subword tokeniza- tion. arXiv preprini urXiv:2106. /2672, 2021.
[50] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[51] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, ł. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models zero-shot learners. arXiv preprint arXiv:2109.01652 2021.
[53] Yang, W. Wang, Wang. Y. Fang, D. Tang, J. Huang, H. Lu, and J. Yao. scBERT as large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data. Nature Machine Intelligence, 4(10))852-8) 866, 2022.
[54] Yu, Z. Wang, Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners image-text foundation models. arXiv preprint arXiv:2205.01911 2022.
[55] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, Pham, A. Ravula, Q. Wang, L. Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297 2020.
[56] Zaina, E. Pérez- Luruz, and G. Lund. Genetics talks to epigenetics? the interplay between sequence variants and chromatin structure. Current genomics, 11(5):359 -379, 2010.
[57] J. Zhou and O. G. Troyanskaya. Predicting effects of noncoding variants with deep learning- based sequence model. Nature methods, 12(10):931 -933,, 2015.
[58] M. Zvyagin, A. Brace, K. Hippe, Y. Deng. B. Zhang, C. O. Bohorquez, A. Clyde, B. Kale, D. Perez-Rivera, H. Ma, et al. GenSLMs: Genome-scan scale language models reveal SARS-CoV. 2 evolutionary dynamics. bioRxiv, pages 2022-10, 2022.
14

---

HyenaDNA
Supplementary Material
Contents
1
Introduction
Preliminaries and Related Work
2.1
Transformers and Attention
2.2
Long Context Strategies in Genomics
2.3
Large Convolutional Models
3
HyenaDNA Long-Range Genomic Foundation Models
3.1
The HyenaDNA Model
Model
3.2
Training Long Sequence Models
3.3
Downstream Adaptation
Experiments
4.1
Pretraining on the Human Genome
4.2
Single Nucleotide Resolution
4.3
In-context Learning for Genomic Sequences
4.4
Ultralong-Range Genomics
4.4.1
Chromatin Profile Prediction
4.4.2
Biotype Embeddings
4.4.3
Species Classification
10
5
Conclusion
10
Appendix: Experimental Details
17
A.1
Pretraining Details
17
A.2
Short-Range Genomics Details
18
A.2.1
GenomicBenchmarks experiment
18
A.2.2 Ablations on the GenomicBenehmarks
18
A.2.3
Downstream prediction tasks for Nucleotide Transformer benchmark
19
A.2.4 Ablations the Nucleotide Transformer benchmarks
20
A.3
In-Context Learning Details
20
A.4
Chromatin Profile Details
22
A.5
Biotype Embeddings Analysis Details
23
15

---

A.6 Long-range Species Classification Details
24
16

---

A Appendix: Experimental Details
In the following sections we provide further details for each experiment. Across all experiments, we use Pytorch and Pytorch Lightning. We train on mix of Nvidia GPUs with A100s, V100s, and T4s. Unless otherwise stated, we use cross entropy loss for our objective. Our repository made public here:
A.1 Pretraining Details
Table A.I: Hyperparameter settings for Hyena pretraining (select models).
Layers
2
8
Width
128
256
128
256
256
Params (M)
0.44
1,6
0.87
3.3
6.6
Max seq. len.
64k
64k
64k
64k
1M
Optimizer
AdamW
Optimizer momentum
81. _82 0.9, 0.999
Learning rate
1.5 - 6e-4
LR Scheduler
Cosine decay
Batch size
64 - 256
Global steps
10 20k
Weight decay (model)
0.1
Weight decay (Hyena layers)
Embed dropout
0.1
Residual dropout
0
Data For pretraining, we use single human reference genome [22], and leverage the training and validation intervals (start and end) from [1]. During training, sample interval and obtain sequence of length by adjusting the intervals on both ends. For the test set, we use chromosomes 14 and X. exclusively, and sample non-overlapping sequences of length L.
Model We design suite parameter efficient architectures with depths between 2 and 8 layers, Hyena blocks of Order-N 2, and width 128 256. The MLP expansion factor (reverse bottleneck) 4x the width. See Fig. 1.3 for the block architecture of HyenaDNA. The parameter counts range from 400k to 6.6M, trained a representative subset of the models we trained. Note: we use different pretrained model sizes depending the downstream task to prevent overfitting. When selecting which pretrained model to use for a downstream task, we found that a pretrained sequence length of 2 to 4x the downstream max sequence length results in the best performance.
sequence lengths between 1,024 and 1M. Tab. A.1 highlights
Training We pretrain each model for 10-20k global steps. For models trained on longer sequences, this translates more tokens being used, as each sample contains more tokens. For example, the largest model with context length 1M was trained on 2T tokens over 4 weeks. We adjust the "accu- mulate_gran batches" argument in Pytorch Lightning keep the global batch size consistent across models and sequence lengths. See Tab. A.1 for hyperparameter details.
Training efficiency We compare pretraining compute resources and GPU-hours to reach compet- itive performance on the short-range tasks for several baselines and HyenaDNA models, shown in Tab. A.2.
Table A.2: Pretraining GPU & runtime comparison for short-range models.
DNABERT NUCLEOTIDE TRANSFORMER
HyenaDNA
HyenaDNA
Params
110M
2.5B
436K
1.6M
GPUs
8-2080 TI
128-A 128-A100-800 100-80GB
1-A100-40GB mins
1-A100-40GB 80 mins
Wall clock
25 days
28 days
GPU-hrs
12,000
215,000
1.3
1.3
17

---

A.2
Short-Range Genomics Details
A.2.1
GenomicBenchmarks experiment
Data The GennomicBenhmarks [23] includes 8 datasets designed for sequence-level classification tasks that involve predicting regulatory elements, along with one binary species task. The bench- marks provided for the baseline model include two sets results: one obtained with Pytorch and the other with TensorFlow. Since our code base is implemented in Pytorch, we compare our results with the Pytorch-based benchmarks.
Model Our backbone is pretrained 2 layer HyenaDNA model with width 128, trained on quence length 1024. We pool along the sequence dimension to obtain a classification token, and attach simple linear decoder head. The baseline CNN, described by [23], uses uses embed- ding layer, 3 convolutional layers with number of filters: 16, 8, and 4. uses batch norm and max pooling after each convolutional layer, followed by dense layers. It is trained for 10 epochs with batch size 64. The mode sizes range from 120k to 520k, depending on sequence length chosen.
Table A.3: GenomicBenchmarks hyperparameters for HyenaDNA and the baseline Transformer (GPT from 4.1), which uses FlashAttention [11].
TRANSFORMER
HyenaDNA
Layers
2
Width
128
128
Parameters
529k
436k
Learning rate
1-6e
1-6e -4
Weight decay (model)
0-0.2
0-0.2
Weight decay (Hyena layers)
0
Embed dropout
0-0.2
0.0-0.3
Resid dropout
0-0.2
0-0.3
Num heads
8
Optimizer
AdamW 81, 8g = 0.9, 0.999 Cosine decay 128-1024
Optimizer momentum
LR scheduler
Batch size
Training epoch
100
Reverse complement aug.
true/false 200-4800
Sequence lengths
The primary hyperparameters we sweep across include: learning rate, global batch size, dropout, weight decay, and a reverse complement augmentation. See Tab. A.3 for ranges of hyper- paramters used.
Training
A.2.2 Ablations the GenomicBenchmarks
To better understand how specific design choices in the HyenaDNA model effect performance, perform a series of ablation experiments on the GenomicBenchmarks.
Prefraining: We train HyenaDNA from scratch and compare with the pretrained version. The pre- trained models provide mild to moderate gains likely due to the benchmarks being near saturation already.
Tokenization: We train Hye using a k-mer tokenizer (k=6) to isolate the effect of the single nucleotide tokenizer. The k-mer tokenizer drops performance significantly across on majority of the datasets (by as much as 10 accuracy points), while boosting one dataset (Human Enhancer Ensembl). Therefore, the single nucleotide tokenization appears to be significant component of the HyenaDNA model.
Bidirectional: To ablate the impact of using causal model, we implemented bidirectional ver- sion of DNe and trained from scratch on the GenomicBenchmarks (i.c. without masked
18

---

Table A.4: GenomicBenchmarks Top-1 accuracy (%) GPT is the causal Transformer from 4.1,
NyenaDNA k-mer uses 6-mer tokenizer, and HyenaDNA bidirection is bidirectional version of the Hyena operator.
HyenaDNA k-mer
HyenaDNA bidirection
MODEL
GPT
GPT
HyenaDNA HyenaDNA
DNABERT
Pretrained
no
yes
no
yes
no
no
yes
Mouse Enhancers
79.3
79.3
84.7
85.1
81.8
80.6
66,9
Coding VS Intergenomic
89.3
91.2
90,9
91.3
86.7
90.3
92.5
Human VS Worm
94.8
96.6
96,4
96.6
92.9
95.9
96.5
Human Enhancers Cohn
67.7
72.9
72.9
74.2
69.8
72.1
74.0
Human Enhancers Ensembl
79.0
88.3
85.7
89.2
88.0
85.9
85.7
Human Regulatory
90.2
91.8
90.4
93.8
89.1
88.1
Human Nontata Promoters
85.2
90.1
93.3
96.6
83.5
88.5
85.6
Human OCR Ensembl
68.3
79.9
78.8
80.9
70.2
75.3
75.1
language model pretraining). The bidirectional version degraded performance on 7 of 8 datasets compared to the standard causal HyenaDNA (also from scratch), average by 3.8 accuracy points.
The bidirectional NNA was implemented by using a circular FFT convolution. This involved manipulating the padding the input sequence before performing the FFT convolution. Previously, we zero padded the input on the right side by length L (the sequence length). For bidirectionality, pad by 1/2 L the left and right side of the input, effectively providing bidirectional ceptive field (due to the circular convolution). This is one of many possible ways to implement a bidirectional version of Hyena.
A.2.3 Downstream prediction tasks for Nucleotide Transformer benchmark
Following the Nucleotide Transformer [10], collected datasets from four different sources [21, 35, 34,46]. 46].
Promoter The promoter dataset included box-containing and ANTA-box-lacking promot- Tasks involved predicting promoters with TATA-box. identifying promoters lacking TATA- box, and distinguishing between both promoter categories and non-promoter sequences. The pro- moter datasets mouse genomes. Promoter sequences were extracted from regions 249 nucleotides upstream and 50 nucleotides downstream of the transcription start sites.
obtained from the Eukaryotic Promoter Database (EPDnew)* for human and
Enhancer For the enhancer prediction task, used the dataset from [21] containing DNA quences classified into strong enhancers, weak enhancers, and non-enhancers. The tasks involved binary classification to distinguish enhancer sequences from non-enhancer sequences and identify specific enhancer types.
Epigenetic Marks the epigenetic marks prediction task, we used the dataset from [35, 36] to predict nucleosome occupancy and modification states in the yeast genome. In 10 binary classifica- tion tasks, the model had to discriminate between DNA regions that were occupied by histones or not. The 10 tasks varied based the types of histones investigated, including unmodified histones HB and H4, as well as histones modified by either acetylation (H3K9ac, H3K14ac) or methylation (H3K4me1, H3K4me2, H3K4me3, H3K36me3, H3K79me3).
Splice Site For the splice site prediction task, DNA sequences from over 100 organisms were used to predict whether the sequences contain donor or acceptor splice sites [46]. Donor splice sites denote the beginning of intron and acceptor splice sites the end of intron. During RNA splicing, these sites are recognized by the spliceosome, a complex molecular machine that enables the removal of introns from the gene.
"bapss//epo.eppt.cc/iideexxp
19

---

Table A.5: Hyperparameter ranges used to fine-tune HyenaDNA for all Nucleotide transformer datasets. Exact hyperparameters per dataset can be found in our code repository.
HyenaDNA
Layers
Width
256
Parameters
1.6M
Optimizer
Adam W
Optimizer momentum
81. B2 = 0.9, 0.999
Training epoch
100 256-1024
Batch size
Learning rate
2c-4 to lc-3
LR scheduler
Cosine decay 0-0.2
Weight decay (model)
Weight decay (Hyena layers)
Embed dropout
0-0.2
Resid dropout
0-0.2
Reverse complement aug.
true/false 200-600
Sequence lengths
Preprocessing The Nucleotide Transformer study did not provide their exact train-test splits, cept for the enhancer dataset. Therefore, we generated our own train-test splits using a 90:10 ratio. For the promoter dataset, negative samples not available, and had to be generated following the procedure described by [34].
Model & Training For the architecture, we use a HyenaDNA model with 2 layers and width 256, and trained on sequences length 1024. We average across the tokens obtain a single classification token. For each task, we replaced the model head and fine-tuned the weights of the entire model (1.6M parameters). In contrast, the Nucleotide Transformer uses parameter-officient fine-tuning technique that introduces new weights and fine-tunes only the newly added weights, while keeping the initial model weights frozen, presumably due its large size of 500M to 2.5B parameters. The corresponding HyenaDNA hyperparameter ranges used for training each task are reported in Table A.5.
A.2.4 Ablations on the Nucleotide Transformer benchmarks
We perform additional ablations on the Nucleotide Transformer benchmarks to assess the impact of pretraining, well attention HyenaDNA, as shown in shown in Table A.6. We observed that pretraining has greater effect on the more challenging tasks (and as sequences become longer, shown in A)) On the more challenging tasks (histone marks, datasets starting with "H"), pre- training boosts NNA metrics by up to 21 MCC points on H3K4me3. For simpler tasks (with higher baseline scores) such as the splice sites and promoter tasks, the gain was lower to accu- racy points), as these were already near saturation in performance.
A.3 In-Context Learning Details
Background A key premise of foundation models is that they are able to learn new tasks with little to no new training data [4]. Recent advances in language modeling have demonstrated that language foundation models often adopt the behaviors necessary to perform new tasks in-context [6]. Here, information about the task that is to be performed, such as examples of respective inputs and targets, added to the input of the model. By conditioning their prediction the provided context, language foundation models are generally able to perform the task without any changes to their parameters.
key challenge for in-context learning with HyenaDNA its limited vocabulary, which com- posed of only few nucleotides, and does not provide any vocabulary for novel downstream tasks, such as class labels. To explore the potential for in-context learning in genomics, we use two vari- ants of in-context learning, both using brief tuning phase to introduce HyenaDNA to the concept of
20

---

Table A.6: Pretraining & Attention ablations on the Nucleotide Transformer (NT) bench- marks. The Matthews correlation coefficient (MCC) is used as the performance metric for the enhancer and epigenetic marks dataset, and the Fl-score is used for the promoter and splice site dataset.
MODEL
NT
GPT
HyenaDNA 1.6M
HyenaDNA
PARAMS
2.5B
1.6
1.6M
PRETRAIN
yes
yes
yes
no
Enhancer
58.0
59.3
62.6
58.6
Enhancer types
47.4
51.9
55.7
48.4
H3
81.4
75.8
81.7
79.9
13K4me1
55.9
38.7
57.1
43.4
H3K4me2
32.6
28.8
53,9
34.5
H3K4me3
42.1
28.3
61.2
40.2
H3K9ac
57.5
49.2
65.1
52.6
H3K 14ac
55,0
41.6
66.3
48.0
H3K36me3
63.2
47.8
65.3
53.4
H3K79me3
64.2
58.9
71.6
59.7
H4
82.2
77.7
79.6
79.1
H4ac
50.1
36.4
63.7
43.5
Promoter all
97.4
96.3
96.5
96.1
Promoter non non-TAT: TATA
97.7
96.6
96.6
96.5
Promoter TATA
96.4
96.6
96.7
96.1
Splice acceptor
99.0
97.6
96.6
96.6
Splice donor
98.4
98.1
97.3
96.5
Splice all
98.3
98.0
97.9
97.3
classification with its existing vocabulary. As test bed for this exploration, use datasets from the GenomicBenshmarks and a HyenaDNA pretrained on sequences 160k length sequences.
In the first experiment, we apply soft prompting approach [27] by adding sequence of tuneable
tokens to the input inself. In the second experiment, we explore few-shot learning approach [6] to in-context learning by adding & demonstrations (DNA sequence and its label) for each class of
dataset as input to the model. indicate classes, we make use of HyenaDNA's existing vocabulary by indicating classes with specific nucleotides. For binary classification, we indicate classes with the nucleotides "A" and "N", while additionally utilising nucleotide "G" for three-way classification. During model tuning, thereby optimise the same necl-oicle prediction loss as used during pretraining. See Table A.7 7 for an overview the optimisation settings.
Erhancers-Cohn
Fromatassromotars
Enhancers- Ensembi Erasmmmr--Enmmmii
Engulbll.Rggulbtrry
Ocr-Ensemll
Accurecy
0.75
shots
0.50
0.25
16
0.00
32
100
200
100 #TTuning samples
100
200
100
200
4Tanning samples
#Tuning samples
#Tunnin samples
TTuniig samples
Figure A.I: Few-shot prompting: HyenaDNA's performance on new tasks generally improves with the number of tuning samples, but is less clear when isolating the number of k-shot demonstrations. With less tuning samples, the number of k-shot demonstrations do improve performance. As tuning samples increase, the number of k-shot demonstrations start to improve performance.
Soft prompting details For each dataset, we prepend sequence of n (2 to 32k) learnable tokens T. E Ra each of dimension d, the input sequences X of the model: (Te, X, SEP), where "SEP" indicates the separation token. We optimise these tuneable tokens for a maximum of 20 training epochs on the dataset's training data while keeping all other model parameters fixed. We stop training early if the model's validation loss does not improve for two epochs. After this tuning phase, we evaluate the model's performance on the dataset's full validation data. For an overview of the results of this experiment, Fig. 4.2 of the main text.
21

---

Few-shot prompting details For each dataset, we prepend set of & (0 to 32, 0 indicates regular fine-tuning) examples of each class of a dataset (so-called "shots") to an input sequence:
X: 1 SEP, Y1, SEP; X2, SEP, V2 SEP, X, SEP),
where X; indicates an example sequence of class with label Y; (exemplified for two- wwy clas- sification task). We tune the model on (2 to 256) such k-shot samples before evaluating its per- formance on the dataset's full validation data. For an overview of the results of this experiment, see Fig. A.I.
Table A.7: Optimization settings for in-context learning.
SOFT PROMPTING
FEW-SHOT PROMPTING
Optimizer
Adam
AdamW
Optimizer momentum (31, (82)
0.9, 0.999
0.9, 0.999
Learning Rate
0.001
0.0001
Batch Size
16
2
Weight Decay (model)
0
0
Weight Decay (Hyena layers)
0
Resid dropout
0
0
Embed dropout
0.1
0.1
Reverse complement aug.
true
false
I.R-schedule
Plateau
A.4 Chromatin Profile Details
Background Variations in non-coding regions of the genome account for the majority of disease and other trait-associated single-nucleotide polymorphisms (SNPs). For example, whilst not directly altering the sequence of encoded protein, SNP in non-coding region can affect the expression of downstream genes by inducing a change in the epigenetic state [56]. Therefore predicting epige- netic markers from given sequence is important task in the context of quantifying the functional effects of non-coding variants. Previously DeepSEA [57], a deep convolutional sequence model, has been introduced to predict chromatin features directly from non-coding sequences.
Data The authors of DeepSEA [57] compiled a dataset of 919 chromatin features from [15] and [42] including 690 TF binding profiles for 160 different TFs, 125 DHS and 104 HM profiles. The original DeepSEA dataset consists of 1000 base pair (bp) sequences from the hg19 human reference genome [8] with corresponding 919-dimension multi-label target vectors. Each label corresponds to the presence/absence of a peak in a given chromatin feature within the central 200 bp region of the sequence. The 400 bp flanking regions of the sequence provide broader contextual information which is beneficial to the task. Training and testing sets are split by chromosome and are strictly non-overlapping. In total, there are 2.2 M training samples and 227,512 samples from chromosomes 8 and 9 are held-out for testing. We use the DeepSEA chromatin profile prediction task to evaluate HyenaDNA models with varying context window. We use LiftOver [26] to convert the original DeepSEA dataset to hg38 coordinates and expand flanking regions about the central 200 bp bin symmetrically up to 8000 bp. Approximately 0.5% of samples are filtered in cases where LiftOver fails or the resulting translated sequence has a different length.
Model We fine-tune several models consisting of a pretrained HyenaDNA encoder, a sequence- level pooling layer and a fully-connected decoder to perform multilabel sequence classification. We compare HyenaDNA against benchmarks set by DeepSEA, a convolutional sequence model, and BigBird [55]. a sparse attention based language model. The authors of BigBird fine-tune on the DeepSEA dataset with input sequences extended to 8000 bp (asymmetrically about the center- point by -5000 and +3000 bp). Notably BigBird utilizes a byte-pair encoding tokenization scheme whereas HyenaDNA uses a single-character tokenizer and DeepSEA uses one-hot encodings. For the shortest range model (1k), we average across all tokens to perform sequence-level pooling. Whereas in the longer context model (8k) we find that extracting the last token in the sequence as the input to the fully-connected decoder performs better. We also find that for the longer context
22

---

model using an encoder pretrained on sequences larger than those used in fine-tuning was beneficial. The hyperparameters of the models used these experiments are shown Table A.8. Note that we reduced the depth and of models with increasing context window due to limitations compute cost/time.
Results The performance of the fine-tuned NNe models summarised in Table 4.3. We find that the smallest sequence length model (1024 bp) outperforms both DeepSEA and BigBird on TF and DHS prediction. We find that the model pretrained 32k sequences with only 4 layers and fine-tuned on 8k sequences outperforms BigBird on the long range HM task but suffers from degraded performance on the short range tasks. However, we postulate that this performance loss may be recovered by increasing the depth of the model. We also remark that our models contain 5-30x fewer parameters compared DeepSEA and BigBird.
Table A.8: Chromatin profile model settings. HyenaDNA hyperparameter settings used in the chromatin profile prediction experiments (fine-tuning).
Hye
Sequence length
1024
8k
Context window
1024
32770
Width
256
256
Layers
Pooling method
Average 6.6
Last token 3.5
Parameters (M)
Optimizer
AdamW
Adam W
Optimizer momentum
81, = 0.9, 0.999
81, 82
0.9, 0.999
Weight decay (model)
0.1
0.1
Weight decay (Hyena layers)
Embed dropout
0.1
0.1
Learning rate
6e-4
6e-4
Batch size
64
64
Epochs
50
A.5 Biotype Embeddings Analysis Details
Background Sequence embeddings are useful in reducing dimensionality and capturing semantic relationships into fixed length vectors. We analyze pretrained embedding quality from NNA and show that learns biologically informed features. We utilize linear probing, freezing the weights on a pretrained model and attaching linear classification head to predict biotype sequences. We also use t-SNE visualize clusterings that emerge from the embeddings.
Data The Ensembl database [9] is a comprehensive resource for gene and transcript annotations such as biotypes. Ensembl biotypes are a classification system, based on a combination of experi- mental evidence and computational predictions, that summarises the high-level functional properties of genes and transcripts. For example, biotype classes may annotate whether gene is protein-coding encodes a long non-coding RNA; if gene is disrupted homologue of known protein coding gene (pseudogene) and by what mechanism it is produced; or the role of a small non-coding RNA such post- trasecranccipti modification of other RNAs in the cell nucleus. We use biotype anno- tations to qualitatively visualize the clustering of gene embeddings into functional groups. We con- struct multi-classfication task using the top 10 most frequent biotype annotations as multi-class target labels which we predict from the unsupervised embeddings to assess how well biological function is encoded in the embedding space.
Model & Training We use a frozen pretrained HyenaDNA model consisting of 8 layers and width 256 pretrained on sequences of length 160k. To extract sequence-level embeddings, we average along the sequence dimension in the final encoder layer. For comparison we also construct embed- dings using DNABERT (5-mer) and Nucleotide Transformer. We construct embeddings for genes in the Ensembl dataset up to length of 160k. For genes with sequence lengths exceeding the context
23

---

window of the encoder, we chunk the sequence and average the embeddings over the chunks. We utilize an XGBoost [7] classifier to perform the supervised multicclasification task on the embed- dings. The hyperparameters used are shown in Table A.9.
Table A.9: Hyperparameters. Overview of XGBoost hyperparameters used in biotype multi- classifier.
Estimators 1000
Max depth
Learning rate
0.1
Objective
softmax
Results As shown in 4.4, HyenaDNA achieves the highest Fl score the biotype classification task indicating that its embeddings contain features that are informative of biological function. No- tably, HyenaDNA achieves this using the much smaller embedding space dimension of 256, com- pared to DNABERT and Nucleotide Transformer, which produce embeddings of dimension 1029 and 1280, respectively.
A.6 Long-range Species Classification Details
Table A.10: Hyperparameter ranges for ultra-long range species classification task. Transformer uses FlashAttention [1]].
TRANSFORMER
Hye
Layers Sequence length
2
2
2
8
1024
32768
1024
32768
250000
450000
Width
128
128
128
256
Parameters (M)
0.5
4.5
0.4
0.4
6.6
6.6
Num heads
8
8
Learning rate
6e
6e
6
3e
6e
-4
Optimizer
Adam 31. 8g 0.9, 0.999
Optimizer momentum
LR scheduler
Cosine decay
Weight decay (model)
0.1
Weight decay (Hyena layers)
Embed dropout
Resid dropout
Batch size
128 256
Training epoch
200
Reverse complement aug.
False
Background Given a genetic sequence randomly sampled from a set of different species, success- ful identification of the source species requires model to learn a distinct mutational profile for each species. The more locations for discriminative mutations a model can consider, the more successful it should be at this task. We can arbitrarily tune this task's difficulty by including higher number of species increasing the evolutionary similarity the included species, and thus represents helpful setting for measuring long context reasoning abilities for DNA sequence models.
Data We select five species for this task: human (homo sapien), lemur (lemur catta), mouse (mus musculus), pig (sus scrofa), and hippo (hippopotamus amphibius). We hold out four chromosomes from each species (chromosome numbers 1, 3, 12, and 13) for evaluation, and use the rest of each species' chromosomes for training.
Model We compare Hyena DNA against baseline Transformer, which uses Flash Attention [11] the mixing layer instead of Hyena operator. We use 2 and layer models, depending on se- quence length. For HNenaDNA, we train on sequence lengths of 1 32k, 250k, 450k and M.. For

---

Top 1% accuracy for Transformer, we limit sequence lengths to 1k and 32k due to the quadratic increase in training time, making training infeasible on our hardware. See Table A.10 for model sizes and hyperparamters.
Training We use pretrained models from 4.1, trained on various lengths between 1k to 1M nu- cleotides, and fine-tune them using a linear decoder head. We either pool across all tokens (Ik and 32k models) or use the last token for classification (250k 1M models). We randomly sample a (species, chromosome, sequence start, sequence end) tuple at each training step, with uniform probability across all species and non-held-out chromosomes. If sequence's starting location on a chromosome such that the end of that sequence would exceed the length of the chromosome, then we pad the sequence with N's to its full intended length. For evaluation, we randomly sample a (species, chromosome, sequence start, sequence end) tuple from our held-out evaluation set of chromosomes, and record the overall Top-1 5-way accuracy of our model (i.c. fraction of sequences correctly classified).
At sequence length 450k, we use the sequence length warm-up scheduler described in 3.2 on HyenaDNA. This involves gradually increasing the length of sequences fed the model during fine-tuning from 1k to 450k. We observe better convergence and higher overall peak accuracy with this strategy, as shown in 3.2.
Table A.11: Pretraining vs scratch on 5-way species classification. HyenaDNA by sequence length.
HyenaDNA
LENGTH
SCRATCH
PRETRAINED
53.9
61.1
32k
70.7
93.4
250k
65.7
97.9
450k
71.4
99.4
Pretraining ablation For species classification, pretraining becomes more important for longer sequences. This is in-line with our observation that for harder tasks (including longer sequences), pretraining becomes more important. At sequence length 250k and 450k, the scratch vs. pretraining gap is 30+ accuracy points.
25


Reviews:
[{"review_id": "23SppH4hIw", "reviewer": "Reviewer_YJ4C", "summary": "This manuscript applies Hyena, a neural operator based on implicitly parametrized long convolutions and data-controlled gating, to the domain of DNA modelling. \nThe subquadratic complexity of Hyena enables scaling to context lengths of up to 450,000 at single nucleotide resolution. \nThis represents a significant improvement over previous methods based on dense attention, increasing pre-training sequence length by more than two orders of magnitude.\nPre-training is done using a sequence length warm-up scheme, such that sequences grow longer as training progresses.\nThe resulting model is benchmarked on a variety of downstream tasks using different adaptation methods.", "questions": "- Why is HyenaDNA pre-trained autoregressively rather than non-autoregressively, as was done in Nucleotide Transformers? Both methods would be feasible since downstream adaptation of HyenaDNA always includes a fine-tuning phase.\n- Considering the limited vocabulary of just 4 nucleotides, it's somewhat unexpected that the reported perplexity score for a high-performing long-context model like HyenaDNA would be this high. Could you provide some insight on this? To underscore the significance of pre-training further, it would be valuable if you could supplement the evaluation of pre-trained embeddings using probing classifiers with an additional experiment where HyenaDNA is trained directly from scratch on the downstream datasets.\n- The manuscript mentions that both a long context and single nucleotide resolution are crucial. However, K-mer tokenization could help reduce the temporal dimension, and previous work has shown that a higher k leads to improved results (Ji et al., 2021). Therefore, it would be beneficial to include an experiment that compares HyenaDNA models pre-trained with different tokenization schemes.\n\n### References\n- Ji, Y., Zhou, Z., Liu, H., & Davuluri, R. V. (2021). DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformatics, 37(15), 2112-2120.\n\n### Minor comments\n\n- Possible Typo in Table 4.1 row 2 (Coding vs. Intergenomic): the improvement over baseline performance (+3.5) does not match with the reported base performance. Both Base and HyenaDNA are reported to have the same accuracy. \n- Typo on line 543 : sequence-level instead of sequence-leel\n- Typo on line 544: that instead of taht", "limitations": "The authors address the limitations of their method and experiments in the manuscript.\nPossible malicious uses of HyenaDNA are not addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. (**Originality**) HyenaDNA is the first work to apply Hyena, a novel breed of implicitly parametrized long convolutions, to DNA modeling.\n2. (**Originality**) The authors experiment with two adaptation strategies commonly used in NLP to apply the pre-trained model to novel tasks.\n3. (**Quality**) The method is evaluated on a large number of genomic datasets introduced as part of related work. This firmly situates HyenaDNA within the existing literature.\n4. (**Significance**) The proposed model outperforms state-of-the-art attention-based methods on the majority of presented downstream tasks with considerably fewer parameters and scales to extremely large context sizes of 450,000 tokens at single nucleotide resolution.\n5. (**Clarity**) Overall, the manuscript is written clearly, and the arguments are easy to follow.\n6. (**Clarity**) Architecture and hyperparameter details are clearly presented.\n7. (**Clarity**) Including background task descriptions for the Nucleotide Transformer, the Biotype Classification, and the Chromatin Profile Prediction benchmarks are helpful for understanding the evaluation protocol.", "weaknesses": "1. (**Originality**) Beyond the application of Hyena to the DNA domain, the manuscript offers few new technical insights that could be of interest to the machine learning community. Sequence length warm-up is an established technique to stabilize training instabilities, and soft prompting, as well as few-shot fine-tuning, are commonly used adaptation strategies in the natural language processing community.\n2. (**Quality**) Results are presented without any measure of deviation over multiple repetitions. This is true even for smaller scale fine-tuning experiments.\n3. (**Quality**) For the GenomicsBenchmark, the authors should report more details about the CNN baseline (parameter count, number of layers, etc.), as well as training details and hyperparameters for AttnDNA. If the same hyperparameters are used for both HyenaDNA and AttnDNA, please state this explicitly.\n4. (**Clarity**) Few-shot adaptation includes a tuning phase, making this adaptation method more similar to few-shot fine-tuning than in-context learning. Relevant literature should be cited.\n5. (**Clarity**) The results reported in Table 4.1 for the Nucleotide Transformer do not seem to match with those reported in the original paper in Supplementary Table 6. Please explain this deviation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review of our work. We’re glad they appreciate the strength of the results, the originality and significance of the application, and the clarity of the manuscript. Below, we address the reviewer's concerns about the contributions, design, performance, and tokenization.\n\n### ML community contributions\n\nThis work is representative of many problems in ML, namely on how to efficiently capture statistics that include both local and long-scale dependencies, local spatial structure of objects and large-scale correlations of those objects in images, local motion and long action sequences in video, and structure within phrases and long-range correlation in text. \n\nRegarding technical relevance, we would like to highlight a few observations and insights we think are relevant to the ML community, and that could be used as a recipe for similar tasks.\n\n**Ultralong sequence training:** Training on 1M context length data gives rise to unique instability challenges, which depart from the typical scaling of model size or datasets. Sequence length warmup has been explored in the NLP community in the past, notably by only up to 2k long sequences [Li et al., 2022] and for small Transformers [Press et al., 2020]. Many questions remain about the schedule of increases (in particular for long sequences), learning rate adjustment, and how it affects token efficiency at ultralong sequences. We share one set of strategies and findings in genomics that were performant.\n\n**Single character training:** In nLP, the use of single characters consistently performs worse than aggregated tokens [Yu et al., 2023, Tay et a., 2021, Kalchbrenner et al., 2016]. Our work contributes a successful recipe for single character training that surpasses aggregated-character methods.\n\n**Model vs vocabulary size:** Our work raises novel questions on the tradeoff between model size vs vocabulary size. Given that our design uses a small vocabulary and model size (orders of magnitude smaller) than previous SotA genomic FMs, it begs the question of whether a similar design (single character tokens with Hyena) can be applied to natural language, and if these findings can be transferred.\n\n### Clarity concerns\n\n- **GenomicBenchmarks CNN baseline details:**  We will include more details of the CNN baseline model in the appendix section A.2.1. As described by the original authors, the CNN uses an embedding layer, 3 conv layers with number of filters: 16, 8, and 4. It uses batch norm and max pooling after each conv layer, followed by 2 dense layers. It is trained for 10 epochs with batch size 64. The mode sizes range from 120k to 520k, depending on sequence length chosen.\n\n- **Hyperparameters:** We indeed use different hyperparameters between HyenaDNA and AttnDNA, and will clarify this in supplement section A.2.1.\n\n- **Few-shot adaptation vs in-context learning:**  We will make the distinction more clear with citations.\n\n- **Nucleotide Transformer Table 4.1 results:**  Thank you for pointing out the discrepancy. The Nucleotide Transformer paper has 2 sets of results: one in Figure 2 of the main paper, and one in Supplementary Table 6. We originally used Figure 2 in the main paper, but after speaking with the authors, the Supplementary Table 6 indeed does fit our procedure more closely by using the best test set results. We will update values in our comparison in the manuscript, and is already reflected in the common response in Table R2. Notably, this does not change the ranking of SotA performance per dataset for HyenaDNA.\n\n### Questions\n\n**1. Autoregressive vs bidirectional:** HyenaDNA was pretrained autoregressively (causal) because the initial results showed better downstream performance over a bidirectional Hyena (see ablation in our common response) that we experimented with. The autoregressive training also offered additional appealing properties:\n\n- The Hyena model is naturally flexible to variable length sizes over BERT-style models that need to be explicitly fed variable lengths during training, since the causal model “sees” different lengths incrementally. \n\n- Additionally, we wanted to explore the use of in-context (ICL) learning in genomics, as we believe ICL has driven a lot of innovation in natural language. An autoregressive model is more amenable to current ICL methods that leverage next token prediction for class prediction, for example.\n\n**2. DNA perplexity**: We too were surprised by the relatively high perplexity for the given vocabulary size. We think this is a challenge inherent to the domain of genomics, and provide a few relevant data points:\n\n- Biologically, genomes carry “junk” DNA that is difficult to decipher and or not informative. One rationale for why this occurs can be viewed through an evolutionary lens, in that, as mutations occur over time, a mutation is less deleterious if it occurs in “junk” DNA, as opposed to informative DNA [Zhang el al., 2012, Ohno et al.,1972]. Knowing whether a sequence is “junk” (informative) or not is still an open question, making genomics overall particularly challenging.\n\n- [Rajarajeswari et al] showed DNA compression algorithms that sought to reduce bits per character (BPC, convertible to perplexity) in a similar range. They achieved an equivalent perplexity of 1.58 vs our 1.54, with a lower score as better.\n\n- [Benegas et al., 2022] trained a genomic plant Transformer (not FM) using single nucleotides, and observed similar perplexity. \n\n**3. Pretraining vs. scratch:**  Please see the common response for ablations on pretraining vs. finetuning.\n**4. K-mer tokenization:** In the ablation presented in the common response in Table R1, we observed consistent (and significant) degradation in performance on the GenomicBenchmarks when using 6-mer tokenization on HyenaDNA and training from scratch, from 2 to 10 accuracy points.\n\n**Limitations**\nWe will address the potential for malicious uses of HyenaDNA in the camera-ready. "}, {"review_id": "akYlJs1OCX", "reviewer": "Reviewer_LTPa", "summary": "The authors train the Hyena operator model on the human genome and adapt it to downstream tasks in computational biology.\n", "questions": "Figure 3.2 - Do longer sequences simply take longer to do a forward and backward pass? You show wall time. Are the number of steps relatively closer?\n\nHow much is the result of your model’s result is its architecture versus using tunable prompting? What would happen if you used tunable prompting for the Nucleotide Transformer?\n\nHow are the F1 and MCC scores for Table 4.2 generated? Which values are you comparing?\n\nHow was the finetuning in section 4.4.1 done? Is it not soft-prompting?\n", "limitations": "Generally, I find the term “Foundation Model” to be ambiguous and unnecessarily buzzwordy. HyenaDNA is a large, pretrained model. Even though the authors have pretrained the model, it has to be finetuned and quite heavily engineered differently for each subtask. But, I guess this is what the field is wanting to say.\n\nWhile I do like that the important aspects of the paper are emphasized, some of the color, boldness, different font, and italicization can be toned down a bit. For example: “Therefore, having both long-range context and single nucleotide resolution simultaneously” doesn’t need to be blue and bolded. Also, what is the author’s intent in using blue text versus bold text?\n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Lots of clever things about this work that I really like:\n\nVery good use case of Hyena model with very long-range dependencies\nCurriculum learning is very clever and makes sense. Figure 3.2 is great to show this\nI think Table 4.4 with the use of XGBoost is very simple but effective in getting a statistic about the utility of an embedding. With XGBoost, you are going to get a pretty reasonable and comparable every time you use it.\n\nThe model capacity shown by perplexity and sequence length is good.\n\nI really really like that it is able to work at single base pair resolution.\n\nI’m impressed and happy that section 4.4.3 uses held-out chromosomes for training and testing (rather than random sampling). Since you are working with multiple species, are there still some aspect of data leakage in train/test sets because of synteny?\n\nI think this is impressive being only trained on the human genome. I’m excited to see this model trained on all genomes!\n", "weaknesses": "I think the GenomicsBenchmark claims about SotA could be improved. Definitions of SotA should be based on architecture, not the datasets. Since the dataset is newer than previously benchmarked models, I would recommend the authors evaluate the GenomicsBenchmark across other SotA models or architectures (like DNABERT and Nucleotide Transformer, which are mentioned in this paper). Alternatively the SotA claim is weak at best, and misleading at worst.\n\nI think section 4.4.3 could have a very simple baseline of a kmer count model. Can a simple kmer based model do just as well, or poorly? I’m sure it is very fast.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review of our work and are glad they appreciate many of the contributions including the long-range capabilities, single nucleotide resolution, and the curriculum learning introduced. Below, we address the concerns the reviewer made about the GenomicBenchmarks baselines, and include additional ablations requested.\n\n### GenomicBenchmark Baselines\n\nIn the common response above, we added ablation results to address concerns the reviewer (and others made), including:\n\n- Finetuning a stronger baseline using DNABERT, which achieved SotA on 1 of 8 datasets (with HyenaDNA outperforming on the other 7 datasets).\n- HyenaDNA trained with a K-mer (K=6) tokenizer to compare the effect of K-mers vs single character tokenizers. See Table R1 in the common response. The K-mer tokenizer consistently degrades performance over the single nucleotide tokenizer (2-10 accuracy points).\n\n**Planned:** We plan to finish finetuning the Nucleotide Transformer model on the GenomicBenchmarks in the coming weeks. As these models are very large (500M to 2.5B parameters), we did not yet have the compute resources to complete training. We did finetune 1 of the 8 datasets for the Mouse Enhancer task. The 500M Nucleotide Transformer underperformed HyenaDNA by about 10 accuracy points (85.1 vs 75.2). If accepted, we can include the remaining 7 datasets in the camera-ready.\n\n### Questions\n\n**Longer sequence time:** Indeed, longer sequences do require more time for forward and backward passes.  See figure 4.1 in the manuscript for the wall time (forward and backward pass) by sequence length.  We compare times for both HyenaDNA and its attention counterpart. \n\n**Data leakage / synteny:** That’s a really interesting point. It’s possible “data” leakage from this evolutionary process can occur, and it would be challenging to account for explicitly. We take comfort in that in the aim of the species classification task, we’re interested in the relative performance between long sequence lengths, over the absolute performance (in this example).\n\n**Tuneable Prompting vs. Finetuning:** To be clear, all of our short-range tasks in section 4.2, as well as 4.4, use **standard finetuning**.  This includes the chromatin profile task in 4.4.1 (as inquired by the reviewer). This makes the comparison between previous architectures more clear.  The tuneable prompting approach is a standalone set of experiments in section 4.3 only, and considered “self-contained”. We’ll be sure to make this distinction more explicit in the manuscript, thank you for pointing this out.\n\n**Tuneable prompting on Nucleotide Transformer:**  For comparing baseline models with tuneable prompting, this unfortunately would require a significant redesign of their models. The most important constraint is the context length, which HyenaDNA opens up, but previous methods are limited to 512 or 1000 tokens. HyenaDNA uses up to 32k tuneable tokens, in addition to adding multiple samples within the context, which would simply not be possible on DNABERT and Nucleotide Transformer. That may be a really interesting future research direction, and we hope we’ve inspired others to pursue this.\n\n\n\n\n**Foundation models terminology:** We acknowledge that this can be a contentious terminology to some. We took the stance that most effectively conveys what we hope and believe our model can do, which is to learn useful and general representations that can be applied to downstream tasks.\n\n**Styling / bold font:** Thank you for bringing this up, we initially wanted to bold and highlight text we thought were key takeaways per section. We’ll be sure to readdress the format as to not be so distracting.\n\n**F1 and MCC metrics:** The 18 datasets in the Nucleotide Transformer were sequence-level classification tasks (binary and 3-way). Using the model prediction for a DNA sequence, we’re able to determine a true/false positive or true/false negative with the label. F1 and MCC metrics are standard statistical methods used by the original authors of the Nucleotide Transformer.\n\nFormally, these are calculated as:\n\n$$F1 = 2 \\cdot \\frac{precision \\cdot recall}{precision + recall}$$\n\n$$ MCC = \\frac{TP \\cdot TN - FP \\cdot FN } {\\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}} $$\n\nImplementation-wise, we used predefined metrics from the scikit-learn package to calculate our F1 and MCC metrics. For the F1 score, we used the f1_score function [https://scikit-learn.org/stable/modules/model_evaluation.html#precision-recall-f-measure-metrics] with the average flag set to macro. And for the MCC values, we used the matthews_corrcoef function [https://scikit-learn.org/stable/modules/model_evaluation.html#matthews-corrcoef]. "}, {"review_id": "FLvAqRKfEb", "reviewer": "Reviewer_nDFb", "summary": "This paper introduces HyenaDNA, a genome foundation model based on the Hyena architecture that replaces attention layers with implicit convolutions.  Though being 2500 times smaller, it achieves better performance than the state-of-the-art model.", "questions": "- How does AttnDNA perform on the datasets used by Nucleotide Transformers?\n- Can you implement Nucleotide Transformer and DNABERT on the GenomicBank datasets for comparison?", "limitations": "In sum, the work is technically sound and empirically strong. However, based on the above thoughts, I tend to reject it for now. If the concerns are clearly solved, I would be happy to recommend a strong acceptance.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written.\n- The proposed method achieves very strong performance with very few parameters and is able to process ultra-long dna sequences.\n- This paper investigates in-context learning and tunable prompting to the area of dna language model and show great performance improvements.", "weaknesses": "- The comparison over baselines is unfair. \n    - Since Hyena is fundamentally different from the Transformer-based architecture, simply comparing the number of parameters does not accurate reflects the model efficiency. I think it would be more accurate the compare the wall-clock time and GPU time used in both pre-training and downstream evaluation. Actually, according to the figure A.2, the model is slower than its attention-based variants when inputs are shorted than $10^4$, which is very common in genome analysis tasks. Therefore, it is very interested to see how efficient the model is on each downstream task compared to DNABERT and Nucleotide Transformers.\n    - All the previous models use standard or parameter-efficient fine-tuning for downstream evaluation, I think it is important to also fine-tune HyenaDNA on each downstream tasks to show how good it is as a *backbone* model and for fair comparison with other models. Also, it is interesting to see how the baseline models performs with the  tunable prompting.\n\n- This paper introduces a very strong model with a series of fancy techniques, however, as an academical publication, it fails to provide enough insights and explanations to the community about the superiority of its performance.\n    - According to the Hyena paper, the Hyena architecture achieves a similar level of performance (slightly better) as the transformer-based model (e.g., GPT) with the same number of parameters in general language modeling. Therefore, the fact that it dominants transformer-based models in the dna language modeling indicates the existence of **fundamental errors** of all the existing works. If this is true, what are the errors? I can think of a few possibilities:\n        - Masked language modeling vs casual language modeling\n        - k-mer tokenization vs character-level tokenization\n        - attention mechanism fails to capture the semantics of DNA. (however, the AttnDNA performs very well)\n    - To be honest, I don't think any of the above can lead to such significant improvements. Thus, throughout ablation study and experiments under the same setting is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review of our work. We’re glad they appreciate the writing clarity, strong empirical results and the exploration of in-context learning and tunable prompting in genomics. Below we address the weaknesses and questions the reviewer described. We’re happy to follow up with any further clarifications the reviewer may have.\n\n### Pretrain and Finetune Runtime Comparison\n\nThank you for suggesting to compare runtime during pretraining and finetuning. We’ve provided these results for HyenaDNA and baseline models in Table R4 and R5 in the common response, and will add a section on efficiency in the appendix of the manuscript. We share part of the efficiency discussion we plan to include below as well.\n\n- Indeed, FlashAttention is faster than HyenaDNA on shorter sequences. Fortunately, HyenaDNA required far less pretraining time and data to more than make up for this, as noted by the high GPU-hour difference in Table R4.\n\n- We agree that many existing genomic tasks are shorter range, though we believe this to be due to the *constraints* of current sequence models - and that demand for long-range models is prevalent (eg gene expression, histone modification, splice prediction, chromatin accessibility and structure). As computational biologists see that longer-range models are possible, the emphasis on genomic benchmarks will likely shift to long-range tasks, shifting the advantage quickly to subquadratic scaling models.\n\n### Insights into Hyena’s performance\n\nWe’re glad the reviewer is interested in understanding where the HyenaDNA gains come from. We performed a series of ablations (details and results in Tables R1 to R3 of the common response) to help understand the effects of some of the factors the reviewer suggested, including:  k-mer tokenization vs single character, attention vs Hyena, and causal vs. bidirectional. \n\nThe results of the ablation suggest each of these design choices contributed to gains over previous genomic FMs, and will be included in the manuscript in section 4 and the appendix. It may suggest fundamental errors (as the reviewer noted), or simply design choices that have yet to be explored in genomics and biology. We’re excited to further investigate other existing assumptions that may exist at the intersection of biology and machine learning.\n\n### Finetune comparison (as a backbone model)\n\nWe wanted to clarify that in all downstream tasks (except for section 4.3 on in-context learning), we perform standard finetuning with HyenaDNA. This enables a fairer comparison between previous genomic FMs. We hope this alleviates the reviewer’s concern about whether HyenaDNA is used as a backbone in a similar fashion to baselines.\n\n**Tuneable prompting on Baseline models:**  For comparing baselines models with tuneable prompting, this unfortunately would require a significant redesign of their models. The most important constraint is the context length, which HyenaDNA opens up, but previous methods are limited to 512 or 1000 tokens. HyenaDNA uses up to 32k tuneable tokens, in addition to adding multiple samples within the context, which would simply not be possible on DNABERT and Nucleotide Transformer. That may be a really interesting future research direction, and we hope we’ve inspired others to pursue this.\n\n### Questions\n\n**1. AttnDNA on the Nucleotide Transformer (NT) datasets:** As suggested by the reviewer, we added results for AttnDNA on the NT datasets in the common response in Table R2. In all tasks, AttnDNA underperforms HyenaDNA (see common response for more analysis).\n\nTo summarize the main takeaways, AttnDNA was able to perform well on simpler tasks ( promoter prediction), but struggled significantly on the more challenging histone mark tasks (the ones starting with “H”) as compared to HyenaDNA, and with very large gaps. This suggests that the Hyena operator itself also contributes to the boost in performance over previous attention-based genomic FMs.\n\n**2. DNABERT finetuning:** As suggested, we finetuned DNABERT as a stronger baseline on the GenomicBenchmarks, shown in the common response in Table R1. Indeed DNABERT does reach SotA on 1 of 8 datasets, while HyenaDNA retains top performance on 7 of 8 datasets.\n\n**Planned:** We plan to finish finetuning the Nucleotide Transformer model on the GenomicBenchmarks in the coming weeks. As these models are very large (500M to 2.5B parameters), we did not yet have the compute resources to complete training. We did finetune 1 of the 8 datasets for the Mouse Enhancer task. The 500M Nucleotide Transformer underperformed HyenaDNA by about 10 accuracy points (85.1 vs 75.2). If accepted, we can include the remaining 7 datasets in the camera-ready.\n\n\nThank you again for the suggestions to improve our work, they’ve helped us better understand and communicate the differences between our methods and previous genomic FMs.  \n"}, {"review_id": "CJqbWKqA6G", "reviewer": "Reviewer_DWDg", "summary": "This paper presents HyenaDNA, an advanced genomic foundation model leveraging the Hyena language model's capabilities, which are based on implicit convolutions. The authors highlight the limitations of prior Transformer-based genomic models, which have been constrained by token lengths and therefore impeded accurate modeling of long-range genetic interactions. These models also relied on tokenizers or fixed k-mers, which resulted in loss of single nucleotide resolution and crucial genetic variations. HyenaDNA addresses these issues. The model's unique abilities, including its sub-quadratic scaling in sequence length, usage of single nucleotide tokens, and full global context at each layer, mark a significant advancement in genomics.", "questions": "I would raise my rating if the authors resolve my concerns. \n\n- On line 132 , it would be beneficial to showcase $ T_{i, j} $.\n- Could you elaborate a bit more on Figure 3.1, specifically, what the arrows represent?\n- Could you provide detailed information about the baseline CNN model used in the GenomicsBenchmark?\n- Are there any results from training the model from scratch? The connection between the pre-training goal and the downstream task doesn't seem very apparent.\n- How well did the CNN model perform in predicting on the Nucleotide Transformer benchmark?\n- On lines 259 - 260, how would training from scratch for each task fare? If the advantages of pre-training are minimal, using $3200 x$ less pretraining data doesn't present a significant benefit.\n- Could you detail the fine-tuning process in section 4.2?\n- For Figure 4.1, it would be helpful to incorporate the data from table 4.1 to display the \"upper bound\" performance.\n- It would be advantageous to always include a training-from-scratch CNN or dilated CNN on all tasks.\n- The species classification result is quite impressive! Would it be possible to apply any Explainable $\\mathrm{Al}$ (XAI) methods to discern the principles that the model has learned?\n- It would be beneficial to update the supplement to include more details about the Long Convolution. I had to refer to a previous paper to understand this concept.", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this is an excellent paper! I appreciate the application of the Hyena model to DNA sequences. The utilization of advanced sequence modeling to tackle significant challenges in biology is commendable.\n\nThe concept of soft prompting for long-context models is exceptionally innovative and interesting!", "weaknesses": "- To be frank, I believe the evaluation of short sequences may not be crucial for this paper, considering its primary focus on long-range modeling. I would suggest including a comparison with the Enformer model for a more comprehensive perspective. More benchmarks on the long-context task  are necessary. \n\n- Moreover, additional technical details should be incorporated either in the main text or the supplementary materials to facilitate a thorough understanding of the methodology used.\n\nSee more detail in the questions:", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for the thoughtful review. We’re glad the reviewer is excited about the contribution of Hyena to DNA sequences, as well as the novelty of soft prompting for long-context. Below we address concerns the reviewer raised, and clarifications to technical details that were asked. We’re happy to answer any further questions the reviewer may have.\n\n### Short vs long range evaluations\nThe suggestion to apply HyenaDNA to the Enformer [Avsec et al., 2021] task is a good one. We dwelled on applying HyenaDNA to Enformer for quite some time. After much debate, we made the strategic choice to focus on a general foundation model (as opposed to a supervised Enformer task), with the compute budget we had. Our hope was that we could reach more computational biologists who needed more expressive long-range backbone models that could be easily adapted for their specific downstream tasks. We absolutely believe Enformer would be an incredibly exciting application of HyenaDNA and leave that to future research.\n\nThat being said, we do showcase a number of long-range downstream tasks including chromatin profile (8k long), a novel ultralong-range species classification tasks up to 1M tokens, in-context learning tasks that uses over 32k tokens, and biotype classification on up to 160k tokens.\n\nWe primarily included a high number of short-range benchmarks because of their accessibility and clear ability to compare against existing models. We hope that these make the case that HyenaDNA performs well on both short and long range tasks.\n\n### Additional methodology details\n\nThank you for bringing these up. We’ll be sure to add further architecture and training details to section 4 and appendix of the manuscript. We will also release all the code (with colab notebook examples), and model weights.\n\n### Questions\n\n**Model and training clarifications**\n\n- Thanks for pointing these out. We will add clarifications regarding $T_{ij}$ on line 132, more details on the long convolutions in the supplement section, and add an upper bound to the tuneable tokens for table 4.1. $T_h$ is the Toeplitz matrix representation of a convolution with filter $h$. It is a mathematically equivalent representation ($T_h u$ or $h * u$), with definition $T_ij = h_(i - j)$ i.e. T is the matrix with the filter on each row, shifted by the column index.\n\n- We will amend section 3.1 for clarity (also see figure A.1 in the supplement for block diagram of HyenaDNA). To address your specific question, in Figure 3.1, the arrows represent how an input flows through a single Hyena operator (inside). Starting with the input $x$, separate projections are produced using 3 different $W$ matrices (dense layers). This is followed by 3 short convolutional layers, $T_{x2}$, $T_{x1}$, and $T_{v}$, creating $x_{2}$, $x_{1}$, and $v$.  Then $v$ is elem-wise gated by $D_{x1}$ (a diagonal matrix/layer), followed by a long convolution by $T_{h}$, and finally another gate by $x2$.\n\n- We will add further details in the appendix on the GenomicBenchmarks baseline CNN. As described by the original authors, the CNN uses an embedding layer, 3 conv layers with number of filters: 16, 8, and 4. It uses batch norm and max pooling after each conv layer, followed by 2 dense layers. It is trained for 10 epochs with batch size 64. The model sizes range from 120k to 520k, depending on sequence length chosen. We did not apply this basic CNN on the Nucleotide Transformer datasets as it appeared to be a simple demonstration model by the dataset authors. We’ll be glad to add these by the camera-ready if the reviewer believes this would bring clarity into how HyenaDNA performs over previous methods.\n\n- The finetuning procedure in section 4.2 has some additional details in the supplement information section (A.2), but we will certainly provide further details in the camera-ready. Specifically, our finetuning procedure attaches a single linear layer to a pretrained HyenaDNA model (2 layer, d_model=128 and 256), pretrained on sequence length 1k. The output embeddings learned are averaged over all the tokens, and then used to classify a class prediction. Each dataset is finetuned separately and sweeped through a number of hyperparameters (via a grid search). We provide a range of the hyperparameters in Supplemental Tables A.2 GenomicBenchmarks and Table A.3 for the Nucleotide Transformer datasets.\n\n**Scratch/pretraining**\n\n- Thank you for the suggestion - we provided additional experiments training from scratch in the common response in Tables R1 to R3, along with our observations. We will add these to section 4 of the manuscript. The key takeaways were that on simpler tasks, pretrained boosted mildly to moderately for HyenaDNA. \n- As difficulty increased (both the task itself and the sequence length), we observed greater performance gains from pretraining. See the common response and Table R1 and Table R3 for more in depth analysis of pretraining effects on downstream performance.\n\n**Species classification**\nWe’re glad the reviewer found the species results impressive. We certainly think XAI methods can be applied to this task. Though our initial experiment focused primarily on showcasing the relative performance between longer sequence lengths, we think XAI methods applied to HyenaDNA would be an exciting future research direction, especially given the successful history of using convolutions in interpretability in genomics.\n\n\nThanks so much for the thoughtful and very detailed feedback and questions, it’s helped us better communicate our work!\n\n### Citations\n\nAvsec, Žiga, et al. \"Effective gene expression prediction from sequence by integrating long-range interactions.\" Nature methods 18.10 (2021): 1196-1203.\n"}, {"review_id": "61EKnxPquE", "reviewer": "Reviewer_JfVK", "summary": "In this work, the authors propose a new genomic foundational model that can be pretrained on the human reference genome. This model, called HyenaDNA, is built upon the previous model Hyena. The advantage of this model is that it can train on ultra-long sequences (up to 450k) at single nucleotide resolution with significantly fewer parameters. The authors introduce a warm-up technique to stabilize the training procedure, and they also employ a soft prompting method for downstream tasks. The authors show the efficacy of their framework through various experimental settings.", "questions": "1. Line 206, it is not so clear what the embedding step refers to. Also, perhaps to double check the correctness of $x\\in\\mathbb{R}^{L\\times(T+N)}$\n2. It seems that for different experiments, the authors train different HyenaDNA (different hyper-parameters, or using input sequences with different lengths as the pretraining data). Is there any specific reason that the authors use different models instead of e.g. using the biggest model that can handle up to 450k context, for all the experiments? \n3. Table A.3, for different tasks the authors use different hyper-parameters, how these hyper-parameters be chosen? \n4. The authors indicate in the main text their framework is more computationally efficient than Nucleotide Transformer since it has a significantly smaller number of parameters. However, in the appendix the authors mention that for each new task the entire model needs to be fine-tuned. How efficient this fine-tuning is compared to the fine-tuning procedure used in Nucleotide Transformer? How many computer resources are required for fine-tuning HyenaDNA? \n5. The authors compare HyenaDNA with Nucleotide Transformer in 17 datasets. However, in the paper of Nucleotide Transformer there are 18 datasets in total. Why one dataset missing?", "limitations": "The authors have pointed out some limitations of their work in the conclusion section. I don’t think there is any significant negative societal impact of this work.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This work tries to solve an important problem: developing new foundational models for DNA sequences. \n2. Previous genomic foundational models are usually only able to be trained on relatively short sequences (<10kb). It is hard for them to deal with ultra-long sequences, which is necessary to capture long-range dependencies. This work provides a strategy to train the model on these long sequences. \n3. The experimental results shown in the manuscript seem promising. \n4. Overall, the manuscript is well written.", "weaknesses": "1. It is a bit difficult to understand 3.1 especially for people who are not familiar with the previous Hyena manuscript. $x_1$, $x_2$, $v$, $W_{x_{1}}$, $W_{x_{2}}$, $W_{v}$ are not defined in the main text. Line 159, not sure “input time” refers to. Line 162, not sure “projections” refer to. \n\nSome minor points:\n1. Section 2.1, no definition of D, is it equal to 4 representing {A,T,C,G}?\n2. Line 320, A.5 should be A.4\n3. Line 350, A.4 should be A.6\n4. Appendix line 613, Fig. 4.2 should be Fig 4.1. \n5. Line 206 equation, equation number missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review! We’re glad you appreciate the importance of foundation models for DNA sequences, and the clarity of writing of the manuscript. Below we address concerns and clarifications the reviewer raised. We’re happy to answer any further questions the reviewer may have.\n\n### Hyena method clarification\n\nThank you for pointing this out. We’ll be sure to modify the description for readers unfamiliar with the original Hyena work as well. Specifically:\n\nStarting with the input $x$ (a length $L$ sequence of embeddings of size $D$), we apply three different parametric linear operators to obtain “projections” $x_1, x_2, v$, each of the same dimensions of the input. For example, $x_1 = T_{x1} x W_{x1}$ where $T_{x1}$ is the L x L Toeplitz matrix corresponding to a short convolution, and $W_{x1}$ a D x D weight matrix.\nAfter computing the projections, we start applying the Hyena operator recurrently, starting with $v$. First, we compute $ z = D_{x1} v$ where $D_{x1} = diag(x_1)$ (gating), following up with the long convolution $T_h$ and another gate $D_{x2}$. Higher-order Hyena operators (with more projections) continue until all projections have been used in the recurrence, following the same pattern of diagonal and Toeplitz.\nThe phrase “input time” (on line 159) is meant to describe using time (or position in the sequence) as an input to the neural network parametrizing the convolution filter $h$. The inputs are the positions of the filter (kernel), and the outputs are the convolution weights for that position. Note:  the use of “time” is a remnant of signal processing literature (which Hyena draws from), but can also refer to space or a position.\nThe term “projections” refers to a linear projection (applying a linear layer) to an input.\n\n### Minor Points\n\nThank you for catching the fine grained details that were missing here.\n\n1. In section 2.1, D refers to the model embedding dimension. We’ll define this explicitly in the paper.\n2. through 5., we will edit these typos and omissions in the final paper.\n\n### Questions \n\n1. The embedding step uses a standard embedding function (e.g. nn.Embedding in Pytorch). In general, this maps an index value (like 0 through 3, representing indexes of DNA nucleotides) to a learned embedding space of dimension D (typically the model dimension).\n\n1a. Indeed, we do finetune different pretrained models of different sizes depending on the downstream task. There are 2 reasons for this:\n\n**a. Efficiency:**  Using the “right” size model that is sufficient for the downstream task is far more compute efficient, since ultralong-range sequence models require more training time, data and parameters to perform well.\n\n**b. Reduce overfitting:**  the larger models tend to overfit quicker (a common occurrence across machine learning). This is especially true if we’re testing on a set of short-range tasks (<1k long). Using a large model (eg for 450k sequences) will overfit quicker on these short range sequences if there is not a sufficient number of samples. Many of the short range genomic datasets are indeed small, typically 10s of thousands of samples, and as few as 1200 samples.\n\n2. In Table A.3. For the Nucleotide Transformer datasets, we performed an extensive hyperparameter sweep via a grid search, a common hyperparameter search strategy. In practice, as we train, we use our intuition about which hyperparameters to adjust based on the training loss curves, for example, if regularization needs to be adjusted to reduce overfitting.\n\n3. Regarding computation efficiency compared to baseline models, the Nucleotide Transformer authors also finetuned on every new downstream task. In comparison though, the Nucleotide Transformer used 8-A100-80GB GPUs for about 50 mins on average to finetune each dataset, as described in their paper. We used a single A100-40GB GPU, for 10 to 30 mins across the same datasets. Concretely, we provide results in Table R4 of the common response for a comparison of GPU-hours for pretraining and finetuning between baseline models and HyenaDNA. We will add these results to the appendix.\n\n4. The 18th dataset of the Nucleotide Transformer (on splice site prediction) had a broken link during the time we were working on the submission. The dataset is now available publicly, and we have updated our results with this dataset (as shown in Table R2 of the common response).  Thanks for pointing this out. We slightly underperformed the Nucleotide Transformer on this task (by 0.4 accuracy points)\n\nThank you again for the thorough review. Please let us know if there are other things we can clarify.\n"}]

Author rebuttal:
### Common Response

We thank the reviewers for their time and in-depth reviews. We believe that addressing the reviewer’s feedback and questions has helped greatly improve the quality of our manuscript.

We are happy to hear the reviewers appreciated the strong empirical performance of HyenaDNA over previous methods **[YJ4C, LTPa, nDFb]**, the novelty/relevance of the application **[YJ4C, LTPa, nDFb, DWDg, JfVK]**, and the clear description of the methods **[YJ4C, nDFb]**.

### Ablations

Multiple reviewers suggested ablations, including downstream performance for scratch vs. pretrained models **[YJ4C, DWDg]**, K-mer vs. single character tokenizers **[YJ4C, LTPa, nDFb]**, causal vs bidirectional **[YJ4C, nDFb]**, and additional model baselines on the GenomicBenchmarks and Nucleotide Transformer datasets **[LTPa, nDFb]**.

Below, we provide a common response with updates and ablation results that were requested by reviewers **[YJ4C, LTPa, nDFb, DWDg]**. In these experiments, we further investigate how each design choice in HyenaDNA contributes to performance gains compared to baseline models.

### Updates

Since our submission, we are pleased to report that we have pretrained an even longer HyenaDNA model: 1M context length (vs 450k in the submission), 500x longer than previous genomic FMs and is 160x faster than attention.

The results from ablations are in the PDF, with a summary below:

- Several new choices in HyenaDNA contribute to gains over previous genomic foundation models (FMs): single character vs. k-mer tokenizer, Hyena operator vs. attention, and the causal vs. bidirectional Hyena.
- Pretraining has a greater effect on the more challenging tasks, and as sequences become longer, eg, by up to 30 accuracy points on species classification at 450k context.
- HyenaDNA uses far less compute for pretraining and finetuning. E.g. the Nucleotide Transformer uses 128 A100 GPUs to pretrain for 1 month, while HyenaDNA used 1 A100 for 80 mins to pretrain a model used on the same downstream tasks.

**We provide further details on the ablations in the sections below.**

### Scratch / Pretraining

Reviewers **[YJ4C, DWDg]** inquired about how much pretraining (vs. scratch) improves downstream performance. To address this, we include training from scratch on 3 groups of datasets: GenomicBenchmarks, Nucleotide Transformer, and Species Classification.

**1. GenomicBenchmarks (Table R1 PDF):** Pretraining boosted HyenaDNA by up to 3.5 acc points, and by 1.8 points on average. HyenaDNA already performed strongly from scratch, which made pretraining gains more difficult. For AttnDNA, pretraining is more important, boosting performance by up to 11.7 acc points and by 4.5 points on average.

**2. Nucleotide Transformer datasets (Table R2 PDF):** 
On the Nucleotide Transformer datasets: On the more challenging tasks (histone marks, datasets starting with “H”), pretraining boosts HyenaDNA metrics by up to 21 MCC points on H3K4me3. For simpler tasks (with higher baseline values) such as the splice sites and promoter tasks, there was less boost (0 to 1 accuracy points). Note: the 18 tasks use a mix of MCC, F1, and accuracy metrics, and so an average comparison is less meaningful.

**3. Long-range species classification (Table R3 PDF):** 
For species classification, pretraining becomes more important for longer sequences, addressing questions from reviewers **[YJ4C and DWDg]** about where pretraining helps in HyenaDNA. At sequence length 250k & 450k, the scratch/pretrain gap is 30+ accuracy points. 

### K-mer tokenization vs single nucleotides

To ablate the impact of the K-mer tokenizer vs. single character **[YJ4C, LTPa]**, we used the same K-mer (K=6) tokenizer from the Nucleotide Transformer model, which had a vocabulary of ~4100. We then trained a scratch HyenaDNA model using this K-mer tokenizer on the GenomicBenchmarks. The K-mer tokenizer degraded performance for every dataset, from 2 to 10 accuracy points compared to (scratch) HyenaDNA with a single character tokenizer (see Table R1 PDF). The K-mer tokenizer is one factor in HyenaDNA’s gain, which we will add to section 4.2.

### Bidirectional vs Causal

To ablate the impact of using a causal model **[YJ4C, nDFb]**, we implemented a bidirectional version of HyenaDNA and trained from scratch on the GenomicBenchmarks. The bidirectional version degraded performance on 7 of 8 datasets compared to the standard causal HyenaDNA (also from scratch), on average by 3.8 accuracy points. See Table R1 PDF. The bidirectional HyenaDNA was implemented by using a circular FFT convolution.

### DNABERT baseline

To compare against a stronger baseline on the GenomicBenchmarks suggested by reviewers **[LTPa and nDFb]**, we also finetune DNABERT. DNABERT is able to reach SotA on 1 dataset (Coding vs Intergenomic), and match on another (Human Enhancers Cohn) with HyenaDNA (Table R1 PDF). DNABERT uses 110M params, while HyenaDNA uses just 400k param. We will add these results to section 4.2.

### AttnDNA finetuning

We finetune the AttnDNA model on the Nucleotide Transformer datasets, Table R2 PDF. AttnDNA and HyenaDNA are causal and use single nucleotide tokens, but AttnDNA significantly underperformed against its Hyena counterpart. This suggests the Hyena operator itself contributes significantly to the overall performance gains of HyenaDNA.

### Pretrain and Finetune Runtime Comparisons

**Pretraining compute:** Reviewers **[LTPa, nDFb, JfVK]** suggested also comparing efficiency by compute resources used (in addition to parameter count). When comparing actual GPU-hrs used for pretraining across baseline models, HyenaDNA is more efficient than baselines. See Table R4 PDF. We will put a full table of results in the updated manuscript.

**Finetuning compute:** We use the GenomicBenchmarks for finetuning, and record the per epoch runtime in Table R5 PDF. The Nucleotide Transformer uses a parameter-efficient finetuning: only 0.5% parameters are updated.


Meta-review: The reviewers agree that the authors' methods achieve strong empirical performance over previous methods, the approach appears novel, and the paper is well written. In response to the reviewers' common ask for a variety of ablation studies, the authors responded with the results of these studies and all reviewers were satisfied with the response. Ultimately, this paper represents a fairly significant engineering success: a foundation model trained on DNA that achieves state of the art performance with a much smaller model but much larger context window. Practitioners working with genomic sequence data are sure to find such a model very valuable.
Decision: Accept (spotlight)