Title: HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
Venue: NeurIPS 2023
Accepted: yes
Keywords: genomics

hyena

foundation models

large language models

transformers
Abstract: Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution (i.e. DNA "characters") where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena’s new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level – an up to 500x increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics for simple adaptation to novel tasks without updating pretrained model weights. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data.1 On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points. Code at https://github.com/HazyResearch/hyena-dna.

HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution

Abstract

Genomic (DNA) sequences encode an enormous amount of information for gene regulation, protein synthesis, and numerous other cellular properties. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution (i.e. DNA "characters") where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena , a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena 's new long-range capabilities, we present HyenaDNA , a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attentionbased models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens , and has full global context at each layer . We explore what longer context enables - including the first use of in-context learning in genomics for simple adaptation to novel tasks without updating pretrained model weights. On a long-range species classification task, HyenaDNA is able to effectively solve the challenge by increasing the context length to 1M without downsampling. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data. 2 On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points, and by as much as +20 accuracy points on enhancer identification. Code available at https://github.com/HazyResearch/hyenadna.

A C G T A C G T

A C G T A C G ?

DNA Sequence

L

Adaptation to

Sequence

ACGTACGT

2 n

*

L

Training

Stages

(n)

Regulatory element type

2 *

L

MLP

x

N

(Single Nucleotide Tokens)

HyenaDNA Pretraining

Hyena

Operator

Sequence length warm-up

for stability & faster training

HyenaDNA

Frozen

1M

…

1 Introduction

Understanding and learning from DNA sequences has long been a goal of biologists and deep learning researchers, as its 'language' encodes instructions essential for all living things [16]. The mapping from DNA instructions, genotypes, to observable function and traits, phenotypes, remains ongoing research effort. Towards this goal, researchers have proposed using foundation models (FMs) in genomics to learn generalizable features from unstructured whole genome data that can then be fine-tuned for a number of tasks including predicting the location and function of genes, identifying regulatory elements, and analyzing the evolution of species [25, 10, 20, 3, 53, 58]. In contrast to protein sequences, which have had successes in protein language models [29, 31, 32, 17, 5, 41, 14], DNA sequences are orders of magnitudes longer (e.g. the human genome is 3.2B nucleotides) with long-range dependencies and interactions that span over 100k+ nucleotides in length [1]. Overcoming the long-range limitations of current generation models could help drive the next wave of innovations in AI-powered drug discovery and therapeutics, and enable genomic FMs to understand and learn in-context whole patient genomes in a personalized way.

Limitations of current models Previous genomic FM approaches have relied on attention-based Transformers [25, 10, 53, 58], but face a number of challenges unique to DNA sequences. The attention mechanism scales quadratically in sequence length, with current genomic FMs pretraining on only 512 to 4,096 tokens as context [25, 58, 10, 55], <0.001% of the human genome. Also prevalent is the reliance on fixed k-mers, akin to DNA 'words', and tokenizers to aggregate meaningful DNA units. However, single nucleotide alterations represent physical analogs where, for example, single nucleotide polymorphisms (SNPs) and mutations can have a profound impact on biological properties including regulatory activity [33]. In contrast, natural language semantics can often be con-

PPL vs Context on the Human Genome

0

200k

400k

600k

800k

1M

2

.

9

3

3

.

1

3

.

2

3

.

3

Sequence Length

Perplexity

2 Layer Transformer

2 Layer HyenaDNA

4 Layer HyenaDNA

8 Layer HyenaDNA

served when single character or word changes occur over very long contexts. Therefore, having both long-range context and single nucleotide resolution simultaneously is critical, and remains a particular challenge in genomics.

Add

Hyena

Operator

Add

MLP

HyenaDNA Block

x

N

Long Convolution

Elem-wise Gate

Dense+Conv

Long Convolution

Elem-wise Gate

x

Hyena Filters

Dense

Dense

Sine Act

x

3

Window

Pos Enc

Elem-wise Gate

Dense+Conv

Dense+Conv

Dense+Conv

Toward longer context models Recently, Hyena [37], a large language model based on implicit convolutions, was shown to match attention in quality while reducing computational time complexity, thereby allowing a longer context to be processed. Hyena uses a parameter-efficient global convolutional filter along with a data-controlled gating mechanism, which enables a context-specific operation over every token. Indeed, Hyena showed that for simple associative recall tasks using synthetic data, a shallow 2 layer model could effectively process context lengths at 131k tokens. We hypothesize that Hyena 's core operations can unlock the potential to capture both the long-range and single nucleotide resolution of real genomic sequences over attention-based approaches. To test this, we explore two questions: (i.) Can a convolutional long-context model be used effectively at single nucleotide resolution? (ii.) What new capabilities could long-context genomic foundations models enable?

HyenaDNA The result of our investigation is HyenaDNA , a genomic FM pretrained on the human reference genome at context lengths up to 1 million tokens at single nucleotide resolution - an up to 500x increase over existing genomic FMs using dense-attention. HyenaDNA scales subquadratically in sequence length (training up to 160x faster than attention at sequence length 1M), uses single nucleotide tokens, and has a global receptive field at each layer. Our contributions include a "full-stack" recipe for building genomic FMs, including architecture design, a warm-up schedule to speed up training on ultralong sequences, and an efficient downstream adaptation procedure based on soft prompting and in-context learning.

Full-stack genomics modeling We start with a decoder-only Hyena architecture pretrained using next nucleotide (token) prediction. We forego standard aggregating tokenizers, using a singlecharacter tokenizer and a minimal DNA vocabulary of 4 nucleotides (plus special tokens). Training stability becomes an issue at ultralong sequences (200k+). To overcome this issue, we introduce a sequence length warm-up scheduler that gradually increases sequence length in stages. At sequence length 450k, training time is reduced by 40%, while boosting accuracy by 7.5 accuracy points on a species classification task. Furthermore, we design downstream adaptation procedures to leverage longer context windows, as simpler and more flexible alternatives to standard fine-tuning in genomics. This includes a novel soft prompt technique where learnable tokens (up to 32k) are injected directly into the input sequence itself, enabling competitive downstream results without the need to update a pretrained model.

Genomic downstream tasks We apply our pretrained HyenaDNA models to 29 diverse downstream genomic tasks to showcase its long-range ability as well as fine-grain resolution. On fine-tuned benchmarks from the Nucleotide Transformer [10], HyenaDNA achieves state-of-the-

art (SotA) on 12 of 18 datasets while using a model with orders of magnitude less parameters and pretraining data (see Tab. 4.2). On the GenomicBenchmarks [23], HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points, and by as much as +20 accuracy points on enhancer function identification. On a novel species classification task, HyenaDNA effectively solves the challenge by increasing the context length to 1 million tokens. In a challenging chromatin profile experiment, a 919-way multi-task, HyenaDNA performs competitively against a larger SotA sparseattention BigBird Transformer [55]. Finally, we analyze the learned embeddings of a pretrained HyenaDNA model by clustering sequences by biotype (gene or transcription type) and compare the results with existing genomic FMs, showing that HyenaDNA can serve as an effective universal featurizer in genomics.

2 Preliminaries and Related Work

2.1 Transformers and Attention

Powering many recent foundation models is the attention mechanism. Given a lengthL sequence x ∈ R L × D , a (single-headed) layer of scaled self-attention [2, 51] is a map from R L × D to R L × D which performs the following operations:

where D is the embedding dimension, W q , W k , W v ∈ R D × D are learnable linear maps and σ indicated row-wise softmax (and optional scaling). Attention computes all pair-wise comparison for every token, and scales as O ( L 2 ) in sequence length. This allows a global context at high resolution, but limits the size of the context on current hardware.

Previous methods to reduce the quadratic cost of attention have used specialized methods to approximate full dense attention [18]. In sparse attention, elements attend only to a subset of all other positions. Alternatively, linear attention methods construct approximations to A ( u ) that can be evaluated in subquadratic time. Both of these classes of methods, however, trade lower time complexity (allowing longer sequences) for loss in expressivity.

2.2 Long Context Strategies in Genomics

To achieve longer context, genomic models have relied on two strategies: i. tokenization and ii. dilation and downsampling. Tokenization is a necessary step in masked language modeling (MLM) with bidirectional Transformer architectures (BERT) [13], a common model in genomics. These tokenizers use fixed k-mers (short overlapping sequences of length k) or frequency-based byte pair encoding (BPE), that attempt to aggregate DNA into meaningful units [25, 55]. Consequently, these aggregation techniques create large new vocabularies (compared to the natural vocabulary of 4 nucleotides) that are less generalizable [49]. The second strategy uses dilated convolutions and downsampling, both of which essentially average or skip elements between weights [18]. A canonical example is the Enformer, which uses dilation and downsampling to reach context lengths of 100k nucleotides to predict gene expression tracks [1]. Common across tokenization, dilation, and downsampling is the sacrifice of single nucleotide resolution to reach longer context.

2.3 Large Convolutional Models

A discrete convolution between an input x of length L and a (learnable) filter h is given by:

where T ∈ R L × L is the Toeplitz matrix corresponding to the convolution. Historically, convolutions have played an important role in deep learning and more broadly signal processing. More recently, it has been shown that by stacking k long convolution layers, where k is parametrized through a function γ θ i.e. k := γ θ ( L ) , one can achieve state-of-the-art performance on a variety of benchmarks involving long sequences, for example the Long Range Arena (LRA) [48, 24, 47, 19]. Different γ θ have been proposed in the literature: state-space models [24, 19], and implicit parametrizations via neural fields [45, 44, 37]. On language, the H -family of implicit convolution language models,

H3 and Hyena , [12, 37] used long convolutions and gating to match Transformer performance in O ( L log 2 L ) time, notably lower than the O ( L 2 ) of attention-based models.

HyenaDNA takes inspiration from these approaches, showing that attention-free, long-context causal models can achieve high performance on downstream genomic tasks. These extended long-range capabilities enable us to explore new paradigms in genomics, such as in-context learning to easily adapt to new tasks without updating pretrained models.

3 HyenaDNA Long-Range Genomic Foundation Models

In this section, we introduce the HyenaDNA approach to long-range genomic sequence modeling. We start with a description of the model architecture, then discuss sequence length warm-up and soft prompting techniques for downstream adaptation.

3.1 The HyenaDNA Model

The HyenaDNA model is a decoder-only, sequence-to-sequence architecture defined by a stack of blocks consisting of a Hyena operator [37], followed by a feed-forward neural network (see Fig. 1.3).

Given an input x ∈ R L ( L denotes sequence length), a Hyena 3 operator can be defined as:

where x 1 , x 2 , v are projections of the input, and T h ∈ R L × L is the Toeplitz matrix constructed from a learnable long convolution filter produced as the output of a neural network, ( T h ) ij = h i -j . The convolution filter values themselves are obtained through a small neural network γ θ taking as input

x

W

v

T

v

W

x

1

T

x

1

W

x

2

T

x

2

D

x

1

T

h

D

x

2

x

1

x

2

v

the time (position) index and optionally positional encodings, h t = γ θ ( t ) , which enable the operator to process very long sequences without growing linearly in the number of parameters. Further, the matrices D x 1 , D x 2 ∈ R L × L are constructed with x 1 , x 2 on the diagonals, and evaluated as elementwise gating. The projections are obtained by applying a dense linear layer and short convolution to the input sequence, as shown in Figure 3.1.

Proposition 3.1. A Hyena operator can be evaluated in O ( L log 2 L ) time.

Efficient evaluation is crucial on settings involving extremely long sequences such as genomics. In the general case where the embedding dimension D > 1 and x ∈ R L × D , the linear projections W x 1 , W x 2 , W v ∈ R D × D are right multiplied to x , and D independent Hyena operators are then applied to each dimension.

3.2 Training Long Sequence Models

Tokenization The subquadratic cost of HyenaDNA in sequence length allows the model to process ultralong sequences directly at the single nucleotide level without the need for frequency-based aggregation tokenizers. This enables fine-grain resolution for both short and long sequences, critical for detecting single nucleotide polymorphisms or mutations and modeling long-range dependencies in gene expression.

We use the natural DNA vocabulary and refer to each nucleotide as a token. The tokens include "A", "G", "C", "T", and "N" (a non-specific nucleotide) and special character tokens for padding, separation, and unknown characters. Tokens are mapped to embedding dimension D .

Sequence length warm-up for ultralong sequences Directly training on long sequences can affect training stability as the variance in gradient increases [28]. Training on shorter sequences initially (followed by longer sequences) was used by [38] to train small scale Transformers and reduce training time, while [28] used sequence length warm-up to address stability on up to 2k tokens.

For ultralong sequences (200k+), we develop a new warm-up schedule that gradually increases the sequence length in stages to improve both stability and decrease training time.

Our sequence length schedule starts at L 1 = 64 , then doubles the window at each stage while keeping the global batch size constant. By doing so, iterations at each consecutive stage will include more tokens, ensuring the scheduler can also act as a form of batch size warm-up. In Fig. 3.2, we observe sequence length scheduling to be particularly important at sequence lengths greater than 450 k, where at this length training time is reduced by 40% and improving ultimate accuracy by 7.5% points for a species classification task described later in section 4.4.3.

3.3 Downstream Adaptation

Tuneable prompting for long-context models

Prompts have been traditionally used to guide the output of a FM [30] by prepending additional context to an input. Expanding on this approach, soft tuneable prompting was introduced to inject learnable tokens (as weights) into the input directly [27] as an alternative to model fine-tuning.

With an extended context length ( L ), we're able to explore new paradigms in adapting FMs after pretraining. Given a downstream task with prompts x p ∈ R T and corresponding labels y p , we prepend N ≤ L -T trainable parameters θ of dimension D after the embedding step:

The resulting sequences x are then processed by the model, and θ is optimized on a loss function involving the input sequence's label y p . Crucially, soft prompting requires utilization of a small subset of prompt and label pairs to optimize θ .

During soft prompting, HyenaDNA only optimizes the parameters of the prompt in the input sequence while keeping all other model parameters fixed. Soft prompting thereby provides a flexible and computationally efficient approach to adapting genomic FMs to new downstream tasks.

4 Experiments

In 4.1, we start with pretraining HyenaDNA on the human reference genome [22]. We then evaluate HyenaDNA on existing short-range (<5k nucleotides) downstream benchmarks in 4.2 to assess the performance of single nucleotide resolution. In 4.3, we explore what new capabilities emerge with longer range genomic modeling in the form of in-context learning. Finally, we push the limits of ultralong context performance in 4.4.

4.1 Pretraining on the Human Genome

We pretrain HyenaDNA on the human reference genome [22] using next nucleotide (token) prediction. Starting with a stack of decoder-only Transformer blocks, we swap attention for the Hyena operator, and compare against a baseline Transformer (GPT) with Flash Attention [11]. We add gradient checkpointing to HyenaDNA to decrease the memory footprint by 3x on longer sequences ( > 160k). We then scale HyenaDNA along dimensions of model depth (2 to 8 layers), width (128 to 256 dimensions), and sequence length (1024 to 1M). At sequence length 1M, HyenaDNA is 160x faster than its Transformer counterpart as shown in Fig. 4.1.

As shown in Fig. 1.2, we observe that as context length increases, perplexity improves during pretraining. However, this improvement comes at the expense of more training time and tokens. For models too shallow to effectively process longer context, perplexity can begin to degrade (increase), observing inflection points with longer sequences. In this way, increasing context can serve as a

0

200

400

20

40

60

80

100

Time (min)

Non-warmup

Warmup

novel regularization dimension. For genomic pretraining, we provide the following guidelines. 1. In optimizing for faster training time, shorter context enable lower perplexity to be reached faster. 2. In optimizing for best overall perplexity, longer context allows for lower perplexity at the cost of training on more tokens. See A.1 for experiment details.

4.2 Single Nucleotide Resolution

Our first downstream tasks use short-range genomic sequences (<5k) aimed at evaluating single nucleotide resolution performance on sequence-level classification using standard fine-tuning.

GenomicBenchmarks We start with the newly released GenomicBenchmarks [23], which is comprised of 8 regulatory element classification datasets with sequence lengths of 200-500, and one up to 4,776. The original baseline model uses a short-range CNN. We fine-tune the pretrained Transformer (GPT) and HyenaDNA from 4.1, both having single nucleotide resolution, as well as the DNABERT model [25]. HyenaDNA sets a new SotA on 7 of 8 datasets and by up to 20% points on the human enhancer identification

10

3

10

4

10

5

10

6

10

1

10

2

10

3

10

4

10

5

160x faster

Sequence Length (log)

millisec (log)

Runtime vs Context (log-scale)

Transformer

HyenaDNA

task, as shown in Tab. 4.1. See A.2 for additional experiment details and ablations.

Nucleotide Transformer

Next, we benchmark against 18 datasets from the Nucleotide Transformer (NT) [10], which includes predicting regulatory elements for enhancers, promoters, epigenetic marks, and splice sites from DNA sequences of length 200-600 nucleotides. We compare against 3 NT base models, which were pretrained using masked language

modeling (BERT) and then fine-tuned. The NT models ranged from 500M to 2.5B parameters, and pretrained on up to 3202 genomes. All NT models use 6-mer sequences of 1000 tokens long. For HyenaDNA , we attach a linear decoder head and fine-tune a pretrained model, surpassing SotA on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data, shown in Tab. 4.2. See A.2 for additional experiment details and ablations.

4.3 In-context Learning for Genomic Sequences

Compared to natural language FMs, which have shown strong success with in-context learning, HyenaDNA 's vocabulary is very small. DNA sequences are also less diverse in structure, e.g. there's no concept of labels or descriptions that follow a DNA sequence. This makes it challenging to perform "pure" in-context learning (relying only on inference), since new concepts such as classification labels would require new symbols. To overcome this limitation and explore the potential for in-context learning in genomics, we make use of two variants of in-context learning: soft prompting and instruction fine-tuning. Each involve a brief tuning phase to introduce the concept of classification using only the existing vocabulary.

Procedure In both variants, we use the GenomicBenchmarks in 4.2, and a HyenaDNA model pretrained on sequence length 160k from 4.1.

In the first experiment, we evaluate a soft prompting approach by prepending a sequence of soft tuneable tokens (2 to 32k) directly in the input sequences. We include a brief tuning phase ( < 20 epochs), updating the soft tokens only, to provide HyenaDNA with the ability to indicate the target classes. To denote classes, we repurpose HyenaDNA 's fixed vocabulary: for binary classification, for example, we indicate the two classes with the letters "A" and "N".

In the second experiment, we evaluate a few-shot learning approach to in-context learning [6] by prepending, consecutively, k (2 to 32) demonstrations of each class and its sequence into the prompt. As before, we encode class labels by the use of individual letters of HyenaDNA 's existing vocabulary. We additionally perform a brief instruction-tuning period [52]

for each dataset to familiarize HyenaDNA with this task structure by tuning the pretrained model on a small subset of the dataset.

Nontata-Promoters

Enhancers-Cohn

Enhancers-Ensembl

Ensembl-Regulatory

75

50

A

25

225 2729 211 213 215

225 27 29 211 213215

22527 29211213 215

225 2729211213 215

2252729211213215

#Tuneable Tokens

#Tuneable Tokens

#Tuneable Tokens

Results In Fig. 4.2, HyenaDNA 's performance on novel tasks improves as more tuneable tokens are added into the input sequences, and saturates close to baseline performance (Tab. 4.1; with the exception of the Human Regulatory dataset). By contrast, we find that increasing k -shot demonstrations to the input does not necessarily improve performance. A higher number of tuning samples is needed before k -shot demonstrations start to boost accuracy as shown in Tab. A.1. See A.3 for experiment details.

4.4 Ultralong-Range Genomics

In our final experimental section, we focus on pushing the limits of using long context effectively in genomics. In 4.4.1, we tackle a challenging 919 binary multi-task against a sparse-attention baseline. In 4.4.2 we analyze the learned embeddings HyenaDNA and its use in clustering long sequences by functional annotation, and in 4.4.3 we showcase a novel ultralong-range species classification task.

4.4.1 Chromatin Profile Prediction

The prediction of chromatin profiles and epigenetic markers from DNA sequences is an important and challenging task to quantify the functional effects of non-coding variants. These variants include single nucleotide changes in DNA that can affect the downstream expression of genes [56]. The DeepSEA dataset [57] is compiled from 919 chromatin features including transcription factor (TF) binding profiles, DNase I-hypersensitive sites (DHS) and histone mark (HM) profiles. For a given sequence, the task is to jointly predict 919 labels cor-

responding to the chromatin profile (similar to peak detection) of a central region of the sequence, indicating the presence of such functional effects. The input also includes flanking regions that provide broader contextual information needed to incorporate long-range interactions. We fine-tune our pretrained HyenaDNA models from 4.1 and perform competitively against a DeepSea CNN and the SotA sparse attention BigBird [55] baselines using 5-30 × fewer parameters. See A.4 for experiment details.

4.4.2 Biotype Embeddings

Sequence embeddings, colored by biotype

Nucleotide Transformer

HyenaDNA

IncRNA

Unprocessed Pseudogene

snRNA

miRNA

TEC

snoRNA

MiscRNA

Next, we analyze the pretrained embeddings from HyenaDNA and compare them with DNABERT [25] and the Nucleotide Transformer [10]. We encode sequences of human genes corresponding to different biological function annotations obtained from the Ensembl dataset known as biotypes [9]. In cases where the length of the input exceeds the context window of the encoder, the sequence is chunked (by the max length of the encoder) and averaged.

We fit the embeddings using an XGBoost [7] classifier on the 10 most frequent biotypes, and apply tSNE [50] for visualization. As shown in 4.3, distinct clusterings emerge visually, while quantitatively, HyenaDNA produces the highest F1 score in biotype classification (with a much smaller model), indicating that during pretraining, HyenaDNA learns informative features related to biological function.

4.4.3 Species Classification

The majority of the genome is conserved across species - humans and non-human primates, for example, have <10% sequence divergence [43], making them difficult to discriminate. This allows us to to design an ultralong-range sequence modeling task to test whether a model can determine the source species of a random genetic sequence. To train, we randomly sample DNA sequences from 5 different species, and fine-tune pretrained HyenaDNA and Transformer models from 4.1 to predict the species label. We observe in Tab. 4.5 that both models struggle on shorter sequences of length 1024 , but performance improves with longer contexts as the distinct mutational profile of each species becomes more evident. HyenaDNA effectively solves the task by using a context length of 450 k to 1 million, where Transformer cannot due to infeasible training time limitations. See A.6 for experiment details.

5 Conclusion

Summary We presented HyenaDNA , a genomic foundation model pretrained on the human reference genome with context lengths up to 1 million tokens at single nucleotide resolution - an up to 500x increase over previous genomic FMs using dense-attention. HyenaDNA is able to learn generalizable features that can then be finetuned for tasks including identifying regulatory elements and on a 919-way chromatin profile prediction task. We also explored the first use of in-context learning in genomics to enable simpler adaptation to downstream tasks without any updates to pretrained weights.

Limitations and Future Work While demonstrating competitive results and introducing novel capabilities, it is worth noting that HyenaDNA was pretrained on only one human reference genome. Incorporating genomes of multiple humans and species could increase generalizability in learned features and reduce bias. Furthermore, our current focus in this study was exclusively on DNA sequences. Extending our framework to incorporate other

biological or chemical sequences, such as proteins and drug molecules, has the potential to unlock multi-modal capabilities similar to those observed in natural language and vision FMs [39, 40, 54].

With respect to model size, HyenaDNA is significantly smaller than previous genomic FMs and was pretrained using up to 8 Nvidia A100 (80GB) GPUs. We expect increasing model size, and compute, may lead to additional long-range capabilities. Notably, with model parallelism, it becomes feasible to extend the context length by orders of magnitude beyond this current work, and leave that open to future research.

Furthermore, beyond discriminative applications, the use of long context models in generative tasks unlocks exciting prospects for the design of synthetic regulatory elements, genes and protein complexes. In conclusion, the continued advancements of long-range sequence models with single nucleotide resolution hold great promise in driving innovation in genomic research and unraveling the complexities of biological systems.

Acknowledgments

We would like to thank Guatam Machiraju, Elliott Epstein, Archis Joglekar, Jared Dunnmon, Nazim Bouatta and Anshul Kundaje for helpful discussion and feedback on earlier drafts, and Together for providing the compute used to train models in this paper. We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); US DEVCOM ARL under No. W911NF-21-2-0251 (Interactive Human-AI Teaming); ONR under No. N000141712266 (Unifying Weak Supervision); ONR N00014-20-1-2480: Understanding and Applying Non-Euclidean Geom-

etry in Machine Learning; N000142012275 (NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), Department of Defense (DoD) through the National Defense Science and Engineering Graduate Fellowship (NDSEG) Program, and members of the Stanford DAWN project: Facebook, Google, and VMWare. This work is supported by NSF (1651565), AFOSR (FA95501910024), ARO (W911NF-21-1-0125), ONR, DOE (DE-SC0022222), CZ Biohub, and Sloan Fellowship. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.

References

Ž. Avsec, V. Agarwal, D. Visentin, J. R. Ledsam, A. Grabska-Barwinska, K. R. Taylor, Y. Assael, J. Jumper, P. Kohli, and D. R. Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods , 18(10):1196-1203, 2021.

D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 , 2014.

G. Benegas, S. S. Batra, and Y. S. Song. DNA language models are powerful zero-shot predictors of non-coding variant effects. bioRxiv , pages 2022-08, 2022.

R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

N. Brandes, D. Ofer, Y. Peleg, N. Rappoport, and M. Linial. ProteinBERT: a universal deeplearning model of protein sequence and function. Bioinformatics , 38(8):2102-2110, 2022.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining , pages 785-794, 2016.

D. M. Church, V. A. Schneider, T. Graves, K. Auger, F. Cunningham, N. Bouk, H.-C. Chen, R. Agarwala, W. M. McLaren, G. R. Ritchie, et al. Modernizing reference genome assemblies. PLoS biology , 9(7):e1001091, 2011.

F. Cunningham, J. E. Allen, J. Allen, J. Alvarez-Jarreta, M. R. Amode, I. M. Armean, O. Austine-Orimoloye, A. G. Azov, I. Barnes, R. Bennett, et al. Ensembl 2022. Nucleic acids research , 50(D1):D988-D995, 2022.

H. Dalla-Torre, L. Gonzalez, J. Mendoza-Revilla, N. L. Carranza, A. H. Grzywaczewski, F. Oteri, C. Dallago, E. Trop, H. Sirelkhatim, G. Richard, M. Skwark, K. Beguir, M. Lopez, and T. Pierrot. The Nucleotide Transformer: Building and evaluating robust foundation models for human genomics. bioRxiv , 2023.

T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems , 2022.

T. Dao, D. Y. Fu, K. K. Saab, A. W. Thomas, A. Rudra, and C. Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052 , 2022.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

A. Elnaggar, M. Heinzinger, C. Dallago, G. Rehawi, Y. Wang, L. Jones, T. Gibbs, T. Feher, C. Angerer, M. Steinegger, et al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intelligence , 44 (10):7112-7127, 2021.

ENCODE Project Consortium. An integrated encyclopedia of dna elements in the human genome. Nature , 489(7414):57, 2012.

ENCODE Project Consortium. Expanded encyclopaedias of DNA elements in the human and mouse genomes. Nature , 583:699-710, 2020.

N. Ferruz, S. Schmidt, and B. Höcker. ProtGPT2 is a deep unsupervised language model for protein design. Nature communications , 13(1):4348, 2022.

Q. Fournier, G. M. Caron, and D. Aloise. A practical survey on faster and lighter transformers. ACM Computing Surveys , 2021.

D. Y. Fu, E. L. Epstein, E. Nguyen, A. W. Thomas, M. Zhang, T. Dao, A. Rudra, and C. Ré. Simple hardware-efficient long convolutions for sequence modeling. arXiv preprint arXiv:2302.06646 , 2023.

D. Gankin, A. Karollus, M. Grosshauser, K. Klemon, J. Hingerl, and J. Gagneur. Speciesaware DNA language modeling. bioRxiv , pages 2023-01, 2023.

Q. Geng, R. Yang, and L. Zhang. A deep learning framework for enhancer prediction using word embedding and sequence generation. Biophysical Chemistry , 286:106822, 2022.

Genome Reference Consortium. Genome reference consortium human build 38 (grch38). National Center for Biotechnology Information, 2013. URL https://www.ncbi.nlm. nih.gov/assembly/GCF_000001405.26/ .

K. Gresova, V. Martinek, D. Cechak, P. Simecek, and P. Alexiou. Genomic Benchmarks: A collection of datasets for genomic sequence classification. bioRxiv , 2022.

A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 , 2021.

Y. Ji, Z. Zhou, H. Liu, and R. V. Davuluri. DNABERT: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome. Bioinformatics , 37 (15):2112-2120, 2021.

W. J. Kent, C. W. Sugnet, T. S. Furey, K. M. Roskin, T. H. Pringle, A. M. Zahler, and D. Haussler. The human genome browser at ucsc. Genome research , 12(6):996-1006, 2002.

B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 , 2021.

C. Li, M. Zhang, and Y. He. The stability-efficiency dilemma: Investigating sequence length warmup for training GPT models. In Advances in Neural Information Processing Systems , 2022.

Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, A. dos Santos Costa, M. Fazel-Zarandi, T. Sercu, S. Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv , 2022.

P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35, 2023.

A. Madani, B. Krause, E. R. Greene, S. Subramanian, B. P. Mohr, J. M. Holton, J. L. Olmos Jr, C. Xiong, Z. Z. Sun, R. Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology , pages 1-8, 2023.

J. Meier, R. Rao, R. Verkuil, J. Liu, T. Sercu, and A. Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural Information Processing Systems , 34:29287-29303, 2021.

J. Nasser, D. T. Bergman, C. P. Fulco, P. Guckelberger, B. R. Doughty, T. A. Patwardhan, T. R. Jones, T. H. Nguyen, J. C. Ulirsch, F. Lekschas, K. Mualim, H. M. Natri, E. M. Weeks, G. Munson, M. Kane, H. Y. Kang, A. Cui, J. P. Ray, T. M. Eisenhaure, R. L. Collins, K. Dey, H. Pfister, A. L. Price, C. B. Epstein, A. Kundaje, R. J. Xavier, M. J. Daly, H. Huang, H. K. Finucane, N. Hacohen, E. S. Lander, and J. M. Engreitz. Genome-wide enhancer maps link risk variants to disease genes. Nature , 593:238-243, 2021.

M. Oubounyt, Z. Louadi, H. Tayara, and K. T. Chong. DeePromoter: Robust promoter predictor using deep learning. Frontiers in Genetics , 10, 2019.

T. H. Pham, D. H. Tran, T. B. H. Ho, K. Satou, and G. Valiente. Qualitatively predicting acetylation and methylation areas in DNA sequences. Genome Informatics , 16(2):3-11, 2005.

D. K. Pokholok, C. T. Harbison, S. Levine, F. Lewitter, D. K. Gifford, and R. A. Young. Genome-wide map of nucleosome acetylation and methylation in yeast. Cell , 122(4):517-527, 2005.

M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and C. Ré. Hyena Hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866 , 2023.

O. Press, N. A. Smith, and M. Lewis. Shortformer: Better language modeling using shorter inputs. arXiv preprint arXiv:2012.15832 , 2020.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021.

R. Rao, J. Meier, T. Sercu, S. Ovchinnikov, and A. Rives. Transformer protein language models are unsupervised structure learners. Biorxiv , pages 2020-12, 2020.

Roadmap Epigenomics Consortium. Integrative analysis of 111 reference human epigenomes. Nature , 518(7539):317-330, 2015.

J. Rogers and R. A. Gibbs. Comparative primate genomics: emerging patterns of genome content and dynamics. Nature Reviews Genetics , 15(5):347-359, 2014.

D. W. Romero, R.-J. Bruintjes, J. M. Tomczak, E. J. Bekkers, M. Hoogendoorn, and J. C. van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. arXiv preprint arXiv:2110.08059 , 2021.

D. W. Romero, A. Kuzina, E. J. Bekkers, J. M. Tomczak, and M. Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102.02611 , 2021.

N. Scalzitti, A. Kress, R. Orhand, T. Weber, L. Moulinier, A. Jeannin-Girardon, O. Collet, Pierre anf Poch, and J. D. Thompson. Spliceator: multi-species splice site prediction using convolutional neural networks. BMC Bioinformatics , 22(1):1-26, 2021.

J. T. Smith, A. Warrington, and S. W. Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933 , 2022.

Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006 , 2020.

Y. Tay, V. Q. Tran, S. Ruder, J. Gupta, H. W. Chung, D. Bahri, Z. Qin, S. Baumgartner, C. Yu, and D. Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. arXiv preprint arXiv:2106.12672 , 2021.

L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 , 2021.

F. Yang, W. Wang, F. Wang, Y. Fang, D. Tang, J. Huang, H. Lu, and J. Yao. scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data. Nature Machine Intelligence , 4(10):852-866, 2022.

J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022.

M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems , 33:17283-17297, 2020.

S. Zaina, E. L. Pérez-Luque, and G. Lund. Genetics talks to epigenetics? the interplay between sequence variants and chromatin structure. Current genomics , 11(5):359-367, 2010.

J. Zhou and O. G. Troyanskaya. Predicting effects of noncoding variants with deep learningbased sequence model. Nature methods , 12(10):931-934, 2015.

M. Zvyagin, A. Brace, K. Hippe, Y. Deng, B. Zhang, C. O. Bohorquez, A. Clyde, B. Kale, D. Perez-Rivera, H. Ma, et al. GenSLMs: Genome-scale language models reveal SARS-CoV2 evolutionary dynamics. bioRxiv , pages 2022-10, 2022.

Contents

HyenaDNA

Supplementary Material

A.6

Long-range Species Classification Details

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

24

A Appendix: Experimental Details

In the following sections we provide further details for each experiment. Across all experiments, we use Pytorch and Pytorch Lightning. We train on a mix of Nvidia GPUs with A100s, V100s, and T4s. Unless otherwise stated, we use a cross entropy loss for our objective. Our repository is made public here: https://github.com/HazyResearch/hyena-dna.

A.1 Pretraining Details

Data For pretraining, we use a single human reference genome [22], and leverage the training and validation intervals (start and end) from [1]. During training, we sample an interval and obtain a sequence of length L by adjusting the intervals on both ends. For the test set, we use chromosomes 14 and X, exclusively, and sample non-overlapping sequences of length L .

Model We design a suite of parameter efficient architectures with depths between 2 and 8 layers, Hyena blocks of Order-N = 2, and width 128 to 256. The MLP expansion factor (reverse bottleneck) is 4x the width. See Fig. 1.3 for the block architecture of HyenaDNA . The parameter counts range from 400k to 6.6M, trained on sequence lengths between 1,024 and 1M. Tab. A.1 highlights a representative subset of the models we trained. Note: we use different pretrained model sizes depending on the downstream task to prevent overfitting. When selecting which pretrained model to use for a downstream task, we found that a pretrained sequence length of 2 to 4x the downstream max sequence length results in the best performance.

Training Wepretrain each model for 10-20k global steps. For models trained on longer sequences, this translates to more tokens being used, as each sample contains more tokens. For example, the largest model with context length 1M was trained on 2T tokens over 4 weeks. We adjust the "accu-mulate_grad_batches" argument in Pytorch Lightning to keep the global batch size consistent across models and sequence lengths. See Tab. A.1 for hyperparameter details.

Training efficiency We compare pretraining compute resources and GPU-hours to reach competitive performance on the short-range tasks for several baselines and HyenaDNA models, shown in Tab. A.2.

A.2 Short-Range Genomics Details

A.2.1 GenomicBenchmarks experiment

Data The GenomicBenchmarks [23] includes 8 datasets designed for sequence-level classification tasks that involve predicting regulatory elements, along with one binary species task. The benchmarks provided for the baseline model include two sets of results: one obtained with Pytorch and the other with TensorFlow. Since our code base is implemented in Pytorch, we compare our results with the Pytorch-based benchmarks.

Model Our backbone is a pretrained 2 layer HyenaDNA model with width 128, trained on sequence length 1024. We pool along the sequence dimension to obtain a classification token, and attach a simple linear decoder head. The baseline CNN, as described by [23], uses uses an embedding layer, 3 convolutional layers with number of filters: 16, 8, and 4. It uses batch norm and max pooling after each convolutional layer, followed by 2 dense layers. It is trained for 10 epochs with batch size 64. The mode sizes range from 120k to 520k, depending on sequence length chosen.

Training The primary hyperparameters we sweep across include: learning rate, global batch size, dropout, weight decay, and a reverse complement augmentation. See Tab. A.3 for ranges of hyperparamters used.

A.2.2 Ablations on the GenomicBenchmarks

To better understand how specific design choices in the HyenaDNA model effect performance, we perform a series of ablation experiments on the GenomicBenchmarks.

Pretraining: Wetrain HyenaDNA from scratch and compare with the pretrained version. The pretrained models provide mild to moderate gains - likely due to the benchmarks being near saturation already.

Tokenization: Wetrain HyenaDNA using a k-mer tokenizer (k=6) to isolate the effect of the single nucleotide tokenizer. The k-mer tokenizer drops performance significantly across on a majority of the datasets (by as much as 10 accuracy points), while boosting one dataset (Human Enhancer Ensembl). Therefore, the single nucleotide tokenization appears to be a significant component of the HyenaDNA model.

Bidirectional: To ablate the impact of using a causal model, we implemented a bidirectional version of HyenaDNA and trained from scratch on the GenomicBenchmarks (i.e. without masked

language model pretraining). The bidirectional version degraded performance on 7 of 8 datasets compared to the standard causal HyenaDNA (also from scratch), on average by 3.8 accuracy points.

The bidirectional HyenaDNA was implemented by using a circular FFT convolution. This involved manipulating the padding on the input sequence before performing the FFT convolution. Previously, we zero padded the input on the right side by length L (the sequence length). For bidirectionality, we pad by 1 / 2 L on the left and right side of the input, effectively providing a bidirectional receptive field (due to the circular convolution). This is one of many possible ways to implement a bidirectional version of Hyena.

A.2.3 Downstream prediction tasks for Nucleotide Transformer benchmark

Following the Nucleotide Transformer [10], we collected datasets from four different sources [21, 35, 34, 46].

Promoter The promoter dataset included TATA-box-containing and TATA-box-lacking promoters. Tasks involved predicting promoters with a TATA-box, identifying promoters lacking a TATAbox, and distinguishing between both promoter categories and non-promoter sequences. The promoter datasets were obtained from the Eukaryotic Promoter Database (EPDnew) 4 for human and mouse genomes. Promoter sequences were extracted from regions 249 nucleotides upstream and 50 nucleotides downstream of the transcription start sites.

Enhancer For the enhancer prediction task, we used the dataset from [21] containing DNA sequences classified into strong enhancers, weak enhancers, and non-enhancers. The tasks involved binary classification to distinguish enhancer sequences from non-enhancer sequences and identify specific enhancer types.

Epigenetic Marks In the epigenetic marks prediction task, we used the dataset from [35, 36] to predict nucleosome occupancy and modification states in the yeast genome. In 10 binary classification tasks, the model had to discriminate between DNA regions that were occupied by histones or not. The 10 tasks varied based on the types of histones investigated, including unmodified histones H3 and H4, as well as histones modified by either acetylation (H3K9ac, H3K14ac) or methylation (H3K4me1, H3K4me2, H3K4me3, H3K36me3, H3K79me3).

Splice Site For the splice site prediction task, DNA sequences from over 100 organisms were used to predict whether the sequences contain donor or acceptor splice sites [46]. Donor splice sites denote the beginning of an intron and acceptor splice sites the end of an intron. During RNA splicing, these sites are recognized by the spliceosome, a complex molecular machine that enables the removal of introns from the gene.

Preprocessing The Nucleotide Transformer study did not provide their exact train-test splits, except for the enhancer dataset. Therefore, we generated our own train-test splits using a 90:10 ratio. For the promoter dataset, negative samples were not available, and had to be generated following the procedure described by [34].

Model & Training For the architecture, we use a HyenaDNA model with 2 layers and width 256, and trained on sequences of length 1024. We average across the tokens to obtain a single classification token. For each task, we replaced the model head and fine-tuned the weights of the entire model (1.6M parameters). In contrast, the Nucleotide Transformer uses a parameter-efficient fine-tuning technique that introduces new weights and fine-tunes only the newly added weights, while keeping the initial model weights frozen, presumably due to its large size of 500M to 2.5B parameters. The corresponding HyenaDNA hyperparameter ranges used for training each task are reported in Table A.5.

A.2.4 Ablations on the Nucleotide Transformer benchmarks

We perform additional ablations on the Nucleotide Transformer benchmarks to assess the impact of pretraining, as well as attention vs. HyenaDNA , as shown in shown in Table A.6. We observed that pretraining has a greater effect on the more challenging tasks (and as sequences become longer, shown in A.11). On the more challenging tasks (histone marks, datasets starting with 'H'), pretraining boosts HyenaDNA metrics by up to 21 MCC points on H3K4me3. For simpler tasks (with higher baseline scores) such as the splice sites and promoter tasks, the gain was lower (0 to 1 accuracy points), as these were already near saturation in performance.

A.3 In-Context Learning Details

Background Akey premise of foundation models is that they are able to learn new tasks with little to no new training data [4]. Recent advances in language modeling have demonstrated that language foundation models can often adopt the behaviors necessary to perform new tasks in-context [6]. Here, information about the task that is to be performed, such as examples of respective inputs and targets, are added to the input of the model. By conditioning their prediction on the provided context, language foundation models are generally able to perform the task without any changes to their parameters.

A key challenge for in-context learning with HyenaDNA is its limited vocabulary, which is composed of only a few nucleotides, and does not provide any vocabulary for novel downstream tasks, such as class labels. To explore the potential for in-context learning in genomics, we use two variants of in-context learning, both using a brief tuning phase to introduce HyenaDNA to the concept of

classification with its existing vocabulary. As a test bed for this exploration, we use 5 datasets from the GenomicBenchmarks and a HyenaDNA pretrained on sequences of 160 k length sequences.

In the first experiment, we apply a soft prompting approach [27] by adding a sequence of tuneable tokens to the input inself. In the second experiment, we explore a few-shot learning approach [6] to in-context learning by adding k demonstrations (DNA sequence and its label) for each class of a dataset as input to the model. To indicate classes, we make use of HyenaDNA 's existing vocabulary by indicating classes with specific nucleotides. For binary classification, we indicate classes with the nucleotides "A" and "N", while additionally utilising nucleotide "G" for three-way classification. During model tuning, we thereby optimise the same next-nucleotide prediction loss as used during pretraining. See Table A.7 for an overview of the optimisation settings.

Enhancers-Cohn

Nontata-Promoters

Enhancers-Ensembl

Ensembl-Regulatory

Eval.Accuracy

0.75

shots

0.50

0

0.25

2

16

0.00

32

0

100

200

0

100

200

0

100

200

0

100

200

0

100

200

#Tuning samples

#Tuning samples

#Tuning samples

#Tuning samples

#Tuning samples

Soft prompting details For each dataset, we prepend a sequence of n ( 2 to 32 k) learnable tokens T e ∈ R n × d , each of dimension d , to the input sequences X of the model: { T e , X, SEP } , where "SEP" indicates the separation token. We optimise these tuneable tokens for a maximum of 20 training epochs on the dataset's training data while keeping all other model parameters fixed. We stop training early if the model's validation loss does not improve for two epochs. After this tuning phase, we evaluate the model's performance on the dataset's full validation data. For an overview of the results of this experiment, see Fig. 4.2 of the main text.

Few-shot prompting details For each dataset, we prepend a set of k ( 0 to 32 , 0 indicates regular fine-tuning) examples of each class of a dataset (so-called "shots") to an input sequence:

where X i indicates an example sequence of class i with label Y i (exemplified for a two-way classification task). We tune the model on n ( 2 to 256 ) such k -shot samples before evaluating its performance on the dataset's full validation data. For an overview of the results of this experiment, see Fig. A.1.

A.4 Chromatin Profile Details

Background Variations in non-coding regions of the genome account for the majority of disease and other trait-associated single-nucleotide polymorphisms (SNPs). For example, whilst not directly altering the sequence of an encoded protein, a SNP in a non-coding region can affect the expression of downstream genes by inducing a change in the epigenetic state [56]. Therefore predicting epigenetic markers from a given sequence is an important task in the context of quantifying the functional effects of non-coding variants. Previously DeepSEA [57], a deep convolutional sequence model, has been introduced to predict chromatin features directly from non-coding sequences.

Data The authors of DeepSEA [57] compiled a dataset of 919 chromatin features from [15] and [42] including 690 TF binding profiles for 160 different TFs, 125 DHS and 104 HM profiles. The original DeepSEA dataset consists of 1000 base pair (bp) sequences from the hg19 human reference genome [8] with corresponding 919-dimension multi-label target vectors. Each label corresponds to the presence/absence of a peak in a given chromatin feature within the central 200 bp region of the sequence. The 400 bp flanking regions of the sequence provide broader contextual information which is beneficial to the task. Training and testing sets are split by chromosome and are strictly non-overlapping. In total, there are 2.2 M training samples and 227,512 samples from chromosomes 8 and 9 are held-out for testing. We use the DeepSEA chromatin profile prediction task to evaluate HyenaDNA models with varying context window. We use LiftOver [26] to convert the original DeepSEA dataset to hg38 coordinates and expand flanking regions about the central 200 bp bin symmetrically up to 8000 bp. Approximately 0.5% of samples are filtered in cases where LiftOver fails or the resulting translated sequence has a different length.

Model We fine-tune several models consisting of a pretrained HyenaDNA encoder, a sequencelevel pooling layer and a fully-connected decoder to perform multilabel sequence classification. We compare HyenaDNA against benchmarks set by DeepSEA, a convolutional sequence model, and BigBird [55], a sparse attention based language model. The authors of BigBird fine-tune on the DeepSEA dataset with input sequences extended to 8000 bp (asymmetrically about the centerpoint by -5000 and +3000 bp). Notably BigBird utilizes a byte-pair encoding tokenization scheme whereas HyenaDNA uses a single-character tokenizer and DeepSEA uses one-hot encodings. For the shortest range model (1k), we average across all tokens to perform sequence-level pooling. Whereas in the longer context model (8k) we find that extracting the last token in the sequence as the input to the fully-connected decoder performs better. We also find that for the longer context

model using an encoder pretrained on sequences larger than those used in fine-tuning was beneficial. The hyperparameters of the models used in these experiments are shown in Table A.8. Note that we reduced the depth and of models with increasing context window due to limitations on compute cost/time.

Results The performance of the fine-tuned HyenaDNA models are summarised in Table 4.3. We find that the smallest sequence length model (1024 bp) outperforms both DeepSEA and BigBird on TF and DHS prediction. We find that the model pretrained on 32k sequences with only 4 layers and fine-tuned on 8k sequences outperforms BigBird on the long range HM task but suffers from degraded performance on the short range tasks. However, we postulate that this performance loss may be recovered by increasing the depth of the model. We also remark that our models contain 5-30 × fewer parameters compared to DeepSEA and BigBird.

A.5 Biotype Embeddings Analysis Details

Background Sequence embeddings are useful in reducing dimensionality and capturing semantic relationships into fixed length vectors. We analyze pretrained embedding quality from HyenaDNA and show that it learns biologically informed features. We utilize linear probing, freezing the weights on a pretrained model and attaching a linear classification head to predict biotype sequences. We also use t-SNE to visualize clusterings that emerge from the embeddings.

Data The Ensembl database [9] is a comprehensive resource for gene and transcript annotations such as biotypes. Ensembl biotypes are a classification system, based on a combination of experimental evidence and computational predictions, that summarises the high-level functional properties of genes and transcripts. For example, biotype classes may annotate whether a gene is protein-coding or encodes a long non-coding RNA; if a gene is a disrupted homologue of a known protein coding gene (pseudogene) and by what mechanism it is produced; or the role of a small non-coding RNA such as post-transcriptional modification of other RNAs in the cell nucleus. We use biotype annotations to qualitatively visualize the clustering of gene embeddings into functional groups. We construct a multi-classification task using the top 10 most frequent biotype annotations as multi-class target labels which we predict from the unsupervised embeddings to assess how well biological function is encoded in the embedding space.

Model & Training We use a frozen pretrained HyenaDNA model consisting of 8 layers and width 256 pretrained on sequences of length 160k. To extract sequence-level embeddings, we average along the sequence dimension in the final encoder layer. For comparison we also construct embeddings using DNABERT (5-mer) and Nucleotide Transformer. We construct embeddings for genes in the Ensembl dataset up to a length of 160k. For genes with sequence lengths exceeding the context

window of the encoder, we chunk the sequence and average the embeddings over the chunks. We utilize an XGBoost [7] classifier to perform the supervised multi-classification task on the embeddings. The hyperparameters used are shown in Table A.9.

Table A.9: Hyperparameters. Overview of XGBoost hyperparameters used in biotype multiclassifier.

Results As shown in 4.4, HyenaDNA achieves the highest F1 score on the biotype classification task indicating that its embeddings contain features that are informative of biological function. Notably, HyenaDNA achieves this using the much smaller embedding space dimension of 256, compared to DNABERT and Nucleotide Transformer, which produce embeddings of dimension 1029 and 1280, respectively.

A.6 Long-range Species Classification Details

Background Given a genetic sequence randomly sampled from a set of different species, successful identification of the source species requires a model to learn a distinct mutational profile for each species. The more locations for discriminative mutations a model can consider, the more successful it should be at this task. We can arbitrarily tune this task's difficulty by including a higher number of species or increasing the evolutionary similarity of the included species, and thus it represents a helpful setting for measuring long context reasoning abilities for DNA sequence models.

Data We select five species for this task: human ( homo sapien ), lemur ( lemur catta ), mouse ( mus musculus ), pig ( sus scrofa ), and hippo ( hippopotamus amphibius ). We hold out four chromosomes from each species (chromosome numbers 1, 3, 12, and 13) for evaluation, and use the rest of each species' chromosomes for training.

Model We compare HyenaDNA against a baseline Transformer, which uses Flash Attention [11] in the mixing layer instead of a Hyena operator. We use 2 and 8 layer models, depending on sequence length. For HyenaDNA , we train on sequence lengths of 1k, 32k, 250k, 450k and 1M. For

Transformer, we limit sequence lengths to 1k and 32k due to the quadratic increase in training time, making training infeasible on our hardware. See Table A.10 for model sizes and hyperparamters.

Training We use pretrained models from 4.1, trained on various lengths between 1k to 1M nucleotides, and fine-tune them using a linear decoder head. We either pool across all tokens (1k and 32k models) or use the last token for classification (250k - 1M models). We randomly sample a (species, chromosome, sequence start, sequence end) tuple at each training step, with uniform probability across all species and non-held-out chromosomes. If a sequence's starting location on a chromosome is such that the end of that sequence would exceed the length of the chromosome, then we pad the sequence with N's to its full intended length. For evaluation, we randomly sample a (species, chromosome, sequence start, sequence end) tuple from our held-out evaluation set of chromosomes, and record the overall Top-1 5-way accuracy of our model (i.e. fraction of sequences correctly classified).

At sequence length 450k, we use the sequence length warm-up scheduler described in 3.2 on HyenaDNA . This involves gradually increasing the length of sequences fed to the model during fine-tuning from 1k to 450k. We observe better convergence and higher overall peak accuracy with this strategy, as shown in 3.2.

Pretraining ablation For species classification, pretraining becomes more important for longer sequences. This is in-line with our observation that for harder tasks (including longer sequences), pretraining becomes more important. At sequence length 250k and 450k, the scratch vs. pretraining gap is 30+ accuracy points.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose a new genomic foundational model that can be pretrained on the human reference genome. This model, called HyenaDNA, is built upon the previous model Hyena. The advantage of this model is that it can train on ultra-long sequences (up to 450k) at single nucleotide resolution with significantly fewer parameters. The authors introduce a warm-up technique to stabilize the training procedure, and they also employ a soft prompting method for downstream tasks. The authors show the efficacy of their framework through various experimental settings.\n\nSTRENGTHS:\n1. This work tries to solve an important problem: developing new foundational models for DNA sequences. \n2. Previous genomic foundational models are usually only able to be trained on relatively short sequences (<10kb). It is hard for them to deal with ultra-long sequences, which is necessary to capture long-range dependencies. This work provides a strategy to train the model on these long sequences. \n3. The experimental results shown in the manuscript seem promising. \n4. Overall, the manuscript is well written.\n\nWEAKNESSES:\n1. It is a bit difficult to understand 3.1 especially for people who are not familiar with the previous Hyena manuscript. $x_1$, $x_2$, $v$, $W_{x_{1}}$, $W_{x_{2}}$, $W_{v}$ are not defined in the main text. Line 159, not sure “input time” refers to. Line 162, not sure “projections” refer to. \n\nSome minor points:\n1. Section 2.1, no definition of D, is it equal to 4 representing {A,T,C,G}?\n2. Line 320, A.5 should be A.4\n3. Line 350, A.4 should be A.6\n4. Appendix line 613, Fig. 4.2 should be Fig 4.1. \n5. Line 206 equation, equation number missing\n\nQUESTIONS:\n1. Line 206, it is not so clear what the embedding step refers to. Also, perhaps to double check the correctness of $x\\in\\mathbb{R}^{L\\times(T+N)}$\n2. It seems that for different experiments, the authors train different HyenaDNA (different hyper-parameters, or using input sequences with different lengths as the pretraining data). Is there any specific reason that the authors use different models instead of e.g. using the biggest model that can handle up to 450k context, for all the experiments? \n3. Table A.3, for different tasks the authors use different hyper-parameters, how these hyper-parameters be chosen? \n4. The authors indicate in the main text their framework is more computationally efficient than Nucleotide Transformer since it has a significantly smaller number of parameters. However, in the appendix the authors mention that for each new task the entire model needs to be fine-tuned. How efficient this fine-tuning is compared to the fine-tuning procedure used in Nucleotide Transformer? How many computer resources are required for fine-tuning HyenaDNA? \n5. The authors compare HyenaDNA with Nucleotide Transformer in 17 datasets. However, in the paper of Nucleotide Transformer there are 18 datasets in total. Why one dataset missing?\n\nLIMITATIONS:\nThe authors have pointed out some limitations of their work in the conclusion section. I don’t think there is any significant negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents HyenaDNA, an advanced genomic foundation model leveraging the Hyena language model's capabilities, which are based on implicit convolutions. The authors highlight the limitations of prior Transformer-based genomic models, which have been constrained by token lengths and therefore impeded accurate modeling of long-range genetic interactions. These models also relied on tokenizers or fixed k-mers, which resulted in loss of single nucleotide resolution and crucial genetic variations. HyenaDNA addresses these issues. The model's unique abilities, including its sub-quadratic scaling in sequence length, usage of single nucleotide tokens, and full global context at each layer, mark a significant advancement in genomics.\n\nSTRENGTHS:\nOverall, this is an excellent paper! I appreciate the application of the Hyena model to DNA sequences. The utilization of advanced sequence modeling to tackle significant challenges in biology is commendable.\n\nThe concept of soft prompting for long-context models is exceptionally innovative and interesting!\n\nWEAKNESSES:\n- To be frank, I believe the evaluation of short sequences may not be crucial for this paper, considering its primary focus on long-range modeling. I would suggest including a comparison with the Enformer model for a more comprehensive perspective. More benchmarks on the long-context task  are necessary. \n\n- Moreover, additional technical details should be incorporated either in the main text or the supplementary materials to facilitate a thorough understanding of the methodology used.\n\nSee more detail in the questions:\n\nQUESTIONS:\nI would raise my rating if the authors resolve my concerns. \n\n- On line 132 , it would be beneficial to showcase $ T_{i, j} $.\n- Could you elaborate a bit more on Figure 3.1, specifically, what the arrows represent?\n- Could you provide detailed information about the baseline CNN model used in the GenomicsBenchmark?\n- Are there any results from training the model from scratch? The connection between the pre-training goal and the downstream task doesn't seem very apparent.\n- How well did the CNN model perform in predicting on the Nucleotide Transformer benchmark?\n- On lines 259 - 260, how would training from scratch for each task fare? If the advantages of pre-training are minimal, using $3200 x$ less pretraining data doesn't present a significant benefit.\n- Could you detail the fine-tuning process in section 4.2?\n- For Figure 4.1, it would be helpful to incorporate the data from table 4.1 to display the \"upper bound\" performance.\n- It would be advantageous to always include a training-from-scratch CNN or dilated CNN on all tasks.\n- The species classification result is quite impressive! Would it be possible to apply any Explainable $\\mathrm{Al}$ (XAI) methods to discern the principles that the model has learned?\n- It would be beneficial to update the supplement to include more details about the Long Convolution. I had to refer to a previous paper to understand this concept.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces HyenaDNA, a genome foundation model based on the Hyena architecture that replaces attention layers with implicit convolutions.  Though being 2500 times smaller, it achieves better performance than the state-of-the-art model.\n\nSTRENGTHS:\n- The paper is clearly written.\n- The proposed method achieves very strong performance with very few parameters and is able to process ultra-long dna sequences.\n- This paper investigates in-context learning and tunable prompting to the area of dna language model and show great performance improvements.\n\nWEAKNESSES:\n- The comparison over baselines is unfair. \n    - Since Hyena is fundamentally different from the Transformer-based architecture, simply comparing the number of parameters does not accurate reflects the model efficiency. I think it would be more accurate the compare the wall-clock time and GPU time used in both pre-training and downstream evaluation. Actually, according to the figure A.2, the model is slower than its attention-based variants when inputs are shorted than $10^4$, which is very common in genome analysis tasks. Therefore, it is very interested to see how efficient the model is on each downstream task compared to DNABERT and Nucleotide Transformers.\n    - All the previous models use standard or parameter-efficient fine-tuning for downstream evaluation, I think it is important to also fine-tune HyenaDNA on each downstream tasks to show how good it is as a *backbone* model and for fair comparison with other models. Also, it is interesting to see how the baseline models performs with the  tunable prompting.\n\n- This paper introduces a very strong model with a series of fancy techniques, however, as an academical publication, it fails to provide enough insights and explanations to the community about the superiority of its performance.\n    - According to the Hyena paper, the Hyena architecture achieves a similar level of performance (slightly better) as the transformer-based model (e.g., GPT) with the same number of parameters in general language modeling. Therefore, the fact that it dominants transformer-based models in the dna language modeling indicates the existence of **fundamental errors** of all the existing works. If this is true, what are the errors? I can think of a few possibilities:\n        - Masked language modeling vs casual language modeling\n        - k-mer tokenization vs character-level tokenization\n        - attention mechanism fails to capture the semantics of DNA. (however, the AttnDNA performs very well)\n    - To be honest, I don't think any of the above can lead to such significant improvements. Thus, throughout ablation study and experiments under the same setting is necessary.\n\nQUESTIONS:\n- How does AttnDNA perform on the datasets used by Nucleotide Transformers?\n- Can you implement Nucleotide Transformer and DNABERT on the GenomicBank datasets for comparison?\n\nLIMITATIONS:\nIn sum, the work is technically sound and empirically strong. However, based on the above thoughts, I tend to reject it for now. If the concerns are clearly solved, I would be happy to recommend a strong acceptance.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors train the Hyena operator model on the human genome and adapt it to downstream tasks in computational biology.\n\nSTRENGTHS:\nLots of clever things about this work that I really like:\n\nVery good use case of Hyena model with very long-range dependencies\nCurriculum learning is very clever and makes sense. Figure 3.2 is great to show this\nI think Table 4.4 with the use of XGBoost is very simple but effective in getting a statistic about the utility of an embedding. With XGBoost, you are going to get a pretty reasonable and comparable every time you use it.\n\nThe model capacity shown by perplexity and sequence length is good.\n\nI really really like that it is able to work at single base pair resolution.\n\nI’m impressed and happy that section 4.4.3 uses held-out chromosomes for training and testing (rather than random sampling). Since you are working with multiple species, are there still some aspect of data leakage in train/test sets because of synteny?\n\nI think this is impressive being only trained on the human genome. I’m excited to see this model trained on all genomes!\n\nWEAKNESSES:\nI think the GenomicsBenchmark claims about SotA could be improved. Definitions of SotA should be based on architecture, not the datasets. Since the dataset is newer than previously benchmarked models, I would recommend the authors evaluate the GenomicsBenchmark across other SotA models or architectures (like DNABERT and Nucleotide Transformer, which are mentioned in this paper). Alternatively the SotA claim is weak at best, and misleading at worst.\n\nI think section 4.4.3 could have a very simple baseline of a kmer count model. Can a simple kmer based model do just as well, or poorly? I’m sure it is very fast.\n\nQUESTIONS:\nFigure 3.2 - Do longer sequences simply take longer to do a forward and backward pass? You show wall time. Are the number of steps relatively closer?\n\nHow much is the result of your model’s result is its architecture versus using tunable prompting? What would happen if you used tunable prompting for the Nucleotide Transformer?\n\nHow are the F1 and MCC scores for Table 4.2 generated? Which values are you comparing?\n\nHow was the finetuning in section 4.4.1 done? Is it not soft-prompting?\n\nLIMITATIONS:\nGenerally, I find the term “Foundation Model” to be ambiguous and unnecessarily buzzwordy. HyenaDNA is a large, pretrained model. Even though the authors have pretrained the model, it has to be finetuned and quite heavily engineered differently for each subtask. But, I guess this is what the field is wanting to say.\n\nWhile I do like that the important aspects of the paper are emphasized, some of the color, boldness, different font, and italicization can be toned down a bit. For example: “Therefore, having both long-range context and single nucleotide resolution simultaneously” doesn’t need to be blue and bolded. Also, what is the author’s intent in using blue text versus bold text?", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript applies Hyena, a neural operator based on implicitly parametrized long convolutions and data-controlled gating, to the domain of DNA modelling. \nThe subquadratic complexity of Hyena enables scaling to context lengths of up to 450,000 at single nucleotide resolution. \nThis represents a significant improvement over previous methods based on dense attention, increasing pre-training sequence length by more than two orders of magnitude.\nPre-training is done using a sequence length warm-up scheme, such that sequences grow longer as training progresses.\nThe resulting model is benchmarked on a variety of downstream tasks using different adaptation methods.\n\nSTRENGTHS:\n1. (**Originality**) HyenaDNA is the first work to apply Hyena, a novel breed of implicitly parametrized long convolutions, to DNA modeling.\n2. (**Originality**) The authors experiment with two adaptation strategies commonly used in NLP to apply the pre-trained model to novel tasks.\n3. (**Quality**) The method is evaluated on a large number of genomic datasets introduced as part of related work. This firmly situates HyenaDNA within the existing literature.\n4. (**Significance**) The proposed model outperforms state-of-the-art attention-based methods on the majority of presented downstream tasks with considerably fewer parameters and scales to extremely large context sizes of 450,000 tokens at single nucleotide resolution.\n5. (**Clarity**) Overall, the manuscript is written clearly, and the arguments are easy to follow.\n6. (**Clarity**) Architecture and hyperparameter details are clearly presented.\n7. (**Clarity**) Including background task descriptions for the Nucleotide Transformer, the Biotype Classification, and the Chromatin Profile Prediction benchmarks are helpful for understanding the evaluation protocol.\n\nWEAKNESSES:\n1. (**Originality**) Beyond the application of Hyena to the DNA domain, the manuscript offers few new technical insights that could be of interest to the machine learning community. Sequence length warm-up is an established technique to stabilize training instabilities, and soft prompting, as well as few-shot fine-tuning, are commonly used adaptation strategies in the natural language processing community.\n2. (**Quality**) Results are presented without any measure of deviation over multiple repetitions. This is true even for smaller scale fine-tuning experiments.\n3. (**Quality**) For the GenomicsBenchmark, the authors should report more details about the CNN baseline (parameter count, number of layers, etc.), as well as training details and hyperparameters for AttnDNA. If the same hyperparameters are used for both HyenaDNA and AttnDNA, please state this explicitly.\n4. (**Clarity**) Few-shot adaptation includes a tuning phase, making this adaptation method more similar to few-shot fine-tuning than in-context learning. Relevant literature should be cited.\n5. (**Clarity**) The results reported in Table 4.1 for the Nucleotide Transformer do not seem to match with those reported in the original paper in Supplementary Table 6. Please explain this deviation.\n\nQUESTIONS:\n- Why is HyenaDNA pre-trained autoregressively rather than non-autoregressively, as was done in Nucleotide Transformers? Both methods would be feasible since downstream adaptation of HyenaDNA always includes a fine-tuning phase.\n- Considering the limited vocabulary of just 4 nucleotides, it's somewhat unexpected that the reported perplexity score for a high-performing long-context model like HyenaDNA would be this high. Could you provide some insight on this? To underscore the significance of pre-training further, it would be valuable if you could supplement the evaluation of pre-trained embeddings using probing classifiers with an additional experiment where HyenaDNA is trained directly from scratch on the downstream datasets.\n- The manuscript mentions that both a long context and single nucleotide resolution are crucial. However, K-mer tokenization could help reduce the temporal dimension, and previous work has shown that a higher k leads to improved results (Ji et al., 2021). Therefore, it would be beneficial to include an experiment that compares HyenaDNA models pre-trained with different tokenization schemes.\n\n### References\n- Ji, Y., Zhou, Z., Liu, H., & Davuluri, R. V. (2021). DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformatics, 37(15), 2112-2120.\n\n### Minor comments\n\n- Possible Typo in Table 4.1 row 2 (Coding vs. Intergenomic): the improvement over baseline performance (+3.5) does not match with the reported base performance. Both Base and HyenaDNA are reported to have the same accuracy. \n- Typo on line 543 : sequence-level instead of sequence-leel\n- Typo on line 544: that instead of taht\n\nLIMITATIONS:\nThe authors address the limitations of their method and experiments in the manuscript.\nPossible malicious uses of HyenaDNA are not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}