SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Eric Nguyen\*, Michael Poli\*, Marjan Faizi\*, Armin W. Thomas\*, Callum Birch Sykes\*, Michael Wornow\*, Aman Patel\*, Clayton Rabideau\*, Stefano Massaroli\*, Yoshua Bengio\*, Stefano Ermon\*, Stephen A. Baccus\*, Christopher Re\*,†
Genomic (DNA) sequences encode an enormous amount of information for gene regulation, protein synthesis, and numerous other cellular properties. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context ( $<0.001\%$  of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution (i.e. DNA "characters") where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to  $500x$  increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to  $160x$  faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics for simple adaptation to novel tasks without updating pretrained model weights. On a long-range species classification task, HyenaDNA is able to effectively solve the challenge by increasing the context length to 1M without downsampling. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data. On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points, and by as much as +20 accuracy points on enhancer identification. Code available at https://github.com/HazyResearch/hyena-dna.
*Equal contribution. † Equal senior authorship. 1Stanford University. 2Harvard University. 3SynTensor. 4Mila and Université de Montréal.
${}^{2}$  On benchmarks from Nucleotide Transformer,HyenaDNA uses a model with  ${1500}\mathrm{x}$  fewer parameters (2.5B vs 1.6M) and  ${3200}\mathrm{x}$  less pretraining data (3202 vs 1 human reference genome).
Figure 1.1: HyenaDNA recipe for long-range foundation models in genomics. The HyenaDNA architecture is a simple stack of Hyena operators [37] trained using next token prediction. (See Fig. 1.3 for block diagram of architecture). We introduce a new sequence length scheduling technique to stabilize training, and provide a method to leverage the longer context length to adapt to novel tasks without standard fine-tuning by filling the context window with learnable soft prompt tokens.
Understanding and learning from DNA sequences has long been a goal of biologists and deep learning researchers, as its "language" encodes instructions essential for all living things [16]. The mapping from DNA instructions, genotypes, to observable function and traits, phenotypes, remains ongoing research effort. Towards this goal, researchers have proposed using foundation models (FMs) in genomics to learn generalizable features from unstructured whole genome data that can then be fine-tuned for a number of tasks including predicting the location and function of genes, identifying regulatory elements, and analyzing the evolution of species [25, 10, 20, 3, 53, 58]. In contrast to protein sequences, which have had successes in protein language models [29, 31, 32, 17, 5, 41, 14], DNA sequences are orders of magnitudes longer (e.g. the human genome is 3.2B nucleotides) with long-range dependencies and interactions that span over  $100\mathrm{k}+$  nucleotides in length [1]. Overcoming the long-range limitations of current generation models could help drive the next wave of innovations in AI-powered drug discovery and therapeutics, and enable genomic FMs to understand and learn in-context whole patient genomes in a personalized way.
Limitations of current models Previous genomic FM approaches have relied on attention-based Transformers [25, 10, 53, 58], but face a number of challenges unique to DNA sequences. The attention mechanism scales quadratically in sequence length, with current genomic FMs pretraining on only 512 to 4,096 tokens as context [25, 58, 10, 55],  $< 0.001\%$  of the human genome. Also prevalent is the reliance on fixed k-mers, akin to DNA "words", and tokenizers to aggregate meaningful DNA units. However, single nucleotide alterations represent physical analogs where, for example, single nucleotide polymorphisms (SNPs) and mutations can have a profound impact on biological properties including regulatory activity [33]. In contrast, natural language semantics can often be con
served when single character or word changes occur over very long contexts. Therefore, having both long-range context and single nucleotide resolution simultaneously is critical, and remains a particular challenge in genomics.
PPL vs Context on the Human Genome
Figure 1.2: Pretraining on the human reference genome using longer sequences leads to better perplexity (improved prediction of next token).
Figure 1.3: HyenaDNA block architecture. A Hyena operator is composed of long convolutions and element-wise gate layers. The gates are fed projections of the input using dense layers and short convolutions. The long convolutions are parameterized implicitly via an MLP that produces the convolutional filters. The convolution itself is evaluated using a Fast Fourier Transform convolution with time complexity  $\mathcal{O}(L\log_2L)$ .
Toward longer context models Recently, Hyena [37], a large language model based on implicit convolutions, was shown to match attention in quality while reducing computational time complexity, thereby allowing a longer context to be processed. Hyena uses a parameter-efficient global convolutional filter along with a data-controlled gating mechanism, which enables a context-specific operation over every token. Indeed, Hyena showed that for simple associative recall tasks using synthetic data, a shallow 2 layer model could effectively process context lengths at 131k tokens. We hypothesize that Hyena's core operations can unlock the potential to capture both the long-range and single nucleotide resolution of real genomic sequences over attention-based approaches. To test this, we explore two questions: (i.) Can a convolutional long-context model be used effectively at single nucleotide resolution? (ii.) What new capabilities could long-context genomic foundations models enable?
HyenaDNA The result of our investigation is HyenaDNA, a genomic FM pretrained on the human reference genome at context lengths up to 1 million tokens at single nucleotide resolution - an up to  $500\mathrm{x}$  increase over existing genomic FMs using dense-attention. HyenaDNA scales subquadratically in sequence length (training up to  $160\mathrm{x}$  faster than attention at sequence length 1M), uses single nucleotide tokens, and has a global receptive field at each layer. Our contributions include a "full-stack" recipe for building genomic FMs, including architecture design, a warm-up schedule to speed up training on ultralong sequences, and an efficient downstream adaptation procedure based on soft prompting and in-context learning.
Full-stack genomics modeling We start with a decoder-only Hyena architecture pretrained using next nucleotide (token) prediction. We forego standard aggregating tokenizers, using a single-character tokenizer and a minimal DNA vocabulary of 4 nucleotides (plus special tokens). Training stability becomes an issue at ultralong sequences  $(200\mathrm{k} + )$ . To overcome this issue, we introduce a sequence length warm-up scheduler that gradually increases sequence length in stages. At sequence length  $450\mathrm{k}$ , training time is reduced by  $40\%$ , while boosting accuracy by 7.5 accuracy points on a species classification task. Furthermore, we design downstream adaptation procedures to leverage longer context windows, as simpler and more flexible alternatives to standard fine-tuning in genomics. This includes a novel soft prompt technique where learnable tokens (up to  $32\mathrm{k}$ ) are injected directly into the input sequence itself, enabling competitive downstream results without the need to update a pretrained model.
Genomic downstream tasks We apply our pretrained HyenaDNA models to 29 diverse downstream genomic tasks to showcase its long-range ability as well as fine-grain resolution. On fine-tuned benchmarks from the Nucleotide Transformer [10], HyenaDNA achieves state-of-the
art (SotA) on 12 of 18 datasets while using a model with orders of magnitude less parameters and pretraining data (see Tab. 4.2). On the GenomicBenchmarks [23], HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points, and by as much as +20 accuracy points on enhancer function identification. On a novel species classification task, HyenaDNA effectively solves the challenge by increasing the context length to 1 million tokens. In a challenging chromatin profile experiment, a 919-way multi-task, HyenaDNA performs competitively against a larger SotA sparse-attention BigBird Transformer [55]. Finally, we analyze the learned embeddings of a pretrained HyenaDNA model by clustering sequences by biotype (gene or transcription type) and compare the results with existing genomic FMs, showing that HyenaDNA can serve as an effective universal featurizer in genomics.
2.1 Transformers and Attention
Powering many recent foundation models is the attention mechanism. Given a length- $L$  sequence  $x \in \mathbb{R}^{L \times D}$ , a (single-headed) layer of scaled self-attention [2, 51] is a map from  $\mathbb{R}^{L \times D}$  to  $\mathbb{R}^{L \times D}$  which performs the following operations:
$$
\mathsf {A} (x) = \sigma \left(x \mathsf {W} _ {q} \mathsf {W} _ {k} ^ {\top} x ^ {\top}\right), \quad y = \mathsf {A} (x) x \mathsf {W} _ {v} \tag {2.1}
$$
where  $D$  is the embedding dimension,  $\mathsf{W}_q,\mathsf{W}_k,\mathsf{W}_v\in \mathbb{R}^{D\times D}$  are learnable linear maps and  $\sigma$  indicated row-wise softmax (and optional scaling). Attention computes all pair-wise comparison for every token, and scales as  $\mathcal{O}(L^2)$  in sequence length. This allows a global context at high resolution, but limits the size of the context on current hardware.
Previous methods to reduce the quadratic cost of attention have used specialized methods to approximate full dense attention [18]. In sparse attention, elements attend only to a subset of all other positions. Alternatively, linear attention methods construct approximations to  $A(u)$  that can be evaluated in subquadratic time. Both of these classes of methods, however, trade lower time complexity (allowing longer sequences) for loss in expressivity.
2.2 Long Context Strategies in Genomics
To achieve longer context, genomic models have relied on two strategies: i. tokenization and ii. dilation and downsampling. Tokenization is a necessary step in masked language modeling (MLM) with bidirectional Transformer architectures (BERT) [13], a common model in genomics. These tokenizers use fixed k-mers (short overlapping sequences of length  $\mathrm{k}$ ) or frequency-based byte pair encoding (BPE), that attempt to aggregate DNA into meaningful units [25, 55]. Consequently, these aggregation techniques create large new vocabularies (compared to the natural vocabulary of 4 nucleotides) that are less generalizable [49]. The second strategy uses dilated convolutions and downsampling, both of which essentially average or skip elements between weights [18]. A canonical example is the Enformer, which uses dilation and downsampling to reach context lengths of 100k nucleotides to predict gene expression tracks [1]. Common across tokenization, dilation, and downsampling is the sacrifice of single nucleotide resolution to reach longer context.
2.3 Large Convolutional Models
A discrete convolution between an input  $x$  of length  $L$  and a (learnable) filter  $h$  is given by:
$$
y _ {t} = (h * x) _ {t} = \sum_ {t ^ {\prime} = 0} ^ {L - 1} h _ {t - t ^ {\prime}} x _ {t ^ {\prime}} \quad \text {o r e q u i v a l e n t l y} \quad y = \mathsf {T} x. \tag {2.2}
$$
where  $\mathsf{T} \in \mathbb{R}^{L \times L}$  is the Toeplitz matrix corresponding to the convolution. Historically, convolutions have played an important role in deep learning and more broadly signal processing. More recently, it has been shown that by stacking  $k$  long convolution layers, where  $k$  is parametrized through a function  $\gamma_{\theta}$  i.e.  $k \coloneqq \gamma_{\theta}(L)$ , one can achieve state-of-the-art performance on a variety of benchmarks involving long sequences, for example the Long Range Arena (LRA) [48, 24, 47, 19]. Different  $\gamma_{\theta}$  have been proposed in the literature: state-space models [24, 19], and implicit parametrizations via neural fields [45, 44, 37]. On language, the H-family of implicit convolution language models,
H3 and Hyena, [12, 37] used long convolutions and gating to match Transformer performance in  $\mathcal{O}(L\log_2L)$  time, notably lower than the  $\mathcal{O}(L^2)$  of attention-based models.
HyenaDNA takes inspiration from these approaches, showing that attention-free, long-context causal models can achieve high performance on downstream genomic tasks. These extended long-range capabilities enable us to explore new paradigms in genomics, such as in-context learning to easily adapt to new tasks without updating pretrained models.
3 HyenaDNA Long-Range Genomic Foundation Models
In this section, we introduce the HyenaDNA approach to long-range genomic sequence modeling. We start with a description of the model architecture, then discuss sequence length warm-up and soft prompting techniques for downstream adaptation.
3.1 The HyenaDNA Model
The HyenaDNA model is a decoder-only, sequence-to-sequence architecture defined by a stack of blocks consisting of a Hyena operator [37], followed by a feed-forward neural network (see Fig. 1.3).
Given an input  $x\in \mathbb{R}^L$  ( $L$  denotes sequence length), a Hyena operator can be defined as:
$$
(x _ {1}, x _ {2}, v) \mapsto \mathsf {H} (x _ {1}, x _ {2}) v
$$
$$
\mathsf {H} \left(x _ {1}, x _ {2}\right) = \mathrm {D} _ {x _ {2}} \mathrm {T} _ {h} \mathrm {D} _ {x _ {1}} \tag {3.1}
$$
where  $x_{1}, x_{2}, v$  are projections of the input, and  $\mathsf{T}_h \in \mathbb{R}^{L \times L}$  is the Toeplitz matrix constructed from a learnable long convolution filter produced as the output of a neural network,  $(\mathsf{T}_h)_{ij} = h_{i-j}$ . The convolution filter values themselves are obtained through a small neural network  $\gamma_{\theta}$  taking as input
Figure 3.1: The Hyena operator is a combination of long convolutions  $\mathsf{T}$  and data-controlled gating  $\mathsf{D}$ , and can be a drop-in replacement for attention.
the time (position) index and optionally positional encodings,  $h_t = \gamma_\theta(t)$ , which enable the operator to process very long sequences without growing linearly in the number of parameters. Further, the matrices  $\mathsf{D}_{x_1}, \mathsf{D}_{x_2} \in \mathbb{R}^{L \times L}$  are constructed with  $x_1, x_2$  on the diagonals, and evaluated as element-wise gating. The projections are obtained by applying a dense linear layer and short convolution to the input sequence, as shown in Figure 3.1.
Proposition 3.1. A Hyena operator can be evaluated in  $\mathcal{O}(L\log_2L)$  time.
Efficient evaluation is crucial on settings involving extremely long sequences such as genomics. In the general case where the embedding dimension  $D > 1$  and  $x \in \mathbb{R}^{L \times D}$ , the linear projections  $\mathsf{W}_{x_1}, \mathsf{W}_{x_2}, \mathsf{W}_v \in \mathbb{R}^{D \times D}$  are right multiplied to  $x$ , and  $D$  independent Hyena operators are then applied to each dimension.
3.2 Training Long Sequence Models
Tokenization The subquadratic cost of HyenaDNA in sequence length allows the model to process ultralong sequences directly at the single nucleotide level without the need for frequency-based aggregation tokenizers. This enables fine-grain resolution for both short and long sequences, critical for detecting single nucleotide polymorphisms or mutations and modeling long-range dependencies in gene expression.
We use the natural DNA vocabulary and refer to each nucleotide as a token. The tokens include "A", "G", "C", "T", and "N" (a non-specific nucleotide) and special character tokens for padding, separation, and unknown characters. Tokens are mapped to embedding dimension  $D$ .
Sequence length warm-up for ultralong sequences Directly training on long sequences can affect training stability as the variance in gradient increases [28]. Training on shorter sequences initially (followed by longer sequences) was used by [38] to train small scale Transformers and reduce training time, while [28] used sequence length warm-up to address stability on up to 2k tokens.
<sup>3</sup>We discuss  $D = 1$  and order 2 Hyena operators for simplicity.
For ultralong sequences  $(200\mathrm{k}+)$ , we develop a new warm-up schedule that gradually increases the sequence length in stages to improve both stability and decrease training time.
Our sequence length schedule starts at  $L_{1} = 64$ , then doubles the window at each stage while keeping the global batch size constant. By doing so, iterations at each consecutive stage will include more tokens, ensuring the scheduler can also act as a form of batch size warm-up. In Fig. 3.2, we observe sequence length scheduling to be particularly important at sequence lengths greater than  $450\mathrm{k}$ , where at this length training time is reduced by  $40\%$  and improving ultimate accuracy by  $7.5\%$  points for a species classification task described later in section 4.4.3.
Figure 3.2: Sequence length warm-up reduces the training time of HyenaDNA at sequence length  $450\mathrm{k}$  by  $40\%$  and boosts accuracy by 7.5 points on species classification.
3.3 Downstream Adaptation
Tuneable prompting for long-context models
Prompts have been traditionally used to guide the output of a FM [30] by prepending additional context to an input. Expanding on this approach, soft tuneable prompting was introduced to inject learnable tokens (as weights) into the input directly [27] as an alternative to model fine-tuning.
With an extended context length  $(L)$ , we're able to explore new paradigms in adapting FMs after pretraining. Given a downstream task with prompts  $x_{p} \in \mathbb{R}^{T}$  and corresponding labels  $y_{p}$ , we pretend  $N \leq L - T$  trainable parameters  $\theta$  of dimension  $D$  after the embedding step:
$$
x \leftarrow \operatorname {c o n c a t} \left[ \operatorname {e m b e d} \left(x _ {p}\right), \theta \right], \quad x \in \mathbb {R} ^ {L \times (T + N)} \tag {3.2}
$$
The resulting sequences  $x$  are then processed by the model, and  $\theta$  is optimized on a loss function involving the input sequence's label  $y_{p}$ . Crucially, soft prompting requires utilization of a small subset of prompt and label pairs to optimize  $\theta$ .
During soft prompting, HyenaDNA only optimizes the parameters of the prompt in the input sequence while keeping all other model parameters fixed. Soft prompting thereby provides a flexible and computationally efficient approach to adapting genomic FMs to new downstream tasks.
In 4.1, we start with pretraining HyenaDNA on the human reference genome [22]. We then evaluate HyenaDNA on existing short-range (<5k nucleotides) downstream benchmarks in 4.2 to assess the performance of single nucleotide resolution. In 4.3, we explore what new capabilities emerge with longer range genomic modeling in the form of in-context learning. Finally, we push the limits of ultralong context performance in 4.4.
4.1 Pretraining on the Human Genome
We pretrain HyenaDNA on the human reference genome [22] using next nucleotide (token) prediction. Starting with a stack of decoder-only Transformer blocks, we swap attention for the Hyena operator, and compare against a baseline Transformer (GPT) with Flash Attention [11]. We add gradient checkpointing to HyenaDNA to decrease the memory footprint by  $3\mathrm{x}$  on longer sequences  $(>160\mathrm{k})$ . We then scale HyenaDNA along dimensions of model depth (2 to 8 layers), width (128 to 256 dimensions), and sequence length (1024 to 1M). At sequence length 1M, HyenaDNA is  $160\mathrm{x}$  faster than its Transformer counterpart as shown in Fig. 4.1.
As shown in Fig. 1.2, we observe that as context length increases, perplexity improves during pretraining. However, this improvement comes at the expense of more training time and tokens. For models too shallow to effectively process longer context, perplexity can begin to degrade (increase), observing inflection points with longer sequences. In this way, increasing context can serve as a
novel regularization dimension. For genomic pretraining, we provide the following guidelines. 1. In optimizing for faster training time, shorter context enable lower perplexity to be reached faster. 2. In optimizing for best overall perplexity, longer context allows for lower perplexity at the cost of training on more tokens. See A.1 for experiment details.
4.2 Single Nucleotide Resolution
Our first downstream tasks use short-range genomic sequences  $(< 5\mathrm{k})$  aimed at evaluating single nucleotide resolution performance on sequence-level classification using standard fine-tuning.
GenomicBenchmarks We start with the newly released GenomicBenchmarks [23], which is comprised of 8 regulatory element classification datasets with sequence lengths of 200-500, and one up to 4,776. The original baseline model uses a short-range CNN. We fine-tune the pretrained Transformer (GPT) and HyenaDNA from 4.1, both having single nucleotide resolution, as well as the DNABERT model [25]. HyenaDNA sets a new SotA on 7 of 8 datasets and by up to  $20\%$  points on the human enhancer identification
task, as shown in Tab. 4.1. See A.2 for additional experiment details and ablations.
Figure 4.1: Runtime (forward & backward pass) for Transformer & HyenaDNA: 2 layers, width=128, gradient checkpoint, batch size=1, A100 80GB. At 1M tokens HyenaDNA is 160x faster than Transformer.
Nucleotide Transformer Next, we benchmark against 18 datasets from the Nucleotide Transformer (NT) [10], which includes predicting regulatory elements for enhancers, promoters, epigenetic marks, and splice sites from DNA sequences of length 200-600 nucleotides. We compare against 3 NT base models, which were pretrained using masked language
Table 4.1: GenomicBenchmarks Top-1 accuracy (%) for pretrained HyenaDNA, DNABERT and Transformer (GPT from 4.1), and the previous SotA baseline CNN (scratch).
DATASET	CNN	DNABERT	GPT	HYENADNA
Mouse Enhancers	69.0	66.9	80.1	85.1
Coding vs Intergenomic	87.6	92.5	88.8	91.3
Human vs Worm	93.0	96.5	95.6	96.6
Human Enhancers Cohn	69.5	74.0	70.5	74.2
Human Enhancers Ensembl	68.9	85.7	83.5	89.2
Human Regulatory	93.3	88.1	91.5	93.8
Human Nontata Promoters	84.6	85.6	87.7	96.6
Human OCR Ensembl	68.0	75.1	73.0	80.9
modeling (BERT) and then fine-tuned. The NT models ranged from 500M to 2.5B parameters, and pretrained on up to 3202 genomes. All NT models use 6-mer sequences of 1000 tokens long. For HyenaDNA, we attach a linear decoder head and fine-tune a pretrained model, surpassing SotA on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data, shown in Tab. 4.2. See A.2 for additional experiment details and ablations.
4.3 In-context Learning for Genomic Sequences