SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A.2.3 Downstream prediction tasks for Nucleotide Transformer benchmark
Following the Nucleotide Transformer [10], we collected datasets from four different sources [21, 35, 34, 46].
Promoter The promoter dataset included TATA-box-containing and TATA-box-lacking promoters. Tasks involved predicting promoters with a TATA-box, identifying promoters lacking a TATABOX, and distinguishing between both promoter categories and non-promoter sequences. The promoter datasets were obtained from the Eukaryotic Promoter Database (EPDnew) for human and mouse genomes. Promoter sequences were extracted from regions 249 nucleotides upstream and 50 nucleotides downstream of the transcription start sites.
Enhancer For the enhancer prediction task, we used the dataset from [21] containing DNA sequences classified into strong enhancers, weak enhancers, and non-enhancers. The tasks involved binary classification to distinguish enhancer sequences from non-enhancer sequences and identify specific enhancer types.
Epigenetic Marks In the epigenetic marks prediction task, we used the dataset from [35, 36] to predict nucleosome occupancy and modification states in the yeast genome. In 10 binary classification tasks, the model had to discriminate between DNA regions that were occupied by histones or not. The 10 tasks varied based on the types of histones investigated, including unmodified histones H3 and H4, as well as histones modified by either acetylation (H3K9ac, H3K14ac) or methylation (H3K4me1, H3K4me2, H3K4me3, H3K36me3, H3K79me3).
Splice Site For the splice site prediction task, DNA sequences from over 100 organisms were used to predict whether the sequences contain donor or acceptor splice sites [46]. Donor splice sites denote the beginning of an intron and acceptor splice sites the end of an intron. During RNA splicing, these sites are recognized by the spliceosome, a complex molecular machine that enables the removal of introns from the gene.
4https://epd.epfl.ch//index.php
Table A.5: Hyperparameter ranges used to fine-tune HyenaDNA for all Nucleotide transformer datasets. Exact hyperparameters per dataset can be found in our code repository.
HyenaDNA
Layers	2
Width	256
Parameters	1.6M
Optimizer	AdamW
Optimizer momentum	β1, β2 = 0.9, 0.999
Training epoch	100
Batch size	256-1024
Learning rate	2e-4 to 1e-3
LR scheduler	Cosine decay
Weight decay (model)	0-0.2
Weight decay (Hyena layers)	0
Embed dropout	0-0.2
Resid dropout	0-0.2
Reverse complement aug.	true/false
Sequence lengths	200-600
Preprocessing The Nucleotide Transformer study did not provide their exact train-test splits, except for the enhancer dataset. Therefore, we generated our own train-test splits using a 90:10 ratio. For the promoter dataset, negative samples were not available, and had to be generated following the procedure described by [34].
Model & Training For the architecture, we use a HyenaDNA model with 2 layers and width 256, and trained on sequences of length 1024. We average across the tokens to obtain a single classification token. For each task, we replaced the model head and fine-tuned the weights of the entire model (1.6M parameters). In contrast, the Nucleotide Transformer uses a parameter-efficient fine-tuning technique that introduces new weights and fine-tunes only the newly added weights, while keeping the initial model weights frozen, presumably due to its large size of 500M to 2.5B parameters. The corresponding HyenaDNA hyperparameter ranges used for training each task are reported in Table A.5.
A.2.4 Ablations on the Nucleotide Transformer benchmarks
We perform additional ablations on the Nucleotide Transformer benchmarks to assess the impact of pretraining, as well as attention vs. HyenaDNA, as shown in shown in Table A.6. We observed that pretraining has a greater effect on the more challenging tasks (and as sequences become longer, shown in A.11). On the more challenging tasks (histone marks, datasets starting with "H"), pretraining boosts HyenaDNA metrics by up to 21 MCC points on H3K4me3. For simpler tasks (with higher baseline scores) such as the splice sites and promoter tasks, the gain was lower (0 to 1 accuracy points), as these were already near saturation in performance.
A.3 In-Context Learning Details
Background A key premise of foundation models is that they are able to learn new tasks with little to no new training data [4]. Recent advances in language modeling have demonstrated that language foundation models can often adopt the behaviors necessary to perform new tasks in-context [6]. Here, information about the task that is to be performed, such as examples of respective inputs and targets, are added to the input of the model. By conditioning their prediction on the provided context, language foundation models are generally able to perform the task without any changes to their parameters.
A key challenge for in-context learning with HyenaDNA is its limited vocabulary, which is composed of only a few nucleotides, and does not provide any vocabulary for novel downstream tasks, such as class labels. To explore the potential for in-context learning in genomics, we use two variants of in-context learning, both using a brief tuning phase to introduce HyenaDNA to the concept of
Table A.6: Pretraining & Attention ablations on the Nucleotide Transformer (NT) benchmarks. The Matthews correlation coefficient (MCC) is used as the performance metric for the enhancer and epigenetic marks dataset, and the F1-score is used for the promoter and splice site dataset.
MODEL	NT	GPT	HyenaDNA	HyenaDNA
PARAMS	2.5B	1.6M	1.6M	1.6M
PRETRAIN	yes	yes	yes	no
Enhancer	58.0	59.3	62.6	58.6
Enhancer types	47.4	51.9	55.7	48.4
H3	81.4	75.8	81.7	79.9
H3K4me1	55.9	38.7	57.1	43.4
H3K4me2	32.6	28.8	53.9	34.5
H3K4me3	42.1	28.3	61.2	40.2
H3K9ac	57.5	49.2	65.1	52.6
H3K14ac	55.0	41.6	66.3	48.0
H3K36me3	63.2	47.8	65.3	53.4
H3K79me3	64.2	58.9	71.6	59.7
H4	82.2	77.7	79.6	79.1
H4ac	50.1	36.4	63.7	43.5
Promoter all	97.4	96.3	96.5	96.1
Promoter non-TATA	97.7	96.6	96.6	96.5
Promoter TATA	96.4	96.6	96.7	96.1
Splice acceptor	99.0	97.6	96.6	96.6
Splice donor	98.4	98.1	97.3	96.5
Splice all	98.3	98.0	97.9	97.3
classification with its existing vocabulary. As a test bed for this exploration, we use 5 datasets from the GenomicBenchmarks and a HyenaDNA pretrained on sequences of  $160\mathrm{k}$  length sequences.
In the first experiment, we apply a soft prompting approach [27] by adding a sequence of tuneable tokens to the input itself. In the second experiment, we explore a few-shot learning approach [6] to in-context learning by adding  $k$  demonstrations (DNA sequence and its label) for each class of a dataset as input to the model. To indicate classes, we make use of HyenaDNA's existing vocabulary by indicating classes with specific nucleotides. For binary classification, we indicate classes with the nucleotides "A" and "N", while additionally utilising nucleotide "G" for three-way classification. During model tuning, we thereby optimise the same next-nucleotide prediction loss as used during pretraining. See Table A.7 for an overview of the optimisation settings.
Figure A.1: Few-shot prompting: HyenaDNA's performance on new tasks generally improves with the number of tuning samples, but is less clear when isolating the number of  $k$ -shot demonstrations. With less tuning samples, the number of  $k$ -shot demonstrations do not improve performance. As tuning samples increase, the number of  $k$ -shot demonstrations start to improve performance.
Soft prompting details For each dataset, we prepend a sequence of  $n$  (2 to 32k) learnable tokens  $T_{e} \in \mathbb{R}^{n \times d}$ , each of dimension  $d$ , to the input sequences  $X$  of the model:  $\{T_{e}, X, SEP\}$ , where "SEP" indicates the separation token. We optimise these tuneable tokens for a maximum of 20 training epochs on the dataset's training data while keeping all other model parameters fixed. We stop training early if the model's validation loss does not improve for two epochs. After this tuning phase, we evaluate the model's performance on the dataset's full validation data. For an overview of the results of this experiment, see Fig. 4.2 of the main text.
Few-shot prompting details For each dataset, we prepend a set of  $k$  (0 to 32, 0 indicates regular fine-tuning) examples of each class of a dataset (so-called "shots") to an input sequence:
$$
X: \quad \left\{X _ {1}, \operatorname {S E P}, Y _ {1}, \operatorname {S E P}, X _ {2}, \operatorname {S E P}, Y _ {2}, \operatorname {S E P}, X, \operatorname {S E P} \right\},
$$
where  $X_{i}$  indicates an example sequence of class  $i$  with label  $Y_{i}$  (exemplified for a two-way classification task). We tune the model on  $n$  (2 to 256) such  $k$ -shot samples before evaluating its performance on the dataset's full validation data. For an overview of the results of this experiment, see Fig. A.1.
Table A.7: Optimization settings for in-context learning.
SOFT PROMPTING	FEW-SHOT PROMPTING
Optimizer	AdamW	AdamW
Optimizer momentum (β1, β2)	0.9, 0.999	0.9, 0.999
Learning Rate	0.001	0.0001
Batch Size	16	2
Weight Decay (model)	0	0
Weight Decay (Hyena layers)	0	0
Resid dropout	0	0
Embed dropout	0.1	0.1
Reverse complement aug.	true	false
LR-schedule	Plateau	-
A.4 Chromatin Profile Details
Background Variations in non-coding regions of the genome account for the majority of disease and other trait-associated single-nucleotide polymorphisms (SNPs). For example, whilst not directly altering the sequence of an encoded protein, a SNP in a non-coding region can affect the expression of downstream genes by inducing a change in the epigenetic state [56]. Therefore predicting epigenetic markers from a given sequence is an important task in the context of quantifying the functional effects of non-coding variants. Previously DeepSEA [57], a deep convolutional sequence model, has been introduced to predict chromatin features directly from non-coding sequences.
Data The authors of DeepSEA [57] compiled a dataset of 919 chromatin features from [15] and [42] including 690 TF binding profiles for 160 different TFs, 125 DHS and 104 HM profiles. The original DeepSEA dataset consists of 1000 base pair (bp) sequences from the hg19 human reference genome [8] with corresponding 919-dimension multi-label target vectors. Each label corresponds to the presence/absence of a peak in a given chromatin feature within the central 200 bp region of the sequence. The 400 bp flanking regions of the sequence provide broader contextual information which is beneficial to the task. Training and testing sets are split by chromosome and are strictly non-overlapping. In total, there are  $2.2\mathrm{M}$  training samples and 227,512 samples from chromosomes 8 and 9 are held-out for testing. We use the DeepSEA chromatin profile prediction task to evaluate HyenaDNA models with varying context window. We use LiftOver [26] to convert the original DeepSEA dataset to hg38 coordinates and expand flanking regions about the central 200 bp bin symmetrically up to 8000 bp. Approximately  $0.5\%$  of samples are filtered in cases where LiftOver fails or the resulting translated sequence has a different length.
Model We fine-tune several models consisting of a pretrained HyenaDNA encoder, a sequence-level pooling layer and a fully-connected decoder to perform multilabel sequence classification. We compare HyenaDNA against benchmarks set by DeepSEA, a convolutional sequence model, and BigBird [55], a sparse attention based language model. The authors of BigBird fine-tune on the DeepSEA dataset with input sequences extended to 8000 bp (asymmetrically about the centerpoint by -5000 and +3000 bp). Notably BigBird utilizes a byte-pair encoding tokenization scheme whereas HyenaDNA uses a single-character tokenizer and DeepSEA uses one-hot encodings. For the shortest range model (1k), we average across all tokens to perform sequence-level pooling. Whereas in the longer context model (8k) we find that extracting the last token in the sequence as the input to the fully-connected decoder performs better. We also find that for the longer context
model using an encoder pretrained on sequences larger than those used in fine-tuning was beneficial. The hyperparameters of the models used in these experiments are shown in Table A.8. Note that we reduced the depth and of models with increasing context window due to limitations on compute cost/time.
Results The performance of the fine-tuned HyenaDNA models are summarised in Table 4.3. We find that the smallest sequence length model (1024 bp) outperforms both DeepSEA and BigBird on TF and DHS prediction. We find that the model pretrained on  $32\mathrm{k}$  sequences with only 4 layers and fine-tuned on 8k sequences outperforms BigBird on the long range HM task but suffers from degraded performance on the short range tasks. However, we postulate that this performance loss may be recovered by increasing the depth of the model. We also remark that our models contain  $5 - 30\times$  fewer parameters compared to DeepSEA and BigBird.
Table A.8: Chromatin profile model settings. HyenaDNA hyperparameter settings used in the chromatin profile prediction experiments (fine-tuning).
HyenaDNA
Sequence length	1024	8k
Context window	1024	32770
Width	256	256
Layers	8	4
Pooling method	Average	Last token
Parameters (M)	6.6	3.5
Optimizer	AdamW	AdamW
Optimizer momentum	β1, β2 = 0.9, 0.999	β1, β2 = 0.9, 0.999
Weight decay (model)	0.1	0.1
Weight decay (Hyena layers)	0	0
Embed dropout	0.1	0.1
Learning rate	6e-4	6e-4
Batch size	64	64
Epochs	50	50
A.5 Biotype Embeddings Analysis Details
Background Sequence embeddings are useful in reducing dimensionality and capturing semantic relationships into fixed length vectors. We analyze pretrained embedding quality from HyenaDNA and show that it learns biologically informed features. We utilize linear probing, freezing the weights on a pretrained model and attaching a linear classification head to predict biotype sequences. We also use t-SNE to visualize clusterings that emerge from the embeddings.
Data The Ensembl database [9] is a comprehensive resource for gene and transcript annotations such as biotypes. Ensembl biotypes are a classification system, based on a combination of experimental evidence and computational predictions, that summarises the high-level functional properties of genes and transcripts. For example, biotype classes may annotate whether a gene is protein-coding or encodes a long non-coding RNA; if a gene is a disrupted homologue of a known protein coding gene (pseudogene) and by what mechanism it is produced; or the role of a small non-coding RNA such as post-transcriptional modification of other RNAs in the cell nucleus. We use biotype annotations to qualitatively visualize the clustering of gene embeddings into functional groups. We construct a multi-classification task using the top 10 most frequent biotype annotations as multi-class target labels which we predict from the unsupervised embeddings to assess how well biological function is encoded in the embedding space.
Model & Training We use a frozen pretrained HyenaDNA model consisting of 8 layers and width 256 pretrained on sequences of length 160k. To extract sequence-level embeddings, we average along the sequence dimension in the final encoder layer. For comparison we also construct embeddings using DNABERT (5-mer) and Nucleotide Transformer. We construct embeddings for genes in the Ensembl dataset up to a length of 160k. For genes with sequence lengths exceeding the context
window of the encoder, we chunk the sequence and average the embeddings over the chunks. We utilize an XGBoost [7] classifier to perform the supervised multi-classification task on the embeddings. The hyperparameters used are shown in Table A.9.
Table A.9: Hyperparameters. Overview of XGBoost hyperparameters used in biotype multi-classifier.
Estimators	1000
Max depth	3
Learning rate	0.1
Objective	softmax
Results As shown in 4.4, HyenaDNA achieves the highest F1 score on the biotype classification task indicating that its embeddings contain features that are informative of biological function. Notably, HyenaDNA achieves this using the much smaller embedding space dimension of 256, compared to DNABERT and Nucleotide Transformer, which produce embeddings of dimension 1029 and 1280, respectively.
A.6 Long-range Species Classification Details
Table A.10: Hyperparameter ranges for ultra-long range species classification task. Transformer uses FlashAttention [11].
TRANSFORMER	HyenaDNA
Layers	2	2	2	2	8	8
Sequence length	1024	32768	1024	32768	250000	450000
Width	128	128	128	128	256	256
Parameters (M)	0.5	4.5	0.4	0.4	6.6	6.6
Num heads	8	8	-	-	-	-
Learning rate	6e-5	6e-4	6e-5	3e-4	6e-5	6e-4
Optimizer	AdamW
Optimizer momentum	β1, β2 = 0.9, 0.999
LR scheduler	Cosine decay
Weight decay (model)	0.1
Weight decay (Hyena layers)	0
Embed dropout	0.1
Resid dropout	0
Batch size	128 - 256
Training epoch	200
Reverse complement aug.	False
Background Given a genetic sequence randomly sampled from a set of different species, successful identification of the source species requires a model to learn a distinct mutational profile for each species. The more locations for discriminative mutations a model can consider, the more successful it should be at this task. We can arbitrarily tune this task's difficulty by including a higher number of species or increasing the evolutionary similarity of the included species, and thus it represents a helpful setting for measuring long context reasoning abilities for DNA sequence models.
Data We select five species for this task: human (homo sapien), lemur (lemur catta), mouse (mus musculus), pig (sus scrofa), and hippo (hippopotamus amphibius). We hold out four chromosomes from each species (chromosome numbers 1, 3, 12, and 13) for evaluation, and use the rest of each species' chromosomes for training.
Model We compare HyenaDNA against a baseline Transformer, which uses Flash Attention [11] in the mixing layer instead of a Hyena operator. We use 2 and 8 layer models, depending on sequence length. For HyenaDNA, we train on sequence lengths of 1k, 32k, 250k, 450k and 1M. For
Transformer, we limit sequence lengths to 1k and 32k due to the quadratic increase in training time, making training infeasible on our hardware. See Table A.10 for model sizes and hyperparameters.
Training We use pretrained models from 4.1, trained on various lengths between 1k to 1M nucleotides, and fine-tune them using a linear decoder head. We either pool across all tokens (1k and 32k models) or use the last token for classification (250k - 1M models). We randomly sample a (species, chromosome, sequence start, sequence end) tuple at each training step, with uniform probability across all species and non-held-out chromosomes. If a sequence's starting location on a chromosome is such that the end of that sequence would exceed the length of the chromosome, then we pad the sequence with N's to its full intended length. For evaluation, we randomly sample a (species, chromosome, sequence start, sequence end) tuple from our held-out evaluation set of chromosomes, and record the overall Top-1 5-way accuracy of our model (i.e. fraction of sequences correctly classified).
At sequence length  $450\mathrm{k}$ , we use the sequence length warm-up scheduler described in 3.2 on HyenaDNA. This involves gradually increasing the length of sequences fed to the model during fine-tuning from 1k to  $450\mathrm{k}$ . We observe better convergence and higher overall peak accuracy with this strategy, as shown in 3.2.
Table A.11: Pretraining vs scratch on 5-way species classification. Top  $1\%$  accuracy for HyenaDNA by sequence length.
HyenaDNA
LENGTH	SCRATCH	PRETRAINED
1k	53.9	61.1
32k	70.7	93.4
250k	65.7	97.9
450k	71.4	99.4
Pretraining ablation For species classification, pretraining becomes more important for longer sequences. This is in-line with our observation that for harder tasks (including longer sequences), pretraining becomes more important. At sequence length 250k and 450k, the scratch vs. pretraining gap is  $30+$  accuracy points.