SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Compared to natural language FMs, which have shown strong success with in-context learning, HyenaDNA's vocabulary is very small. DNA sequences are also less diverse in structure, e.g. there's no concept of labels or descriptions that follow a DNA sequence. This makes it challenging to perform "pure" in-context learning (relying only on inference), since new concepts such as classification labels would require new symbols. To overcome this limitation and explore the potential for in-context learning in genomics, we make use of two variants of in-context learning: soft prompting and instruction fine-tuning. Each involve a brief tuning phase to introduce the concept of classification using only the existing vocabulary.
Procedure In both variants, we use the GenomicBenchmarks in 4.2, and a HyenaDNA model pretrained on sequence length 160k from 4.1.
In the first experiment, we evaluate a soft prompting approach by preponding a sequence of soft tuneable tokens (2 to 32k) directly in the input sequences. We include a brief tuning phase (< 20 epochs), updating the soft tokens only, to provide HyenaDNA with the ability to indicate the target classes. To denote classes, we repurpose HyenaDNA's fixed vocabulary: for binary classification, for example, we indicate the two classes with the letters "A" and "N".
In the second experiment, we evaluate a few-shot learning approach to in-context learning [6] by prepending, consecutively,  $k$  (2 to 32) demonstrations of each class and its sequence into the prompt. As before, we encode class labels by the use of individual letters of HyenaDNA's existing vocabulary. We additionally perform a brief instruction-tuning period [52]
Table 4.2: Nucleotide Transformer (NT) Benchmarks The Matthews correlation coefficient (MCC) is used as the performance metric for the enhancer and epigenetic marks dataset, and the F1-score is used for the promoter and splice site dataset.
MODEL	NT	NT	NT	HyenaDNA
PARAMS	500M	2.5B	2.5B	1.6M
# OF GENOMES	1	3,202	850	1
Enhancer	53.5	59.3	58.0	62.6
Enhancer types	48.5	50.0	47.4	55.7
H3	73.7	77.6	81.4	81.7
H3K4me1	35.8	44.5	55.9	57.1
H3K4me2	28.1	30.0	32.6	53.9
H3K4me3	26.3	28.1	42.1	61.2
H3K9ac	46.2	50.8	57.5	65.1
H3K14ac	37.7	47.1	55.0	66.3
H3K36me3	46.7	53.3	63.2	65.3
H3K79me3	57.7	59.2	64.2	71.6
H4	76.2	78.9	82.2	79.6
H4ac	34.4	42.3	50.1	63.7
Promoter all	95.4	96.6	97.4	96.5
Promoter non-TATA	95.6	96.9	97.7	96.6
Promoter TATA	94.8	95.8	96.4	96.7
Splice acceptor	96.5	98.5	99.0	96.6
Splice donor	97.2	98.2	98.4	97.3
Splice all	97.2	97.8	98.3	97.9
for each dataset to familiarize HyenaDNA with this task structure by tuning the pretrained model on a small subset of the dataset.
Figure 4.2: Filling long-context with soft tuneable tokens. HyenaDNA is able to learn new tasks in-context when adding a sequence of tuneable tokens to the input sequences. Longer sequences of tuneable tokens lead to better performance.
Results In Fig. 4.2, HyenaDNA's performance on novel tasks improves as more tuneable tokens are added into the input sequences, and saturates close to baseline performance (Tab. 4.1; with the exception of the Human Regulatory dataset). By contrast, we find that increasing  $k$ -shot demonstrations to the input does not necessarily improve performance. A higher number of tuning samples is needed before  $k$ -shot demonstrations start to boost accuracy as shown in Tab. A.1. See A.3 for experiment details.
4.4 Ultralong-Range Genomics
In our final experimental section, we focus on pushing the limits of using long context effectively in genomics. In 4.4.1, we tackle a challenging 919 binary multi-task against a sparse-attention baseline. In 4.4.2 we analyze the learned embeddings HyenaDNA and its use in clustering long sequences by functional annotation, and in 4.4.3 we showcase a novel ultralong-range species classification task.
4.4.1 Chromatin Profile Prediction
The prediction of chromatin profiles and epigenetic markers from DNA sequences is an important and challenging task to quantify the functional effects of non-coding variants. These variants include single nucleotide changes in DNA that can affect the downstream expression of genes [56]. The DeepSEA dataset [57] is compiled from 919 chromatin features including transcription factor (TF) binding profiles, DNase I-hypersensitive sites (DHS) and histone mark (HM) profiles. For a given sequence, the task is to jointly predict 919 labels cor
Table 4.3: Chromatin profile prediction Median AUROC computed over three categories: Transcription factor binding profiles (TF), DNase I-hypersensitive sites (DHS) and histone marks (HM).
MODEL	PARAMS	LEN	AUROC
TF	DHS	HM
DeepSEA	40 M	1k	95.8	92.3	85.6
BigBird	110 M	8k	96.1	92.1	88.7
HyenaDNA	7 M	1k	96.4	93.0	86.3
3.5 M	8k	95.5	91.7	89.3
responding to the chromatin profile (similar to peak detection) of a central region of the sequence, indicating the presence of such functional effects. The input also includes flanking regions that provide broader contextual information needed to incorporate long-range interactions. We fine-tune our pretrained HyenaDNA models from 4.1 and perform competitively against a DeepSea CNN and the SotA sparse attention BigBird [55] baselines using  $5 - 30\times$  fewer parameters. See A.4 for experiment details.
4.4.2 Biotype Embeddings
Sequence embeddings, colored by biotype
Figure 4.3: Embedding visualisation. t-SNE of the embeddings generated by DNABERT, Nucleotide Transformer and HyenaDNA coloured by Ensembl biotype annotations.
Next, we analyze the pretrained embeddings from HyenaDNA and compare them with DNABERT [25] and the Nucleotide Transformer [10]. We encode sequences of human genes corresponding to different biological function annotations obtained from the Ensembl dataset known as biotypes [9]. In cases where the length of the input exceeds the context window of the encoder, the sequence is chunked (by the max length of the encoder) and averaged.
We fit the embeddings using an XGBoost [7] classifier on the 10 most frequent biotypes, and apply t-SNE [50] for visualization. As shown in 4.3, distinct clusterings emerge visually, while quantitatively, HyenaDNA produces the highest F1 score in biotype classification (with a much smaller model), indicating that during pretraining, HyenaDNA learns informative features related to biological function.
Table 4.4: Embedding quality Weighted F1 classification score on 10 biotypes.
MODEL	PARAMS	LEN	F1
DNABERT	110 M	512	64.6
NT	500 M	6k	66.5
HyenaDNA	7 M	160k	72.0
4.4.3 Species Classification
The majority of the genome is conserved across species – humans and non-human primates, for example, have  $< 10\%$  sequence divergence [43], making them difficult to discriminate. This allows us to design an ultralong-range sequence modeling task to test whether a model can determine the source species of a random genetic sequence. To train, we randomly sample DNA sequences from 5 different species, and fine-tune pretrained HyenaDNA and Transformer models from 4.1 to predict the species label. We observe in Tab. 4.5 that both models struggle on shorter sequences of length 1024, but performance improves with longer contexts as the distinct mutational profile of each species becomes more evident. HyenaDNA effectively solves the task by using a context length of 450k to 1 million, where Transformer cannot due to infeasible training time limitations. See A.6 for experiment details.
Summary We presented HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths up to 1 million tokens at single nucleotide resolution - an up to  $500\mathrm{x}$  increase over previous genomic FMs using dense-attention. HyenaDNA is able to learn generalizable features that can then be finetuned for tasks including identifying regulatory elements and on a 919-way chromatin profile prediction task. We also explored the first use of in-context learning in genomics to enable simpler adaptation to downstream tasks without any updates to pretrained weights.
Limitations and Future Work While demonstrating competitive results and introducing novel capabilities, it is worth noting that HyenaDNA was pretrained on only one human reference genome. Incorporating genomes of multiple humans and species could increase generalizability in learned features and reduce bias. Furthermore, our current focus in this study was exclusively on DNA sequences. Extending our framework to incorporate other
Table 4.5: Species classification Top-1 accuracy (%) for 5-way classification (human, lemur, mouse, pig, hippo). The  $\times$  symbol indicates infeasible training time.
MODEL	LEN	ACC
Transformer	1k	55.4
HyenaDNA	1k	61.1
Transformer	32k	88.9
HyenaDNA	32k	93.4
Transformer	250k	X
HyenaDNA	250k	97.9
Transformer	450k	X
HyenaDNA	450k	99.4
Transformer	1M	X
HyenaDNA	1M	99.5
biological or chemical sequences, such as proteins and drug molecules, has the potential to unlock multi-modal capabilities similar to those observed in natural language and vision FMs [39, 40, 54].
With respect to model size, HyenaDNA is significantly smaller than previous genomic FMs and was pretrained using up to 8 Nvidia A100 (80GB) GPUs. We expect increasing model size, and compute, may lead to additional long-range capabilities. Notably, with model parallelism, it becomes feasible to extend the context length by orders of magnitude beyond this current work, and leave that open to future research.
Furthermore, beyond discriminative applications, the use of long context models in generative tasks unlocks exciting prospects for the design of synthetic regulatory elements, genes and protein complexes. In conclusion, the continued advancements of long-range sequence models with single nucleotide resolution hold great promise in driving innovation in genomic research and unraveling the complexities of biological systems.
We would like to thank Guatam Machiraju, Elliott Epstein, Archis Joglekar, Jared Dunnmon, Nazim Bouatta and Anshul Kundaje for helpful discussion and feedback on earlier drafts, and Together for providing the compute used to train models in this paper. We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); US DEVCOM ARL under No. W911NF-21-2-0251 (Interactive Human-AI Teaming); ONR under No. N000141712266 (Unifying Weak Supervision); ONR N00014-20-1-2480: Understanding and Applying Non-Euclidean Geom
etry in Machine Learning; N000142012275 (NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), Department of Defense (DoD) through the National Defense Science and Engineering Graduate Fellowship (NDSEG) Program, and members of the Stanford DAWN project: Facebook, Google, and VMWare. This work is supported by NSF (1651565), AFOSR (FA95501910024), ARO (W911NF-21-1-0125), ONR, DOE (DE-SC0022222), CZ Biohub, and Sloan Fellowship. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
[1] Ž. Avsec, V. Agarwal, D. Visentin, J. R. Ledsam, A. Grabska-Barwinska, K. R. Taylor, Y. Assael, J. Jumper, P. Kohli, and D. R. Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18(10):1196-1203, 2021.
[2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
[3] G. Benegas, S. S. Batra, and Y. S. Song. DNA language models are powerful zero-shot predictors of non-coding variant effects. bioRxiv, pages 2022–08, 2022.
[4] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
[5] N. Brandes, D. Ofer, Y. Peleg, N. Rappoport, and M. Linial. ProteinBERT: a universal deep-learning model of protein sequence and function. Bioinformatics, 38(8):2102-2110, 2022.
[6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm SIGkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[8] D. M. Church, V. A. Schneider, T. Graves, K. Auger, F. Cunningham, N. Bouk, H.-C. Chen, R. Agarwala, W. M. McLaren, G. R. Ritchie, et al. Modernizing reference genome assemblies. PLoS biology, 9(7):e1001091, 2011.
[9] F. Cunningham, J. E. Allen, J. Allen, J. Alvarez-Jarreta, M. R. Amode, I. M. Armean, O. Austine-Orimoloye, A. G. Azov, I. Barnes, R. Bennett, et al. Ensembl 2022. Nucleic acids research, 50(D1):D988-D995, 2022.
[10] H. Dalla-Torre, L. Gonzalez, J. Mendoza-Revilla, N. L. Carranza, A. H. Grzywaczewski, F. Oteri, C. Dallago, E. Trop, H. Sirelkhatim, G. Richard, M. Skwark, K. Beguir, M. Lopez, and T. Pierrot. The Nucleotide Transformer: Building and evaluating robust foundation models for human genomics. bioRxiv, 2023.
[11] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022.
[12] T. Dao, D. Y. Fu, K. K. Saab, A. W. Thomas, A. Rudra, and C. Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2022.
[13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[49] Y. Tay, V. Q. Tran, S. Ruder, J. Gupta, H. W. Chung, D. Bahri, Z. Qin, S. Baumgartner, C. Yu, and D. Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. arXiv preprint arXiv:2106.12672, 2021.
[50] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[51] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
[53] F. Yang, W. Wang, F. Wang, Y. Fang, D. Tang, J. Huang, H. Lu, and J. Yao. scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data. Nature Machine Intelligence, 4(10):852-866, 2022.
[54] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[55] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297, 2020.
[56] S. Zaina, E. L. Pérez-Luque, and G. Lund. Genetics talks to epigenetics? the interplay between sequence variants and chromatin structure. Current genomics, 11(5):359-367, 2010.
[57] J. Zhou and O. G. Troyanskaya. Predicting effects of noncoding variants with deep learning-based sequence model. Nature methods, 12(10):931-934, 2015.
[58] M. Zvyagin, A. Brace, K. Hippe, Y. Deng, B. Zhang, C. O. Bohorquez, A. Clyde, B. Kale, D. Perez-Rivera, H. Ma, et al. GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics. bioRxiv, pages 2022–10, 2022.
HyenaDNA
1 Introduction 2
2 Preliminaries and Related Work 4
2.1 Transformers and Attention 4
2.2 Long Context Strategies in Genomics 4
2.3 Large Convolutional Models 4
3 HyenaDNA Long-Range Genomic Foundation Models 5
3.1 The HyenaDNA Model 5
3.2 Training Long Sequence Models 5
3.3 Downstream Adaptation 6
4 Experiments 6
4.1 Pretraining on the Human Genome 6
4.2 Single Nucleotide Resolution 7
4.3 In-context Learning for Genomic Sequences 7
4.4 Ultralong-Range Genomics 8
4.4.1 Chromatin Profile Prediction 8
4.4.2 Biotype Embeddings 9
4.4.3 Species Classification 10
A Appendix: Experimental Details 17
A.1 Pretraining Details 17
A.2 Short-Range Genomics Details 18
A.2.1 GenomicBenchmarks experiment 18
A.2.2 Ablations on the GenomicBenchmarks 18
A.2.3 Downstream prediction tasks for Nucleotide Transformer benchmark 19
A.2.4 Ablations on the Nucleotide Transformer benchmarks 20
A.3 In-Context Learning Details 20
A.4 Chromatin Profile Details 22
A.5 Biotype Embeddings Analysis Details 23
A.6 Long-range Species Classification Details 24
A Appendix: Experimental Details
In the following sections we provide further details for each experiment. Across all experiments, we use Pytorch and Pytorch Lightning. We train on a mix of Nvidia GPUs with A100s, V100s, and T4s. Unless otherwise stated, we use a cross entropy loss for our objective. Our repository is made public here: https://github.com/HazyResearch/hyena-dna.
A.1 Pretraining Details
Table A.1: Hyperparameter settings for HyenaDNA pretraining (select models).
Layers	2	2	4	4	8
Width	128	256	128	256	256
Params (M)	0.44	1.6	0.87	3.3	6.6
Max seq. len.	64k	64k	64k	64k	1M
Optimizer	AdamW
Optimizer momentum	β1, β2 = 0.9, 0.999
Learning rate	1.5 - 6e-4
LR Scheduler	Cosine decay
Batch size	64 - 256
Global steps	10 - 20k
Weight decay (model)	0.1
Weight decay (Hyena layers)	0
Embed dropout	0.1
Residual dropout	0
Data For pretraining, we use a single human reference genome [22], and leverage the training and validation intervals (start and end) from [1]. During training, we sample an interval and obtain a sequence of length  $L$  by adjusting the intervals on both ends. For the test set, we use chromosomes 14 and X, exclusively, and sample non-overlapping sequences of length  $L$ .
Model We design a suite of parameter efficient architectures with depths between 2 and 8 layers, Hyena blocks of Order-N = 2, and width 128 to 256. The MLP expansion factor (reverse bottleneck) is 4x the width. See Fig. 1.3 for the block architecture of HyenaDNA. The parameter counts range from 400k to 6.6M, trained on sequence lengths between 1,024 and 1M. Tab. A.1 highlights a representative subset of the models we trained. Note: we use different pretrained model sizes depending on the downstream task to prevent overfitting. When selecting which pretrained model to use for a downstream task, we found that a pretrained sequence length of 2 to 4x the downstream max sequence length results in the best performance.
Training We pretrain each model for 10-20k global steps. For models trained on longer sequences, this translates to more tokens being used, as each sample contains more tokens. For example, the largest model with context length 1M was trained on 2T tokens over 4 weeks. We adjust the "accumulate_grad_batches" argument in Pytorch Lightning to keep the global batch size consistent across models and sequence lengths. See Tab. A.1 for hyperparameter details.
Training efficiency We compare pretraining compute resources and GPU-hours to reach competitive performance on the short-range tasks for several baselines and HyenaDNA models, shown in Tab. A.2.
Table A.2: Pretraining GPU & runtime comparison for short-range models.
DNABERT	NUCLEOTIDE TRANSFORMER	HyenaDNA	HyenaDNA
Params	110M	2.5B	436K	1.6M
GPUs	8-2080 TI	128-A100-80GB	1-A100-40GB	1-A100-40GB
Wall clock	25 days	28 days	80 mins	80 mins
GPU-hrs	12,000	215,000	1.3	1.3
A.2 Short-Range Genomics Details
A.2.1 GenomicBenchmarks experiment
Data The GenomicBenchmarks [23] includes 8 datasets designed for sequence-level classification tasks that involve predicting regulatory elements, along with one binary species task. The benchmarks provided for the baseline model include two sets of results: one obtained with Pytorch and the other with TensorFlow. Since our code base is implemented in Pytorch, we compare our results with the Pytorch-based benchmarks.
Model Our backbone is a pretrained 2 layer HyenaDNA model with width 128, trained on sequence length 1024. We pool along the sequence dimension to obtain a classification token, and attach a simple linear decoder head. The baseline CNN, as described by [23], uses uses an embedding layer, 3 convolutional layers with number of filters: 16, 8, and 4. It uses batch norm and max pooling after each convolutional layer, followed by 2 dense layers. It is trained for 10 epochs with batch size 64. The mode sizes range from  $120\mathrm{k}$  to  $520\mathrm{k}$ , depending on sequence length chosen.
Table A.3: GenomicBenchmarks hyperparameters for HyenaDNA and the baseline Transformer (GPT from 4.1), which uses FlashAttention [11].
TRANSFORMER	HyenaDNA
Layers	2	2
Width	128	128
Parameters	529k	436k
Learning rate	1-6e-4	1-6e-4
Weight decay (model)	0-0.2	0-0.2
Weight decay (Hyena layers)	-	0
Embed dropout	0-0.2	0.0-0.3
Resid dropout	0-0.2	0-0.3
Num heads	8	-
Optimizer	AdamW
Optimizer momentum	β1, β2 = 0.9, 0.999
LR scheduler	Cosine decay
Batch size	128-1024
Training epoch	100
Reverse complement aug.	true/false
Sequence lengths	200-4800
Training The primary hyperparameters we sweep across include: learning rate, global batch size, dropout, weight decay, and a reverse complement augmentation. See Tab. A.3 for ranges of hyperparameters used.
A.2.2 Ablations on the GenomicBenchmarks
To better understand how specific design choices in the HyenaDNA model effect performance, we perform a series of ablation experiments on the GenomicBenchmarks.
Pretraining: We train HyenaDNA from scratch and compare with the pretrained version. The pretrained models provide mild to moderate gains - likely due to the benchmarks being near saturation already.
Tokenization: We train HyenaDNA using a k-mer tokenizer  $(k = 6)$  to isolate the effect of the single nucleotide tokenizer. The k-mer tokenizer drops performance significantly across on a majority of the datasets (by as much as 10 accuracy points), while boosting one dataset (Human Enhancer Ensembl). Therefore, the single nucleotide tokenization appears to be a significant component of the HyenaDNA model.
Bidirectional: To ablate the impact of using a causal model, we implemented a bidirectional version of HyenaDNA and trained from scratch on the GenomicBenchmarks (i.e. without masked
Table A.4: GenomicBenchmarks Top-1 accuracy (%) GPT is the causal Transformer from 4.1, HyenaDNA k-mer uses a 6-mer tokenizer, and HyenaDNA bidirection is a bidirectional version of the Hyena operator.
MODEL	GPT	GPT	HyenaDNA	HyenaDNA	HyenaDNA k-mer	HyenaDNA bidirection	DNABERT
Pretrained	no	yes	no	yes	no	no	yes
Mouse Enhancers	79.3	79.3	84.7	85.1	81.8	80.6	66.9
Coding vs Intergenomic	89.3	91.2	90.9	91.3	86.7	90.3	92.5
Human vs Worm	94.8	96.6	96.4	96.6	92.9	95.9	96.5
Human Enhancers Cohn	67.7	72.9	72.9	74.2	69.8	72.1	74.0
Human Enhancers Ensembl	79.0	88.3	85.7	89.2	88.0	85.9	85.7
Human Regulatory	90.2	91.8	90.4	93.8	90.2	89.1	88.1
Human Nontata Promoters	85.2	90.1	93.3	96.6	83.5	88.5	85.6
Human OCR Ensembl	68.3	79.9	78.8	80.9	70.2	75.3	75.1
language model pretraining). The bidirectional version degraded performance on 7 of 8 datasets compared to the standard causal HyenaDNA (also from scratch), on average by 3.8 accuracy points.
The bidirectional HyenaDNA was implemented by using a circular FFT convolution. This involved manipulating the padding on the input sequence before performing the FFT convolution. Previously, we zero padded the input on the right side by length  $L$  (the sequence length). For bidirectionality, we pad by  $1/2L$  on the left and right side of the input, effectively providing a bidirectional receptive field (due to the circular convolution). This is one of many possible ways to implement a bidirectional version of Hyena.