BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks
Authors: Frederikke Isa Marin
Felix Teufel
Marc Horlacher
Dennis Madsen
Dennis Pultz
Ole Winther
Wouter Boomsma
Venue: ICLR 2024 poster (2024)
Keywords: Biological sequence analysis
enhancer annotation
gene finding
gene annotation
Language model
genome modelling
benchmark
LLM
embeddings
representations
DNA
Primary area: datasets and benchmarks
TL;DR: A dataset and downstream tasks for benchmarking emerging DNA language models with realistic and biologically meaningful tasks

Abstract: The genome sequence contains the blueprint for governing cellular processes. 
  While the availability of genomes has vastly increased over the last decades, experimental annotation of the various functional, non-coding and regulatory elements encoded in the DNA sequence remains both expensive and challenging. This has sparked interest in unsupervised language modeling of genomic DNA, a paradigm that has seen great success for protein sequence data. 
  Although various DNA language models have been proposed, evaluation tasks often differ between individual works, and might not fully recapitulate the fundamental challenges of genome annotation, including the length, scale and sparsity of the data. In this study, we introduce **BEND**, a **BEN**chmark for **D**NA language models, featuring
  a collection of realistic and biologically meaningful downstream tasks defined on the human genome.
  We find that embeddings from current DNA LMs can approach performance of expert methods on some tasks, but only capture limited information about long-range features.
  BEND is available at https://github.com/frederikkemarin/BEND.

Published as a conference paper at ICLR 2024
BEND: BENCHMARKING DNA LANGUAGE MODELS ON BIOLOGICALLY MEANINGFUL TASKS
Frederikke L. Marin Felix Teufel***; Mare Horlacher', Dennis Madsen',
Dennis Pultz!, Ole Winther*,, Wouter Boomsma²
'Bioinformatics & Design, Enzyme Research, Novozymes A/S *Department of Computer Science, University of Copenhagen DDigital Science & Innovation, Novo Nordisk A/S
*Department of Biology, University of Copenhagen $ 'Computational Health Center, Helmholtz Center Munich 'TU Compute, Technical University Denmark
ABSTRACT
The genome sequence contains the blueprint for governing cellular processes. While the availability of genomes has vastly increased over the last decades, ex- perimental annotation of the various functional, non-coding and regulatory ele- ments encoded in the DNA sequence remains both expensive and challenging. This has sparked interest in unsupervised language modeling of genomic DNA, paradigm that has seen great success for protein sequence data. Although various DNA language models have been proposed, evaluation tasks often dif- fer between individual works, and might not fully recapitulate the fundamen- tal challenges of genome annotation, including the length, scale and sparsity of the data. In this study. we introduce BEND, a Benchmark for DNA language models, featuring collection of realistic and biologically meaningful down- stream tasks defined on the human genome. We find that embeddings from cur- rent DNA LMs can approach performance of expert methods on some tasks, but only capture limited information about long-range features. BEND is available at https: / github........ com wrroem/rreeriiieee wtt.//reeerrieriii.... www.kkkamaaaakke..eekae
INTRODUCTION
Within the last two decades, the cost of sequencing whole genomes has significantly decreased, hav- ing led to an extraordinary wealth of genomic DNA sequences. This has improved our understand- ing of genetic variation among human genomes and introduced genomes of hitherto understudied species. However, the generation of experimental data annotate and understand these genomic sequences has not kept pace.
At the same time, Natural Language Processing (NLP) has demonstrated the power of large-scale models to capture signals in sequences by masking and reconstructing them in a self-supervised manner. The success of masked language modeling (MLM) has extended to the biological domain Rao et al. (2019); Bepler & Berger (2021): Madani et al. (2023); Rives et al. (2019), with protein language models (pLMs) now being widely used for prediction tasks on protein sequences. The availability of unlabeled genomic sequences and, in many organisms, limited labeled data appear to make language modeling a natural fit for DNA. DNA language models (LMs) have indeed started to emerge, but while the paradigms of NLP have been easy to transfer to proteins, the same may be true for modeling genomes, as they present unique challenges: signals can have an extremely long length range, high-signal regions are sparse, and even those regions the density of signal is lower compared to proteins.
In this paper, we present BEND, a Benchmark for DNA Language Models, collection of realistic and biologically meaningful downstream tasks. BEND aims to provide a standardized set of tasks that measure the ability of LMs to capture the intricacies of genomic data, and to help advance this nascent field. In summary, BEND contributes:
"Equal contribution

---

Published as a conference paper at ICLR 2024
Seven curated tasks and datasets, probing understanding of different DNA functional elements over a variety of length scales.
Experiments covering DNA LMs from six different sources. To knowledge, this represents first evaluation of all publicly available self-supervised DNA LMs suitable for the human genome together with appropriate baseline methods.
An adaptable benchmarking framework for preparing embeddings and training lightweight supervised models.
Result: DNA LMs approach expert method performance some tasks. However, no LM consistently outperforms all others, and reasoning over very long contexts, as c.g. required for finding enhancers, is still challenging.
Result: DNA LMs can learn distinct features masked language modeling. Some LMs' embeddings primarily capture information about gene structure, while others focus noncoding regions.
2 BACKGROUND
2.1 EUKARYOTIC DNA ORGANIZATION AND TERMINOL.OGY
In order to facilitate understanding how different prediction tasks relate to various aspects of the genome, we briefly discuss the fundamental structure and function of eukaryotic genomic DNA (Figure 1). DNA is a linear polymer of four nucleotide bases, which are represented by the four letters A. G and T. It consists of two complementary strands that form a double helix by base pairing the bases A, T, and C, G respectively.
Genomic DNA physically organized in a bierarchical manner. The DNA polymer is coiled around histone proteins, which reduces its physical length and plays role in regulation. A complex of 8 histone proteins together with coiled DNA is called nucleosome. Nucleosomes further condense to form chromatin fibers, which occur in compact (closed) loose (open) form. This controls the accessibility of the DNA sequence to the transcriptional machinery, a process tightly regulated by chemical modifications of the histones (Bannister & Kouzarides, 2011). 1). Chromatin can form loops, which allows regions distant in the sequence to be close in physical space. DNA appears in independent modules called chromosomes, which are typically millions of base pairs (bp) in length.
The genome contains genes, segments that are transcribed to RNA molecules and potentially trans- lated to proteins. Protein-coding genes are structured introns and exons. For expression, a gene is first transcribed to pre-mRNA molecule, and introns are removed via splicing. This combines the exons to one contiguous sequence that encodes the protein. Flanking nucleotides the RNA that do not code for the protein are called untranslated regions (UTRs) and can have regulatory function. In addition, genes are associated with regulatory regions such as promoters, enhancers, silencers and insulators that modulate their expression. Some elements, such as promoters, may lie in close prox- imity to the start of the gene, the transcription start site (TSS). Others can appear multiple thousands bp away from the gene, but mediate their effect by physical proximity.
2.2 LANGUAGE MODELING FOR BIOMOLECULAR SEQUENCES: FROM PROTEINS TO DNA
Over the last years, language modeling has achieved breakthroughs in representation learning for protein property and structure prediction, with transformer-baste pLMs emerging as powerful foun- dation models, capable of learning long-range interactions fully unsupervised (Rives et al., 2019; Elnaggar et al., 2022; Lin et al., 2023). The development of pLMs benefitted from the availability of standardized, representative benchmarks, such TAPE (Rao al., 2019) and PEER (Xu al., 2022), as well as long-running protein machine learning tasks with an emphasis on fair benchmark- ing measure progress (Kryshtafovych et al., 2021: Zhou et al., 2019).
While LMs have been extremely successful for modeling proteins, key differences between the two types of macromolecules hinder their widespread adoption for DNA. typical protein consists of 400-500 amino acids, which are represented as tokens from an alphabet of size 20. The analogy of amino acid tokens with word tokens in NLP, as well as the fact that size of inputs to pLMs and NLP models on the same order of magnitude, made methods developed for NLP directly transferable to protein data, with little to no methodological adaption required (Rao et al., 2020; Elnaggar et al., 2022). The alphabet of DNA is significantly smaller (4 tokens), while the same time sequences,
2

---

Published as a conference paper at ICLR 2024
150bp
Exon
Intron
20, 000, 000bp
Promoter
Enhancer
ATACCATTC CCAGGTTAGC
50, 000bp
H 1bp
500bp
5, 000bp
1 150bp
Figure 1: The organization of eukaryotic genomic DNA. Numbers are indicative examples for the human genome. Genes are structured as alternating introns (average: 5,400 bp) and exons (average: 170 bp), and have promoter regulatory element before their TSS. Enhancers can be thousands of bp away from the gene. DNA is wrapped around histone proteins and densely packed as chromosome.
such those of genes, considerably longer and have no naturally defined border, e.g. the position of the most distant relevant regulatory element is typically unknown. In contrast, protein sequences are naturally self-contained and, being the final gene product, have a significantly higher information density. Together, sparsity and long sequences pose unique challenges to DNA LMs.
2.3
RELATED WORKS
2.3.1 DNA LANGUAGE MODELS
The first available DNA LM was DNABERT (Ji et al., 2021), 12-layer BERT (Devlin et al., 2018) model trained on sequences of length 512 from the human genome. Sequences were tokenized as k-mers using a sliding window, DNABERT was evaluated by fine-tuning on tasks comprising promoter, transcription factor (TF) binding site and splice site (SS) prediction.
growing number of DNA LMS has been proposed since the release of DNABERT, These include the Genomic Pretrained Network (GPN) (Benegas et al., 2023), FloraBERT (Levy al., 2022), the Nucleotide Transformer (NT) (Dalla-Torre et al., 2023), Species-aware LM (Gankin et al., 2023), GENA-LM (Fishman et al., 2023), DNABERT-2 (Zhou et al., 2023) and HyenaDNA (Poli et al., 2023). With the exception of HyenaDNA, models were trained using the MLM objective, but differ in their model architectures, tokenization strategies and training data.
GPN uses dilated convolution layers rather than transformer model. It showed strong performance for zero-shot prediction of variant effects in the A. thaliana genome it was trained on. Qualitative results showed that GPN captures information about gene structure and motifs of binding sites.
Nucleotide Transformer introduced the first large-scale raasformer-basee DNA LMs. All NT mod- els share the same architecture, but differ in their number of training genomes and model parameters. Models were trained on either the human reference genome, 3,202 different genetically diverse hu- man genomes or a selection of 850 genomes from a range of species. To increase the receptive field of the model, sequences were tokenized as 6-mers, allowing for processing sequences of up to 5,994 bp in length. A second generation of multispecies models released later (NT-V2) extended the input length to 12,282 bp. The NT models were evaluated on tasks comprising promoter, SS, histone modification and enhancer prediction with context length of up to 600 bp.
GENA-LM (Fishman et al., 2023) proposed multiple medium meei LMs trained on human and multi- species genomes based on BERT and the BigBird (Zaheer et al., 2020) architecture for long se- quences. Byte-Pair Encoding (BPE) was used for tokenization to further increase the receptive field, enabling an input length of about 36,000 bp. Models were evaluated on tasks comprising promoter, SS, enhancer, chromatin profile and polyadenylation site prediction. While covering the same bio- logical phenomena, tasks were defined differently than in NT. Similarly, DNABERT-2 (Zhou et al., 2023) replaced DNABERT's k-mer tokenizer with BPE and pre-trained multi-species genomes, while GROVER (Sanabria et al., 2023) adopted BPE for the human genome.
Predating NT and GENA-LM, FloraBERT (Levy et al., 2022) proposed pre-training on 93 different plant genomes to enable transfer learning for predicting gene expression. However, FRR was
3

---

Published as a conference paper at ICLR 2024
trained exclusively on promoter-containing sequences. As this requires features to already be anno- tated in the genome, it can be considered a departure from the paradigm of fully self-supervised learning. Similarly, Gankin et al. (2023) pre-trained on 3' UTRs from 1,500 fungal genomes. Species information was made explicit by providing a species label with each sequence to the model.
HyenaDNA (Nguyen et al., 2023) introduced collection of autoregressive LMs, trained using the next token prediction objective at single-nucleotide resolution on the human genome. The Hyena LM architecture (Poli et al., 2023) enabled scaling input lengths of up to I million nucleotides. HyenaDNA models were evaluated by fine-tuning on NT's supervised tasks and the Genomic Bench- marks (Grešová et al., 2023) collection, outperforming NT on the majority of tasks.
A number of DNA LMs were proposed without making trained models available. These comprise the original BigBird (Zaheer et al., 2020). GeneBERT, which includes the prediction of ATAC- seq signals in the pre-training stage. MoDNA (An et al., 2022), with a motif prediction task as an additional objective, the BERT- LOGO (Yang et al., 2021), and Revolution (Cheng et al., 2023), which adopts convolutions with circular padding.
2.3.2 SUPERVISED LEARNING ON DNA
Developing models genomic DNA sequences for the prediction of properties and understand- ing of transcriptional regulation has long been a central task of computational genomics research. The availability of large-scal scale functional genomics data and advancements in deep learning tech- niques have brought progress in predicting various genomic features directly from DNA sequences. DeepBind (Alipanahi et al., 2015) and DeepSEA (Zhou & Troyanskaya, 2015) were two of the first methods leveraging shallow CNNs for predicting TF binding and chromatin features, respectively. DeepCpGi (Angermueller et al., 2017) predicts DNA methylation via a CNN/GRU architecture. Bas- set (Kelley et al., 2016) and ChromTranster (Salvatore et al., 2023) model chromatin state in a cell type specific manner by predicting the presence of DNase-I peaks. Using chromatin state as an aux- iliary input, DeepChrome (Singh et al., 2016) predicts gene expression via multi-modal learning on DNA sequence and histone mark information.
Recently, methods for predicting gene expression have leveraged information across thousands of functional genomic tracks by training 2018) and Enformer (Avsec et al., 2021) demonstrated sfate-of-the-arr performance for gene expres- sion prediction from DNA sequence alone, by integrating genomic information across up to 200 kilobases and multi-task training across several genome-wide functional tasks, including D activity and CAGE signal prediction. Similarly, Sei (Chen et al., 2022) models cis-regulatory TF binding, chromatin accessibility and histone modification profiles across a large range of cell types.
large-scale, multi-task fashion. Basenji (Kelley et al.,
2.3.3 BENCHMARK COLLECTIONS ON DNA
Genomic Benchmarks (Grešová et al., 2023) features balanced classification tasks DNA quences with median lengths ranging from 200 to 2.381 bp. The benchmark covers the classification of functional elements and the prediction of sequence's origin. The element classification tasks defined on human DNA, with one task covering D. melanogaster additionally. For each task, only performance of a baseline supervised neural network model was reported.
DNABERT-2 introduced Genome Understanding Evaluation (GUE), a collection of classification tasks ranging from 70 1,000 bp. On the human genome, it includes classification of promoter, SS and TF binding sequences. It covers other species with TF binding task on mouse, a histone mod- ification task on yeast and a Covid variant classification task on viruses. DNABERT, DNABERT-2 and NT were evaluated. No non-LM task-specific baselines are included in GUE.
The authors of NT provide public leaderboard for their tasks, comprising promoter (hu- man/mouse), enhancer (human), SS (human/multispecies) and histone modification (yeast) pre- diction with lengths ranging from 300 to 600 bp. NT is compared to DNABERT, DÑABERT-2, HyenaDNA and Enformer. No task-specific baselines are included in the leaderboard.
2.3.4 MOTIVATION OF BEND
While existing DNA LMs have reported good performance the tasks on which they were evalu- ated, evaluation strategies to date have shown limited consistency across individual works, with GUE constituting the most recent attempt benchmarking important to ensure that benchmark tasks reflect the complexity and characteristics real-world genome analysis. In practice, genomes vast, and functional regions are sparsely distributed
equal terms. Beyond comparability, it is
4

---

Published as a conference paper at ICLR 2024
Table 1: Overview of the tasks included in the benchmark, Nucleotide-wise tasks require the pre- diction of sequence of labels with the same length as the input. In sequence- wise tasks the whole input sequence be classified. In binned tasks, multiple nucleotides share label.
Type (4 labels)
Evaluation trains/aaites))
Task
Samples Length range
Metric
Source
Nucleotide-wise Multiclass (9)
GENCODE
Gene linding
5,976
14,000 bp
4784551597
MCC
(Frunkish et al., 2021)
Binned (128bp)
Fulco
(2019). Gasperini et (2019).
Enhuncer unnolation
Binary
100,096 bp
AUPRC
Enformer (Avsce et al., 2021)
Chromatin accessibility
Sequence wise Multilabel (125)
2,005,617
512
279,422/722.552
1,354,042/
AUROC
ENCODE Project Consortium (2012)
Histone modification
Multilabel (18) Soquence-wise
Sequencewiie
612,081
512 bp
70.801/120,667
420,713/
AUROC
ENCODE Project Consortiam (2012)
CpG metiylation
Multilabel (7)
959,039
512 bp
189.717/106.277
743,0951
AUROC
ENCODE Project Consortium (2012)
Noncoding variaut
Sequence-wise
DecpEE
effects (expression)
Binary
105,263
512 bp
zero-shot
AUROC
(Zhon & Troyanskaya, 2015)
Noncoding variant effects (disease)
Sequence-wise Binary
Clin (Landrum ot al., 2020)
295,495
512 bp
zero-shot
AUROC
throughout the genome. While there are tasks DNA that are inherently local, such classifying functional regions (e.g. classifying TF binding sites), it needs to be recognized that such tasks do not allow to evaluate model's understanding of the genome over longer ranges.
Therefore, focusing solely on tasks on short sequences, such as distinguishing promoter from non- promoter sequences, falls short of evaluating the extent to which a model's 's representations capture complex features of genomic organization, preventing us from measuring benefits of modeling the genome with larger context windows. For instance, reporting performance on predicting SSs. which can be done on short sequences, does not allow us to evaluate how useful a model would be for gene finding over longer ranges, common task in genome annotation.
To provide more comprehensive assessment, BEND proposes genomic tasks that rely less prior knowledge of feature positions and require reasoning over potentially long contexts. The tasks cover range of length scales, selected to be both biologically relevant and to cover a variety of DNA properties. The tasks explore representations at different resolutions, requiring modelling of DNA single bp resolution as well over longer stretches (Table 1). We establish our benchmark on the human genome, as it offers ample experimental data for the derivation of tasks, has a complex organization, and was the focus of most published DNA LMs.
TASKS AND DATASETS
We introduce the collection of tasks included in BEND. For each task, we additionally provide a datasheet (Gebru et al., 2018) in section A.1. All tasks are provided in bed format, listing the genome coordinates of samples (A.2). This makes it convenient to include more flanking context without reprocessing the data, should future works find it useful take more bp into account.
3.1 GENE FINDING
Definition Gene finding is a multiclass problem where each nucleotide is either part of an exon (Er/R). intron (Lr/n), donor (Dryn) or acceptor (Apyn) splice site or a noncoding region (N C". The F/R subscript denotes whether the gene is located on the forward or reverse strand. Biological relevance Annotating genes and identifying coding sequences is key step in genome annotation and protein discovery. It requires a model to use local context to identify correct reading frames and codon structure, while using longer range signals to propagate the location of SS dis- tant bp between SS, and correctly annotate them as lying in introns or exons. Introns vary in length from a few hundred to several thousand bp, requiring an L.M to understand long-range dependencies. Data GENCODE (Frankish et al., 2021) gene annotations were processed to construct sequences of nucleotide labels y E {Ep, DF, IF, f, AF, ER, DR. JR. AR, NC² for each gene. Detailed pro- cessing is laid out in A.1.1. Samples were partitioned at 80% identity following AUGUSTUS' recommendations (Stanke & Waack, 2003). It should be noted that there a large label imbalance as there is only one donor and acceptor site per intron segment.
Metric We compute the multi-class Matthews correlation coefficient (MCC) (Gorodkin, 2004) over all bp. The MCC is used as it is robust the inherently highly uneven label ratios of this task.
5

---

Published as a conference paper at ICLR 2024
3.2 ENHANCER ANNOTATION
Definition Enhancer annotation is the problem of finding enhancer regions for given gene. We define enhancer annotation as a binary classification task. Given a sequence of gene-adjacent ge- nomic DNA that contains enhancers, a binary label indicating whether it is part of an enhancer needs to be predicted for each segment of 128bp.
Biological relevance Enhancers are short, noncoding segments that contribute to regulating gene expression. They can be located anywhere from a few thousand to a million bp away from their target gene and work by being brought into physical proximity to the gene's ' promoter. Their anno- tation is a highly challenging task that requires detection of long-range interactions.
Data Experimentally validated enhancer-gene pairs were taken from CRISPR interference experi- ments (Fulco et al. (2019); Gasperini et al. (2019) and paired with the main TSS of each gene from Avsec al. (2021). We extracted a sequence of 100,096 bp centered on the TSS for each gene. Each 128bp were annotated with a binary label E {0. 1 indicating whether the bin is part of an enhancer, yielding label sequence of length 782. Detailed processing is laid out A.1.2. Samples were partitioned based on chromosomes.
Metric The AUPRC is computed over all labels. As the number of samples is too limited for measuring performance robustly on a single test split, we perform 10-fold cross-validation in order to evaluate performance over all samples.
3.3 CHROMATIN ACCESSIBILITY PREDICTION
Definition Chromatin accessibility prediction is multilabel task where sequences are classified as being in open or closed chromatin across a range of cell types.
Biological relevance Dynamically modulating chromatin accessibility is a key mechanism for the cell type specific regulation of gene expression, as binding the transcription machinery is highly dependent on the accessibility of DNA elements, including promoters, enhancers and TSS.
Data DNase I typersensitive sites were obtained from ENCODE (ENCODE Project Consortium, 2012) for 125 cell types. Following the preprocessing of Kelley et al. (2016), segments of length 512 bp were labeled with binary vectors E (0, 1)125, with yi = if the chromatin is open for the i'th cell type. Detailed processing is laid out in A.1.3. Samples were partitioned based on chromosomes. Metric The AUROC is computed for each label and averaged.
3.4 HISTONE MODIFICATION PREDICTION
Definition Histone modification prediction is multilabel task, where the histones which are part of the nucleosomes a given DNA sequence are labeled with one or more histone marks. Biological relevance Histone proteins are key the organisation of DNA into chromatin. Mod- ifications of histones modulate chromatin structure and thus contribute to regulating chromatin ac- cessibility and gene expression. Histone modification prediction requires modeling local binding of TFs as well as long-range regulation, such as by distant enhancers.
Data Histone ChiP-seq data for 11 histone marks and replicates in the K562 cell line was obtained from ENCODE. Detailed processing is laid out in 1.1.4 and follows the methodology of 3.3. Each sample a sequence of length 512 bp with a label vector y E {0, 1 1 such that y, = histone bound to this sequence carries mark i. Samples were partitioned based on chromosomes. Metric
The AUROC computed for each label and averaged.
3.5 CpG METHYLATION PREDICTION
Definition CpG methylation prediction is multilabel classification task, where given CpG site is either methylated or unmethylated in different cell lines.
Biological relevance Methylation of cytosine nucleotides in CpG sites is a prominent form of epigenetic modification and plays a key role in the repression gene expression.
Data Bisulfite sequencing data for 7 human cell lines was obtained from ENCODE. Detailed processing is laid out 1.5. Each sample a sequence of length 512 bp centered on the CpG site with label vector y {0, 1)7, such that y if the C is methylated. Samples were partitioned based on chromosomes.
Metric The AUROC computed for each label and averaged.
3.6 NONCODING VARIANT EFFECTS (EXPRESSION AND DISEASE)
Definition Predicting variant effects is a binary problem, where single-bp mutations are classified as either having an effect not. We treat classification zero-shot task, using the cosine distance

---

Published as a conference paper at ICLR 2024
Table 2: Overview of the LMs applicable to the human genome included in the benchmark.
Model
Seq length
Trained on
Architecture
Source
AWD-LSTM
Infinite"
Multispecies
RNN
This work
Dilated ResNet
10,000
Human Ref
CNN
This work
Nucleotide Transformer
5,994
Human Ref
BERT
Dalla-Torr et al. (2023)
Nucleotide Transformer
5,994
Multispecies
BERT
Dalla-Tome Torre et al. (2023)
Nucleotide Transformer
5,994
1000 Genomes project"
BERT
Dalla-Tore et al. (2023)
Nucleotide Transformer V2
12.282
Multispecies
BERT
Dalla-Toree et (2023)
DNABERT
512
Human Ref
BERT
ct (2021)
DNABERT-2
Infinite
Multispecies
BERT
Zhou al. (2023)
GENA-LM
4,500
1000 Genomes project
BERT
Fishman et (2023)
GENA-LM
36,000
1000
Genomes project
BigBird
Fishman et al. (2023)
HyenaDNA
1,000,000
Human Ref
Hyena
Nguyen et al. (2023)
HyenaDNA
1,000
Human Ref
Hyena
Nguyen et al. (2023)
GROVER
8,160
Iumaan Ref
BERT
Sanabria et al. (2023)
As the LSTM compresses all preceding tokens into a single hidden state. it can technically process infinite sequences, even
though trained finite lengths and might not learnt exploit such long contexts.
DNABERT-2 uses ALiBi (Press et al., 2022) to encode position, which can technically scale to sequence length. In practice, model was trained on finite lengths and the authors recommend embedding sequences below 10,000 bp. No explicit length was reported in bp. The indicated number was derived by considering 510 BPE tokens of size 16.
d Schneider et al. (2017), MeVean et al. (2012)
in embedding space between variant nucleotide and its reference nucleotide the prediction score. Biological relevance Single-bp variants in noncoding regions can have functional consequences by altering gene expression levels or causing discase. This task probes the LM's understanding of local context and potentially the structure of regulatory motifs. We focus on noncoding regions, as coding variant effects can be predicted with high accuracy by modeling the mutation in the resulting protein sequence (Frazer et al., 2021).
Data For expression variants, we adapt the DeepSEA dataset (Zhou & Troyanskaya, 2015). For discase-associated variants, we process Clin Var (Landrum et al., 2020). We apply Ensembl VEP (McLaren et al., 2016) to categorize variants by genomic regions into consequence types. De- tailed processing is laid out in A.1.6 and A.1.7. Each variant is genomic position with mutation I E (A, C, G, T) and a label E (0, 1). The adjacent 512 bp serve as embedding context.
Metric We compute the AUROC. Additionally, we report separate AUROCs for the variant sequence types to gain further insight into what genomic features are driving performance.
MODELING
Language Models We benchmark available LMs suitable for the human genome (Table 2). Checkpoint selection criteria are laid out in A.6.2. Additionally, train two simple baseline DNA LMs: An AWD-LSTM (Merity et al., 2017) model trained on three species, and a dilated CNN similar GPN (Benegas al., 2023), trained on the human genome. The model differs from GPN in the parameter count and the length of training sequences (A.6.1).
Downstream model We train lightweight supervised two-layer CNN model with 64 channels on top of the LM embeddings for each task. LM weights are kept frozen and are not fine-tuned. For LMs with reduced output length due to tokenization, embeddings sequence length (A.6.3). For sequence-level tasks, we apply average pooling after the last CNN layer. For the enhancer annotation task, the number of channels was reduced to prevent overfitting. No downstream model is trained for variant effect prediction, as the cosine distance of the LM embeddings directly serves the zero-shot predictor.
upsampled the original
Supervised baselines For each task, we train two supervised models without pre-training. For direct comparison of raw and embedded DNA, we train the two-layer CNN on one-hot encoded sequences. For chromatin accessibility, histone modificaton and CpG methylation prediction, we train the Basset model (Kelley et al., 2016), which was specifically designed for modeling genome- wide functional genomics data. For gene finding and enhancer annotation, we train the ResNet CNN model on one-hot encoded sequences. For variant effect prediction, supervised models are trained. For all tasks where Basset is not applicable, we report the performance of a previously published task-specific expert method on the benchmark dataset put LM performance into context.
7

---

Published as a conference paper at ICLR 2024
Table 3: Results on all tasks. The best performing DNA LM for each task highlighted in bold.
Gene finding
Enhancer annolation
Chromatia accessibilitt
Histone modification
CpG Meliylation
Variunt effects
Variant effects (disease)
(expression)
Expert method
AUGUSTUS
0.80
ENFORMER
0.07
BASSET
BASSET
0.93 BASSET
0.70 DEEPSEA
0.56 DEEPSEA
Fully supervised
ResNet
CNN
0.00
0:46
0.03
0:06
0.75
0.76
0.84
ResNetLM
0.36
0.02
0.82
0.77
0.87
0.55
0.55
AWD-LSTM
0.05
0.03
0.69
0.74
0.81
0.53
0.11
NT-H
0.41
0.05
0.74
0.76
0.88
0.55
0.48
NE-M8
01688
0.06
0.54
Pre-trained
NT-1000G
0.49
0.04
0.77
0.89
0.45
0.49
NT-V2
0.64
0.05
0.80
0.76
0.91
0.48
0.48
DNABERT
0.20
0.03
0.85
0.79
0.91
0.60
0.56
DNABERF-2
0:43
0.03
0.81
0.78
0.90
0.19
0.51
GENA-LM BERT
0.52
0.49
0.55
GENA-L.M RigBird
0.39
0.04
0.78
0.91
0.49
0.52
HycmaDNA large
0.35
0.03
0.84
0.76
0.91
0.51
0.45
HyemaDNA ting
0.10
0.02
0.78
0.76
0.86
0.47
0.44
GROVER
0.28
0.03
0.82
0.77
0.89
0.56
0.51
RESULTS
Gene finding DNA LMs show promising performance for gene finding (Table 3). The two-layer CNN baseline fails to learn, possibly due to its inherent limitation to local context. However, the same CNN is able to achieve varying levels of performance when using LM embeddings, suggesting that embeddings capture some long-range information. NT-MS and NT-V2 outperform all other models by a wide margin, but still do not approach the highly specialized AUGUSTUS (Stanke & Waack, 2003) gene finding model. This highlights that while more specialized downstream models are still needed to accurately predict gene structure, using pre-trained DNA LM embeddings presents promising avenue to attain good performance. Computing individual performance metries across all classes (Table A8) reveals that although there is high variance in the performance across all classes, some embeddings capture splice sites fairly considering their low frequency. HyenaDNA- large, although being the only LM whose context length fully covers the input length of the task, only shows modest performance.
Enhancer annotation All investigated models perform poorly on this task. Enhancer annotation is an extremely difficult task due to the length scale, sparsity of the signal, and small dataset, which pose challenges for all investigated models. Although the supervised baseline has a large enough receptive field to detect the long-range interaction, the size of the dataset is prohibitive for perfor- mance. The performance of Enformer (Avsee et al., 2021) (A.7) is comparable on this task, but must be noted that this is an unsupervised method that was not trained directly on enhancer data. Rather, it infers their locations from learning to predict other genome annotations. Predicting gene- specific enhancers from sequence alone without considering supporting experimental data input therefore remains a highly difficult problem. While this task already proves to be highly challeng- ing for current models the given length scales, we note that biology is even more complex, with enhancers potentially being millions of bp away.
Chromatin accessibility DNABERT shows the highest performance, on par with the specialized Basset model (0.85). All other LMs perform worse this task, offering no advantage over Basset.
Histone modification NT-MS and DNABERT show the highest performance (0.74), outperform- ing Basset (0.72). This suggests that LM embeddings can improve performance for histone modifi- cation prediction, albeit at marginal levels.
CpG methylation NT-MS performs best (0.92) on all included cell lines (Table A11), but is out- performed by Basset (0.93). DNABERT, GENA-LM and HyenaDNA-larg also perform competi- tively, indicating that embeddings capture information about CpG island methylation patterns.
Variant effect prediction DNA LMs show some signal for unsupervised prediction of noncoding variant effects. As the two datasets focus on different genomic regions, we only see limited con- sistency between the expression and disease variant tasks, with DNABERT and NT-MS performing best respectively. While being worse than the supervised DeepSEA method, DNABERT matches DeepSEA's unsupervised performance the expression dataset (AUROC 0.6, A.7). On the disease
8

---

Published as a conference paper at ICLR 2024
dataset, multiple LMs approach DeepSEA's Disease Impact Score, with NT-MS outperforming it. When dissecting performance by variant types, we find that the performance of NT-MS is driven by variants affecting splice sites and introns (Table A13). While splice sites can be considered noncoding DNA, they are not the focus of DeepSEA, which models chromatin features, and shows stronger performance in UTRs and up- downstream regions. Similar to the results on the expres- sion dataset, we find that DNABERT outperforms NT-MS in such regions, suggesting that the two LMs learned distinct sequence features during pre-training. As all other NT models show weaker performance on variants affecting gene structure, this could be consequence of the model's large size and multi-species pre-training. However, we do not see similarly strong performance in the multi-species DNABERT-2 and NT-V2.
DISCUSSION
We find that currently available DNA LMs already show promising performance on some tasks over fully supervised baselines, but do not offer consistent improvements over all included tasks and can fall short of surpassing specialized existing prediction methods. Overall, we find that NT-MS is strong default LM, but is in some tasks inferior the much smaller DNABERT. Interestingly, while both models trained using the MLM objective, we find that they learned distinct genomic features during pre-training. With the pre-training data and the tokenization strategy being the key architectural difference, these choices may deserve more attention in future DNA LMs.
For modeling functional genomics data, DNA LMs only show limited utility. In direct comparison to the Basset model trained on the same data, LM embeddings fail to yield consistent improvements in performance when only using a two-layer CNN.
On the gene finding task, we observe that NT-MS with a simple two-layer CNN shows promising performance compared to the specialized AUGUSTUS, which was found to be the state of the art in a recent benchmark (Scalzitti al., 2020). This suggests that future more sophisticated LM-bbsed gene finders might become method of choice for this problem. The result also indicates that current DNA LMs are capable of modeling long-range dependencies to some extent.
Probing LMs at even longer ranges in the enhancer annotation task reveals that long-range under- standing still needs improvement for sparse problems with limited data. This highlights a key issue facing DNA IMss Not only is there a need for long-range modeling to improve our understanding of the genome, as demonstrated by Avsec et al. (2021), but it also raises fundamental question as to whether current LM training objectives will lead to the incorporation of such distant, sparse signals, whether the local sequence context is all that is required for sequence reconstruction and some level of supervision is needed. Since BEND is not inherently tied to an LM objective, our standardized benchmark may also prove useful for evaluating eventual DNA representation models that follow a different paradigm.
LIMITATIONS AND OUTLOOK
As the curation of a comprehensive benchmark task collection requires experimental ground-truth data to be available, and most published models are trained on human data, we focused BEND on the human genome. BEND aims at comparing the effectiveness of different model architectures and training strategies for learning representations from genomic data, under the assumption that other, similarly structured genomes should behave comparably under self-supervision. However, an important question that remains unanswered is whether DNA LMs can aid with generalization across different organisms. In the future, we hope to extend the benchmark to other, diverse organisms, so that generalization power can be tested in transinr- setting, i.e. by training task given organism, and evaluating performance on another.
In BEND, we benchmarked to what extent embeddings capture features that can be leveraged by downstream models for prediction. This approach is fully agnostic regarding the underlying LM's methodology and scales to models any size. Other works proposed to fine-tune LMs on tasks directly. While this potentially conflates a representation's 's content with the inductive bias of a model architecture for a given task, fine-tuning may yield performance gains beyond the results observed in this work (Nguyen et al., 2023; Zhou al., 2023). Another aspect to be investigated in the future is to dive deeper into how LMs learn features during pre- traininngg as done previously for protein LMs ( (Vig et al., 2021).

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGEMENTS AND DISCLOSURE OF FUNDING
This work was funded in part by Innovation Fund Denmark (0153-00197B), the Novo Nordisk Foundation through the MILS Center (Basic Machine Learning Research in Life Science, NNF200C0062006), the Pioneer Centre for AI (DRNF grant number PI), and the Danish Data Science Academy, which funded by the Novo Nordisk Foundation (NNT2ISA0069429) and VIL- LUM FONDEN (40516).
This work was supported by the Helmholtz Association under the joint research school "Munich School for Data Science (MUDS)".
We would like to acknowledge and thank Ziga Avsec, David Kelley, as well as the rest of the authors behind the Enformer model Avsec et al. (2021), for providing the set transcription start sites used in the enhancer annotation task.
REFERENCES
Babak Alipanahi, Andrew Delong, Matthew Weirauch, and Brendan J Frey. Predicting the se- quence specificities of DNA- and RNA-binding proteins by deep learning. Nature Biotechnol- ogy, 33(8):831-838 August 2015. ISSN 1087-0156, 1546-1696. doi: 10. 10.1038/nbt.3300 URL https: /ww::////wwwwww./ nature cwm wamm//aticcleseeee /btt 3300.
Weizhi An, Yuzhi Guo, Yatao Bian, Hehuan Ma, Jinyu Yang, Chunyuan Li, and Junzhou Huang. MoDNA: motif-oriented pre-training for DNA language model. In Proceedings of the 13th ACM International Conference on Bioinformaties, Computational Biology and Health Informatics, pp. 1-5, Northbrook Illinois, August 2022. ACM. ISBN 978- 978-1-4503--986-77, doi: 10. 155/353550 3545512. URL https: /tl///l.//// acm. wttp/////oooooeeoo/ ht///oo/110// 11 145 /1t.//...111//3511..1 354555111885555..
Christof Angermueller, Heather J. Lee, Wolf Reik, and Oliver Stegle. DeepCpG: accurate prediction of single-cell DNA methylation states using deep learning. Genome Biology, 18(1):67, December 2017. ISSN 1474-760X, doi: 10. 10.118866130059000 189-z. URL http: h /gweo/eemmooogggggooooooooo bioeedeentral com coom/articlese
Adam Auton, Gonçalo R. Abecasis, David M. Altshuler, Richard M. Durbin, Gonçalo R. Abeca- sis, David R. Bentley, Aravinda Chakravarti, Andrew G. Clark, Peter Donnelly, Evan E. Eichler, Paul Flicek, Stacey B. Gabriel, Richard A. Gibbs, Eric D. Green, Matthew F. Hurles, Bartha M. Knoppers, Jan O. Korbel, Eric S. Lander, Charles Lee, Hans Lehrach, Elaine R. Mardis, Ga- bor T. Marth, Gil A. Vcanan, Deborah A. Nickerson, Jeanette P. Schmidt, Stephen T. Sherry, Jun Wang, Richard K. Wilson, Richard A. Gibbs, Eric Boerwinkle, Harsha Doddapaneni, Yi Han, Viktoriya Korchina, Christie Kovar, Sandra Lee, Donna Muzny, Jeffrey G. Reid, Yiming Zhu, Jun Wang. Yuqi Chang, Qiang Feng. Xiaodong Fang. Xiaosen Guo, Min Jian, Hui Jiang, Xin Jin, Tianming Lan, Guoqing Li, Jingxiang Li, Yingrui Li, Shengmao Liu, Xiao Liu, Yao Lu, Xuedi Ma, Meifang Tang, Bo Wang, Guangbiao Wang, Honglong Wu, Renhua Wu, Xun Xu, Ye Yin, Dandan Zhang, Wenwei Zhang, Jiao Zhao, Meiru Zhao, Xiaole Zheng, Eric S. Lander, David M. Altshuler, Stacey B. Gabriel, Namrata Gupta, Neda Gharani, Lorraine H. Toji, Norman P. Gerry, Alissa M. Resch, Paul Flicek, Jonathan Barker, Laura Clarke, Laurent Gil, Sarah E. Hunt, Gavin Kelman, Eugene Kulesha, Rasko Leinonen, William M. McLaren, Rajesh Radhakrishnan, Asier Roa, Dmitriy Smirnov, Richard E. Smith, Ian Streeter, Anja Thormann, Iliana Toneva, Brendan Vaughan, Xiangqun theng-Bradley, David R. Bentley, Russell Grocock, Sean Humphray, Terena James, Zoya Kingsbury, Hans Lehrach, Ralf Sudbrak, Marcus W. Albrecht, Vyacheslav S. Am- stislavskiy, Tatiana A. Borodina, Matthias Lienhard, Florian Mertes, Mare Sultan, Bernd Timmer- mann, Marie-Laure Yaspo, Elaine R. Mardis, Richard K. Wilson, Lucinda Fulton, Robert Fulton, Stephen T. Sherry, Victor Ananiev, Zinaida Belaia, Dimitriy Beloslyudtsev, Nathan Bouk, Chao Chen, Deanna Church, Robert Cohen, Charles Cook, John Garner, Timothy Hefferon, Mikhail Kimelman, Chunlei Liu, John Lopez, Peter Meric, Chris O' Sullivan, Yuri Ostapchuk, Lon Phan, Sergiy Ponomarov, Valerie Schneider, Eugene Shekhtman, Karl Sirotkin, Douglas Slotta, Hua Zhang, Gil A. McVean, Richard M. Durbin, Senduran Balasubramaniam, John Burton, Petr Danecek, Thomas M. Keane, Anja Kolb- Kolb-Kokocinn Kokocinski. Shane McCarthy, James Stalker, Michael Quail, Jeanette P. Schmidt, Christopher J. Davies, Jeremy Gollub, Teresa Webster, Brant Wong, Yiping Zhan, Adam Auton, Christopher L. Campbell, Yu Kong, Anthony Marcketta. Richard A. Gibbs, Fuli Yu, Lilian Antunes, Matthew Bainbridge, Donna Muzny, Aniko Sabo, Zhuoyi Huang.
10

---

Published as a conference paper at ICLR 2024
Jun Wang. Lachlan J. M. Coin, Lin Fang, Xiaosen Guo, Xin Jin, Guoqing Li, Qibin Li, Yingrui Li, Zhenyu Li, Haoxiang Lin, Binghang Liu, Ruibang Luo, Haojing Shao, Yinlong Xie, Chen Ye, Chang Yu, Fan Zhang, Hancheng Zheng, Hongmei Zhu, Can Alkan, Elif Dal, Fatma Kahveci, Ga- bor'T. Marth, Erik P. Garrison, Deniz Kural, Wan-Ping Lee, Wen Fung Leong, Michael Stromberg, Alistair N. Ward, Jiantao Wu, Mengyao Zhang. Mark Daly, Mark A. DePristo, Robert E. Hand- saker, David M. Altshuler, Eric Banks, Gaurav Bhatia, Guillermo del Angel, Stacey B. Gabriel, Giulio Genovese, Namrata Gupta, Heng Li, Seva Kashin, Eric S. Lander, Steven A. McCarroll, James C. Nemesh, Ryan E. Poplin, Seungtai C.
Lihm, Vladimir Makarov, Andrew G. Clark, Srikanth Gottipati, Alon Keinan, Juan L. Rodriiuezz Flores, Jan O. Korbel, Tobias Rausch, Markus H. Fritz, Adrian M. Stütz, Paul Flicek, Kathryn Beal, Laura Clarke, Avik Datta, Javier Herrero, William M. McLaren, Graham R. S. Ritchie, Richard E. Smith, Daniel Zerbino, Xi- Pardis C. Sabeti, Ilya Shlyakhter, Stephen F. Schaffner, Joseph Vitti, David N. Cooper, Edward V. Ball, Peter D. Stenson, David R. Bentley, Bret Barnes, Markus Bauer, R. Keira Cheetham, Anthony Cox, Michael Eberle, Sean Humphray, Scott Kahn, Lisa Murray, John Peden, Richard Shaw, Eimear E. Kenny, Mark A. Batzer, Miriam K. Konkel, Jeri- lyn A. Walker, Daniel G. MacArthur, Monkol Lek, Ralf Sudbrak, Vyacheslav S. Amstislavskiy, Ralf Herwig, Elaine R. Mardis, Li Ding, Daniel C. Koboldt, David Larson, Kai Ye, Simon Gravel, The 1000 Genomes Project Consortium, Corresponding authors, Steering committee. Produc- tion group, Baylor College of Medicine, BGI-Shenzhen, Broad Institute of MIT and Harvard, Coriell Institute for Medical Research, European Bioinformatics Institute European Molecular Biology Laboratory. Illumina, Max Planck Institute for Molecular Genetics, McDonnell Genome Institute at Washington University, US National Institutes of Health, University of Oxford, Well- come Trust Sanger Institute, Analysis group, Affymetrix, Albert Einstein College of Medicine, Bilkent University, Boston College, Cold Spring Harbor Laboratory. Cornell University, Euro- pean Molecular Biology Laboratory, Harvard University, Human Gene Mutation Database, Icahn School of Medicine at Mount Sinai, Louisiana State University. Massachusetts General Hospital, McGill University, and NIH National Eye Institute. A global reference for human genetic vari- ation. Nature, 526(7571):68--
angqun Zheng-Bradie
October 2015. ISSN 1476-4687. doi: 10.1038/nature1553 15393. URL httptttp:/// www. nature com/ ar http//atuurreeee 5393. Number: 7571 Publisher: Nature Publishing Group.
Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R. Ledsam, Agnieszka
Kyle R. Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R. Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature Meth-
ods, 18110):116-1223, October 2021. ISSN 1548-7105. doi: 10. 100000886441522-001--122224
URL https: www. nature com aat.aaatiiileeeeeee 020--9222221--25-----1- Publisher: Nature Publishing Group.
Number: 10
Andrew Bannister and Tony Kouzarides. Regulation of chromatin by histone modifications. Cell Research, 21(3):81-3355 March 2011. ISSN 1748-7838. doi: 10.1038/cr.211..2.. URL
101///1..11011/1111 ht /tt::://ooiii www.or/gg...o.o.ooooooo/ 1038/ ///////cr../ . http//........21 20.1.22.
Gonzalo Benegas, Sanjit Singh Batra, and Yun S. Song. DNA language models are powerful zero- shot predictors of genome- wide variant effects. bioRxiv, pp. 002..88.22.504766, January 2023. URL httt::/biioxxii org/ httg///otee///ral//////e
doi: 10.1 10111002...8..2..5001111 04/ 12/ 2022. 08. 22. 504706 abstract.
Tristan Bepler and Bonnie Berger. Learning the protein language: Evolution, structure, and function. Cell Systems, 12(6):654-669. June 2021. ISSN 24054712. doi: 10.1016/ Acels.2021.0550111 URL
52405471221002033
Kathleen M. Chen, Aaron K. Wong, Olga G. Troyanskaya, and Jian Zhou. A sequence-based global map of regulatory activity for deciphering human genetics. Nature Genetics, 54(7):940-949
July 2022. ISSN 1061-4036, 1546-1718. doi: 110884441588002-0110000 //www /wwww
URL https:
/ 02200102-22
com/
Lei Cheng, Tong Yu, Tero Aittokallio, Jukka Corander, Ruslan Khalitov, and Zhirong Yang. Self- supervised learning for DNA sequences with cireular dilated convolutional networks. preprint, Bioinformatics, February 2023. URL http: /ttp::://bixxiv...mm org/ httg:/ooook/////1...11
11

---

Published as a conference paper at ICLR 2024
Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Hassan Sirelkhatim, Guillaume Richard, Marcin Skwark, Karim Beguir, Marie Lopez, and Thomas Pierrot. The Nucleotide Trans- former: Building and Evaluating Robust Foundation Models for Human Genomics. bioRxiv, doi: 21t.0///222...11.22222.22 biorxiv. org http://wwww..cttoetee///////////
pp. 2023.01.11 1,53669,, January 2023.
URL http:// 01 11 w2w6792367992 ..s.................tttt abst
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2018. doi: 10.48550/ARXIV.8811 04805. 2.
org/oogggooooo abs/ 04805. Publisher: arXiv Version Number:
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger. Debsindhu Bhowmik, and Burkhard Rost. ProfTrans: Toward Understanding the Language of Life Through Self-Supervised Learning. IEEE Transactions on Puttern Analysis and Machine Intelligence 44(10):7122-1177, October 2022. ISSN 0162- 8828, 2160-9292, 1939-3539. doi: 10. 10L1109/TAAMM2221133055555 URL https: htttps://ieeex::tee htte/eeeeepeeeeeeeeeeeeeeeeeeeee ieee org hwwp///ccumeeeeeem /
ENCODE Project Consortium, An integrated encyclopedia of DNA elements in the human genome. Nature, 489(7414):57-74, September 2012. ISSN 1476-4687. doi: 10. 10.103//natrre 1247.
Veniamin Fishman, Yuri Kuratov, Maxim Petrov, Aleksei Shmelev, Denis Shepelin, Nikolay Chekanov, Olga Kardymon, and Mikhail Burtsev. GENA-LM: A Family of Open-Sourc Foun-
dational Models for Long DNA Sequences, June 2023. URL https: / www. biorxiv. Section: New
content/10.r 111.1/0..11//2222. 01w..023.....1.4455444444444 Results.
12. 544594v1.
Pages: 2023.06.
Adam Frankish, Mark Diekhans, Irwin Jungreis, Julien Lagarde, Jane Loveland, Jonathan M Mudge, Cristina Sisu, James C Wrightt Joel Armstrong, If Barnes, Andrew Berry, Alexandra Bignell, Carles Boix, Silvia Carbonell Sala, Fiona Cunningham. Tomás Di Domenico, Sarah Donaldson, Ian T Fiddes, Carlos García Girón, Jose Manuel Gonzalez, Tiago Grego, Matthew Hardy, Thibaut Hourlier, Kevin L Howe, Toby Hunt, Osagie G Izuogu, Rory Johnson, Fergal Martin, Laura Martínez, Shamika Mohanan, Paul Muir, Fabio C Navarro, Anne Parker, Baikang Pei, Fernando Pozo, Ferriol Calvet Riera, Magali Ruffier, Bianca M Schmitt, Eloise Stapleton, Marie-Marthe Suner, Irina Sycheva, Barbara Us7czynska-Ratajeza Maxim Y Wolf, Jinuri Xu, Yucheng T Yang, Andrew Yates, Daniel Zerbino, Yan Zhang, Jyoti S Choudhary, Mark Ger- stein, Roderic Guigó. Tim J Hubbard, Manolis Kellis, Benedict Paten, Michael L Tress, and Paul Flicek. GENCODE 2021. Nucleic Acids Research, 49(D1))1911--D922 January 2021. ISSN 0005-1048. doi: 10. 1..10/9/////aaaa000 URL https: http:////oooooo//////// /00 1093/nart gkaa1087.
Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K. Min, Kelly Brock, Yarin Gal, and Debora Marks. Disease variant prediction with deep generative models of lutionary data. Nature, 599(7883):91-95, November 2021. ISSN 1476-4687. doi: 10. s41586-021 021002211-008888 URL https:tpp////...... //ooi org. 1038 20088//41186655 024-04443-888
Charles P. Fulco, Joseph Nasser, Thouis R. Jones, Glen Munson, Drew T. Bergman, Vidya Subra- manian, Sharon R. Grossman, Rockwell Anyoha, Benjamin R. Doughty. Tejal A. Patwardhan, Tung H. Nguyen, Michael Kane, Elizabeth M. Perez, Neva C. Durand, Caleb A. Lareau, Elena K. Stamenova, Erez. Lieberman Aiden, Eric S. Lander, and Jesse M. Engreitz. Activity-by-comtact model of enhancer-promoter regulation from thousands of CRISPR perturbations. Nature Genet- ics,
December 2019, ISSN 1546-1718. doi: 10. (088//4155880019-5533-888888 URL https::// www. nature cow/articles/....ocoooo s 41588 01 0538-0030000000 0. Publisher: Nature Publishing Group.
Number: 12
Dennis Gankin, Alexander Karollus, Martin Grosshauser, Kristian Klemon, Johannes Hingerl, and Julien Gagneur. Species-aware DNA language modeling. bioRxiv, pp. 2023.01.26.525670, Jan- uary 2023. doi: 10.1101/2023.01..66 525670. URL http::/bbocxivv.rrg// early/2223/01//7//0002
26. 525670 abstract.
12

---

Publisher: Published as a conference paper at ICLR 2024
Molly Gasperini, Andrew J. Hill, José L. McFaline- Figueroa, Beth Martin, Seungsoo Kim, Melissa D. Zhang, Dana Jackson, Anh Leith, Jacob Schreiber, William Noble, Cole Trap- nell, Nadav Ahituv, and Jay Shendure. A Genome-wide Framework for Mapping Gene Regula- tion via Cellular Genetic Screens. Cell, 176(1):377-330.e1)) January 2019. ISSN 0092-8674. doi: 10.10166j.cel122118.111 atticie/pp11S00023771111111
URL https: /www. sci sncedirrets com com/sciences
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé, and Kate Crawford. Datasheets for Datasets. 2018. doi: 10.48550/ARXIV.180) 09010. CRLhttts:/// /ttps://aaxvvvvvo 8.
/is0//1803 09010. Publisher: arXiv Version Number:
Gorodkin. Comparing two K-category assignments by K-category correlation coefficient. Computational Biology and Chemistry, 28(5):367-374, December 2004. ISSN 1476-9271. doi: http:/fj.commpooocceeee00...... 9.60 URL https: hwwww//wwwo.ooo////o/ sci wcweecediiiee..cc....c... rect. com/ sccience/artiicl/ppii///
(0066692711000000 04000799.
Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Šimeček, and Panagiotis Alexiou. Ge- nomic benchmarks: a collection of datasets for genomic sequence classification. BMC Ge- nomic Data, 24(1):25, May 2023. ISSN 2730-6844. doi: (0.11866122283-02--001 https:// /dw:::////iii////io doii..i....//... /10 11 00096//12263333 000-00002-0--2000 01123-8.
URL
Benjamin C. Hitz, Jin- Wook Lee, Otto Jolanki, Meenakshi S. Kagda, Keenan Graham, Paul Sud, Idan Gabdank, J. Seth Strattan, Cricket A. Sloan, Timothy Dreszer, Laurence D. Rowe, Nikhil R. Podduturi, Venkat S. Malladi, Esther T. Chan, Jean M. Davidson, Marcus Ho, Stuart Miyasato, Matt Simison, Forrest Tanaka, Yunhai Luo, Ian Whaling, Eurie I.. Hong, Brian T. Lee, Richard Sandstrom, Eric Rynes, Jemma Nelson, Andrew Nishida, Alyssa Ingersoll, Michael Buckley, Mark Frerker, Daniel S Kim, Nathan Boley, Diane Trout, Alex Dobin, Sorena Rahmanian, Dana Wyman, Gabriela Balderrama-Gutieree Fairlie Reese, Neva C. Durand, Olga Dudchenko, David Weisz, Suhas P. Rao, Alyssa Blackburn, Dimos Gkountaroulis, Mahdi Sadr, Moshe Olshansky, Yossi Eliaz, Dat Nguyen, Ivan Bochkov, Muhammad Saad Shamim, Ragini Mahajan, Erez Aiden, Tom Gingeras, Simon Heath, Martin Hirst, James Kent, Anshul Kundaje, Ali Mortazavi, Bar- bara Wold, and J. Michael Cherry. The ENCODE Uniform Analysis Pipelines. preprint, Bioinfor- matics, April 2023. URL http: /ttt:://bbioxxv.. wwwp:llvvooooooooe/eeeeee /ttp://kuuu//o///..011///2/22222 04. 04. 03.6336...
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformat- ics, 37(15)211222113 2220, August 2021. ISSN 1367-4803. doi: 1..100//iiinfommatcs/aaaaS URL https::tp//......... / /doii www.ioor//oo.ooooooooo.ooooo orgg 0 093 /tt/biiiiinttee werooinfrrmmm formarrtttt.....oo cs cs/sbsbbbt....
Meenakshi S. Kagda, Bonita Lam, Casey Litton, Corinn Small, Cricket A. Sloan, Emma Spragins, Forrest Tanaka, Ian Whaling, Idan Gabdank, Ingrid Youngworth, Seth Strattan, Jason Hilton, Jennifer Jou, Jessica Au, Jin-Wook Lee, Kalina Andreeva, Keenan Graham, Khine Lin, Matt Simison, Otto Jolanki, Paul Sud, Pedro Assis, Philip Adenekan, Eric Douglas, Mingjie Li, Pedro Assis, Keenan Graham, Paul Sud, Stuart Miyasato, Weiwei Zhong, Yunhai Luo, Zachary Myers, Michael Cherry, and Benjamin C. Hitz. Data navigation on the ENCODE portal. 2023. doi: 488550/ARXI..330 .000000 00006. URL / 230.. 200.///000..000...... 00006. arXiv Version Number: 2.
David R. Kelley, Jasper Snoek, and John I.. Rinn. Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks. Genome Research, 26(7):99--999, July 2016. ISSN 1549-5469. doi: 10.1101//r.200535.155
David R. Kelley, Yakir A. Reshef, Maxwell Bileschi, David Belanger, Cory Y. McLean, and Jasper Snock. Sequential regulatory activity prediction across chromosomes with convolutional neural networks. Genome Research, 28(5).399-750, May 2018. ISSN 1088-9051, 1549-5469, doi: 10. 100//r
1101/g:.228119.117 URL http: /gen:///////mmm////ee//e hww.moom...hhhllp. cshlp. haarr///ooopp//////o 227819.117.
Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelis, and John Moult. Crit- ical assessment of methods of protein structure prediction (Cusp)-Round XIV. Proteins:
13

---

January Published as a conference paper at ICLR 2024
Structure. Function, and Bioinformatics, 89(12):1077-1677, 2021. ISSN 1097-0134. doi: 10. 1002/prot.22377. URL htttttss:////nnnnee eprint: /tttps::/ooLLteeeibbaary// wiley. com/ wwwp/////aa//.../00000000
prot. 26237.
Melissa Landrum, Shanmuga Chitipiralla, Garth R Brown, Chao Chen, Baoshan Gu, Jennifer Hart, Douglas Hoffman, Wonhee Jang, Kuljeet Kaur, Chunlei Liu, Vitaly Lyoshin, Zenith Maddipatla, Rama Maiti, Joseph Mitchell, Nuala O'Leary, George R Riley, Wenyao Shi, George Zhou, Valerie Schneider, Donna Maglott, J Bradley Holmes, and Brandi L Kattman. Clin Var: improvements to accessing data. Nucleic Acids Research, 48(D1):D835-D844 January 2020. ISSN 0305-1048. httpst:: http///oiiiaa http://i..ogg.gg///// 1tth://1.1111///11 www.///rr///ee/ narg
doi: 10. 10.1093/nar/gk7772
Richard Leslie, Christopher O'Donnell, and Andrew Johnson. GRASP: analysis of geno- type-phenotype results from 1390 genome-wide association studies and corresponding open access database. Bioinformatics, 30(12)i1185-1194 June 2014. ISSN 1367-4803, doi:
10.1092/bionffommaiic/bbuuu URL https:://ddtee http:e/://bbb.oggg/0.../////aao. btu273.
Benjamin Levy. Zihao Xu, Liyang Zhao, Karl Kremling, Ross Altman, Phoebe Wong, and Chris Tanner. FloraBERT: cross-species transfer learning withattention- neural networks for geneexpression prediction. preprint, In Review, August 2022. URL https: //wp:///ww
researchsquare. coomartiice//re 1
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Zaral-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scaa prediction of atomic-level protein structure with a language model. Science, 379(6637):1123-11 March 2023. ISSN 0036-8075, 1095-9203. doi: 10. 2112///ceene..de255777 URL https : / /ww.. /10/ 1126 stt/scienccc ad22.574.
org/
Yunhai Luo, Benjamin C. Hitz, Idan Gabdank, Jason A. Hilton, Meenakshi S. Kagda, Bonita Lam, Zachary Myers, Paul Sud, Jennifer Jou, Khine Lin, Ulugbek Baymuradov, Keenan Graham, Casey Litton, Stuart R. Miyasato, J. Seth Strattan, Otto Jolanki. Jin- Wook Lee, Forrest Y. Tanaka, Philip Adenekan, Emma O' Neill, and Michael Cherry. New developments on the Encyclopedia of DNA Elements (ENCODE) data portal. Nucleic Acids Research, 48(D1): 2020. ISSN 1362-4962. doi: 10. n0.109//aar/g221066.
Ali Madani, Ben Krause, Eric R. Greene, Subu Subramanian. Benjamin P. Mohr, James M. Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z. Sun, Richard Socher, James S. Fraser, and Nikhil Naik. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, January 2023. ISSN 1087-0156, 1546- 1696. doi: 10. 1878//41587-022-0166188 URL https:/ /www. nature..te.... com/articces 011157-022--01 01688-11181
William McLaren, Laurent Gil, Sarah E. Hunt, Harpreet Singh Riat, Graham R. S. Ritchie, Anja Thormann, Paul Flicek, and Fiona Cunningham. The Ensembl Variant Effect Predictor. Genome Biology, 17(1):12,, June 2016. ISSN 1474-760X. doi: 10. 0111130009--11-9744444 URL
https::/dos /ooi.. ooggorgg/ooooo /10 11e.11166/80 00111///1039001-000000
Gil A. McVean, David M. Altshuler (Co-Chair), Richard M. Durbin (Co-Chair), Gonçalo R. Abeca- sis, David R. Bentley, Aravinda Chakravarti, Andrew G. Clark, Peter Donnelly, Evan E. Eichler, Paul Flicek, Stacey B. Gabriel, Richard A. Gibbs, Eric D. Green, Matthew E. Hurles, Bartha M. Knoppers, Jan O. Korbel, Eric Lander, Charles Lee, Hans Lehrach, Elaine R. Mardis, Gabor T. Marth, Gil A. Veana Deborah A. Nickerson, Jeanette P. Schmidt, Stephen T. Sherry, Jun Wang, Richard K. Wilson, Richard A. Gibbs (Principal Investigator), Huyen Dinh, Christie Kovar, San- dra Lee, Lora Lewis, Donna Muzny, Jeff Reid, Min Wang. Jun Wang (Principal Investigator), Xiaodong Fang, Xiaosen Guo, Min Jian, Hui Jiang, Xin Jin, Guoqing Li, Jingxiang Li, Yingrui Li, Zhuo Li, Xiao Liu, Yao Lu, Xuedi Ma. Zhe Su, Shuaishuai Tai, Meifang Tang, Bo Wang, Guangbiao Wang. Honglong Wu, Renhua Wu, Ye Yin, Wenwei Zhang. Jiao Zhao, Meiru Zhao, Xiaole Zheng, Yan Zhou, Eric S. Lander (Principal Investigator), David M. Altshuler, Stacey B. Gabriel (Co-Chair), Namrata Gupta, Paul Flicek (Principal Investigator). Laura Clarke. Rasko Leinonen, Richard E. Smith, Xiangqun Brangn,, David R. Bentley (Principal Investigator),
14

---

Published as a conference paper at ICLR 2024
Russell Grocock, Sean Humphray, Terena James, Zoya Kingsbury, Hans Lehrach (Principal In- vestigator), Ralf Sudbrak Leader)) Marcus W. Albrecht, Vyacheslav S. Amstislavskiy, Ta- tiana A. Borodina, Matthias Lienhard, Florian Mertes, Mare Sultan, Bernd Timmermann, Marie- Laure Yaspo, Stephen T. Sherry (Principal Investigator), Gil A. Vc (Principal Investigator), Elaine R. Mardis (Co-Principal Investigator) (Co-Chair), Richard K. Wilson (Co-Principal In- vestigator), Lucinda Fulton, Robert Fulton, George M. Weinstock, Richard M. Durbin (Princi- pal Investigator), Senduran Balasubramaniam, John Burton, Petr Danecek, Thomas M. Keane, Anja Kolo-kokocinski, Shane McCarthy, James Stalker, Michael Quail, Jeanette P. Schmidt (Principal Investigator), Christopher J. Davies, Jeremy Gollub, Teresa Webster, Brant Wong, Yiping Zhan, Adam Auton (Principal Investigator), Richard A. Gibbs (Principal Investigator), Fuli Yu (Project Leader), Matthew Bainbridge, Danny Challis, Uday S. Evani, James Lu, Donna Muzny, Uma Nagaswamy, Jeff Reid, Aniko Sabo, Yi Wang, Jin Yu, Jun Wang (Principal Inves- tigator), Lachlan J. M. Coin, Lin Fang, Xiaosen Guo. Xin Jin, Guoqing Li, Qibin Li, Yingrui Li. Zhenyu Li. Haoxiang Lin, Binghang Liu, Ruibang Luo. Nan Qin, Haojing Shao, Bingqiang Wang, Yinlong Xie, Chen Ye, Chang Yu, Fan Zhang, Hancheng Zheng, Hongmei Zhu, Gabor T. Marth (Principal Investigator), Erik P. Garrison, Deniz Kural, Wan-Ping Lee, Wen Fung Leong, Alistair N. Ward, Jiantao Wu, Mengyao Zhang, Charles Lee (Principal Investigator), Lauren Grif- fin. Chih-Heng Hsieh, Ryan E. Mills, Xinghua Shi, Marcin von Grotthuss, Chengsheng Zhang. Mark J. Daly (Principal Investigator), Mark A. DePristo (Project Leader), David M. Altshuler, Eric Banks, Gaurav Bhatia, Mauricio O. Carneiro, Guillermo del Angel, Stacey B. Gabriel, Giulio Genovese, Namrata Gupta, Robert E. Handsaker, Chris Hartl, Eric S. Lander, Steven A. Mc- Carroll, James C. Nemesh, Ryan E. Poplin, Stephen F. Schaffner, Khalid Shakir, Seungtai C. Yoon (Principal Investigator), Jayon Lihm, Vladimir Makarov, Hanjun Jin (Principal Investiga- tor), Wook Kim, Ki Cheol Kim, Jan O. Korbel (Principal Investigator), Tobias Rausch, Paul Flicek (Principal Investigator), Kathryn Beal, Laura Clarke, Fiona Cunningham, Javier Herrero, William M. McLaren, Graham R. S. Ritchie, Richard E. Smith, Xiangqun Zheng-Braee Bradley, An- drew G. Clark (Principal Investigator), Srikanth Gottipati, Alon Keinan, Juan L. Rodriguez- Flores, Pardis C. Sabeti (Principal Investigator), Sharon R. Grossman, Shervin Tabrizi, Ridhi Tariyal, David N. Cooper (Principal Investigator), Edward V. Ball, Peter D. Stenson, David R. Bentley (Principal Investigator), Bret Barnes, Markus Bauer, R. Keira Cheetham, Tony Cox, Michael Eberle. Sean Humphray, Scott Kahn, Lisa Murray, John Peden, Richard Shaw, Kai Ye (Principal Investigator), Mark A. Batzer (Principal Investigator), Miriam K. Konkel, Jeri- lyn A. Walker, Daniel G. MacArthur (Principal Investigator), Monkol Lek, Sudbrak (Project Leader), Vyacheslav S. Amstislavskiy, Ralf Herwig, Mark D. Shriver (Principal Investigator), Carlos D. Bustamante (Principal Investigator), Jake K. Byrnes, Francisco M. De La Vega, Simon Gravel, Eimear E. Kenny, Jeffrey M. Kidd, Phil Lacroute. Brian K. Maples, Andres Moreno- Estrada, Fouad Zakharia, Eran Halperin (Principal Investigator), Yael Baran, David W. Craig (Principal Investigator), Alexis Christoforides, Nils Homer, Tyler Izatt, Ahmet A. Kurdoglu, Shri- pad A. Sinari, Kevin Squire. Stephen T. Sherry (Principal Investigator), Chunlin Xiao, Jonathan Sebat (Principal Investigator)) Vineet Bafna, Kenny Ye, Esteban G. Burchard (Principal Inves- tigator), Ryan D. Hernandez (Principal Investigator), Christopher R. Gignoux, David Haussler (Principal Investigator), Sol J. Katzman, W. James Kent, Bryan Howie, Andres Ruiz-Linares (Principal Investigator), The 1000 Genomes Project Consortium, Corresponding Author, Steering committee, Production group: Baylor College of Medicine, BGI-Shenzheri Broad Institute of MIT and Harvard, European Bioinformatics Institute, Illumina, Max Planck Institute for Molec- ular Genetics, US National Institutes of Health, University of Oxford, Washington University in St Louis, Wellcome Trust Sanger Institute, Analysis group: Affymetrix, Albert Einstein Col- lege of Medicine, Boston College, Brigham and Women's Hospital, Cold Spring Harbor Labora- tory, Dankook University, European Molecular Biology Laboratory, Cornell University, Harvard University, Human Gene Mutation Database, Leiden University Medical Center, Louisiana State University, Massachusetts General Hospital, Pennsylvania State University, Stanford University, Tel-Aviv University, Translational Genomics Research Institute, San Diego University of Califor- nia, San Francisco University of Califomia, Santa Cruz University of California, University of Chicago, University College London, and University of Geneva. An integrated map of genetic variation from 1,092 human genomes. Nature, 491 November 2012. ISSN 1476- 4687. doi: 10. 1038//trr 1632. URL https: / doi org/10. wtt.g//....///////.... w0w.0///nuueeleeee 1332.
Leader),
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and Optimizing LSTM Language Models. 2017. doi: 10.48550/ARXII.1708.222 URL. https: /awxiv..rggg/////
15

---

Published as a conference paper at ICLR 2024
abs/ abs/1708. 02182. Publisher: arXiv Version Number: 1.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Woroww, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, and Chris Ré. HyenaDNA: Long-Range Genomic Sequence Modeling Single Nu- cleotide Resolution, 2023, doi:
URL
abss22305 15794. Publisher: arXiv Version Number: 1.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena Hierarchy: Towards Larger Convo- lutional Language Models, April 2023. URL arXiv:2302. 10866 [es].
Ofir Press, Noah Smith, and Mike Lewis. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation, April 2022. URL http:::/aaxxv..rrg//./ 12409. arXiv:2108. 12409 [cs].
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S. Song. Evaluating Protein Transfer Learning with TAPE. Advances in neu- ral information processing systems, 32:9689-9701, December 2019. ISSN 1049-5258. URL https: / hwwww/ ncwi nllm nih.
Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer pro- tein language models are unsupervised structure learners. preprint, Synthetic Biology, December 1101 1101/20200020 12. 15....... 422761.
2020. URL http: http:::////iii...///// wtt.://rrr/oooo/..ooooo..o http:////i....1111111111111111.1
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. preprint, Synthetic Biology, April 2019. URL httt::::::/oor/ee wtg/lookupp lookup doi 100 1101/ 622803.
Marco Salvatore, Marc Horlacher, Annalisa Marsico, Ole Winther, and Robin Andersson. Trans- fer learning identifies sequence determinants of cell-type specifie regulatory element acces- sibility. NAR Genomics and Bioinformatics, 5(2):lad026,, March 2023. iSSN 2631-9268. doi: 10. 10.1003//arrgahhlaad0226 URL. https: /ttp:///aaaamma/// oup com wtw. http://aagaabartrrra ctte /ooi/10.100 200///00..0333//rrra///////////2
Melissa Sanabria, Jonas Hirsch, and Anna R. Poetsch. The human genome's 's vocabulary proposed by the DNA language model GROVER, September 2023. URL https:ttt//// //ww: /www.wwww......../...... biorxiv. orviv.
org/ ccontent/10) 1101 07 19. 549677v2.
Pages: 2023.07.19.54961 Section:
New Results.
Nicolas Scalzitti, Anne leannin-Girardon, Pierre Collet, Olivier Poch, and Julie D. Thomp- son. A benchmark study of ab initio gene prediction methods in diverse eukaryotic organisms. BMC Genomics, 21(1):293, December 2020. ISSN 1471-2164. doi: 10.1186//128644000--707-- URL https: /bwcgeeemicccsscccc..//..m...mce bi miieedeentr.... com/ 0tt.://e//.0..111112222---000000
Valerie A. Schneider, Tina Graves-Lindsay Kerstin Howe, Nathan Bouk, Kitts, Terence D. Murphy, Kim D. Pruitt, Françoise Thibaud-Nissen. Derek Albracht, Robert S. Fulton, Milinn Kremitzki, Vincent Magrini, Chris Markovic, Sean MeGrath, Karyn Meltz Stein- berg, Kate Auger, William Chow, Joanna Collins, Glenn Harden, Timothy Hubbard, Sarah Pelan, Jared T. Simpson, Glen Threadgold, James Torrance, Jonathan M. Wood, Laura Clarke, Sergey Koren, Matthew Boitano, Paul Peluso, Heng Li, Chen-Shan Chin, Adam M. Phillippy, Richard Durbin, Richard K. Wilson, Paul Flicek, Evan E. Eichler, and Deanna M. Church. Evalua- tion of GRCh38 and de novo haploid genome assemblies demonstrates the enduring quality of the reference assembly. Genome Research, 27(5):849-866 May 2017. ISSN 1549-5469, doi: 10. 10.1100/g:2133611.116
Chen, Paul A.
Ritambhara Singh, Jack Lanchantin, Gabriel Robins, and Yanjun Qi. DeepChrome: deep- learning for predicting gene expression from histone modifications. Bioinformatics, 32(17): 1639-i648, September 2016. ISSN 1367-4803, 1367-4811. doi: 10.109//bioinormmatee
16

---

Published as a conference paper at ICLR 2024
btw427. URL http://tttt..eaa...........//ota
1639/2450757.
Mario Stanke and Stephan Waack. Gene prediction with hidden Markov model and a new intron submodel. Bioinformatics (Oxford, England), 19 Suppl 2::221-225,, October 2003. ISSN 1367-
4811. doi: 10. 10.1009/bioonnforaaaisssh/.
Felix Teufel, Magnús Halldór Gíslason, José Juan Almagro Armenteros, Alexander Rosenberg Johansen, Ole Winther, and Henrik Nielsen. GraphPart: homology partitioning for biologi- cal sequence analysis. NAR Genomics and Bioinformaties, 5(4):lqad088, October 2023. ISSN 2631-9268. doi: 10. /aggg///g/// URL / cup. nar/naagab/ article/dee 10. 0 http////gg/////////////////////
Jesse Vig, Ali Madani, Lav R. Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. BERTology Meets Biology: Interpreting Attention in Protein Language Models, March 2021. URL http://// arxiv. org/abgg// 1/1///006....22
arXiv:2006, 15222 [cs, q-bio].
Minghao Xu. Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Ma Chang, Runcheng Liu, and Jian Tang. PEER: Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding. Advances in Neural Information Processing Systems, 35:35156-35173, De- cember 2022. URL. https: stttps:/prroceediiie ngs
ngs. newrips cc seterrrffllepppppeeeee http://aeeeettt88222200011666663
Abstract -attastss_
and html.
Meng Yang. Haiping Huang, Lichao Huang, Nan Zhang, Jihong Wu, Huanming Yang, and Feng Mu. LOGO, contextualized pre-trained language model of human genome flexibly adapts to various downstream tasks by fine-tuning. preprint, In Review, August 2021. URL https:
/ www. /ttp:/ www.:rrreeaaccusuueeaeaeo com/ article/rccceeeeeeeo
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Al- berti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang. and Amr Ahmed. Big Bird: Transformers for Longer Sequences. In Advances in Neural Information Processing Systems, volume 33, pp. 17283-17297. Curran Asso- ciates, Inc., 2020. URL https: /ttps:://prreeeee http///pooeeddeeeeee neuri neurips. cc pccppappr///20//////cpp httpr//2220hah////// 28512dd1
a Abst
Jian Zhou and Olga G Troyanskaya. Predicting effects of noncoding variants with deep learn- bag-based sequence model. Nature Methods, 12(10):331-344 October 2015. ISSN 1548-7091, 1548-7105. doi: 10.1038/mmeth.5477 URL https: /www nature www./rtiiiaeeeeeeeeeeeee.e h.ntth35447.
Naihui Zhou, Yuxiang Jiang, Timothy R. Bergquist, Alexandra J. Lee, Balint Z. Kacsoh, Alex W. Crocker, Kimberley A. Lewis, George Georghiou, Huy N. Nguyen, Md Nafiz Hamid, Larry Davis, Tunca Dogan, Volkan Atalay, Ahmet S. Rifaioglu, Alperen Dalkıran, Rengul Cetin Atalay, Chengxin Zhang, Rebecca L. Hurto, Peter L. Freddolino. Yang Zhang, Prajwal Bhat, Fran Supek, José M. Fernández, Branislava Gemovic, Vladimir R. Perovic, Radoslav S. Davidovié, Neven Sumonja, Nevena Veljkovic, Ehsaneddin Asgari, Mohammad R.K. Mofrad, Giuseppe Profiti, Castrense Savojardo, Pier Luigi Martelli, Rita Casadio, Florian Boecker, Heiko Schoof, Indika Kahanda, Natalie Thurlby, Alice C. McHardy, Alexandre Renaux, Rabie Saidi, Julian Gough, Alex A. Freitas, Magdalena Antezak, Fabio Fabris, Mark N. Wass. Jie Hou, Jianlin Cheng. Zheng Wang, Alfonso E. Romero, Alberto Paccanaro, Haixuan Yang. Tatyana Goldberg, Chen- guang Zhao, Liisa Holm, Petri Törönen, Alan Medlar. Elaine Zosa, Itamar Borukhov, llya Novikov, Angela Wilkins, Olivier Lichtarge, Po-Han Chi, Wei-Cheng Tseng, Michal Linial, Pe- ter W. Rose, Christophe Dessimoz, Vedrana Vidulin, Saso Dzeroski, Ian Sillitoe, Sayoni Das, Jonathan Gill Lees, David T. Jones, Cen Wan, Domenico Cozzetto, Rui Fa, Mateo Torres, Alex Warwick Vesztrocy, Jose Manuel Rodriguez, Michael L. Tress, Marco Frasca. Marco Notaro, Giu- liano Grossi, Alessandro Petrini, Matteo Re, Giorgio Valentini, Marco Mesiti, Daniel B. Roche, Jonas Reeb, David W. Ritchie, Sabeur Aridhi, Seyed Ziaeddin Alborzi. Marie-Domintne vignes, Da Chen Emily Koo, Richard Bonneau, Vladimir Gligorijević, Meet Barot, Hai Fang, Stefano Toppo, Enrico Lavezzo, Marco Falda, Michele Berselli, Silvio C.E. Tosatto, Marco Car- raro, Damiano Piovesan, Hafeez Ur Rehman, Qizhong Mao, Shanshan Zhang, Slobodan Vucetic,
De-
17

---

Published as a conference paper at ICLR 2024
Gage S. Black, Dane Jo, Erica Suh, Jonathan B. Dayton, Dallas J. Larsen, Ashton R. Om- dahl, Liam J. McGuffin Danielle A. Brackenridge, Patricia C. Babbitt, Jeffrey M. Yunes, Paolo Fontana, Feng Zhang, Shanfeng Zhu, Ronghui You, Zihan Zhang, Suyang Dai, Shuwei Yao, Weidong Tian, Renzhi Cao, Caleb Chandler, Miguel Amezola, Devon Johnson, Jia-Ming Chang, Wen-Hung Liao, Yi-Wei Liu, Stefano Pascarelli, Yotam Frank, Robert Hoehndorf, Maxat Kul- manov, Imane Boudellioua, Gianfranco Politano, Stefano Di Carlo, Alfredo Benso, Kai Hakala, Filip Ginter, Farrokh Mehryary, Suwisa Kaewphan, Jari Björne. Hans Moen, Martti E.E. Tolva- nen, Tapio Salakoski, Daisuke Kihara, Aashish Jain, Tomislav Smuc, Adrian Altenhoff, Asa Ben-Hur, Burkhard Rost, Steven E. Brenner, Christine A. Orengo, Constance J. Jeffery, Gio- vanni Bosco, Deborah Hogan, Maria Martin, Claire O'Donovan, Sean D. Mooney, Casey S. Greene, Predrag Radivojac, and Iddo Friedberg. The CAFA challenge reports improved protein function prediction and new functional annotations for hundreds of genes through experimen- tal screens. Genome Biology, 20(1):244, November 2019. ISSN 1474-760X. $13059-019-1835-88 URL https:::p. / wtte:/.rgg/.....///..1111
doi: 10.1 019-1835---1111111111111111----1 1835-8 8.
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. DNABERT- 2: Efficient Foundation Model and Benchmark For Multi-Species Genome, June 2023. URL http://aat
15006. arXiv:2306. 15006 [cs, q-bio].
18

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 DATASET DOCUMENTATION
We document datasets that were established in this work following the Datasheets for Datasets framework (Gebru et al., 2018), discussing Motivation, Composition, Collection process, Prepro- cessing and Uses, as appropriate. As no new experimental data was acquired within this study, and discussing the original experimental protocols would exceed the scope of datasheet, we limit the Collection sections to listing all relevant sources, where experimental procedures are documented. We omit Distribution and Maintenance as these are identical for each dataset.
A.1.1 GENE FINDING
Motivation The dataset was created to benchmark the performance of models on the gene finding task. Given a DNA sequence, model predicts the structure of the gene, classifying nucleotides as introns, exons, splice sites and noncoding regions.
Composition Instances are the coordinates of human genes including flanking con- text, together with mucleotide-level labels. There are 9 different labels e {Er, Iy, Ap, ER, DR. IR. AR,NAR,NN .NC} denoting exons. donor splice sites, introns, ac- ceptor splice sites and noncoding nucleotides. F and R denote whether the gene lies the forward or reverse strand. There are a total of 5,976 instances with instance lengths ranging from 1,433 to 14,000 nucleotides (Figure A1). The dataset is sample of instances, selected based on the transcript support level of the genes. Label sequences are complete without missing data. A recommended data split is included. The dataset depends on the human reference genome GRCh38.
Collection All data was acquired from GENCODE release 44 (Frankish et al., 2021).
Preprocessing Label sequences were generated from g files downloaded from GENCODE (Frankish et al., 2021). Only HAVANA protein coding gene annotations that were tagged with a transcript support level I or from GENCODE as well as level I or 2 confidence, meaning that the transcript is experimentally verified, were considered. For genes with alternative splicing, only the transcript with the best level of experimental support was chosen. In cases where support was equal, random transcript was chosen. For each transcript, flanking context to include was sampled at random. Following AUGUSTUS' recommendations' for training and testing gene finding models, the data was split so that no pair of instances in different partitions shares more than 80% sequence identity of the mature protein. GraphPart (Teufel et al., 2023) with Needleman- Wunsch global sequence alignments was used for splitting 80% sequence identity into train (80% of the data), test and validation (10% each).
Uses The specific dataset was established in this study and not used before. Data from GENCODE has seen widespread use.
A. 1.2 ENHANCER ANNOTATION
Motivation This dataset created to benchmark the performance of models in annotating the correct enhancer segment. Given a DNA sequence starting at the transcription start site of a gene and encompassing the enhancer, each nucleotide is classified based binary task into enhancer or non-enhancer.
Composition Instances are coordinates in the human genome, covering 100,096 nucleotides each, associated with binary label sequences of length 782. Instances are centered on the transcription start site of gene and extend in both directions symmetrically, containing the enhancer element side (Figure A2). In the label sequence, each label applies to binned segment of 128 bp. The segment is labeled 1 if it contains a nucleotide lying in the enhancer element, and 0 otherwise (Figure A3). Some genes have multiple enhancer elements. In these cases all enhancer elements are labelled in one sample.
Collection Enhancer locations for genes of interest are obtained from the CRISPR interfer- ence (CRISPRi) experiments of Fulco et al. (2019) and Gasperini et al. (2019) (GEO acces- sion GSE120861) via Avsec et (2021). CRISPRi experiments perturb candidate enhancer
'https: :/tpss:///crrr /vtr.v/euuuueee wisc.icc... wec..ddussimoo. wwt.//oonnaaaiiiaaaaaaaaaaaaaaa http://fnf.ommttttae/g/rr///////
nutorial20 2tt.a101/////11 //training.n....../ html htmml
19

---

Published as a conference paper at ICLR 2024
and record whether the perturbation resulted in a change in gene expression. These experiments thereby directly measure the connection of an enhancer element to a specific gene. Following Avsec et al., consider enhancers that had expression change "validated". Enhancer-gent pairs that were predicted by the activity-by-com by-catact (ABC) method only were not considered ex- perimentally validated and excluded. For each gene, the predicted main transcription start site was obtained directly from Avsec et al. (2021).
Preprocessing All non-validated gene-enhancer pairs were discarded, as were all pairs with over 50,048 bp between the enhancer element and the transcription start site. Samples were split chromosome-wir wwise into 10 partitions for ross-validation (1: chr?, chr8, chr18; 2: chr10, chrX, chr13; 3: chr14, 14, chr22, chró; 4: chr20, chr3; 5: chrll, chr 6: chr19; 7: chr4, chr5; 8: chr chr21, chr2; 9: chrl, chr16; 10: chr17, chr9).
Uses The binned label sequences over 100,096 bp were established in this work. The same under- lying enhancer-gene pairs were amongst the ones used in Avsec et al. (2021).
A.1.3 CHROMATIN ACCESSIBILITY PREDICTION
Motivation The data was created benchmark the performance of models on the chromatin accessibility prediction task. Given a DNA sequence, a model predicts whether the sequence is in open or closed chromatin in different cell types.
Composition Instances are coordinates in the human genome, covering 512 nucleotides each, associated with a binary label vector y E {0, 10 indicating whether the DNA is in open (1) closed (0) chromatin in 125 cell types (Table A2). This state is determined experimentally by whether the window of 512 nucleotides contains a DNAse hypersensitive site. There are 2,005,617 instances.
Collection Data was obtained from ENCODE (ENCODE Project Consortium, 2012; Luo et al., 2020; Kagda et al., 2023; Hitz et al., 2023). We downloaded DNase hypersensitivity peaks for 125 cell types in bed format.
Preprocessing The preprocessing followed Kelley et al. (Kelley et al., 2016). Peaks were ex- tended from 512 bp from their midpoint, and peaks overlapping by less than 200bp were merged greedily. When peaks of two more cell types were merged, the resulting sample was annotated with multiple cell type labels. Samples were split chromosome- wise into test (chrl, chr8, chr9; 372,153 samples), validation (chr2, chr4; 279,422 samples) and train (remaining chromsomes; 1,354,042 samples).
Uses The specific dataset was established in this study and not used before. Data from ENCODE has seen widespread use, and comparable datasets were originally created in (Kelley et al., 2016).
A.1.4 HISTONE MODIFICATION PREDICTION
Motivation This dataset benchmarks the ability of models to predict post- tastttranllatinna modifica- tions of Histone proteins. Given a DNA sequence, a model is tasked to predict which histone- modifications are present in the underlying nucleosome.
Composition Instances are coordinates in the human genome, covering 512 nucleotides each, associated with a binary label vector of size 18, indicating whether given histone mark (Table A1) is present (1) or not (0).
Collection Data was obtained from ENCODE (ENCODE Project Consortium, 2012). Narrow peaks files of 18 Histone ChiP-seq experiments was gathered from ENCODE (ENCODE Project Consortium, 2012) in bed format.
Preprocessing Following Kelley et al. (2016), peaks were extended from to 512 bp from their midpoint, with peaks overlapping by less than 200bp being merged greedily. When peaks of two or more ChIP-seq experiments were merged, the resulting sample was annotated with the label of covered by multiple experiments. Samples
each experiment. Note that some Histone marks were split chromosome-wise into test (chrl, chr8, chr9; 120,567 samples), validation (chr2, chr4; 70,801 samples) and train (remaining chromsomes; 420,713 samples).
Uses The specific dataset was established in this study and not used before. is based on publicly available Histone ChiP-seq dataset from the ENCODE project, has seen widespread use.
20

---

Published as a conference paper at ICLR 2024
A.1.5 CPG METHYLATION
Motivation This dataset benchmarks the ability of models to predict the methylation of CpG sites. Methylation is an epigenetic modification of DNA that can affect sequence's activity and repress gene expression. The methylation of a C to form 5-methylcytosine in CpG sites is the most prominent type of methylation.
Composition Instances are coordinates in the human genome, covering 512 nucleotides each, associated with a binary label vector of size 19, indicating whether the CpG site at the center of the segment methylated (1) not (0) in a given cell line (Table A3).
Collection We gathered 7 human cell line whole-genome shotgun bisulfite sequencing (WGBS) experiments from ENCODE (ENCODE Project Consortium, 2012) and processed the "methyla- tion state at CpG" bed files. To select cell lines, experiments marked in ENCODE as "Extremely low coverage" or "Insufficient coverage" were excluded.
Preprocessing We removed all CpG sites that lie on non-standard chromosomes and that have a variant the respective sample genome that does not match the reference genome. Following DeepCpG (Angermueller et al., 2017), we removed CpG sites that are covered by less than 4 reads. CpG sites that had at least 90% methylated reads were labeled as methylated, sites with less than 10% methylated reads were labeled unmethylated, remaining sites were discarded. We took the common subset of CpG sites passing the filtering criteria in all 7 experiments. Sites that were not measured in all experiments were discarded, obtaining 959,039 sites in total. CpG sites were extended with flanking context to yield 51 512bp windows centered on the CpG site. Samples were split by chromosomes (test: chr4, chr13, chr19, chr21 106,227 samples; validation: chr5, chr9, chr22 109,717 samples; remainder train 743,095 samples).
Uses The specific dataset was established in this study and not used before. is based on publicly available WGBS data from the ENCODE project which has seen widespread use.
A.1.6 NONCODING VARIANT EFFECTS (EXPRESSION)
Motivation The dataset created to benchmark the zero-shot noncoding variant effect pre- diction performance of models. Given a reference nucleotide, and a mutated nucleotide, two embeddings are computed and their cosine distance used as the predictor.
Composition Instances are single nucleotide polymorphisms (SNP), genetic coordinates with reference nucleotide Iref E [A, C. G, т} and a variant Tear E {A, C. G, T} together with a bi- nary label y E (0. }} indicating whether the SNP has an effect on gene expression (1) or is genetic background variation (0). We use the same SNPs included in DeepSEA (Zhou & Troyanskaya, 2015). The discovery of such functional SNPs, so-called eQTLs (Expression quantitative trait loci) is done through large-scale genetics studies that link genetic variation to gene expression. There are 98,221 background SNPs and 8,000 variants with effect in total. As this is a zero-shot task, split is required. The dataset depends on the human reference genome GRCh38. eQTLs were collected from GRASP (Leslic et al., 2014) and background SNPs from the 1000 Genomes Project (Ma al., 2012). The dataset is a subsample of SNPs present in these databases. While the 1000 Genomes Project aims at faithfully representing human genetic variation, it might still suffer from ethnicity biases (Table A6). The GRASP database is biased towards eQTLs ob- served in individuals with european ancestry (Table A7).
Collection Genomic coordinates for SNPs were taken from DeepSEA (Zhou & Troyanskaya, 2015).
Preprocessing As the original genomic coordinates refer to the previous reference genome GRCh37, we used LiftOver to transfer the coordinates to the current reference GRCh38. Any coordinates that could not mapped were discarded. Variants where the original reference nu- cleotide does not match the nucleotide at the indicated position in GRCh38 were removed. We only use SNPs included in fold O. We applied Ensembl VEP (McLaren et al., 2016) to categorize variants by consequence. VEP infers the consequence of variant by comparing variant's po- sition to the reference genome annotation, determining what type of sequence region (Table A4) the variant lies in. To obtain one consequence per variant, use VEP's --most_sseeeeeeeee returning the consequence with the potentially most severe effect on function. In DeepSEA, the adjacent 1,000 bp served context for classification. As this exceeds the maximum context length
flag,
21

---

Published as a conference paper at ICLR 2024
of some of the benchmarked models, and chunking inputs is not meaningful strategy when ad- jacent bps serve only as context for an unsupervised embedding, we use 512 bp instead. As this is zero-shot task, split is performed, with the full dataset serving as test set.
Uses The same SNPs on GRCh37 were originally used in DeepSEA for both unsupervised (zero- shot) and supervised variant effect prediction.
A.1.7 NONCODING VARIANT EFFECTS (DISEASE)
Motivation The dataset created to benchmark the zero-shot noncoding variant effect pre- diction performance of models. Given a reference nucleotide, and a mutated nucleotide, two embeddings computed and their distance is used the predictor.
Composition Instances are single nucleotide polymorphisms (SNP), genetic coordinates with a reference nucleotide Fref {A, C. T) and a variant Faar {A, C. G, T) together with binary label E {0, 1} indicating whether the SNP is benign (0) or pathogenic (I), There are 274.399 benign and 21,524 pathogenic SNPs in total. As this is zero-shot task, no split is required. The dataset depends on the human reference genome GRCh38.
Collection SNPs annotated as (likely) benign or pathogenic were collected from Clin using the caariant_svuue .summary file from 2023-07-02 (Landrum et al., 2020). We collected all variants annotated as single nucleotide vari antt with review status of at least one star.
Preprocessing To subset Clin Var for noncoding variants, we first discarded all variants that are annotated as being protein in Clin itself. To further remove variants whose molecular effect might not be annotated in Cai, we compared each SNP to GENCODE release 43 (Frankish et al., 2021). All SNPs that were found to be in a CDS, start codon or stop codon were considered coding and removed. We omit SNPs in the mitochondrial genome ("chromosome M") as they are incompatible with the DeepSEA literature baseline. Following Frazer et al. (2021), the annotations "Likely pathogenic", "Pathogenic" and "Likely benign", "Benign" were combined to yield binary labels. We applied Ensembl VEP (McLaren et al., 2016) to categorize variants by consequence. VEP infers the consequence of variant by comparing variant's position to the reference genome annotation, determining what type of sequence region (Table A5) the variant lies in. To obtain one consequence per variant, we use VEP's most most_seevere flag, returning the consequence with the potentially most severe effect on function. The adjacent 512 bp serve as context for embedding. As this is zero-shot task, no split is performed, with the full dataset serving as test set.
Uses The specific dataset was established in this study and not used before. Data from Clin Var has seen widespread use for variant effect prediction.
Volidadion
150
100
2000
4000
6000
8000
10000
12000
34000
Lenghh [bn]
Figure A1: Length distribution of samples in the gene finding dataset.
22

---

Published as a conference paper at ICLR 2024
30
30-
Pu
prion
Figure A2: Distance to main TSS distribution the enhancer elements the enhancer annotation dataset.
Court
20
300
1003
1300
.000
2500
Enhancer Lzngth (bp)
Figure A3: Length distribution of the enhancer elements in the dataset.
Table A1: Detailed label composition of the histone modification multilabel dataset (n=625,229).
ENCODE modification
Label ID
# positive instances
% positive
10127me3.K566)
41,506
6.64%
13K99c_K562
93,261
14.92%
H3K9me3.KS 3_5622
25,295
4.05%
H3K4mel K562
98,678
15.78%
H3K9ac K562
35,382
5.66%
13562
5
92,587
14.81%
H3K36me3 K562
71,400
11.42%
13K36me3.K550
7
69,975
11.19%
#4K20mel.K562
38,312
6.13%
H3K27mec.KK52
133,535
21.36%
H3K4me3 K562
10
21,717
3,47%
H3K4me3 K562
11
19,706
3.15%
13K4me3_K566
12
29,394
4.70%
H3K4me3.K550
40,934
6,55%
H3K79me2.K562
14
67,714
10.83%
H3K4me2.K566
15
59,069
9.45%
H3K27ac.K500
16
42,993
6.88%
H2AFZ K562
107,810
17.24%
Table A2: Detailed label composition of the chromatin accessibility multilabel dataset
(n=2,062,128) 128))
23/
ENCODE cell line Label ID positive instances % positive
8988T
184,985
8.97%

---

Published as a conference paper at ICLR 2024
ENCODE cell line
Label ID
# positive instances
% positive
AoSMC
158,918
7.71%
Chorion
2
171,737
8.33%
CLL
89,723
4.35%
Fibrobl
4
394,288
19.12%
FibroP
249,221
12.09%
Gliobla
6
158,628
7.69%
GM12891
7
135,186
6.56%
GM12892
8
149,741
7.26%
GM18507
9
109,689
5.32%
GM19238
10
142,111
6.89%
GM19239
11
120,883
5.86%
GM19240
174,077
8.44%
H9ES
13
154,898
7.51%
HeLa-S3JIFNa4h
14
109,698
5.32%
Hepatocytes
15
164,799
7.99%
HPDE6-E6E7
16
132,643
6.43%
B.MM_embl
17
123,566
5.99%
HTR8svn
18
122,358
5.93%
Huh-7.5
19
172,276
8.35%
Huh-7
20
142,675
6,92%
iPS
21
192,872
9.35%
Ishikawa Estradiol
22
131,324
6.37%
Ishikawa 40HTAM
23
133,612
6.48%
LNCaP Landoonn
24
138,434
6.71%
MCF-7_Hypoxia
25
146,053
7.08%
Medullo
26
218,010
10.57%
Melano
27
276,645
13.42%
Myometr
28
165,059
8.00%
Osteobl
29
367,127
17.80%
PanIsletD
30
198,709
9.64%
PanIslets
31
172,141
8.35%
pHTE
32
262,572
12.73%
ProgFib
33
201,038
9,75%
RWPE1
146,568
7.11%
Stellate
35
157,369
7.63%
T-47D
140,932
6.83%
CD4_Th0
37
195,611
9.49%
Urothelia
136,076
6.60%
Urothelia.UT189
39
169,356
8.21%
AG04449
163,835
7.94%
AG04450
41
145,390
7.05%
AG09309
42
198,670
9.63%
AG09319
43
139,005
6,74%
AG10803
44
168,529
8.17%
AoAF
45
171,356
8.31%
BE2-C
46
172,185
8.35%
BJ
47
160,706
7.79%
Caco-2
48
118,338
5.74%
CD20+
100,298
4.86%
CD34+
50
158,606
7.69%
CMK
129,859
6.30%
GM06990
52
88,680
4.30%
GM12864
132,999
6,45%
GM12865
54
139,644
6.77%
H7-hESC
55
263,281
12.77%
HАc
56
177,288
8.60%
HAEpiC
57
201,958
9,79%
24

---

Published as a conference paper at ICLR 2024
ENCODE cell line
Label ID
# positive instances
% positive
HA-h
197,746
9.59%
HA-sp
59
188,882
9.16%
HBMEC
197,261
9.57%
HCF
61
171,925
8.34%
HCFaa
62
182,166
8.83%
HCM
63
190,478
9.24%
HConF
64
150,615
7.30%
HCPEpiC
65
207.114
10.04%
HCT-116
66
110,464
5.36%
HEEpiC
206,638
10.02%
HFF
68
189,177
9.17%
HFF-Myc
206,882
10.03%
HGF
70
143,241
6.95%
HIPEpiC
71
222,312
10.78%
HL-60
72
158,336
7.68%
HMF
73
176,498
8.56%
HMVEC-dAd
74
120,737
5.85%
HMVEC-dBI-Ad
75
159,641
7.74%
HMVEC-dBI-Neo
76
164,741
7.99%
HMVEC-dLy-Ad
77
124,355
6.03%
HMVEC-dLy-Neo
149,601
7.25%
HMVEC-dNco
79
137,163
6.65%
HMVEC-LBB
167,109
8.10%
HMVEC-LLy
81
141,044
6.84%
HNPCEpiC
209,477
10.16%
HPAEC
83
119,805
5.81%
HPAF
185,109
8.98%
HPaLF
85
168,839
8.19%
HPF
86
151,615
7.35%
HRCEPiC
87
189,381
9.18%
HRE
88
184,386
8.94%
HRGEC
89
134,424
6.52%
HRPEpiC
90
224,149
10.87%
HVMF
167,746
8.13%
Jurkat
92
155,987
7.56%
Monocytes-CDI4+
131,745
6.39%
NB4
94
140,287
6.80%
NH-A
95
188,983
9.16%
NHDF-Ad
96
227,566
11.04%
NHDF-nco
97
185,464
8,99%
NHLF
98
203,663
9.88%
NT2-D1
99
179,350
8.70%
PANC-I
100
114,230
5.54%
PEBC
101
164,299
7.97%
RPTEC
102
166,607
8.08%
SAEC
103
195,586
9,48%
SKMC
104
203,116
9,85%
SK-N-MC
105
142,957
6.93%
SK-N-SH_RA
106
86,739
4.21%
Th2
107
86,210
4.18%
WERI-Rb-1
108
188,325
9.13%
WI-38
109
163,827
7.94%
WI-38 40HTAM
110
202,173
9.80%
A549
111
161,511
7.83%
GM12878
112
168,725
8.18%
H1-hESC
113
241,281
11.70%
HeLa-S3
114
183,717
8.91%
25

---

Published as a conference paper at ICLR 2024
ENCODE cell line
Label ID
# positive instances
% positive
HepG2
113
180,213
8.74%
HMEC
116
321,049
15.57%
HSMM
117
291,793
14.15%
HSMMtube
118
304,753
14.78%
HUVEC
119
179,245
8.69%
K562
120
190,083
9.22%
LNCaP
121
291,954
14.16%
MCF-7
122
188,759
9.15%
NHEK
123
201,376
9.77%
Th1
124
293,092
14.21%
26

---

Published as a conference paper at ICLR 2024
Table A3: Detailed label composition of the CpG methylation multilabel dataset.
ENCODE cell line
Label % methylated
SK-N-SH
83%
GM23248
84%
A549
83%
HepG2
81%
HUES64
91 %
GM23248
84%
HeLa-S3
84%
Table A4: VEP variant consequence categories in the expression variant effects dataset.
Consequence
Background
eQTL
% eQTL
Intron variant
55,710
5,002
8.24%
Intergenic variant
22,465
753
3.24%
Upstream gene variant
5,760
579
9.13%
Downstream gene variant
4.146
435
9.50%
Regulatory region variant
3,762
248
6.18%
Noncoding transcript
variant
2,757
342
11.03%
3' UTR variant
1,599
426
21.03%
5' UTR variant
408
54
11.69%
TF binding site variant
410
23
5.31%
Splice region variant
99
30
23.26%
splice polypyrimidine tract variant
85
18
17.48%
Missense variant
51
15.00% 4
Splice donor region variant
27
12.90%
Synonymous variant
20
16.67%
Splice donor variant
13
2
13.33%
Splice donor 5th base variant
6
53.85% 7
Splice acceptor variant
5
0.00% 0
mature miRNA variant
0
0.00%
Stop lost variant
1
50.00% 1
27

---

Published as a conference paper at ICLR 2024
Table A5: VEP variant consequence categories in the disease variant effects dataset.
Consequence
Benign
Pathogenic
% Pathogenic
Intron variant
138,023
188
0.14%
Splice region variant
40,040
320
0.79%
splice polypyrimidine tract variant
39,501
185
0.47%
Noncoding transcript exon variant
23,651
70
0.30%
3' UTR variant
20,407
34
0.17%
5' UTR variant
6,933
63
0.90%
Upstream gene variant
2,245
19
0.84%
Splice donor region variant Splice donor 5th base variant Downstream gene variant
1,744
312
15.18%
507
553
52.17%
268
4
1.47%
262
0.00%
Splice acceptor variant Splice donor variant
194
9,086
97.91% %
189
10,622
98.25%
mature miRNA variant
39
2.50%
Intergenic variant
19
5.00%
Regulatory region variant
10
0
0.00%
Synonymous variant
0.00% 0
Missense variant
0
0.00%
TF binding site variant
0
0.00%
Population statisties of the 1000 Genomes Project (Phases and 3) The data is based on Supplementary Information Table from Auton et al. (2015).
Table A6:
Population
Count
Gambian in Western Division, The Gambia Mandinka
113
Mende in Sierra Leone
85
Esan Nigeria
99
Colombian in Medellin, Colombia
174
Peruvian Lima, Peru
85
Punjabi in Lahore, Pakistan
96
Iberian populations in Spain Toscani in Italy
121
205
Mexican Ancestry in Los Angeles, California Sri Lankan Tamil in the UK Indian Telugu in the UK British in England and Scotland Yoruba in Ibadan, Nigeria Japanese in Tokyo, Japan
130
102
102
180
196
193
Utah residents (CEPH) with Northern and Western European ancestry
184
Han Chinese Beijing, China
200
Chinese Dai in Xishuangbanna, China Luhya in Webuye, Kenya
93
196
Gujarati Indians in Houston, TX
103
African Ancestry in Southwest US
122
Finnish in Finland
192
Han Chinese South
205
Kinh in Ho Chi Minh City, Vietnam
99
Bengali in Bangladesh
86
Puerto Rican in Puerto Rico
159
African Caribbean in Barbados
96
28

---

Published as a conference paper at ICLR 2024
Table A7: Population statistics of the eQTLs in the GRASP 2.0.0.0 database. GRASP combines results from 2,082 individual studies. The ancestry information (cwasancest:yDessripp..ooo is recorded on study-wide level.
Ancestry
eQTLs
European
446,403
Mixed
128,301
Unspecified
111,218
Guropean/Unspeeee
11,376
African
2067
Native
205
29

---

Published as a conference paper at ICLR 2024
A.2 FORMATTING
Building upon established standards in genomics, we curate all tasks in the same format for ease of reuse. Typically, it is not necessary to store DNA sequences X explicitly for each task, as many tasks will refer to the same reference genome. Therefore. for each task, we list the genome coordinates for each sample in a bed genome annotation file. Splits and labels are also stored these files, unless they are too complex to he stored in text format and are provided in hdf5 file that shares its index with the bed file. The bed-based format also makes it convenient to include more flanking context of the segments to be predicted without reprocessing the data, should future works find useful to take more bp into account.
Code to extract DNA sequences from the reference genome with the bed coordinates, dataload- ers, models and config files is available on Github https///aoonyooooo...ppponsseiee 8C42/READMEE.dd)
A.3 LICENSE
As far as applicable, our contributions are licensed CC BY 4.0. As new data was generated in this study, the respective use/redistribution agreements and any copyright claims on the under- lying data sources (GENCODE (Frankish et al., 2021), ENCODE (ENCODE Project Consortium, 2012). GRASP (Leslie et al., 2014), 1000 Genomes Project (McVean et al., 2012), Gasperini et al. Fulco et (2019), Avsec et al. (2021)) apply to the provided datasets. Therefore, citation of the original sources is required when using the data provided with BEND. Citations in BibТex format are listed in the BEND repository.
(2019) (GEO accession
A.4 DISTRIBUTION
data available at https: //sp:://id... erda. dk sid/lssdsss wywpparreeehaaeeee __ aNCa00z21Y Code, configs and scripts extract data and run all experiments are provided at uttttps: /ttp::://g/ttbb/////// com/
A.5 SOCIETAL IMPACT
Predictors building upon DNA LMs may prove useful a wide range of biomedical research ap- plications. Moreover, given their promising performance for understanding the effects of variants, future LMs or derived predictors with even higher performance may eventually become relevant for medical applications. If L.-based predictors are used in clinical diagnostics on humans, it is important to ensure that their performance is evaluated over different populations and potential sub- population biases are accounted for. Moreover, should genomes from human individuals that are not publicly released be used for pre-training LMs, it is important to ensure that their consent is obtained.
A.6 LM DETAILS
A.6.1 LMS TRAINED IN THIS WORK
AWD-LSTM We trained an autoregressive AWD-LSTM LM using truncated backpropagation through time with backpropagation window of 100 bps. Starting points were sampled randomly in the genome, and sequences processed until encountering chromosome end, upon which the hidden state was reset. The model was trained on the full genomes of H. sapiens, M. musculus and D. melanogaster with a batch size of 1,024 for I million steps. This represents a minimal multi- species scenario that was selected due to computational constraints. The model has LSTM layers with dimensions 64, 1,024 and 64. Sequences were tokenized on the nucleotide level, yielding an alphabet of size 4. The model was trained on a single NVIDIA RTX 6000 GPU on a local cluster for 35 days.
Dilated ResNet LM We trained dilated CNN with residual connections, which is the same ar- chitecture used by GPN (Benegas al., 2023). Since this model has a large receptive field due to the dilations, we decided to take advantage of this by increasing the length of the training sequences from 512 nucleotides GPN to 10,000 nucleotides here. To trade off the computational require- ments, we reduce the number of hidden channels in the model from 512 to 256. The model was trained by randomly sampling training sequences from contigs of the human reference genome. The reverse complementary of the sampled sequences used with a 50% chance. Two chromosomes were held out for testing and validation respectively. The model was trained with a batch size of 512 for total of 50k steps, using NVIDIA A40 GPUs for 14 days.
30

---

Published as a conference paper at ICLR 2024
A.6.2 LM CHECKPOINT SELECTION
We aimed to cover all DNA LM works that are publicly available and that included the human genome in their pre-training data. For works that introduce more than one pre-trained checkpoint for their proposed LM architectures, we choose a limited number of representative checkpoints in order to make efficient use of available computational resources. Whenever possible, the selection is driven by results and recommendations presented in the original work.
DNABERT We use the checkpoint that tokenizes DNA as overlapping 6-mers. The orig- inal DNABERT paper (Ji et al., 2021) states that the 6-mer checkpoint showed the best performance when finettniig on the included tasks.
Nucleotide Transformer We evaluate the checkpoints trained on the human reference genome (500M parameters), the 1000 Genomes Project (2.5B parameters) and on the set of genomes from multiple species (2.5B parameters).
GENA-LM In order to include one representative checkpoint both for the BigBird and the BERT architectures, we use bert-large-tan and bi gbi rd-base-t2t.
HyenaDNA HyenaDNA provides multiple sizes of the same model architecture trained on the same data. The checkpoints also differ in the length of the sequences they were trained on. We use t tiny-1kk 1k, the smallest checkpoint that was trained on 1,000bp sequences, and large- 1m, the largest checkpoint trained on million bp sequences.
Nucleotide Transformer V2 We evaluate the largest available model with 500M parame- ters.
A.6.3 UPSAMPLING OF EMBEDDINGS
LMs that make use of k-mer or byte-pair encoding (BPE) tokenization strategies return less em- bedding vectors than their original input sequence length. For nucleotide-level prediction tasks, an embedding sequence of equal length to the nuclcotide-wist label sequence is needed. In order to benchmark all LMs equally, regardless of how they tokenize inputs, upsample embeddings. For the 6-mer tokenization employed by NT, we repeat each embedding vector 6 times. For BPE in GENA-LM, DNABERT-2 and GROVER, token's s sequence. DNABERT, which uses overlapping k-mers, returns a reduced number of em- beddings due to the fact that the left and right borders of the sequence there is no k/2 context available to construct a k-mer embedding around the nucleotide. As DNABERT does not perform any padding to correct for this, these initial and terminal k-mer embeddings are missing. We repeat the first and the last embedding to match the original input sequence length. For k=6, we repeat the first embedding two and the last embedding three times.
repeat each token's embedding by the length of the
A.7 TASK DETAILSS
Computations for all tasks were performed on single GPUs of the types RTX 6000, RTX 8000, V100, A40 and A100 on local clusters, depending on availability.
Gene finding CNN models were trained using Adam W with a learning rate of 0.003 and a weight decay of 0.01 for 100 epochs with batch size of 64.
AUGUSTUS performance was evaluated on the test set. For each input sequence, exactly one com- plete gene model was predicted. Since AUGUSTUS only returns the CDS borders as well as the strand, the remaining labels where inferred from the the CDS locations to compare with the ground truth labels. All nucleotides prior to the first CDS and subsequent to the last are labeled as intergenic. Nucleotides between two CDS segments are labeled introns. The first and last nucleotide of each intron is labeled as a donor and acceptor site respectively for genes predicted to be on the positive strand, the negative strand it is reversed (acceptor site is the first nucleotide and donor the last).
Augustus was run with the following settings:
rarddbbbtth --UTR=off --AIUUSTUS__COFTTGPPPTIIIT
-fff-on - eeenemodel-exacttyyeeee
-- - species human sequence. fasta
Histone modification CNN models were trained using AdamW with a learning rate of 0.003 and weight decay of 0.01 for 100 epochs with batch size of 256.
31

---

Published as a conference paper at ICLR 2024
CpG methylation CNN models were trained using Adam W with learning rate of 0.003 and weight decay of 0.01 for 100 epochs with a batch size of 256.
Enhancer annotation CNN models with channel size were trained using Adam W with learn- ing rate of 0.001 and a weight decay of 0.01 for 100 epochs with a batch size of 8. Due to the high label imbalance the data, positive labels were up-weighted in the loss with a weight corresponding to the average fraction of positive to negative labels.
Enformer performance was evaluated using the code provided in the Compute contribution scores section of the Enformer notebook2. For each sample of 100,086 bp, context was expanded bidi- rectionally and Enformer contribution scores were obtained. The scores were trimmed back to the original 100,086 bp and average pooled 128bp, yielding sequence of 782 bins for each sample.
Noncoding variant effects There are two ways of extracting an embedding for a variant sequence: It is possible either take the mean embedding of the full context window, or extract the embedding at the position where the SNP is found. Within BEND, we opted for the latter approach, as we consider it more universally applicable to e.g. autoregressive models where preceding embeddings in the context window cannot contain any information on the variant that comes later in the sequence. For NT, this means taking the embedding of the 6-mer token containing the variant. For DNABERT- 2 and GENA-LM, the embedding of the BPE token containing the variant is used. For DNABERT with 6-mer tokenization, we use the embedding the token that has the mutated residue as its 3rd nucleotide.
As autoregressive models subsequent tokens cannot affect already computed embeddings, we only used an unidirectional context of 512 preceding nucleotides for AWD-LSTM and HyenaDNA.
For the expression dataset, supervised DeepSEA performance was computed from the cross- validated predictions for split 0 available in the supplementary material of the original DeepSEA publication (Zhou & Troyanskaya, 2015). Unsupervised performance could not be recomputed and was taken at 0.6 from DeepSEA's Supplementary Figure 6 for the expression dataset. For the disease dataset, supervised DeepSEA performance was computed by submission to DeepSEA's online ver- sion, using the Bel luga model. The Disease Impact Score (DIS) output was used for benchmarking.
Beluga
enformer//enformmr--ug-eeee ipynb
32

---

Published as a conference paper at ICLR 2024
A.8
EXTENDED RESULTS
Table A8: Gene finding recall and precision per label.
CIIy (I)
Denaey (T)
batevey
En Phocsan
CIV%
Acceptoryy Rovll
Tearvny
Donorry (7)
Inaergralc
Medd
Rocil
lxcall
Petiace
Boad
Produces
Iocal
Reall
Preaace
Photians
Rceall
Peziase
Read
Presecs
Recall
Phochan
AUGUSTIUS
0.90
0.92
0.20
0:66
0.:3
0.57
6.79
0.95
0.85
0.01
0:51
0.85
0.25
0:30
0.80
0.85
0.51
6:51
0.39
0.84
0335
0.66
0.76
CNN
0.
Bm
Bd
0.0
11:88
3.78
015
0.0
01.
0.3
010
013
0.0
0.0
010
1.8
0.34
0581
RoNooLM
0.99
0182
420
18.2
0.43
125
0.8
0.33
0.76
809
0.20
0.0
100
0.35
6.7
0158
84
86
3.19
016
0.7
0.0
0.33
0.61
0.50
00
0.66
NT-1068G
0.76
0.75
0.0
0.58
0.7
0.59
D.01
0.64
0.76
694
0.14
0.63
6.74
0443
0.56
0.43
0.57
67
875
025
DNABERT
0145
12:3
0.34
0.36
0.38
0.56
1252
162
0.36
0.38
035
DNABERTS
0.51
0.63
310
0.23
0.6
0.5
0.5
3.0
0.5
0.60
011
0.65
0.53
0.96
80
100
065
0.5
GENA AM Righird
0.31
0.53
0.15
035
0.59
3.40
B13
0.3
6.56
011
033
6.75
0.51
054
0.4
043
056
611
HlyaalINA
0.25
0.8
0.65
506
0.6
114
0.38
0.62
041
018
052
GROVER
0.31
0.53
0.12
0.29
0.55
3.59
001
0.64
0.45
0.6
0.25
0.5
0.45
0.4
0.5
0.55
048
0.55
Table A9: Chromatin accessibility prediction performance per cell line.
GENN TENALLA A.M Bightrd
NT-1MIIG (2.58)
GENA-LMI BRRK
3lyemDNA large
DN IMAHBRRTE
RcsNet-AA
RyenaDNA
GROVER
CNN
AN
NT-H
NT-M8
NT-V2
DNA
Cell line
89SST
0.86
0.82
0.82
0.85
0.83
0.85
0.84
0.84
0.86
0.85
0.84
0.85
0.84
0.85
0.85
AsSMC
0.75
0.68
0.84
0.78
0.81
0.76
0.83
0.83
Chorlon
0.81
0.78
0.77
681
0.79
0.81
0.0
088
0.80
0.81
CLI.
0.87
0.81
0.0
0.6
0.82
0:84
0.85
0.85
0.88
0.86
1.86
0.84
0.86
0.86
Fibrol
Fibrobl
0.71
0.68
0.65
0.67
0.71
0.75
0:69
0.90
6.71
0,72
0.69
0:70
0.72
0.7%
0.71
0.7
0.70
0.72
0.71
0.76
1.75
0.71
Glisbla
0.86
0.75
071
0.4
0.77
0.82
0,80
GR2
0.86
DRD
GR3
0.8)
0.85
0.8
GM12W91
0,89
0.81
n81
087
0.83
D.85
n.4
0.86
0.89
0.87
0.85
0.88
0.88
0.87
GMT2892
0.88
0.84
0.83
0.87
0:85
0.86
0.85
0886
0.89
0.87
1.86
B.ST
0.86
0.88
0.ST
GM18507
0.74
0.76
0,79
0.85
0.84
GM19238
0.86
0.79
0.27
0.84
0.80
0.82
0.81
6.8
0.87
0.84
0.81
0. 8484
0.82
0.84
GM19239
0.87
0.79
0.77
0.85
0.80
0.82
0.81
0..4
0.88
0.85
1181
8.85
0.82
1.86
0.85
GMI9240
0.8
0.74
0.73
0.80
0.76
0.77
0.76
0.76
0.82
0.79
0.77
0.80
0.77
0.80
0.79
HOES
0.79
0.85
0.83
0.86
IINaa
0.85
0.72
0.70
0.82
0.76
0.81
0.79
0.85
0.81
0.79
0.R2
0.79
0.83
0.80
Hepatocytes
0.72
0.73
0.72
0.75
0.74
0.75
0.74
0.74
0.76
0.76
0.75
0.76
0.74
0.75
0.75
HPDFS-FGR7
0.90
0.75
0.70
0.85
0.77
0.83
0.81
088
D.85
0.84
0.81
0.85
0.82
0.87
0.85
HSMMLeeb
0.77
0.86
0.85
0.67
0.85
0.85
0.88
HTRSsv
0.91
0.76
0.72
0.86
0.78
0.84
0.82
0.85
0.89
0.85
0.82
0.
0.83
0.88
0.86
H.b-775
0881
0.76
0.75
0.81
0.78
0.80
0.79
0.80
0.81
0.79
0.81
0.79
0.82
0.81
Hub-7
0.84
0.77
0.75
0.83
0.78
0.81
0.80
0.81
0,86
0.82
0.80
0.62
0.80
0.84
0.83
0.90
0.89
0.88
0.90
0.90
Maikawu.Extraion
0.85
0.76
0.74
0.83
0.78
0.81
0.79
0.81
0.85
0.81
0.78
0.82
0.79
0.84
0.82
0.85
0.77
0.75
0.83
0.76
0.81
0.80
0.81
0,86
0.82
0.79
0.83
0.80
0.84
0.83
LNCaP _andrognn
0.82
0.76
0.74
0.83
0.77
0.79
0.7%
0:79
0.85
0.81
0.78
0.81
0.78
0.83
0.82
MCF-7.Iyyoxia
0.74
0.81
0.76
0.77
0.60
0,60
Medullo
0.72
0.71
0.69
0.73
0.71
0.72
0.71
0.72
0.75
0.74
0.71
0.74
0.72
0.74
0.73
Melano
0.71
0.65
0.63
0.70
0.66
0.68
0.67
0.68
0.71
0.70
0.67
0.69
0.67
0.70
0.69
Myumetr
0.84
0.71
0.68
0.82
0.75
0.80
0.78
0.60
0.84
0.77
0.61
0.79
0.83
0.81
Osteobl
0.68
0.71
0.71
0.71
0.72
PanIsleD
0.85
0.74
0.68
0.82
0.74
0.79
0.78
D.
0.84
0.81
0.76
0.81
0.79
0.83
0.82
Panislets
0.79
0.75
0.74
0.80
0.77
0.80
0.78
0.79
0.81
0.80
0.79
0.80
0.78
0.50
0.80
pHTE
0.81
0.73
0.70
0.79
0.74
0.78
0.76
0,78
0.81
0.78
0.76
0.78
0.77
0.80
0.79
ProgFib
0.76
0.21
0.83
0.76
0.79
0.81
0.78
0.82
0.80
0.82
RWPEL
0.90
0.74
0.68
0.85
0.76
0.83
0.81
0.83
0.88
0.84
0.80
0.84
0.82
0.87
0.85
Stellate
0.88
0.77
0.71
0.85
0.77
0.82
0.81
0.83
0.87
0.84
0.80
0.84
0.81
0.86
0.84
T-47D
0.81
0.75
0.73
0.79
0.75
0.78
0.77
0.76
0.81
0.77
0.79
0.76
0.79
CD4_Th0
0.75
0.80
0.76
0.79
0.76
0.79
0.79
Urothelia
0.90
0.79
0.76
0.87
0.81
0.85
0.84
0.86
0.89
0.87
0.83
0.86
0.84
0.88
0.87
Urothella_UT119
0.85
0.76
0.73
0.82
0.78
0.81
0.80
0.81
0.85
0.82
0.79
0.81
0.80
0.84
0.83
AG04449
0.90
0.74
0.65
0.83
0.72
0.79
0.76
0.80
0.87
0.81
0.75
0.82
0.78
0.86
0.83
AG04450
0.74
0.66
0.84
0.74
0.78
0.81
0.87
0.82
0.77
0.83
0.79
0.83
AG09309
0.89
0.73
0.65
0.83
0.72
0.78
0.76
0.80
0.87
0.80
0.75
0.82
0.78
0.85
0.82
AGB9319
0.89
0.75
0.67
0.84
0.74
0.80
0.78
0.81
0.67
0.82
0.77
0.83
0.79
0.85
0.83
AG10803
0.74
0.65
0.83
0.78
0.26
0.90
0.87
0.75
0.82
0.96
0.83
AcAF
0.66
0.83
0.73
0.77
0.76
0.78
0.83
BEL.C
0.80
0.73
0.69
0.80
0.72
0.76
0.75
0.77
0.83
0.78
0.73
0.79
0.75
0.81
0.79
BJ
0.89
0.75
0.66
0.83
0.73
0.79
0.77
0.80
0.87
0.81
0.75
0.82
0.78
0.85
0.83
Caco-2
0.91
0.91
0.90
0.92
0.91
0.91
0.91
0.93
0.92
0.90
0.92
0.91
0.92
0.93
CD20+
0.87
0.78
0.23
0.84
0.76
0.90
0.79
0.83
0:88
0.78
0.84
0.80
0.85
0.84
CD34+
0.87
0.75
0.70
0.83
0.73
0.78
0.77
0.82
0.87
0.81
0.75
0.83
0.78
0.84
0.83
CMK
0.81
0.74
0.69
0.81
0.72
0.76
0.74
0.79
0.85
0.78
0.72
0.81
0.76
0.81
0.81
GM06990
0.78
0.83
0.75
0.80
0.74
0.82
0.82
0.67
0.80
0.69
0.72
0.78
0.85
0.79
0.71
0.50
0.76
0.82
0,80
GM12865
0.86
0.74
0.67
0.81
0.70
0.76
0.73
0.79
0.85
0.79
0.71
0.80
0.76
0.83
0
0.81
H7-BESC
0.78
0.72
0.69
0.78
0.73
0.75
0.75
0.76
0.83
0.77
0.71
0.79
0.75
0.79
0.79
HAC
0.88
0.74
0.66
0.83
0.79
0.78
0.81
0.87
0.76
0.83
0.96
0.83
HAEpIC:
0.72
0.63
0.81
0.70
0.26
0.74
0.78
0.86
0.79
0.75
0.80
0.76
0.84
0.81
33

---

Published as a conference paper at ICLR 2024
HA-h
0.88
0.74
0.67
0.83
0.74
0.79
0.78
0.81
0.86
0.81
0.77
0.83
0.79
0
0.83
HA-pp
0.84
0.73
0.68
0.81
0.73
0.78
0.76
0.79
0.65
0.80
0.75
0.80
0.77
0.82
0.80
HEMEC
0.64
0.72
0.800
0.80
0.74
0.66
0:22
1,80
HCFaa
0.89
0.72
0.64
0.83
0.72
0.80
0.77
0.81
0.87
0.75
0.73
HCConF
HCM
0.89
0.89
0.74
0.73
0.67
0.65
0.84
0:82
0:72
0.74
0:78
0376
0.78
0.61
0:61
0.57
0.82
0.30
0.78
1275
0363
0.00
0.77
0.79
0.86
0.35
0.82
HCPE3C
0.88
0.71
0.64
0:71
0:27
0:75
6.79
13:5
0:79
0.76
0.84
HCT-116
0.89
0.73
0.68
0289
0.25
0:86
0.1
0.85
0.88
0.88
0.81
0.5T
9.8
HEEPIC
HEF
0.90
0.73
0.71
0.65
0.62
0.8
0.83
0.71
6.79
0.78
0.77
0.76
0,80
0.80
0.67
0.81
0.76
0.74
0.81
0.75
0.75
0.85
0.85
HFF-Myc
0.71
0.64
681
0:71
0.25
6:78
0:85
0:77
0:26
HGF
0.89
0.76
0.66
0.83
0.73
0.78
0.77
0.80
0.87
0.81
0.75
0.R2
0.78
0.85
0.83
HL-10
HIPERIC
0.88
0.81
0.70
0.72
0.64
0.64
0:81
0.77
0.71
0.78
0.68
0.77
0:79
0.75
03.86
0.80
0.06
1377
0.75
0:81
0.72
0.77
0.34
0.80
0.77
0.81
HMF
0.90
0.74
0.64
6.84
0.73
0:81
0.78
682
0:88
0:85
0.77
0883
0.80
0.87
0.84
HMYEC-dAd
0.89
0.76
0.70
0.85
0.76
0.81
0.811
0.79
0.85
0.88
0.85
0.79
0.8
0.80
0.57
1884
HMVEC-
0.89
0.74
0.06
0.84
0.72
0.76
03.2
0.88
0.81
1.75
0.83
0.75
0.35
HMVEC-dBBNNo
0.66
0.82
0.72
0.76
0.81
0.801
0.75
0.77
0.85
HMIEE-ddIy-d
0.88
0.76
0.68
0.83
0.74
0:29
0.78
6.11
13:57
0:81
0:77 0
0:22
0.800
HMVEC-dNNo
MMVEEC-dly-Ne
0.89
0.89
0.75
0.76
0.67
0.69
0.84
0.84
0.75
0.75
0.79
0
0.77
0.78
03:2
0.82
0.57
03:88
0.82
0.81
0.76
13.44
1
0.72
0.79
0.38
1134
HMVEC-LBI
0.65
0.85
0.72
0.77
0.
0.30
0.81
0.75
0.78
HMIVEC-LLy
0.87
0.68
0.82
0.74
0:78
0.77
0.90
02:66
0:000
0.76
0.28
0.55
0.82
HNPCKBIC
0.89
0.72
0.64
0.83
0.72
0:29
0.77
0.81
0.33
0:81
1.76
0.78
0
HPARC
0.88
0.75
0.68
0.83
0.74
0
0.78
03:2
0387
0.32
1.777
13.4
0.84
0.79
HPAF
0.73
0.65
0.83
0.72
0.77
0.80
0.87
0.81
0.75
0.75
0.78
0.50
HPOLF
0.89
0.75
0.66
0.83
0.73
0:29
0.77
6.0
D386
0:81
9.76
0:78
HPF
HRCFFI
0.90
0.85
0.75
0.72
0.65
0.67
0.85
0.82
0:75
0.72
0.81
0.78
0.79
0.76
0.79
0.82
0.85
0.98
0.88
1275
0.78
13.1
1883
1881
0.70
0.75
01.54
1111
HRR
0.83
0.78
0.81
0.87
0.82
1.57
0.90
0.55
HRGEC
0.88
0.73
0.66
0.83
0:73
0.77
6.82
D286
0:80
9.76
0:78
0.85
HRPF3I
0.83
0.78
0.65
0.80
0.70
0:25
0.74
0.77
0.84
0.70
1.73
1.79
0.76
0.55
0.80
HVMF
0.86
0.74
0.65
0.82
0.71
0.77
0.75
0.78
0.85
0.30
1.72
18:00
0.76
0.33
0.81
Jorkat
0.72
0.65
0.70
0.76
0.84
0.76
0.68
0.76
0.74
0.82
0.59
Mone
0.66
0.74
0.67
0.82
0.71
0:25
0.74
6880
0-88
0:81
0:77
0.84
0.8
NB4
0.87
0.74
0.68
0.83
0.72
0.27
0.75
0.80
0.88
0.81
0.74
1882
0.77
0.85
NHLA
0,89
0.75
0.66
0.84
0.73
0.79
0.77
0.81
0.87
0.76
0363
0.80
0.96
0.84
NHDF-Ad
0.74
0.64
0.81
0.75
0.78
0.55
0.80
0.76
0.84
NHDF-acO
0.87
0.76
0.65
0.82
0:71
0:27
0.26
6.79
D286
0.80
0:77
0.84
0.82
NHLF
0,89
0.74
0.65
0.83
0.78
0.79
0.77
0.81
0.87
0.81
0.76
0.8
0.79
0.86
0.83
NT2DDI
0.82
0.74
0.71
0.81
0.76
0.79
0.78
0.80
0185
0.800
0.776
0.82
0.78
0.82
0.81
PANC-I
0.65
0.79
0.78
0.62
0.84
0.82
PYFC
0.89
0.73
0.64
0.83
0.72
0.29
0.77
0:87
0.75
0.
0.79
0.85
RPTRC
0.84
0.71
0.65
0.81
0.72
0.28
0.75
0.79
0.84
0.79
1.74
0.80
0.77
0.85
0.80
SAFC
0.90
0.71
0.62
0.82
0.71
0.79
0.77
0.80
0.87
0.80
0.75
0.81
0.78
0.85
0.82
SKMC
0.65
0.83
0.76
0.79
0.75
0.81
0.85
0.82
SK-N-MC
0.81
0.74
0.68
0.79
0.71
0:26
0.74
0:77
0.81
0:77
0.75
0.78
0:75
0.80
0.78
SK-K-SHLRA
0.87
0.85
0.81
0.89
0.83
0.85
0.85
0.86
0.90
0.87
0.83
0.88
0.85
0.89
0.89
0.88
Th2
0.86
0.79
0.73
0.84
0.76
0.78
0.78
0.83
1.87
0.82
0.77
0.84
0.81
0.86
0.84
WERI-Rb-1
0.65
0.81
0.72
0.67
0.79
0,80
WI-38
0.89
0.73
0.64
0.83
0.72
0.79
0.77
0.81
0.87
0.81
0.76
0.82
0.78
0.85
0.83
W1-3L4OHTAM
0.84
0.72
0.62
0.81
0.70
0.27
0.75
0.79
0.84
0.79
0.77
0.80
0.77
0.83
0.80
A.49
0.84
0.71
0.67
0.81
0.74
0.79
0.76
0.80
0.84
0.80
0.77
0.80
0.78
0.82
0.81
GM12478
0.69
0.73
0.73
0.78
HI-BRSC
0.6
0.82
0.80
0.85
0.82
0.84
0.84
0.84
0.87
0.84
0.82
D.85
0.84
0.86
0.85
HeLa-S3
0.82
0.70
0.66
0.79
0.71
0.76
0.75
0.77
0.82
0.78
0.74
0.78
0.76
0.80
0.78
HapG2
0.85
0.79
0.78
0.84
0.80
0.83
0.82
0.83
0.86
0.84
0.81
0.84
0.82
0.85
0.84
HMEC
0.68
0.77
0.74
0.76
0.73
0.77
0.77
HSMM
0.84
0.72
0.65
0.80
0.71
0.25
0.74
0.77
0.83
0.78
0.73
0.79
0.76
0.81
0.80
HSMMMutbe
0.83
0.74
0.69
0.80
0.73
0.77
0.76
0.78
0.84
0.79
0.74
0.79
0.77
0.82
0.80
HUVEC
0.86
0.75
0.69
0.83
0.75
0.79
0.78
0.81
0.85
0.77
0.82
0.79
0.84
0.82
K562
0.69
0.78
0.73
0.75
0.72
0.77
0.76
ENCAP
0.74
0.71
0.67
0.75
0.68
0.20
0.69
0.71
0.77
0.73
0.68
0.73
0.70
0.76
0.74
MCF-7
0.80
0.69
0.67
0.77
0.69
0.75
0.72
0.74
0.79
0.75
0.71
0.75
0.72
0.77
0.76
NHEK
0.86
0.72
0.67
0.81
0.74
0.79
0.77
0.80
0.85
0.80
0.76
0.81
0.78
0.83
0.81
ThE
0.74
0.78
0.77
0.78
0.78
Table A10: Histone modification prediction performance per label.
GEMA-LM Righied
NI-100NG (2.50)
GUNA-LM BERT
HNA tin
DyonnDAA large
RoNeb-LM
DNABERT
DNABERF-2
GROVER
Banset
CNN
NT-II
NTAM
NEV2
Modifiation (lubel no.)
H3K27mma (SMuun3KK5622000
0.63
0.67
0.66
0.70
0.69
0.70
0.60
0.68
0.72
0.70
0.70
0.1
0.68
0.70
H339 K562 (1)
0.57
0.85
0.85
0.87
0.86
0.56
0.87
0.56
0.87
0.56
0.86
0.87
0.87
0.87
0.7
H3399m3.. K562
0.74
0.77
0.75
0.84
0.83
0.83
0.83
0.78
0.53
0.84
0.86
0.56
0.79
0.0
0.2
H.344mll.5 1_661
D.65
0.68
0.67
0.71
0.60
13K99<_K30) (4)
0.74
0.75
0.70
0.74
0.73
0.75
0.74
0.74
0.77
0.75
0.75
0.75
074
0.74
0.5
H334mel K562 (5)
0.60
0.80
0.80
0.81
0.81
0.81
0.81
0.50
0.82
0.81
0.80
0.1
081
081
081
H13K3mm33 K562
0.63
0.65
0.62
0.70
0.70
0.74
0.71
0.66
0.70
0.72
0.73
0.54
0.65
0.67
0.69
H3K36me3 K562
0.75
0.78
0.77
0.77
0.77
0.78
0.76
0.77
0.77
H4K30me1_KSS (8)
0.62
0.69
0.60
0.71
0.69
0.71
0.69
0.69
0.72
0.71
0.70
0.72
0.69
0.69
0.1
34

---

Published as a conference paper at ICLR 2024
HI3K27m3 K566) (9)
0.74
0.74
0.75
0.90
0.79
0.50
0.80
0.79
0.90
0.79
0.50
0.50
0.78
0.77
0.80
113K4me3 K562 (10)
0.88
0.89
0.87
0.89
0.88
0.89
0.89
0.89
0.9
0.39
0.89
0.89
0.89
0.89
0.89
H334mm33 K562 (11)
0.89
0.39
0.85
K56mme335666100 (12)
0.54
43334me3 K362 (13)
0.76
0.77
0.72
0.77
0.75
0.77
0.76
0.76
0.99
0.77
0.77
0.78
0.76
0.75
0.77
H33K79mc2 K562 (14)
0.774
0.76
0.75
0.76
0.76
0.76
0.76
0.75
0.76
0.76
0.76
0.77
0.76
0.70
0.76
1334mm22 K362 (15)
0.70
0.72
0.71
0.71
0.31
0.75
0.72
0.72
K3372ac..36620 (16)
0.67
0.71
0.70
0.70
0.71
0.76
0.71
0.72
0.71
0.71
0.72
H2AFZ.X562(17) (17)
0.70
0.71
0.57
0.72
0.71
0.73
0.71
0.71
0.75
0.73
0.73
0.73
0.70
0.70
0.72
Table A1l: CpG methylation prediction performance per cell line.
Model
SK-N-SH ENCFF361KCL
GM23248 ENCHPL/0XY]
A549
HepG2 ENCEFOOPNN
HURS64 ENCFE9S0GMD
GM23248 ENCFES40XVU
HeLa-S3
EN
ENCEFISARAW
Bassct
0.93
0.94
0.93
0.95
0.94
0.93
0.93
ResNet-LM
0.86
0.87
0.86
0.85
0.91
0.87
0.86
AWD-LSTM
0.50
0.80
0.80
0.78
0.39
0.30
0.79
NI-HI
0.57
0.87
0.94
NE-MS
0.92
0.92
NT-1000G (2.58)
0.38
0.88
NI-V2
0.0
0.91
0.90
0.88
0.96
0.9
0.90
DNABERT
0.91
0.91
0.91
0.88
0.96
0.91
0.90
DNABERT-2
0.89
0.89
0.89
0.87
0.96
0.89
0.89
GENA-LM BERT
0.91
0.91
0.89
0.95
0.91
0.90
GENA-LM Bighted
0.00
HyenaDNA
HyenaDNA large
0.91
0.91
0.91
0.88
0.4
0.9
0.90
GROVER
0.53
0.89
0.88
0.86
0.94
0.89
0.8
Table A12:
Variant effect prediction performance (AUROC) on the expression variant effect pre- diction dataset, stratified by variant category. Categories that only have samples of one label were ommitted as no AUC can be determined. For completeness, also AUROCs on categories with very low sample numbers are reported, but should be interpreted with caution.
Noncoding crancrriyt
Spiice pet pap lnnidie
Splic donur Sib bance
Splice denor region
Downstream gene 08-4 (-4551
Begnlatory regiun (=44000)
Upstecann gene (am4, 3399
TV' hadiing dite
Splice regionn
trrac (n-n (1))
Syneas moses
Splice denur (a-155)
(0-401972)
Intergenic
1,5
UTR
UTR
(a-482)
(0-4333
(ne: 229
Missense (0))
24)
(ne: 13)
Stop lrst
(b-2)
Model
DecpSEA
0.70
0.69
0.71
11.11
0.71
0.68
0.64
0.72
0.64
0.55
0.62
0.72
0.46
0.54
0.35
0.83
1.00
ResNe-LM
0.55
0.54
0.56
0.55
0.52
0.51
0.54
0.44
0.46
0.44
0.54
0.49
0.60
0.70
0.50
01.00
1.00
AWD-LSTM
1.53
0.54
0.52
11.55
0.56
0.53
0.51
0.51
0.51
1148
0.51
0.44
1131
0.25
0.38
1137
0.00
NT-H
1155
0.54
0.54
11.55
0.52
0.51
0.09
0.45
0.44
0.51
0.35
0.57
1136
0.71
0.50
0167
1,00
NT-3IS
1.55
0.53
0.54
I155
0.56
0.35
0.57
0.48
0.53
0.54
0.51
0.54
1156
0.65
0.19
0.60
1,00
NI-1990-2250
0.44
0.45
0.43
0.44
0.18
0.46
0.48
0.42
0.57
0.42
0.40
0.14
0.39
0.54
0.27
0.21
1,00
NT-1090G-5000 S00M
0.49
0.48
0.19
0.47
0.50
0.33
0.30
0:45
0.51
0.51
0.48
0,50
0.66
0.29
0.46
0.35
0.00
NT-V2-500M
0.48
0.47
0.66
0.48
0.50
0:06
0.30
0:41
0.51
0.58
0.54
0.34
0.51
0.68
0.77
0.40
0.00
DNABERT
1.60
0.59
0.61
1.60
0.57
6.57
0.55
0.60
0.51
11.1
0.70
0.57
0.79
8.1
0.65
115
0.00
DNABERES
1.49
0.49
0.57
1.49
0.55
0.48
0.48
0.52
0.57
1149
0.47
0.55
I.78
0.59
0.35
1152
1,00
GENA-LM BERT
0.49
0.49
0.50
I150
0.56
0.51
0.31
0.49
0.55
0.51
0.47
0.53
I.27
0.29
0.58
0.60
0.00
GENA-LM Bighard
0.49
0.48
0.18
0.49
0.52
0.51
0.52
0.49
0.53
0.51
0.47
0.53
0.24
0.43
0.35
0.55
0.00
IlyenaDNA large
0.51
0.52
0.50
0.52
0.55
031
0:30
0:49
0LES
0.47
0.54
0.09
0.37
0.26
0.19
0.35
0.00
TyAeaADNA medium (1668)
0.48
0.49
0.47
0.50
0.5
0.69
0:60
0:46
0.6
0.49
0.5
0.51
0.34
0.2
0.19
0.35
0.00
HyAmaDNA medium (4566)
0.30
0.51
0.49
0.52
0.54
0.50
0:90
045
0.48
0.5
0.54
0.39
0.26
0.19
0.38
0.00
HyemaDNA small
0.46
0:47
0.45
0.47
0.50
0.47
0:48
0:47
0.50
0.49
0.50
0.43
0.32
0.25
0.19
0.35
0.00
DyenaDDA thay
0.47
0.48
0.54
0.49
0.11
0.09
0.48
0.45
0.50
0.48
0.49
0.39
0.37
0.35
0.23
0.25
0.00
GROVER
0.55
0.55
0.58
0.55
0.35
0.56
0.56
0.30
0.56
0,41
0.48
0.66
0.36
0.86
0.42
0.74
0.00
35

---

Published as a conference paper at ICLR 2024
Table A
Variant effect prediction performance (AUROC) the disease variant effect prediction dataset, stratified by variant category. Categories that only have samples of one label were ommitted as no AUC can be determined. For completeness, also AUCs categories with very low sample numbers are reported, but should be interpreted with caution.
Splice pntpyyimiddiee trant (n=39) (9966666
Noncoding Noncodingtramerri tramerrpt exon (23,721)
Splice donor region (mm-2. 2,056)
Downstrean Gene (m=274)
Splie acceptor (o=9,280)
Upstream gune (ma)2 2,2444
Mature miRNA
In- 138,111
Splice region 40, (0.3660)
Spiice donor (nE 11.111
20:20,4411
UTR (1am6, (996)
(ne 1 (060))
Intergenic
Intron
3 UTR
Splice
(u=20))
Model
DeepSEA
0.48
0.46
0.73
0.69
0.48
0.58
0.61
0.45
0.41
0.72
0.18
0.92
ResNet-LM
0.51
AWD-LSTM
0.53
0.56
0.59
0.52
0.50
0.48
0.52
0.46
0.47
0.50
0.43
0.40
0.12
0.53
NT-
0.43
0.53
0.49
0.51
0.56
0.51
0.52
0.53
0.52
0.49
0.50
0.53
0.38
0.58
NTMM
NT-1000G-2.58
0.49
0.57
0.54
0.52
0.48
0.51
D.50
0.47
0.45
0.52
0.52
0.45
0.10
0.11
WT-1000--000
0.46
0.53
0.50
0.49
0.40
0.49
0.47
0.41
0.47
0.49
0.36
0.63
NT-V2-500M
0.50
0.49
0.50
0.53
0.52
0,43
0.54
0.21
DNABERT
0.52
0.55
0.47
0.48
0.63
0.54
0.51
0.62
0.58
0.55
0.56
0.62
0.72
0.05
DNABERT-2
0.48
GENA-LM BERT
0.50
0.49
0.48
0.51
0.50
0.56
0.60
0.50
0,41
0.47
0,42
0.50
0.95
1.00
GENA-LM BigBird
0.48
0.48
0.46
0.51
0.52
0.54
0.60
0.46
0.45
01.8
0.43
0.60
1.00
0.89
HyenaDNA large
0.53
HyenaDNA medium 160%
0.52
0.54
0.60
0.54
0.46
0.44
0.51
0.46
0.50
0.47
0.41
0.41
0.00
0.47
HyenaDNA medium 450k
0.53
0.53
0.58
0,51
0.46
0.45
0.50
0.51
0.50
0,47
0.42
0.35
0.00
0.37
HyenaDNA
INyenaDNA tiny
0.53
0.55
0.59
0.53
0.47
0.44
0.52
0.48
0.50
0.48
0.42
0.44
0.08
0.37
GROVER
0.50
0.46
0.42
0.48
0.54
0.49
0.53
0.55
0.52
0.49
0,45
0.42
0.21
0.21
Table A14: Variant effect prediction performance on the disease variant effects prediction dataset with more stringent filtering. Variants labeled as "Likely" ClinVar were omitted, yielding reduced dataset (Benign n=100,233, Pathogenic p=8,188), Similarly to the results on the full dataset, NT-MS outperforms DeepSEA. Additionally, ResNet-L.M and DNABERT show strong per- formance.
Model
AUC
DeepSEA
0.57
ResNet-LM
0.61
AWD-LSTM
0.45
NT-H
0.52
NT-MS
0.74
NT-1000G-22558
0.49
NT-1000G-500M
0.46
NT-V2-500M
0.48
DNABERT
0.62
DNABERT2
0.50
GENA-LM BERT
0.56
GENA-LM BigBird
0.52
HyenaDNA large
0.44
HyenaDNA medium 160k
0.43
HyenaDNA medium 450k
0.44
HyenaDNA small
0.41
HyenaDNA tiny
0.43
GROVER
0.52
36


Reviews:
[{"review_id": "tD89w01qcn", "reviewer": "Reviewer_hBHB", "summary": "Summary of the paper:\nThe paper presents BEND, a benchmark for DNA language models that focuses on biologically meaningful tasks defined on the human genome. The aim is to evaluate the ability of unsupervised language modeling techniques in annotating various functional and regulatory elements within DNA sequences. The results show that while current DNA LMs can rival expert methods in some tasks, they struggle with capturing long-range features in the DNA.", "questions": "I believe this work has significant potential. If the authors address the concerns raised in my comments, I would be inclined to recommend a higher score for this submission. \n\n- Figure 1 appears to inaccurately represent the length of certain genomic features. For example, the majority of exons in the human genome are shorter than 200 base pairs, with an average length between $120-170$ base pairs. Furthermore, promoter lengths usually range from 100 to 1000 base pairs. The diagram should at least offer a rough indication of these lengths. It's also crucial to highlight that exons and introns alternate in their appearance, a detail that the figure should encompass.\n- In the introduction, l'd suggest mentioning foundational works like DeepSEA alongside DeepBind when discussing supervised learning on DNA. These groundbreaking studies have left an indelible mark on the field.\n- Concerning the enhancer annotation, there is some uncertainty regarding the authenticity of the ground truth. The ABC method is based on inference and might not reflect a direct experimental outcome.\n- Clarity is sought on the dataset splitting methodology - why was the data from the pretraining phase included? What about considering a leave-chromosome-out approach to splitting? The methods to prevent data leakage should also be elucidated.\n- The decision to categorize variant effect prediction as a binary problem raises questions. Given the intricate nature of genomic variations and the subsequent implications, framing this as a regression task might be more appropriate.\n- It's crucial, especially with such intricate datasets, that downstream models be evaluated using a leave-chromosome-out strategy to guarantee robust results.\n- Te absence of a simple CNN baseline for the variant effect prediction task is a notable omission. Such a baseline would not only validate the implemented supervised models but also provide an insight into the relative performance of more complex models.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The overarching motivation behind creating BEND, which emphasizes understanding the genome across longer ranges, is commendable.\n- The efforts to collect the benchmark dataset are commendable, and the dedication shown in running benchmarks for so many language models and supervised baselines is admirable.", "weaknesses": "- However, I feel that the task formulation for long sequences lacks depth and isn't entirely persuasive.\n- When benchmarking DNA Language models, it's crucial to explore the intricacies of training at least one of these models from scratch. This would provide a comprehensive insight into their potential and the limitations of pretraining.\n- A side-by-side comparison with a DNA Language model trained from scratch is essential. Such an analysis would give a more rounded perspective on the strengths and shortcomings of the existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LeYoeVDAOf", "reviewer": "Reviewer_cZPk", "summary": "This paper presents a collection of benchmark tasks that are intended to measure the performance of a DNA language model.  The tasks are quite diverse, both in terms of the phenomena they cover and their difficulty level.  Providing such a benchmark is useful, though several recent studies have already done this. The main novel contribution here is including tasks that take very long sequences as input, such as predicting enhancer-promoter interactions.\n\nN.B. I increased my score by one point after reading the reviewer response.", "questions": "Why are enhancers defined to be 128bp?  I think of an enhancer as minimally corresponding to a missing nucleosome, plus its linker, which would be 225-250bp.  Some are significantly larger.\n\nWhy did you choose to frame the enhancer task as annotation rather than matching?  It seems weird to refer to this as detecting \"long-range interactions,\" since the task does not require that a given enhancer actually be operating on the gene at the center of the selected window.  I also don't understand in what sense this annotation framing of the problem can be considered \"more stringent.\"\n\nWhy did you choose not to do fine-tuning for each task?  It's not clear to me that, in practice, anyone would adopt a fully unsupervised approach to this kind of supervised problem.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "I really liked the section that provided biological background -- it was clear and concise.\n\nThe benchmark tasks are well described, and each one is important for a DNA language model to be able to address.\n\nSome of these tasks are more challenging than the ones used in previous studies.", "weaknesses": "Including only tasks from the human genome is problematic.  It seems clear that a good language model of DNA should cover more than just one species.  The most recent competing benchmark (Gresova 2023) includes eight tasks from three different species.\n\nThis benchmark does not improve very much over the Gresova benchmark published this year.\n\nminor:\n\nIt would be good to point out, in Section 2.1, that these descriptions are about eukaryotic genomes.\n\nIntroduce \"secondary structure\" before using the abbreviation \"SS.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5gwgibAHNj", "reviewer": "Reviewer_vBTw", "summary": "The mansucript presents a set of benchmark datasets for evaluating DNA language models (DNA LMs) and establishes baseline performance for these datasets using existing DNA LMs.  It also compares the LM-based approach to a simple direct supervised learning approach.", "questions": "- In Table 3 the authors provide accuracy from the literature.  Not clear how relevant those numbers are since they are based on potentially very different datasets.  It should be noted in the manuscript that those numbers should be taken with a grain of salt.\n\n- The authors note that \"reasoning over very long contexts, as e.g. required for finding enhancers, is still challenging.\"  Is this too much to expect from these models?  (i.e expect them to reason over long sequence contexts). Perhaps we should be content with LMs producing representations that are based on short contexts, and leave long-range contexts for downstream models?\n\n- The authors say the provide \"An adaptable benchmarking framework for preparing embeddings and training lightweight supervised models.\"  This point requires some elaboration - I assume you are referring to the code in the (currently anonymous) github repository.\n\n- You had noted that your benchmark's advantage is a mix of short and long sequence classification; however, I think you could have chosen better problems for long sequence tasks, e.g. gene expression prediction; also, the histone modification prediction task might have benefitted from longer sequence contexts.\n\nMinor comments:\n\n- A comment regarding splice site classification:  You note that \"Moreover, there are cases in which a short-sequence task represents a simplification compared to real-world applications, as exemplified by SS-containing sequences. In genome annotation, classifying SSs is a subproblem of the gene finding task and would typically not be performed on its own.\"  This is mixing the issue of short sequences and how a splice site classifier would be used.  From my experience, SS classification is a relatively easy task that does not really need long sequences; while it is used in gene finding, it has other applications related to gene annotation; the condition-dependent version of the problem stands on its own (see e.g. the recent paper on the pangolin deep learning model).  \n\n- \"The availability of unlabeled genomic sequences and limited labeled data appear to make language modeling a natural fit for DNA.\"\nThere is actually a wealth of labeled data - models like Sei and Enformer are very successful at leveraging large scale labeled data.\n\n- \"Gene-adjacent regulatory elements are referred to as cis, and distant ones as trans.\"  The definition of cis and trans is not a matter of distance!\n\n- Reference missing in 3.3:  and follows the methodology of ??.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Strengths:\n\n- While there exist other recent benchmark datasets for DNA LMs, the authors' datasets are complementary to the existing ones.  In particular, the tasks are more realistic than those in the \"Genomic benchmarks\" paper (but see some caveats below).\n\n- The authors provide a comprehensive comparison of existing DNA LM performance on the benchmark datasets; they also provide a comparison to a trained-from-scratch simple supervised model.", "weaknesses": "Weaknesses:\n\n- The supervised learning baselines are useful, but are rather weak.  For example, in the context of histone modifications, a model such as Sei which is trained on much larger datasets has achieved much higher accuracy (although the results are not strictly comparable).  I expect such a model to perform much better than the DNA LM approach.\n\n- Regarding the enhancer prediction task:  This is a case where additional forms of data such as DNA accessibility and DNA contacts are able to provide additional hints that make this problem more tractable, and likely provide much higher accuracy than reported here (see e.g. the EPCOT model from the Liu lab).\n\n- The gene-finding task is somewhat contrived, as the resulting prediction need post-processing is required to convert those predictions into a coherent gene model.  Also, it's not clear how alternative splicing was handled, as it makes it difficult to assign a strict label to each position.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NqOA6gSbNo", "reviewer": "Reviewer_gFhh", "summary": "This paper is concerned with BEND, a Benchmark for DNA language\nmodels, investigating several biologically down-stream tasks defined on the human genome.\n\nIt is a interesting to read, as it compares a number of frameworks on biological task of interest. The training data seems to mainly be human curated biological data sets. Is this correct? Although it should not be considered an important critique of the submission, I strongly believe that the future of computational biology will be analysis of more complex data sets having a more complete signal. That is, the annotated data is often not sufficient, and richer experimental datasets will facilitate more interesting analyses. The underlying reason is that the biological reality is more complex: there may be multiple start sites, etc. Nevertheless, the described models can certainly serve as a step in this direction\n\nHowever, the paper contains quite a lot of background and it is a benchmarking study of several methods that only available as rxiv papers, according to the references. The call for papers does not spell out how focused the conference is on original research, which makes the fit of the paper  somewhat unclear. I would, however, not priorities this paper in a comparison with the average ICLR paper.", "questions": "What is the performance of non LM models on these tasks?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper provides an interesting comparison of the performance of several frameworks across 5-6 prediction task of relevance to modern biology.", "weaknesses": "This is not original research and a substantial fraction of the paper is concerned with background material.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)