BANANA: a Benchmark for the Assessment of Neural Architectures for Nucleic Acids
Authors: Luca Salvatore Lorello
Andrea Galassi
Paolo Torroni
Venue: ICLR 2022 Submitted (2022)
Keywords: bioinformatics
language modeling
natural language processing
dataset
benchmark
dna
rna

Abstract: Machine learning has always played an important role in bioinformatics and recent applications of deep learning have allowed solving a new spectrum of biologically relevant tasks.
However, there is still a  gap between the  ``mainstream'' AI and the bioinformatics communities. This is partially due to the format of bioinformatics data, which are typically difficult to process and adapt to machine learning tasks without deep domain knowledge.
Moreover, the lack of standardized evaluation methods makes it difficult to rigorously compare different models and assess their true performance.
To help to bridge this gap, and inspired by work such as SuperGLUE and TAPE, we present BANANA, a benchmark consisting of six supervised classification tasks designed to assess language model performance in the DNA and RNA domains. The tasks are defined over three genomics and one transcriptomics languages (human DNA, bacterial 16S gene, nematoda ITS2 gene, human mRNA) and measure a model's ability to perform whole-sequence classification in a variety of setups.
Each task was built from readily available data and is presented in a ready-to-use format, with defined labels, splits, and evaluation metrics.
We use BANANA to test state-of-the-art NLP architectures, such as Transformer-based models, observing that, in general, self-supervised pretraining without external corpora is beneficial in every task.

Under review as a conference paper at ICLR 2022
BANANA: A BENCHMARK FOR THE ASSESSMENT OF NEURAL ARCHITECTURES FOR NUCLEIC ACIDS
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning has always played an important role in bioinformatics and recent applications of deep learning have allowed solving a new spectrum biologically relevant tasks. However, there is still a gap between the "mainstream" Al and the bioinformatics communities. This is partially due to the format of bioinformat- ies data, which are typically difficult to process and adapt to machine learning tasks without deep domain knowledge. Moreover, the lack of standardized evalu- ation methods makes it difficult to rigorously compare different models and assess their true performance. To help to bridge this gap, and inspired by work such as SuperGLUE and TAPE, present BANANA, benchmark consisting of six pervised classification tasks designed to assess language model performance in the DNA and RNA domains. The tasks are defined over three genomics and one tran- scriptomics languages (human DNA, bacterial 16S gene, nematoda ITS2 gene, human mRNA) and measure model's ability to perform whole-sequence clas- sification in a variety of setups. Each task was built from readily available data and is presented in a ready-to-use format, with defined labels, splits, and evalua- tion metries. We use BANANA to test state-of-the NLP architectures, such as Transformer-basee models, observing that, in general, supervuue pretraining without external corpora is beneficial in every task.
INTRODUCTION
Since the advent of sequencing techniques (Sanger et al., 1977). there has been an exponentially increasing availability of data the form of sequences of DNA. RNA, and proteins. Analysis of these sequences has marked many milestones in every field of biology, from the Human Genome Project (Watson| 1990) to the engineering of vaccines for the SARS-COV-2 virus (Chen al. 2020), from climate-change studies (Jansson & Hofmockel, (2020) to forensics (Bianchi & Liō 22007). ). Due the huge amount of data, these advances could have not been achieved without the support of bioinformatics algorithms.
traditionally solved by search algorithms, such as BLAST (Altschul et al., 1990),, which matches sequence against reference database contain- ing annotated entries (and so tasks are solved in terms of similarity). Databases are usually very large and search can be very slow, impacting fields in which timely response is important, such as cancer diagnostics. Indexing strategies or heuristic searches can speed up the querying process, at the expense of space accuracy trade-off.
Many tasks in bioinformatics
The application machine learning methods has provided great contribution to this field, allowing to solve tasks in computationally inexpensive way, with greater robustness to noise, and to discover new associations (Vingal 2014: Larranaga et al. (006)).
In particular, the recent development of Natural Language Processing (NLP) techniques has lead researchers to use advanced neural models for modeling biological sequences, such as proteins, DNA, and RNA (Ji et al. 2021 Zaheer et al. 2020 Wahab al. 2021).
However, the heterogeneity in the databases where biological data are stored and the lack of general consensus the data processing makes it difficult define machine learning tasks, and more in general to approach this domain without having a deep understanding of it. At the same time, there is consensus in the NLP community that to properly evaluate a language model there is the

---

Under review as a conference paper at ICLR 2022
necessity to test in on multiple tasks measure whether it can be broadly applied. For this reason, multi-tasks benchmarks such as SUPERGLUE (Wang et al., 2019a) have been developed. To meet these needs, Rao et al. (0119) have developed TAPE, a benchmark designed to rigorously evaluate the performance of learned protein embeddings through a multitude of tasks, encompassing the domains of structure prediction, evolutionary understanding, and protein engineering, which have similarities with NLP tasks such as named entity recognition and natural language classification.
A different class of languages, not covered by TAPE, ranges over the DNA and RNA domains. These domains are crucial in bioinformatics because they enable researching tumors, subcellular structures and the environment. Accordingly, we present BANANA (Benchmark for the Assessment of Neu- ral Architectures for Nucleic Acids), a benchmark consisting of six classification tasks assessing language understanding performance in the DNA and RNA domains. The tasks are defined over three genomics (human DNA, bacterial 16S gene, nematoda ITS2 gene) and one transeriptomics (human mRNA) language and measure an architecture's 's capability of performing whole-sequence classification in variety of setups. We curate existing data providing datasets that do not impose any specific constraint regarding the approach, creating a benchmark that is versatile and easy to use even for those who do not have knowledge of the domain. Finally, we perform few experiments with advanced neural architecture, so as to provide baselines for future works.
In Section 2 we describe existing benchmarks and draw similarities and differences with our work. Section provides background for our domain of application and highlights the importance of our tasks, which are described in Section [ We evaluate our baselines in Section 5 while Section 6 list related works for each task. Finally, Section concludes.
NATURAL LANGUAGE UNDERSTANDING AND PREVIOUS BENCHMARKS
In the context of unstructured data, such as textual documents, their representation plays a key role in the performance of any machine learning algorithm. Indeed, representations capable of encoding information not only about the lexicon but also about the syntax and the semantics of a sentence, provide the models with additional knowledge that can be exploited in higher-level tasks. The task of learning effective representations is called Natural Language Modelling (NLM), and it is typically addressed through semi-supervised tasks (Devlin et al. 2019). Once learned, the quality of a language model can be evaluated by using it to address more ad advanced NLP tasks that concern the comprehension of the text, set of tasks that falls under the broad definition of Natural Language Understanding (NLU).
Our proposal draws inspiration from the GLUE benchmark (Wang et al. 2019b), a set of super- vised sentence understanding tasks in the English language based on existing datasets. Its authors proposed it with the purpose of stimulating the development of more unified language model, ca- pable of solving range of different linguistic tasks in different domains. Their experimental results showed how models trained in multi-task learning setting outperformed models trained on a single task, and therefore the benefits of sharing knowledge across tasks. But the interest in most of these tasks has quickly decreased with the advent of advanced language models such as BERT, which was capable of easily solving most of them. Therefore, new benchmark called SUPERGLUE (Wang) let al. 2019a) has been later proposed, including new supervised learning tasks that are solvable by humans but are difficult for machines. All the tasks proposed in this benchmark regard the classi- fication of one or multiple sequences of text, and a final score is given based on the average score obtained on every single task. Similarly, our tasks regard the classification of single sequence, and we compute a final score in a similar fashion. Another analogy between these works and our proposal concerns the fact that they are agnostic regarding the pre-train of the architectures and do not include the language modeling task as part of the benchmark.
The TAPE benchmark (Rao al., 2019) fills the need for datasets and standardized evaluation methods for machine learning techniques applied to protein sequences, and it is another source of inspiration for our work. The authors propose to use NLP techniques in the domains of proteins, presenting the self-supervised learning task of modeling the "languages of proteins" using a dataset containing proteins of different organisms and absolving different functions, along with 5 supervised learning tasks concerning 3 major subfields of protein sciences. All these tasks are based on existing datasets, but they are curated and formalized by Rao et al.] The experimental part shows that the self- supervised task is beneficial for all the neural models and that they can successfully address three
2

---

Under review as a conference paper at ICLR 2022
of the five tasks. It also highlights how the performances of each model vary across the different sub-fields. Our work differs from TAPE since we address tasks that span across four different "biological languages" (human DNA, human mRNA, bacterial and archacbacterial DNA, nematoda DNA) and specifically address each of them. Instead, TAPE's tasks focus on multi-lingual settings, which requires the ability generalize over broad spectrum of "protein languages".
3 BIOLOGY BACKGROUND
The central dogma of molecular biology (Crikk 1970) states that, for every living organism, the information is stored in DNA (deoxyribonucleic acid), then it flows to RNA (ribonucleic acid) and finally to proteins, which are the building blocks of life but cannot be transferred from one generation to the next. Both DNA and RNA are composed of sugar-phosphate scaffold and nucleotides (nt), which contain the genetic information. DNA is a double helix composed of two filaments which complementary nucleotides (A, T, C, G) bind to each other forming base pairs (bp). while RNA is single strand of nucleotides (A, U, C, G), capable of folding in different ways
RNA is synthesized (transcribed) by making a complementary copy (using different alphabet, but semantically a 1:1 copy) of a region of DNA (a gene), starting from the transcription start site (TSS) up termination point. In order for the copying machinery to "select" the proper region, some transcription factors (TF) must bind to DNA a region immediately before the TSS, called the promoter of that gene. DNA is compressed into a structure called chromatin, whose folding structure plays an important role in cell differentiation: a liver cell and a neuron share the same DNA, but thanks to different chromatin profiles they appear and behave in completely different ways. These mechanisms are fundamental for life and their malfunction is linked with cancer (Morgan & Shilatifard 2015).
RNA absolves many functions inside the cell (e.g., gene regulation, signaling, etc.) and it is also correlated to cancer (Reddy) 2015) and infections (Fani et al. 202T). Three main types play universal roles across every living organism: ribosomal RNA (rRNA) and transfer RNA (IRNA) are the gears in the machinery for protein synthesis, while messenger RNA (mRNA) constitutes the "recipe" for proteins. Freshly transcribed mRNA filaments migrate each to a specific position inside the cell, known as their subcellular localization (Holt & Bullock, 2009). and then ribosomes translate them to proteins. The analysis of mRNA localization may help to better understand subcellular compartments and lead to a more detailed and nuanced understanding of cellular architecture (Martin & Ephrussi 2009).
Ribosomes translate mRNA into proteins and are made of two subunits: small (SSU) and large (LSU). The bacterial SSU is called 16S and, since evolves with a very slow mutation rate, it can be used to reliably classify bacteria, also for the so-called "microbial dark matter", those bacteria which cannot be classified using traditional methods (Kalmbach et al. 1997), and which can have an effect, for example, on health (Shreiner et al.) 2015) or climate change (Jansson & Hofmockel] 2020). Eukaryotes can be classified by SSU markers (18S) as well, however popular alternatives are the Internal Transcribed Spacer 2 (ITS2), a region between the SSU and the LSU which is transcribed, but then destroyed instead of being part of ribosomes, and the mitochondrial SSU (12S).
Sequencing is the process of reading a sequence of DNA, RNA, or protein, starting from sam- ple. Genomics, transcriptomics, and proteomics are the studies of data (DNA, RNA, and proteins, respectively) sequenced from single organism. When this is done on data sequenced from mul- tiple organisms at the same time, the studies are called metagenomics. metatranseriptomics, and metaproteomics. For example, extracting all the DNA present in environmental sample in der to determine which bacteria live in that environment is a classical example of metagenomics. Sequencing technologies are characterized by various kinds of reading errors, an important one arising during meta-omics sequencing is the production of chimera artifacts. Chimeras are digi- tal sequences that do not exist in the real world, instead they are composed of portions of existing sequences that have been hybridized. They are produced when sequencing with highly parallel technologies is stalled while it is reading sequences that share a similar region. For example, if
'Sice base pairs are complementary. from linguistic perspective, single DNA strand contains the en- tire information, so both bp, for DNA, and nt, for RNA, can be considered the "characters" of nucleic acids languages.
3

---

Under review as a conference paper at ICLR 2022
AACTCTGGA and GGGTCTITT are both stopped and resumed when reading TCT, the sequencer may mistakenly read the chimeras AACTCTTTT and GGGTCTGGA. More complex chimeras may also arise as hybrids of more than two sequences, palindromic sequences, etc. Chimeras are rious problem in metagenomics studies, especially on 16$ genes, constituting even a large portion of sequencer outputs (Wang & Wang. 1997), and can cause misclassification of a population of ganisms. Moreover, detection has been a problem and some public databases have been historically plagued by chimeras mistaken for legitimate sequences (Hugenholtz & Huber 2003).
4 TASKS AND DATASETS
supervised learning bioinformatics tasks related to genomics. They have been chosen they are similar in their formulation: the classification of a single sequence of RNA or DNA. Our aim is to provide a uniform input formulation, to make all the tasks compatible with any NLU model. At the same time, they have been selected to address broad range of aspects and present slightly different characteristics, so that each of them provides a different challenge. Indeed, the tasks vary between binary and multi-class, single and multi-label, flat and hierarchi- cal (Freitas& & Carvalho, 2007). balanced and unbalanced distributions (Wald et al. 2013). Three additional criteria have been used for our selection: 1) the task must be solvable without explicit Nomain-related knowledge: 2) the data must not require excessive computational resources; 3) the data must be already available under a license that allows us to redistribute them
BANANA is a benchmark
Even if our benchmark is based on existing data, such data is rather difficult to use in its original form as a benchmark, due to heterogeneity the samples and labels across tasks, and because their use requires deep knowledge of the bioinformatics domain. Moreover, they do not include negative samples, problem that previous authors have addressed in various ways, without a standardized methodology, as discussed in Section We publicly release curated version of those data, selected and formatted so as to make them approachable by anyone as machine learning tasks but without altering their nature nor their complexity. For each task, we define 3 data splits, allocating 75% of the data for training, 5% for validation, and 20% for testing 3 Additional information regarding the original format of the data and their processing can be found in Appendix B
It's important to note that, due to the nature of the tasks, and unlike traditional NLP, labeling cannot be done by human evaluation and requires years of (possibly expensive) lab experiments which are then merged together. To limit costs, it is not uncommon to gather data and then use traditional bioinformatics algorithms to label some or the entirety of it. These algorithms have been thoroughly validated by the scientific community, and therefore they can be considered reliable gold standard.
For each task, we present brief summary containing the main characteristics of the task, charac- teristics of the dataset, and which metric will be measured for the benchmark. We also include an analogy to natural language processing tasks, to facilitate understanding and to help to bridge the gap between the communities of bioinformatics and NLP. The metrics for each task range from O to I and we propose as the final BANANA score their arithmetic mean. We propose to use this score to quantitatively assess any neural architecture capable of solving these tasks, regardless of how it was initialized. This is not only to allow the evaluation to a broad spectrum of training pro- cesses (single task, self- sulevvuper pre-training, contrastive learning, joint learning, etc.), but also to avoid penalizing families of architectures that could benefit from a given style (c.g. transformers pre-trained with language modeling tasks, CNNs pre-trained as autoencoders).
4.1 PROMOTER DETECTION (Prom))
Type of task. Binary classification on balanced dataset.
Format of data. Sequence of fixed length.
Language. Human DNA. Alphabet: Σ = G).
Objective. Establish whether sequence of DNA contains a promoter or not.
2 Appendix A contains examples of tasks we have considered but discarded.
3 Following ICLR's 's FAQs. after the opening of the discussion forums, we will make comment directed to the reviewers and area chairs and put link to an anonymous repository. In case of acceptance, it will be hosted on the Zenodo website.
4

---

Under review as a conference paper at ICLR 2022
Dataset. 59,194 sequences of length 2048 associated with single binary label.
Main metric. Macro average Fl score.
NLP analogy. Positive/negative sentiment analysis.
4.2
CHROMATIN CLASSIFICATION (ChromC)
Type of task. Multi-label classifications with many classes.
Format of data. Sequences of variable length.
Language. Human DNA. Alphabet: Σ {A, T,T,C,G,
Objective. Determine all the possible ways DNA strand can fold.
Dataset. 4,862,738 sequences of length of at most 1000bp. Each of them is associated with 919 binary labels.
Main metric. Area under ROC curve (AUC).
NLP analogy. Joint classification of multiple related concepts (eg. legal document analysis according to obligations, penalties, and clarity).
4.3
MRNA CLASSIFICATION (mRNAC)
Type of task. Multi-label classifications with few classes.
Format data. Sequences of variable length.
Language. Human mRNA. Alphabet: Σ (A,U,C,G)
Objective. Establish from which subcellular location the sequence comes from; in case of multiple possible source, multiple labels are assigned.
Dataset. 127,208 sequences of length ranging from 33nt to 9,256nt (median length about 4400nt). Each sample is associated with 15 binary labels.
Main metric. AUC.
NLP analogy. Multiple topic classification.
4.4
16S IHERARCHICAL CLASSIFICATION (16sfT)
Type of task. Multi-class hierarchical classification (7 levels) with many samples.
Type of data. Sequences of variable length.
Language. Bacterial and archaebacterial DNA. Alphabet: E (A,T,C,G)
Objective. Provide a complete classification (kingdom, phylum, class, family, order, genus, species) for bacteria from its 16S gene.
Dataset. 1,783,227 sequences of length between 890 and 4000bp. Each of them is associ- ated with 26,793 793 categorical labels organized into levels.
Main metric. The average score between the 7 macro-averaged F1 scores computed for each level of the hierarchy.
NLP analogy. Hierarchical language identification on a large dataset (eg. Indoeuropean Germanic North Sea Germanic English vs. Indoeuropean > Romance > Italo- Dalmatian > Italian).
4.5
ITS2 HIERARCHICAL CLASSIFICATION (its2H)
Type of task. Multi-class hierarchical classification (4 levels) with few samples.
Type of datas Sequences of variable length.
Language. Nematoda (round worms) DNA (TTS2 gene). Alphabet: Σ = (A,
Objective. Provide partial classification (class, family, order, genus) for nematoda from its ITS2 gene.
Dataset. 19,222 sequences of length betweeen 100bp and 400bp Each is associated with 277 categorical labels organized in 4 levels.

---

Under review as a conference paper at ICLR 2022
Main metric. The average score between the 4 macro-averaged Fl scores computed for each level of the hierarchy.
NLP analogy. Hierarchical language identification on a small dataset.
4.6
CHIMERA DETECTION (ChimD)
Type of task. Binary classification with an unbalanced dataset.
Type of data. Sequences of variable length
Language. Bacterial and archehacterial DNA (16S gene). Alphabet: E GA)
Objective. Determine input a real bacterial sample or chimeric artifact.
Dataset. 412,251 sequences, 19,631 1 which labelled as positive. The length of cach sample ranges from 1,11 to 2,435bp.
Main metric. Macro average Fl score.
NLP analogy. Semantic coherence from grammatically correct samples. E.g., "The dog jumped over the fence in order to fetch the stick" and "Transformers can be pre-trained in order to achieve state-of-the-are performance" are two coherent sentences, but their chimera "The dog jumped over the fence in order to achieve state-of-the-art performance" is not, in spite of still preserving a well formed structure thanks the shared portions.
EXPERIMENTAL SETUP
As baselines, we evaluate two neural architectures, with different pre-training configurations. Since benchmark includes only original datasets, it is not possible to compare these approaches to previous state-of-the-arr results. In our experiments, we focused on models capable of extracting token-level contextual embeddings which used to feed a task-specific classification head, as proposed by Devlln et al. (201)). However, we want to underline that even if we have created this benchmark primarily for the evaluation of contextual embedding based architectures, can be used for any type of classifier.
5.1 ENCODING
Unlike natural languages, DNA and RNA sequences are not "naturally" split into words, therefore it necessary define synthetic tokens. popular approach bioinformatics is to use k-mers, which are "words" made of & contiguous characters (Liang) 2012). We perform input tokenization with two-step approach: first, we split the input into k-mers, then we apply byte-pair encoding (Sennrich) et al. (2066) to compress the embedding table and avoid voc-ol-vocabulary tokens. In each of our experiments, we set & = and the size of vocabulary at 32,000. These values were identified as a good compromise between performance and input size preliminary tests.
5.2 MODELS
sparse-attention transformer, "state-of-the-arr baseline for producing contextual embeddings. For each task we test three training approaches: 1) single-language single-step training. 2) single-language two-steps training, 3) multi-lingual two-steps training. The first approach is to directly train different LongFormer model on each task (LONGFORMER-N)
We used LongFormer (Beltagy et al.] 2020).
The second one differs because we introduce a ppe--training step for each model. Firstly we do self-supervised dynamic masked language model pre-training (Liu et al.) 2019) then a supervised fine-tuning on the classification task using smaller learning rate (Devlin et al. 2019). Both steps are performed using only the data provided for that specific task, therefore we address this approach as LONGFORMER-NE single LongFormer, shared across all the tasks, and then fine-tuning different copy of it each of the 6 tasks independently. The pre- vre-training done on a "multilingual" dataset created by randomly sampling 200,000 sequences from each of the six tasks, using oversampling techniques for small datasets. To keep a similar computational budget to the mono-task case, the embedding size of the L MULTI is reduced from 256 to 192 dimensions, and the number of attention heads
The last approach consists of pre-training a

---

Under review as a conference paper at ICLR 2022
Table 1: Models'
performance the benchmark test sets, reported percentage points
(LF=LONGFORMER)
Model
PromD Fl
ChromC AUC
mRNAC AUC
16sH
its2H
ChimD F1
BANANA
F1
F1
AVG
LF-NO
46.3
73.8
70.0
76.4
72.2
86.0
70.8
LF-MONO
74.0
79.1
84.5
76.3
71.2
87.6
78.8
LF-MULTI
77.7
51.7
66.4
69.0
56.7
79.2
66.8
BILSTM-NE No
61.1
51.1
80.4
73.8
60.2
78.8
67.6
BILSTM-MULLE
73.2
51.2
82.2
70.3
74.5
72.3
70.6
RANDOM MAJORITY
50.0
50.0
50.0
3.5
7.5
50.0
35.1
33.3
50.0
50.0
8.0
16.9
33.3
31.9
the backbone from 16 to 12. The other hyperparameters are kept the same as the mono-task case. Additional details are provided in Appendix C
As "traditional" baseline, we trained a three-layer bidirectional LSTM, the hidden state of the last layer constitutes the contextual embeddings provided to the classification heads. Two versions of this baseline were trained separately: one directly on the final task (BILSTM-No) and the other in two stages, following the multilingual setting (BILSTM-MULT)) For comparison, we also provide two deterministic lower-bound baselines: random classification and majority class.
In all cases, the pre-train lasted for maximum of 10 epochs, with early stopping either when 24 hours clapsed or when the perplexity on the validation set dropped below 5. For fine-tuning, we trained for a maximum of 10 epochs, with early stopping in case of overfitting on the training set (measured as sharp drop of performance on validation in the last epoch) or to 12 hours of training.
5.3 SUPERVISED TASKS TRAINING
For the binary classification tasks (PromD and ChimD) the contextual embeddings subject to a 0.1 dropout and then fed to a fully-connected single neuron with sigmoid activation. We use binary cross-entropy loss for PromD and the weighted version and for ChimD account for class imbalance, giving a weight of 0.95 to the positive class and 0.05 to the negative class.
We use similar setting for multi-labed classification tasks (ChromC and mRNAC), but we use different sigmoid layer for each label. The networks are trained using the binary cross-entropy loss averaged across labels. In ChromC task, we address the classes unbalance following Zaheer et al. (2000) assigning a weight of 8 to the positive class and to the negative one because they have empirically proven to improve convergence with respect to the unweighted binary cross-entropy.
To address the hierarchical classification tasks (16sH and its2H) we exploit hierarchical neural at- tention (Galassi et al., 2020) as shown in Figure [ First, we compress contextual embeddings to lower-dimensional space through 1D convolutional layer, meet our limitations in terms of hardware. Then, the 32-dimensional embeddings are used both as keys and values of a multi-head attention layer (Vaswani et al., 2017) for each hicrarchy level. The query element for the top level of the hierarchy (kingdom for 16sH and class for its2H) is the same vector used as keys and values, while for every other hierarchy level, the query element is the output of the previous attention layer. The outputs of each attention module (one for each hierarchy level) are input to fully-connected layer with softmax activation which constitutes the output classification for that level. The labels one-hot encoded and each level of the hierarchy is trained on the average categorical cross-entropy loss across all layers.
5.4 RESULTS
Both the two neural architectures greatly surpass the baselines in almost all the tasks. The ChromC task seems to the more challenging one, with only the LONGFORMER-N and -MONO signif- icantly overcoming the baseline of 50%. ChimD results to be the easiest to solve, with all the

---

Under review as a conference paper at ICLR 2022
Output embedding (highest level)
Output embedding (l-th level)
To lower hierarchy levels
Lineur
Linear
Linear
Concat
Concal
Concat
MatMul
Scaled Dol-Product
Scaled Dot-Product
Scaled Dat Product
Attention
Attention
Attention
Dropout
SaftMax
Linear
Linnor
Linnor
Linear
Linear
Linear
Linear
Linear
Linser
V
K
V
K
Scale
MetMuu
K
Input embedding
Scaled Dot-Product Altention
Figure 1: Hierarchical multi-head attention for 16sH and its2H classification heads, partially based on Vaswani et al. (2017).
models achieving more than 70 percentage points, up to 87%. The LONGFORMER MONO is the model that obtains the best final score, and it also achieves satisfactory results in all the tasks. The LONGFORMER TRORMER-MMLLT outperform the MONO version only on PromD, and is less performant than the other LONGFORMERS on all the other tasks. This may be caused by the fact that a multi-lingual representation is less specialized, but it may also be influenced by the fact that we use a smaller number of attention heads and the fact that the MULTI pre-train is done only on portion of the data used for the MONO pre-train. On the contrary. BILSTM-MULT and BILSTM-No have comparable behavior in most of the tasks, with a pronounced difference only on its2H, where the MULTI approach achieves the better score.
None of the models completely dominates any other, and different tasks are solved better by different models. These heterogeneous performances confirm the claims of Rao et al. (2019): single task is not enough to properly evaluate a trained model and there is a need for multi-task benchmarks such as TAPE and BANANA.
Other than the formal BANANA score, we strongly encourage considering also the environmental impact of the architectures. Indeed, measured by Strubell et al. (2019). the development of -fate-of-the-art solutions often has a massive carbon footprint. With the perspective of a sustainable research, it is important to shift toward a Green AI focused on efficiency, not only accuracy. Small- size architectures are therefore preferable, since the number of parameters can influence the time, and therefore the energy consumption, needed train a model (Schwartz al. 2020). We include details about the size of our models in Appendix C
6 RELATED WORKS
Umarov & Solovyev (017)
the first to train a neural network on promoter detection. They solved the task by using a very small CNN on a limited subset of promoters. They created a dataset of negative samples by choosing coding sequences (i.e., portions always ahead of promoters). The ease with which the task was solved has been contested both by Qian et al. (2018) and Oubounyt al. (2019) since it contains "useful" biases which can help a classifier, like keeping the TSS of the positive classes always in the same position. Oubounyt et al.] have proposed to create negative samples by means of random noise, but we advocate that, from a language modeling perspective, this approach is unsound. Indeed, it shifts the task of recognizing well-formed language (the promoter's s) to a much simpler discrimination task between a coherent sequence of syntagms and an incoherent sequence of symbols. Zaheer et al. (2020) were the first to solve the task pre- traitraining Transformer based architecture, BigBird, on external data (the entire human genome, version hg19), achieving near-perfect performance. However, their approach to negative samples is the same as
8

---

Under review as a conference paper at ICLR 2022
Oubounyt et al. Ji et al. (2021) addressed the task with another Transformer-base architecture and synthesizing two different types of negatives, but without shifting the TSS in the positive samples.
The chromatin classification task has been introduced by Zhou & Troyanskaya (0115) and has gained huge traction in the bioinformatics community. They propose to approximate the continuous domain of possible foldings into a finite number of classes and to address the problem as multi-label classification task. We follow such an approximation, that allows us to avoid two main problems with the general task of chromatin folding. The first one is that predicting the actual shape requires a deep knowledge of the domain and possibly very complex architectures, akin to protein folding (Jumper al., 202T). Additionally, the same sequence, even from the same cell, may change its shape over time, so unlike protein folding, there would be more than one output, possibly of an unknown a priori number. Differently from our benchmark, (Zhou Troyanskaya propose test set that contains only sequences from human chromosomes 8 and 9, possibly biasing the evaluation scores in different ways in the two cases. Zaheer et al. (2200) address the task achieving very good results, but are not directly comparable to ours due to the different data splits.
The tasks belonging to the family of subcellular localization have been solved in many different flavors using various architectures and many different datasets, addressing protein (Sønderby et al.) 2013). mRNA (Yan et al., 2019; Wang et al. 2021: Li et al.) 202I), mitochondrial proteins (Savo-) jardo et al. 2020), etc. This heterogeneity testifies the practical importance of these tasks but makes comparison among architectures impossible. Our proposal of mRNA localization aims to cover this limitation by providing well-defined prototypical task for the entire family.
Marker gene metagenomics (16S and ITS2 on fungal sequences) is an area of active research, also because meta-analyses have revealed the presence of classification biases in traditional pipelines (Nearing et al. 20188 Straub et al. (220). Neural approaches to metagenomics range from traditional architectures to ad-hoc ones: Fiannaca et al. (2018) use CNNs and deep belief networks, while Fioravanti et al. (2018) design Ph- Ph-CNN introducing a new convolutional layer operating on multiple input sequences aggregated by k-nearest neighbors. It is important to note that, like other neural approaches, our hierarchical classification tasks play only a small role in a complete metage- nomics setup. Indeed, it requires to classify millions of sequences in presence of noise, chimeras, and other artifacts, it typically requires domain knowledge and ad-hoc denoising algorithms, e.g., DADA2 (Callahan et al. 2016).
To the best of our knowledge, we are the first to propose the use of neural architectures for the classification of Nematoda ITS2 sequences (which is basically a smaller marker-gene metagenomics task) and chimera detection tasks. These tasks have been solved with well-established algorithms in the bioinformatics community.
Regarding chimera detection, traditional algorithms, like Chimera Slayer (Haas et al. 2011 exploit a database of chimera- free sequences by counting how many results are triggered by the query sequence. UCHIME (Edgar et al. (011)) improves performance by computing the reference database from the sample itself, under the assumption that, in a metagenomic sample, the abundance of legitimate sequences is higher than the one of chimeras.
7 CONCLUSION
We presented BANANA, a new benchmark to assess the capability of neural models to understand biological languages. The purpose of this benchmark of bioinformatics and NLP (and machine learning in general), providing a standardized and reliable tool evaluate state-of-the-an thrt neural techniques on bioinformatics tasks. Our benchmark is based on existing data, which have been curated and formatted into 6 datasets for which we define as many supervised learning tasks. These datasets are easy to use and can be addressed by machine learning and NLP experts without requiring any additional knowledge of the domain. Our experiments with two neural baselines confirm the results of Rao et al. (2019), remarking the usefulness of multiple- tasks benchmark to properly evaluate neural architectures.
narrow the gap between the communities
In future works, we want to investigate multi- multi-lingual lingual pre- vre-training further, with the purpose to develop single neural model that can address successfully all the tasks. We also want pursue more efficient models, and to extend the BANANA score by defining with formal Green AI metrics, capable of assessing environmental impact and efficiency.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Our work is compliant with the ICLR Code of Ethics. Our benchmark includes only data that are already publicly available under licenses that allow us to redistribute them. We report the sources of our data, describe their original format, and explain our formatting procedure in Appendix B
REPRODUCIBILITY STATEMENT
The dataset, as well as the code used to run the experiments, and the code used to process the data will be released publicly in case of acceptance. The dataset will be made publicly available and hosted on the Zenodo website, the code will be put on a publicly available GitHub repository. During the review period all the material will be available the reviewers. Following ICLR's FAQs, after the opening of the discussion forums, we will make a comment directed to the reviewers and area chairs and put a link to an anonymous repository.
Any details regarding our models and training hyper- sypee--srameters are specified in the body of the paper or the Appendices.
REFERENCES
Kessy Abarenkov, R Henrik Nilsson, Karl-Henrik Larsson, Ian Alexander, Ursula Eberhardt, Su- sanne Erland, Klaus Høiland, Rasmus Kjøller, Ellen Larsson, Taina Pennanen, et The unite database for molecular identification of fungi-recent updates and future perspectives. New Phy- tologist, 186(2):281-288 2010.
Stephen F Altschul, Warren Gish, Webb Miller, Eugene W Myers, and David J Lipman. Basic local alignment search tool. Journal of molecular biology, 215(3):403-410, 1990.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long document transformer. CoRR, abs/2004.05150, 2020. URL /tt.ppps//axxx..org////// ab05/
2004. 05150
Lucia Bianchi and Pietro Liò. Forensic dna and bioinformatics. Briefings in bioinformatics, 8(2): 117-128, 2007.
Benjamin Callahan, Paul J McMurdie, Michael Rosen, Andrew W Han, Amy Jo A Johnson, and Susan P Holmes. Dada2: high-resolution sample inference from illumina amplicon data. Nature methods, 13(7):511-583, 2016.
Lisa Helbling Chadwick. The nih roadmap epigenomics program data resource. Epigenomics, 4(3): 317-324, 2012.
Hong-Zhi Chen, Ling-Li Li Tang, Xin-Ling Yu, Jie Zhou, Yun-Feng Chang, and Xiang Wu. Bioinfor- matics analysis of epitope-based vaccine design against the novel sars-cov-2. Infectious diseases of poverty, 91):11110, 2020.
Francis Crick. Central dogma of molecular biology. Nature, 227 1970.
Todd 7. DeSantis, Philip Hugenholtz, Neils Larsen, Mark Rojas, Boin I. Brodie, Keith Keller, Thomas Huber, Daniel Dalevi, Ping Hu, and Gary L Andersen. Greengenes, a chimera-checke 16s rma gene database and workbench compatible with arb. Applied and environmental microbi- ology, 72(7):5099-5072, 2006.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume (Long and Short Pupers). pp. 4171-4186. Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N11-111 1423. URL https:
//aclanthology. wwtp:/oogg///l/i..ee
Robert Edgar, Brian J Haas, Jose C Clemente, Christopher Quince, and Rob Knight. Uchime improves sensitivity and speed of chimera detection. Bioinformatics, 27(16):1194-2000 2011.
10

---

Under review as a conference paper at ICLR 2022
Mona Fani, Milad Zandi, Sacedeh Ebrahimi, Saber Soltani, and Samaneh Abbasi. The role of mirnas in covid-19 19 disease. Future Virology. 16(4):301-30, 2021.
Antonino Fiannaca, Laura La Paglia, Massimo La Rosa, Giovanni Renda, Riccardo Rizzo, Salva- tore Gaglio, Alfonso Urso, et al. Deep learning models for bacteria taxonomic classification of metagenomic data, BMC bioinformatics, 19(7):61-76, 2018.
Diego Fioravanti, Ylenia Giarratano, Valerio Maggio, Claudio Agostinelli, Marco Chierici, Giuseppe Jurman, and Cesare Furlanello. Phylogenetic convolutional neural networks in metage- nomics. BMC bioinformatics, 19(2):1-13, 2018.
Alex Freitas and André Carvalho. tutorial on hierarchical classification with applications in bioin- formatics. Research and trends in data mining technologies and applications, pp. 175-208, 2007.
Andrea Galassi, Marco Lippi, and Paolo Torroni. Attention in natural language processing. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-18, 2020. doi: 10.1109/TNNLS. 2020.3019893.
Brian J Haas, Dirk Gevers, Ashlee M Earl, Mike Feldgarden, Doyle V Ward, Georgia Giannoukos, Dawn Ciulla, Diana Tabbaa, Sarah K Highlander, Erica Sodergren, et al. Chimeric 16s mma sequence formation and detection in sanger and pyrosseueee per amplicons. Genome research, 21(3):994-504, 2011.
Masami Hasegawa, Taka-aki Yano, and Hirohisa Kishino. A new molecular clock of mitochondrial dna and the evolution of hominoids. Proceedings of the Japan Academy, series B, 60(4):95-98, 1984.
Christine E Holt and Simon L Bullock. Subcellular mmna localization in animal cells and why matters. Science, 326(5957:11112-2116 2009.
Philip Hugenholtz and Thomas Huber. Chimeric 16s rdna sequences of diverse origin are accumu- lating in the public databases. International journal of systematic and evolutionary microbiology, 53(1):289-293, 2003.
Wataru Iwasaki, Tsukasa Fukunaga, Ryota Isagozawa, Koichiro Yamada, Yasunobu Maeda, Takashi P Satoh, Tetsuya Sado, Kohji Mabuchi, Hirohiko Takeshima, Masaki Miya, et al. Mitofish and mitoannotator: a mitochondrial genome database of fish with an accurate and automatic an- 2013.
notation pipeline. Molecular biology and evolution,
Janet K Jansson and Kirsten S Hofmockel. Soil microbiomes and climate change. Nature Reviews Microbiology, 18(1):35-46, 2020.
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language genome. Bioinformatics, 37(15):2112-2133 2021.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583- 2021.
Sibylle Kalmbach, Werner Manz, and UIrich Szewzyk. Isolation of new bacterial species from drinking water biofilms and proof of their in situ dominance with highly specific 16s rma probes. Applied and Environmental Microbiology, 63(1 1):4164-41700 1997.
Pedro Larranaga, Borja Calvo, Roberto Santana, Concha Bielza, Josu Galdiano, Inaki Inza, José A Lozano, Rubén Armananzas, Guzmán Santafé, Aritz Pérez, et al. Machine learning in bioinfor- matics. Briefings in bioinformatics, 7(1):86-112 12, 2006.
Jing Li, Lichao Zhang, Shida He. Fei Guo, and Quan Zou. Sublocep: novel ensemble predictor of subcellular localization of cukaryotic mrna based on machine learning. Briefings in Bioinformat- ics, 2021.
11

---

Under review as a conference paper at ICLR 2022
Sai-Kam Li, Wei Zhou, Aldrin Kay- Yue- Yim, Alden King- Yung Leung, Stephen Kwok- Kwok-Wing Wing Tsui, Ting-Fung Chan, and Terrence Kong Lau. Organism-specific mma capture system for application in next-generation sequencing. PLoS One, 8(9):e74286, 2013.
Yanhui Li, Changliang Wang, Zhengqiang Miao, Xiaoman Bi, Deng Wu, Nana Jin, Liqiang Wang, Hao Wu, Kun Qian, Chunhua Li. al. Virbase: a resource for visus-host nerma-associated interactions. Nucleic acids research, +3(D1):D5778DD53 2015.
Wang Liang. Segmenting dna sequence into words based on statistical language model. Nature Precedings, pp. 1-1, 2012. doi: http://...rrr0000000/3///2222220
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11192 2019. URL httptt:tppaii....////// org. org/ abs/////////////////////////s 1abbs/1900 1907. 116072216222
Kelsey C Martin and Anne Ephrussi. mrna localization: gene expression in the spatial dimension. Cell, 136(4):719-730 2009. doi: (e001//..el..20000010000
Marc A Morgan and Ali Shilatifard. Chromatin signatures of cancer. Genes & development, 29(3): 238-249, 2015.
Jacob T Nearing, Gavin M Douglas, André M Comeau, and Morgan GI Langille. Denoising the denoisers: an independent evaluation of microbiome sequence error-correction approaches. PeerJ, 6:e5364, 2018.
Mhaned Oubounyt, Zakaria Louadi, Hilal Tayara, and Kil To Chong. Deepromoter: Robust promoter predictor using deep learning. Frontiers in Geneticx, 10:286, 2019. ISSN 1664-8021. doi: 10. 2589%fgeme.2219.o00g
URLstttss: htthhtpss:////ee //www, Fronti ieriiiiiinin.i. www.eeriiiinnggat.aeeeee.eeeeeee
Egene agene.20133 201.ee20119002200 00286
Dorota Piekna-Przyhylska, Wayne A. Decatur, and Maurille J. Fournier. The 3d mna modification maps database: with interactive tools for ribosome analysis. Nucleic Acids Research, 36(suppL_1): DI78-DI83, 10 2007. ISSN 0305-1048. doi: 10. 10.1093/nar/gkm8555 URL/tttps://doi 10.1093/mat/gkm633
Francesco Piva, Matteo Giulietti, Alessandra Ballone Burini, and Giovanni Principato. Spliceaid 2: a database of human splicing factors expression data and raa target motifs. Human mutation, 33 (1):81-85, 2012.
Ying Qian, Yu Zhang, Bingyu Guo, Shasha Ye, Yuzhu Wu, and Jiongmin Zhang. An improved promoter recognition model using convolutional neural network. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), volume 1, pp. 471-476. IEEE, 2018.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John F. Canny, Pieter Abbecl, and Yun S. Song. Evaluating protein transfer learning with TAPE. In NewrIPS, pp. 9686-9698, 2019, URL DRp
html
Kaladhar B Reddy. Microrna (mirna) cancer. Cancer cell international, 15(1):1-6, 2015.
Frederick Sanger, Steven Nicklen, and Alan R Coulson. Dna sequencing with chain-terminating inhibitors. Proceedings of the national academy of sciences, 74(12):546355467, 1977.
Castrense Savojardo, Niccolò Bruciaferri, Giacomo Tartari, Pier Luigi Martelli, and Rita Casadio. Deepmito: accurate prediction of protein sub-mitochondrial localization using convolutional neu- ral networks. Bioinformatics, 36( (66),46664, 2020.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai. Commun. ACM, 63(12): 63-63, November 2020. ISSN 0001-0782, doi: 10.1145/338183/ URL https: //o::///doo..mm dow.oooo///oooooooooooooooooo 10.1145/338183
12

---

Under review as a conference paper at ICLR 2022
Rico Sennrich, Barry Haddow. and Alexandra Birch. Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics. ACL 2016, August 7-12, 2016, Berlin, Germany, Volume I: Long Pa- doi: 10.18663/1//16-11622 URL
pers. The Association for Computer Linguistics, 2016.
https://dop lagaoioorgg11e 1ttpg//1..11////1111///o
Andrew Shreiner, John Y Kao, and Vincent B Young. The gut microbiome in health and disease. Current opinion in gastroenteology, 31()):69, 2015.
Cricket A Sloan, Esther T Chan, Jean Davidson, Venkat Malladi, J Seth Strattan, Benjamin Hitz, Idan Gabdank, Aditi K Narayanan, Marcus Ho, Brian T Lee, al. Encode data at the encode portal. Nucleic acids research, 44(D 2016.
Søren Kaae Sønderby, Casper Kaae Sønderby, Henrik Nielsen, and Ole Winther. Convolutional Istm networks for subcellular localization of proteins. International Conference on Algorithms for Computational Biology, pp. 68-80. Springer, 2015.
Daniel Straub, Nia Blackwell, Adrian Langarica- Fuentes, Alexander Peltzer, Sven Nahnsen, and Sara Kleindienst. Interpretations of environmental microbial community studies are biased by the selected 16s rma (gene) amplicon sequencing pipeline. Frontiers in microbiology, 11:2652, 2020.
Lungarica-Fuuetes,
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NL.. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3645-3650, Florence, Italy, July 2019. Association for Computational Linguis- URL
tics. doi: 10.18653/v 1/P19- 1355.
1355
Ramzan Kh Umarov and Victor V Solovyev. Recognition of prokaryotic and eukaryotic promoters using convolutional deep learning neural networks. PloS one, 12(2):00171400 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp.
2017.
Susana Vinga. Alignment-free methods computational biology, 2014.
Abdul Wahab, Hilal Tayara, Zhenyu Xuan, and Kil To Chong. Dna sequences performs as nat- ural language processing by exploiting deep learning algorithm for the identification of n4- methylcytosine. Scientific reports, 11(1):1-9, 2021.
Randall Wald, Taghi M Khoshgoftaar, Alireza Fazelpour, and David Dittman. Hidden dependen- cies between class imbalance and difficulty of learning for bioinformatics datasets. In 2013 IEEE 14th International Conference Information Reuse & Integration (IRI), pp. 232-238. IEEE, 2013.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier bench- mark for general-purpose language understanding systems. In NeurlPS, pp. 3261- 3275, 2019a. URL Abttaa/eee22eeeeeeeaeaattaaaeaae
wttpiipp..ppppppeeee hctps/aaaap///11///aa... Abstract.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In ICLR (Poster). OpenReview.net, 2019b. URL httph::://ppeeeeiieeeeee ew. net wtt.moormmiioo
T TJ4km2R5t7
Duolin Wang, Zhaoyue Zhang, Yuexu Jiang, Ziting Mao, Dong Wang, Hao Lin, and Dong Xu. Dm3loc: multi-label mmn subcellular localization prediction and analysis based multi-head self-attention mechanism. Nucleic Acids Research, 49(8)=46-e46, 2021.
GC Wang and Yue Wang. Frequency of formation of chimeric molecules consequence of per coamplification of 16s mma genes from mixed bacterial genomes. Applied and environmental microbiology, 63(12):4645-46 1997.
13

---

Under review as a conference paper at ICLR 2022
James D Watson. The human genome project: past, present, and future. Science, 248(4951):44499 1990.
Xiaowei Xie, Wenbin Ma, Zhou Songyang, Zhenhua Luo, Junfeng Huang, Zhiming Dai, and Yuanyan Xiong. CCSI: database providing chromatin-chromatin spatial interaction infor- mation. Database, 2016, 02 2016. ISSN 1758-0463. doi: 10. 10.1099/datbbbas/baylla. 124. URL
Sltps://doi.org/100 1093 http:///....9//aaaaaaaeeee/baaaa
bav124.
Zichao Yan, Eric Lécuyer, and Mathieu Blanchette. Prediction of mima subcellular localization using deep recurrent neural networks. Bioinformatics, 35(14))1333-1342, 2019.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. In NeurlPS, 2020.
Jian Zhou and Olga G Troyanskaya. Predicting effects of noncoding variants with deep learning- based sequence model. Nature methods, 12(10):931-944 2015.
14

---

Under review as a conference paper at ICLR 2022
Table 2: Dataset composition for each task.
Split
PromD
ChromC
mRNAC
16sH
its2H
ChimD
Train
44,394
3,647,053
95,406
1,337,420
14,416
402,435
Validation
2,959
243,136
6,360
89,161
961
1,963
Test
11,839
972,549
25,442
356,646
3,845
7,853
Total
59,192
4,862,738
127,208
1,783,227
19,222
412,251
A DISCARDED TASKS
While designing this benchmark, we decided to focus on classification tasks that did not require, character-level reasoning, domain knowledge, complex architectures (note that decided to use the neural attention mechanism for our baselines in the hierarchical tasks, but this was not imposed by the nature of the tasks). For these reasons, ruled out every folding task, includ- ing chromatin prediction in terms of spatial conformations or graph representations, RNA structure prediction (Piekna- Dickna-Przybylsse Przybylska et al. 2007), protein folding (Jumper et al., 202T), etc.; every re- gression task, such as TSS localization inside promoter. molecular clock regression (Hasegawal al. 1984), y a full marker-gene metagenomic pipeline (which also has the complications of domain knowledge ledge and the need of processing extremely large inputs), etc.; every alignment task (given set of noisy fragments and a reference database, sort and merge the fragments in order to get a sequence as close as possible to one in the database), including shotgun metagenomics; every scquence-to-
sequence task, such as post-translational RNA modifications and intron splicing (Piva et al. 2012); and every multiple-sequence
sequence classification task, like virus-host RNA interactions (Li et al. 2015) and chromatin-chromatin interactions (Xie et al. 2016).
Another important factor in our decisions was the desire to provide set of tasks varied possi- ble, while still keeping enough correlation to allow multilingual approaches to the benchmark. For this reason, discarded all the protein-related tasks, since linguistic differences arise even the alphabet level (DNA and RNA have symbols, while proteins 22). Moreover, protein tasks tend to require deeper degree of domain knowledge and often rely additional features other than the sequence.
Finally, we discarded tasks that, albeit interesting on biological level, had either too few samples or were too similar to tasks already present in the dataset. For example, 12S mitochondrial metage- nomics (Iwasaki et al., 2013) has only singleton labels. ITS2 fungal metagenomics (Abarenkov al.) 2010 is too similar to other tasks (its sequences are as short our its2H and almost as many labels as our 16sH), other variants of subcellular localization, are too similar to mRNAC.
B SOURCE OF DATA AND DATA PROCESSING
All the data we have included in our benchmark are derived from existing databases and, in most cases, have been processed guarantee structured, fair, and challenging tasks. hereby report the original source of our data, describe in detail their original format, we list all the processing steps have applied. This information is valuable both because guarantees that work reproducible, but also guarantees the absence of biases or other synthetic artifacts in our benchmark. All the code for data processing is publicly available Table reports detailed statistics regarding the final size of our datasets.
PROMOTER DETECTION (Prom))
Source of data. Human promoters from Eukaryotic Promoter Database (EPD) 29,598 sequences of human promoters, 16,000bp long, and with the transcription start site (TSS)
'Following ICI.R's FAQs, after the opening of the discussion forums, we will make comment directed the reviewers and area chairs and put a link to an anonymous repository. The code will put on a GitHub repository in case of acceptance.
hhttss://pdd sateedddpf1.... epf1 ch/1 hte//iddex.php
15

---

Under review as a conference paper at ICLR 2022
centered position 10,000. Each promoter contain one more of the TATA box, Initiator motif, CCAAT box or GC box patterns.
Data Processing. From each sequence we extract subsequences of length 2048bp to create one positive and one negative sample. The positive sample is created by using a 2048bp window centered at the TSS and shifting it by a random offsett but guaranteeing that the TSS is inside the window and has at least a 32bp context on both sides. The negative sample is taken randomly placing the window before or after the positive one, maintaining a distance of at least 2000bp between the two.
CHROMATIN CLASSIFICATION (ChromC)
Original data. Princeton University's 's DeepSea training bundle (Zhou & Troyanskaya
2015): 4,36 sequences of length 1000bp. Each sequence is associated with 919 binary labels which can be grouped into categories according the biological domain (DNAse hypersensitivity sites: 125, Transcription factor binding sites: 690. Histone methylation sites: 104). Labels are computed by splitting the human reference genome (version hg ) into 200bp bins, discarding the ones not associated with transcription factor binding events and classifying them according to the ENCODE (Sloan et al. (016)) and Roadmap Epige- nomics [Chadwick]] (012) databases. Each sample is extended to 1000bp by taking the two 400bp regions adjacent to the classified bin.
Data Processing. The human reference genome contains some small gaps of unknown content, marked as N characters in the sequences. We remove from each sample these gaps, making the sequences of variable length. In order conform to the other tasks, also reshuffle the original dataset and create new train/validationntess splits.
MRNA CLASSIFICATION (MRNAC)
Original source of data. RNALocate 2.0 More than 210,000 RNA-associated subcel- lular localization entries including messenger RNA (mRNA), micro RNA (miRNA) and other non-coding RNA (ncRNA) of various species.
Data Processing. We extracted the human mRNA from the database and kept the annota- tions only for the 15 labels which were represented by at least 50 samples.
16S IIERARCIICAL CLASSIFICATION (16sHl)
About 12 millions quality-checked ribosomal RNA (rRNA) sequences for all the three domains of life (Archaca, Bacteria. and Eukarya). Sequences are divided into small and large ribosomal subumits and are associated with their taxonomical name, up the lowest level for which classification is possible. Unknown labels don't follow a uniform naming system.
Original source of data. Silva 138.1 9
Data Processing. We extracted only the 16S genes (Archaea and Bacteria) from the small subunit database by filtering based on domain names and then retro-transcribed each se- quence from RNA to DNA 10] Each name is processed in order to have a single nomen- clature for unidentified taxa. We encode each label with integer number, using 0 for the unidentified class, and following the alphabetical order for the others. If level is laheled unidentified, all the following levels in the hierarchy i + 1, 2, 2,...,k in the same way, to guarantee a full hierarchy for each sample. We exclude any class for which only one sample is available. Splits are made so guarantee that both the training and test set have at least one sample for each label.
are labeled
httpp://deesea.rrrintttoo.....
https://www.c
138_1
FFro fingerprinting perspective, ribosomal markers (16S, ITS2, etc.) can be read indifferently as DNA RNA, however RNA analysis typically focuses analyzing mRNA, treating rRNA and IRNA as contami- et 2013), in practice, rihosomal markers are analyzed DNA sequences.
16

---

Under review as a conference paper at ICLR 2022
Table 3: Hardware specifications.
Architecture Architecture Architecture 2
CPU
GPU
RAM
Swap
Intel 17-11370H
8Gb n Vidia RTX 3070
40 Gb
64 Gb
AMD Ryzen 7 4800H
6Gb VVdia RTX 2060 Max-
16 Gb
32 Gb
Table Generic hyperparameters for our experiments (N: no pretraining, MO: mono-task pretrain- ing, MU: multilingual pretraining).
Hyperparameter
Value
LongFormer
MLM learning rate
10
Final task learning rate
5 10
Embedding size (N/MO)
256
Embedding size (MU)
192
Hidden layers
2
Attention heads (N/MO)
16
Attention heads (MU)
12
Attention window
32
Attention dropout
0.1
Bidirectional LSTM
Embedding size
256
Nr. hidden units
256
Hidden layers (BL)
3
Dropout between layers
0.1
ITS2 HIERARCHICAL CLASSIFICATION (its2H)
Original source of data. Nemabiome 19,300 sequences of ITS2 genes sampled from larval stages of parasitic worms. Each sequence is associated with taxonomic name de- termined morphologically by inspecting each larva under the microscope.
Data Processing. Same 16sH, but the higher taxa (domain, kingdom, phylum) discarded because they are trivial (unary classification) and the species level is removed since causes the majority of labels to have only one sample.
CHIMERA DETECTION (ChimD)
Original source of data. GreenGenes (DeSantis et al. 2006) discarded chimeras and clean samples At each revision, the database scanned for chimeras and they are removed. Most of the chimeric sequences are still accessible for manual download NCBI using list of chimeric IDs (21,169). Non-chimeric sequences are 1,262,986 bacterial 16S sam- ples.
Data Processing. We downloaded the chimeras from NCBI assigning positive label. For each of them, we randomly sampled 20 bacterial sequences from GreenGenes. Validation and test set splits have balanced (1:1) ratio of positive and negative samples, while the training set contains 20:1 negative to positive ratio.
C HYPER-PARE PARAARAARSEESERS AND HARDWARE ARCHITECTURE
We trained the BiL and LongFormer models on two hardware architectures, specified in Table Tables and report the hyper-parameters used for the design and the training of our models.
httpss http:://wwwwe hwwp///wwwwwmwwbioommmm//mm.o.ab ome ca
database
17

---

Under review as a conference paper at ICLR 2022
Table 5: Task-specific hyperparameters for experiments.
Hyperparameter
PromD
ChromC
mRNAC
16sH
its2H
ChimD
Multilinuual
MLM epochs
10
6
3
Epochs
10
2
5
(Same)
Input tokens
512
128
512
256
256
256
512
Table 6: Trainable parameters of our models.
PromD
ChromC
mRNAC
16sH
its2H
ChimD
No Pretraining, Mono-task Pretraining
BiLSTM
Backbone
12,392,448
12,392,448
9,478,400
12,392,448
7,355,904
12,392,448
Head
262,145
60,228,503
3,932,175
30,709,437
3,719,423
131,073
Total
12,654,593
72,620,951
13,410,575
43,101,885
11,075,327
12,523,521
LongFormer
Backbone
12,465,408
12,367,104 104
9,551,360
12,399,872
7,363,328
12,399,872
Head
131,073
30,114.711
1,966,095
30,701,245
1,859,839
65,537
Total
12,596,481
42,481,815
11,517,455
43,101,117
9,223,167
12,465,409
Multi-lingual Pretraining
BiLSTM
Backbone
12,392,448
12,392,448
12,392.448
12,392,448
12,392,448
12,392.448
Head
262,145
240,911,255
3,932,175
61,398,717
3,719,423
262,145
Total
12,654,593
253,303,703
16,324.623
73,791,165
16,111,871
12,654,593
LongFormer
Backbone
14,503,168
14,503,168
14,503,168
14,503,168
14,503,168
14,503,168
Head
131,073
120,456,087
1,966,095
59,512,509
2,638,079
131,073
Total
14,634,241
134,959,255
16,469,263
74,015,677
17,141,247
14,634,241
Due to our limited resources, each model was trained with maximum batch size of 32 samples, reduced (up to 4 for the multilingual BiL.SMM trained on the 16sH task) case out-of-memory enros.. Both dynamic masked language modeling and final task training were subject to limit of 10 epochs, reduced in case of overfitting or a timeout. We chose the input size as the smallest power of 2 capable of holding the majority of inputs since BPE encoding makes the input size variable. For tasks with greatly varying lengths, such as mRNAC, we decided to truncate inputs longer than the median value. Multilingual models used the maximum available input size (512 tokens). Table 5 summarizes the task-specific hyper-parameters used for training. The embeddings for tasks 16sH and its2H reduced to a 32-dimensional vector, using ID convolutional layer, with Gaussian eтror linear unit activation (kernel_size = 1, layer shown in Figure i instantiated with a dropout rate of 0. and attention heads for each level of the hierarchy. The sizes of each architecture, measured as the number of its trainable parameters, listed in T
= 32). The hierarchical attention
18


Reviews:
[{"review_id": "vMNHz3ylLAj", "reviewer": "Reviewer_Bhcu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\nThe authors do identify a gap in the literature and attempt to fill it with a set of complementary benchmarks. The datasets cover a variety of biological functions and organisms. The neural network baselines they train and evaluate seem to be generally done correctly. I appreciated the discussion on why some datasets were not included. \n\n### Major weaknesses\n\nThe authors justify the need for deep learning benchmarks in the introduction by claiming that they can be faster and more robust than sequence-similarity or search-based methods. However, they do not run any of these methods as baselines on their benchmarks. For example, methods that build an alignment and then infer a tree should be very competitive on the hierarchical classification tasks. The authors mention that bioinformatics methods are often used as a gold standard, but do not mention what methods were used to label their datasets or compare their results and runtimes with neural network based methods. \n\nAt the end of the introduction, no justification is given as to why human DNA, bacterial 16S genes, and nematode ITS2 genes constitute different languages, or why in that case we can later expect pretraining on all of them simultaneously to be beneficial (and indeed, in most cases it’s not!). I recommend leaning less on the analogy to language, as there’s not a good 1-to-1 comparison between different languages and the genomes of different species, and the analogy confuses more than it helps. It's also not described how they are unified for combined pretraining: the authors should explicitly state this. \n\nThe biological background is often unclear and sometimes wrong. In the first paragraph of section 3, too strong a distinction is made between how DNA base pairs but how RNA merely \"folds in different ways\": RNA also base pairs, just not into a double helix. Furthermore, no mention is made of RNA base pairing to DNA. In the second paragraph of section 3, chromatin is only found in eukaryotic cells, not all cells as the text implies. The definitions for genomics, transcriptomics, and proteomics vs metagenomics, metatranscriptomics, and metaproteomics is also wrong. Genomics is simply the study of genes and genomes, whether limited to single organisms or not. By the definition given, deep mutational scanning would count as metagenomics, since the sequences come from multiple single-celled organisms! Finally, there is some debate about whether 16S and other single-gene studies count as metagenomics. Taken together, this is a major weakness for the paper. However, I think it should also be relatively easy to correct the terminology. \n\nFrom the appendix, I think the authors use random or almost-random train/test splits for each task. Standard practice in biology is to avoid having highly similar sequences between train and test, as this can lead to over-estimating generalization performance. See, for example, these papers: https://www.nature.com/articles/s41580-019-0176-5, https://arxiv.org/abs/2006.16189.  In human genomics, standard practice is to split by chromosome: the authors claim that this would bias the results without providing any evidence or further justification. For hierarchical classification, this may not be as important; however, I suspect that simple sequence-similarity-based methods are both efficient and accurate for these. The paper would be improved by using biologically-justified splits, with explanations of why those splits were chosen. \n\nIn NLP (eg BERT) and in other work on pretraining for biological sequences (eg TAPE, ESM), the pretraining step is performed on many more sequences than are present in the final task. Is there any reason to believe that pretraining on the same sequences used as inputs in the final task should be generally helpful? It appears that they are, but the paper would be much stronger if we had an idea of why they are. Could it be that the LF-NO and BiLSTM-NO models in Table 1 have not converged? \n\n\n\n### Minor weaknesses\n\nIs it really necessary to split into kmers and then use byte-pair encoding? Why not just use the original 4-letter alphabet? To be clear, a sentence justifying the byte-pair encoding is sufficient to address this. \n\nIn the introduction, the authors should mention (and cite) recent work on using neural networks to understand glycans, in addition to DNA, RNA, and proteins. See, for example, this [paper](https://www.biorxiv.org/content/10.1101/2020.01.10.902114v1.abstract) and follow-ups. \n\nI would have appreciated a figure describing the tasks. \n\nIs BLAST really too slow for cancer diagnostics? Or did the author have some other search strategy in mind when they claim that database search is too slow in the intro? \n\nMachine learning is not necessarily \"computationally inexpensive,\" as the authors claim in the introduction. \n", "summary_of_the_paper": "The authors identify a need for standard ML benchmarks in genomics in order to bridge the gap between the bioinformatics and deep learning communities and to make it easier for ML practitioners to contribute to genomics as they have in computer vision and natural language processing. The primary contributions of the paper are\n- six curated, standardized supervised learning tasks involving sequences of nucleic acids\n- baseline neural network models evaluated on those baselines\n", "summary_of_the_review": "Although the paper proposes a useful resource for the genomics ML community, there are serious issues with its presentation of the biology, the data splits, and the baselines evaluated that prevent me from recommending acceptance as is. I do believe that these are addressable (although redoing splits and running baselines may be a tall order during the rebuttal period), and am open to revising my score. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qOUcDJeNgan", "reviewer": "Reviewer_dxXz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The tasks are fairly broad, ranging from regulatory genomics, to phylogeny, to technical biases. \n- I appreciate how the authors try to make a connection with the ML tasks in biology to NLP. \n\nLimitations:\n- Motivation isn't clear: why is it desirable to have a single model that has mediocre predictive performance on all tasks, rather than a specialized model SOTA on each task. Tasks in genomics can be so diverse, why should we expect a single, universal nucleic acid representation that is good for all tasks? For instance, multi-cellular organisms carry the same DNA in each cell, but there is a different gene expression/regulation across different cell types. Moreover, there is a different, timed cellular processes, such as development, within each cell type. This spatial and temporal complexity makes me wonder whether it is sensible to expect that a single representation of nucleic acids would be suitable for a diverse set of prediction tasks in biology. Perhaps the authors can add a rationale for this given the complexity of biological systems.\n- The tasks/datasets do not have enough information and also have inaccuracies.\n\t- For promoter detection, what are the positive label data and what are the negative label data? Which cell types do they come from? \n\t- Random splits are one strategy for generating a test set. Another, better option that is practiced in the field is held out chromosomes. \n\t- It appears that each dataset is preprocessed data from papers, but no clear citations are shown.\n\t- Chromatin classification task, is this the deepsea dataset? If so, the objective is wrong -- it is not about determining all the possible ways a DNA strand can fold! Moreover, the main metric is wrong -- AUC is not sensitive to class imbalance (which this dataset suffers from, assuming it is DeepSea). A better metric is AUPR.  \n- There is no intuition for the kinds of features that are important for each task. Regulatory genomic features are very different from phylogeny problems. A better understanding of the features helps design models with better inductive biases. For example, the bi-LSTM model that is used has practically no inductive bias for regulatory genomics tasks, which is why the AUC is 0.51! If the authors would describe the kinds of features that are important for each task, it would be clear that a convolutional network is more suitable for regulatory genomics, or at least applying a convolutional layer prior to the bi-LSTM can significantly boost performance -- this is a common architectural choice first proposed with DANQ. \n- There were no comparisons to supervised models that are SOTA for each task. This would help give context to the performance of these pretraind language models.  \n- For pretrained models, the authors should benchmark against bigbird and DNABERT.  These models were pretrained across the human genome, not just for the task(s) at hand. It would at least be sensible to pretrain on the human genome, then tune the pretraining on the task at hand. \n- Comments on text:\n\t- The second paragraph is an inaccurate portrayal of the field, with too much emphasis on homology search, which is not one of the benchmark tasks in this paper.\n\t- The longformer is not state-of-the-art -- it has yet to be established.\n- It is not clear biology community would care if performance were better for these tasks. There should be a discussion of how labels are experimentally measured and are thus noisy. They are not ground truth. It's not clear what better performance gives after a certain point, because models may have all captured biology, but better performance may reflect a better model of technical biases or measurement noise. Thus other downstream evaluation metrics are needed, such as model interpretability (i.e. motif discovery and localization) or variant effect prediction, both of which are downstream use cases of NNs in regulatory genomics. \n- IMO, a better sales pitch for this work is to emphasize representation learning versus the more traditional one-hot representation and supervised representation learning versus pre-training with self-supervised masked language modeling. Importantly, if the performance of the language models are not comparable to supervised models, then there will be no engagement between the NLP and bio community. ", "summary_of_the_paper": "This paper proposes a set of 6 benchmark tasks to assess the prediction performance of different models in genomics. The motivation is that there lacks standardized datasets and evaluation methods to seamlessly compare different models across different datasets due to the heterogeneity in the databases and the pre-processing required to get it in a form that is suitable for ML analysis. Therefore the goal is to introduce a set of diverse tasks for DNA/RNA to benchmark pre-trained models, similar to what TAPE has done for proteins. There are 3 tasks for regulatory genomics, 2 tasks based on phylogeny, and 1 task based on technical biases. The authors explore a few baseline models, including a Longformer and bi-directional LSTM, either trained directly on the task, pretrained on the task and then fine tuned on the same task, or pretrained on sequences across all tasks and fine tuned on each task individually. The prediction performance was compared using a metric that the authors thought was appropriate. This paper attempts to bridge the NLP community and the comp bio community by framing numerous comp bio tasks with analogies to NLP. ", "summary_of_the_review": "On surface, the presentation and motivation seems great (to a researcher outside of biology). But once you realize data/model performance are limited compared to existing supervised models, it may mislead the NLP community to focus on ill-formed problems and only provide marginal \"relative\" gains with no impact and no engagement with the bio community. There is a lack of clarity of the datasets, i.e. where they come from and basic intuition for the important features. The baseline models are poor choices for (regulatory) genomics, they are not common. Moreover, prediction performance is only one aspect of evaluation. In biology, evaluation needs to incorporate other aspects, such as model interpretability and variant effect predictions (of GWAS or eQTLs). While bridging both communities (NLP and comp bio) is important, this work seems too premature. I encourage the authors to take this feedback to guide them to generate a much needed set of benchmark tasks, baseline models, and better evaluations that would also be of interest to the comp bio community. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "mVCyJ1n_j0", "reviewer": "Reviewer_QrMg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- I applaud the authors for wanting to reach out to the broader ML community with problems in biology and bioinformatics.\n\n- The authors did a good job of defining if a dataset was balanced or not.\n\nCons:\n\n- Section 3 is a bit too long on it's own. I would have preferred if the biological topics were broken up into each of the respective subsections of 4.x; otherwise it is difficult to keep track of the biological relevance of each task.\n\n- The selection of tasks could be further refined. Why not just focus on different chromatin annotation tasks, like 4.2? Why are there two taxonomy tasks? In particular, why was the its2 task chosen? (Also unclear what its2 is, and why it is an important task the ML community should focus on.)\n\n- Ideally, this paper would present baselines from the original algorithms/packages cited for each of the respective works. e.g. for 4.2, how well does DeepSEA do?\n\n- Dataset specific comments:\n\n4.1 - The negative dataset doesn't seem to be GC content matched. This is typically controlled for in ML for DNA sequences. Moreover, promoters typically have many different transcription start sites. Which one are you picking? Using your negative selection strategy, your \"negative\" data most likely has a TSS in it too.\n\n4.2 The \"Objective\" section is incorrect (\"Determine all the possible ways a DNA strand can fold.\") The annotations in the paper are DNAse1 hypersensitivity, TF binding, and histone methylation. While those are impacted by DNA shape, they are defined as 1D chromatin modifications. Descriptions of DNA folding should be reserved for HiC-like data. Moreover, since you recreated the train/test/validation splits, how are you controlling for homologous sequences? Perhaps a better split would be leaving out a chromosome?\n\n4.6 Why not just use the GreenGenes algorithm directly? The accuracy of your algorithm is determined by the performance of GreenGenes chimera-calling. Does that have less of a greenhouse-emission footprint than a deep model? (As you mention in Section 7.)\n\nNeutral:\n\nDataset 4.3 - Why was only mRNA chosen from all the different RNA subclasses? It seems like the database also contains non-messenger RNA data and annotations.\n\n", "summary_of_the_paper": "The authors curate a set of machine learning tasks on non-protein biological sequences and provide baseline results using deep neural network language models.", "summary_of_the_review": "While the authors have curated a number of non-protein biological datasets, I do not think the objectives of the tasks are refined enough, nor are the methods compelling enough, to justify publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eT61PJRmpv8", "reviewer": "Reviewer_xYqo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am not particularly familiar with these classification tasks so I can not speak to their coverage of relevant tasks in the field. From my vantage point, this paper has the following strengths:\n\n1) It provides a benchmark for an interesting problem area that is actively growing.\n2) The tasks are standardized to allow greater reproducibility and comparisons across competing methods. Based on the authors descriptions, this appears to be missing in the field. \n3) The authors provide reasonable baselines. \n\nThe paper has the following weaknesses\n\n1) Unlike TAPE, which deals exclusively with protein sequences, this work proposes benchmarking DNA and RNA-based prediction. This appears to be convolving two different biological tasks (DNA and RNA) that I would think should be treated separately. Even within DNA and RNA there seem to be differences in, for example, human v.s. nematoda vs bacteria vs achaebacteria that complicate whether a single model should really be benchmarked across all of these tasks. Furthermore, the authors state they draw inspiration from GLUE / SuperGLUE which all deal with a single language (English). In summary TAPE and GLUE / SuperGLUE provide benchmarks capable to benchmarking a unified language model across different tasks. This does not feel appropriate for the tasks provided in this paper as there is no single language. Additionally, whereas in NLP one might hope to combine languages, here DNA and RNA represent fundamentally different domains and a better analogy seems to be benchmarking a model on an NLP task (DNA) and a computer vision task (RNA). In summary, while the goal of providing standardized datasets and benchmarks is achieved, the paper seems fundamentally flawed in it's desire to provide a GLUE / TAPE-like benchmarking system capable of evaluating self-supervised embeddings across different tasks. \n\nIn order to fix this the authors should address the following:\n1a) Clarify their use of the term \"multi-lingual\" which appears to be incorrectly attributed to TAPE. Protein sequences are one language and TAPE provides many tasks for that one language just as GLUE provides many tasks for the English language. \n1b) Comment on the mixing of multiple domains into one benchmark and compare / contrast with SuperGLUE and TAPE. Explicitly discuss the issue of a universal language model, which seems inappropriate for this task. Comment on the results of LF-MULTI and BiLISTM-MULTI in this context. Given the issues I pointed out above, it seems unsurprising that these approaches tend to perform worse (e.g. LF-MULTI is worse on average than LF-NO and LF-MONO). \n1c) Currently, this seems to me to be a major issue with this benchmark. I would greatly appreciate any discussion that the authors can provide to help me better understand whether my critiques are fair and how they can or cannot be addressed. \n\nMy remaining critiques are smaller:\n2) The \"Objective\" for 4.2 Chromatin Classification seems vague and I could not understand what the task is or how this becomes a classification task. Perhaps this could be made clear without resorting to the appendix? \n\n3) The section \"Encoding\" was a bit vague. Why are k-mers needed at all instead of just using bases as tokens analogous to how amino acids are used as tokens in TAPE. Furthermore there is no comment on whether there is a universal encoding for both RNA and DNA or whether the tokenization is done separately for each. It appears to be the former, otherwise the MULTI pretraining would not make sense, but this should be more clearly explained and justified in the paper. ", "summary_of_the_paper": "This paper presents a benchmark consisting of six classification tasks that make predictions based on DNA or RNA sequences. The benchmark is an attempt to standardize existing datasets in terms of featurization, train/val/test splits, and performance metrics. It also reduces the amount of domain expertise required to benchmark a model on these tasks. The paper provides two neural network architecture baselines as well as different pre-training strategies and finds that no single model/strategy dominates all of the tasks. ", "summary_of_the_review": "As the paper currently stands, I see major issues with this being a useful benchmark for the community. It appears to be mixing too many domains (e.g. RNA vs DNA, human vs bacteria). I am open to changing my mind after discussing with other reviewers who may know more about this field and after hearing feedback from the authors. I thank the authors for tackling this challenging and often overlooked problem of benchmarking, but I would like to see more care taken in the construction of the benchmark before this paper is ready to be accepted. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SeI-ig2SZOz", "reviewer": "Reviewer_wY3Y", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript is generally clear and well written. Building a\nstandardized benchmark for comparing methods on prediction tasks from\nDNA or RNA is a useful contribution.\n\nI am not sure however that the proposed contribution is significant\nenough for a publication at ICLR. First, I am not immediately\nconvinced by the importance of comparing models on such a large\nvariety of tasks. The problems listed in the proposed benchmark are\nindeed very different, and may call for radically different\napproaches. In other words, it is not clear to me that one should aim\nfor a method that predicts well on all these tasks. To me a more\nrelevant benchmark could focus on a more narrow problem (eg promoter\nor chromatin feature prediction). I may be wrong but I think it would\nbe useful to motivate this point a bit further.\n\nOf note, the kipoi repository (https://kipoi.org/) already partially\naddresses this task, and should be included in the discussion. More\ngenerally, the description of the field could be made more thorough or\ndetailed at some places. For example:\n\n- I disagree with the claim that \"Umarov & Solovyev (2017) were the\n  first to train a neural network on promoter detection\" when DeepBind\n  (Alipanahi et al., 2015) tackled the same problem.\n\n- It could be useful to say a bit more about the longformer model that\n  is used as a baseline, and about contextual embeddings, and why\n  either are expected to help on the tasks chosen in the benchmark. On\n  a related point, I was surprised by the large sequence size used for\n  the promD task (2048), while this specific task is often done on\n  shorter (a few hundreds of bp) sequences, where long range models\n  are typically less useful.\n\n- ChromC seems to rely on the task tackled in the DeepSea (Zhou and\n  Troyanskaya, 2015) paper, where the goal is to predict chromatin\n  features such as transcription factor binding, histone or DNAse\n  features, not DNA folding.\n\n- I would need more explanations on the claim that \"Unlike natural\nlanguages, DNA and RNA sequences are not “naturally” split into words,\ntherefore it is necessary to define synthetic tokens.\". A common\napproach (taken by both deepbind and deepsea) is to directly start\nfrom a one-hot encoded sequence. Working at the kmer level can improve\nthe performance but is not a necessary step.\n", "summary_of_the_paper": "This contribution introduces a benchmark for the evaluation of neural\nnetworks for supervised classification of DNA and RNA molecules. The\nbenchmark consists of six datasets covering DNA and RNA as well as\ndifferent learning tasks (binary classification, multiclass,\nhierarchical), sequence lengths and sample sizes. The authors propose\nbaselines on each of these tasks, one based on an LSTM, the other on a\nrecent transformer architecture.\n", "summary_of_the_review": "While a benchmark for prediction from DNA/RNA is a useful contribution, I find the following main limitations:\n1 - Limited contribution (given the scope of ICLR).\n2 - Not sufficiently justified use of a common benchmark for very diverse tasks.\n3 - Sometimes not detailed/justified enough elements given the field.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While all reviewers applaud the motivation to bridge the gap between machine learning and bioinformatics communities, they also raise a number of concerns regarding the choice of tasks and of baselines, and about the accuracy in their description. They feel the paper is not ready to be published in its current form, and we hope that their comments will help the authors prepare a revised version for the future.
Decision: Reject