BANANA: a Benchmark for the Assessment of Neural Architectures for Nucleic Acids

1 INTRODUCTION . Since the advent of sequencing techniques ( Sanger et al. , 1977 ) , there has been an exponentially increasing availability of data in the form of sequences of DNA , RNA , and proteins . Analysis of these sequences has marked many milestones in every field of biology , from the Human Genome Project ( Watson , 1990 ) to the engineering of vaccines for the SARS-COV-2 virus ( Chen et al. , 2020 ) , from climate-change studies ( Jansson & Hofmockel , 2020 ) to forensics ( Bianchi & Liò , 2007 ) . Due to the huge amount of data , these advances could have not been achieved without the support of bioinformatics algorithms . Many tasks in bioinformatics are traditionally solved by search algorithms , such as BLAST ( Altschul et al. , 1990 ) , which matches a sequence against a reference database containing annotated entries ( and so tasks are solved in terms of similarity ) . Databases are usually very large and search can be very slow , impacting fields in which a timely response is important , such as cancer diagnostics . Indexing strategies or heuristic searches can speed up the querying process , at the expense of a space or accuracy trade-off . The application of machine learning methods has provided a great contribution to this field , allowing to solve tasks in a computationally inexpensive way , with a greater robustness to noise , and to discover new associations ( Vinga , 2014 ; Larranaga et al. , 2006 ) . In particular , the recent development of Natural Language Processing ( NLP ) techniques has lead researchers to use advanced neural models for modeling biological sequences , such as proteins , DNA , and RNA ( Ji et al. , 2021 ; Zaheer et al. , 2020 ; Wahab et al. , 2021 ) . However , the heterogeneity in the databases where biological data are stored and the lack of a general consensus on the data processing makes it difficult to define machine learning tasks , and more in general to approach this domain without having a deep understanding of it . At the same time , there is a consensus in the NLP community that to properly evaluate a language model there is the necessity to test in on multiple tasks to measure whether it can be broadly applied . For this reason , multi-tasks benchmarks such as SUPERGLUE ( Wang et al. , 2019a ) have been developed . To meet these needs , Rao et al . ( 2019 ) have developed TAPE , a benchmark designed to rigorously evaluate the performance of learned protein embeddings through a multitude of tasks , encompassing the domains of structure prediction , evolutionary understanding , and protein engineering , which have similarities with NLP tasks such as named entity recognition and natural language classification . A different class of languages , not covered by TAPE , ranges over the DNA and RNA domains . These domains are crucial in bioinformatics because they enable researching tumors , subcellular structures and the environment . Accordingly , we present BANANA ( Benchmark for the Assessment of Neural Architectures for Nucleic Acids ) , a benchmark consisting of six classification tasks assessing language understanding performance in the DNA and RNA domains . The tasks are defined over three genomics ( human DNA , bacterial 16S gene , nematoda ITS2 gene ) and one transcriptomics ( human mRNA ) language and measure an architecture ’ s capability of performing whole-sequence classification in a variety of setups . We curate existing data providing datasets that do not impose any specific constraint regarding the approach , creating a benchmark that is versatile and easy to use even for those who do not have knowledge of the domain . Finally , we perform few experiments with advanced neural architecture , so as to provide baselines for future works . In Section 2 we describe existing benchmarks and draw similarities and differences with our work . Section 3 provides background for our domain of application and highlights the importance of our tasks , which are described in Section 4 . We evaluate our baselines in Section 5 , while Section 6 list related works for each task . Finally , Section 7 concludes . 2 NATURAL LANGUAGE UNDERSTANDING AND PREVIOUS BENCHMARKS . In the context of unstructured data , such as textual documents , their representation plays a key role in the performance of any machine learning algorithm . Indeed , representations capable of encoding information not only about the lexicon but also about the syntax and the semantics of a sentence , provide the models with additional knowledge that can be exploited in higher-level tasks . The task of learning effective representations is called Natural Language Modelling ( NLM ) , and it is typically addressed through semi-supervised tasks ( Devlin et al. , 2019 ) . Once learned , the quality of a language model can be evaluated by using it to address more advanced NLP tasks that concern the comprehension of the text , in a set of tasks that falls under the broad definition of Natural Language Understanding ( NLU ) . Our proposal draws inspiration from the GLUE benchmark ( Wang et al. , 2019b ) , a set of 9 supervised sentence understanding tasks in the English language based on existing datasets . Its authors proposed it with the purpose of stimulating the development of a more unified language model , capable of solving a range of different linguistic tasks in different domains . Their experimental results showed how models trained in multi-task learning setting outperformed models trained on a single task , and therefore the benefits of sharing knowledge across tasks . But the interest in most of these tasks has quickly decreased with the advent of advanced language models such as BERT , which was capable of easily solving most of them . Therefore , a new benchmark called SUPERGLUE ( Wang et al. , 2019a ) has been later proposed , including new supervised learning tasks that are solvable by humans but are difficult for machines . All the tasks proposed in this benchmark regard the classification of one or multiple sequences of text , and a final score is given based on the average score obtained on every single task . Similarly , our tasks regard the classification of a single sequence , and we compute a final score in a similar fashion . Another analogy between these works and our proposal concerns the fact that they are agnostic regarding the pre-train of the architectures and do not include the language modeling task as part of the benchmark . The TAPE benchmark ( Rao et al. , 2019 ) fills the need for datasets and standardized evaluation methods for machine learning techniques applied to protein sequences , and it is another source of inspiration for our work . The authors propose to use NLP techniques in the domains of proteins , presenting the self-supervised learning task of modeling the “ languages of proteins ” using a dataset containing proteins of different organisms and absolving different functions , along with 5 supervised learning tasks concerning 3 major subfields of protein sciences . All these tasks are based on existing datasets , but they are curated and formalized by Rao et al . The experimental part shows that the selfsupervised task is beneficial for all the neural models and that they can successfully address three of the five tasks . It also highlights how the performances of each model vary across the different sub-fields . Our work differs from TAPE since we address tasks that span across four different “ biological languages ” ( human DNA , human mRNA , bacterial and archaebacterial DNA , nematoda DNA ) and specifically address each of them . Instead , TAPE ’ s tasks focus on multi-lingual settings , which requires the ability to generalize over a broad spectrum of “ protein languages ” . 3 BIOLOGY BACKGROUND . The central dogma of molecular biology ( Crick , 1970 ) states that , for every living organism , the information is stored in DNA ( deoxyribonucleic acid ) , then it flows to RNA ( ribonucleic acid ) and finally to proteins , which are the building blocks of life but can not be transferred from one generation to the next . Both DNA and RNA are composed of a sugar-phosphate scaffold and nucleotides ( nt ) , which contain the genetic information . DNA is a double helix composed of two filaments in which complementary nucleotides ( A , T , C , G ) bind to each other forming base pairs ( bp ) , while RNA is a single strand of nucleotides ( A , U , C , G ) , capable of folding in different ways1 . RNA is synthesized ( transcribed ) by making a complementary copy ( using a different alphabet , but semantically a 1:1 copy ) of a region of DNA ( a gene ) , starting from the transcription start site ( TSS ) up to a termination point . In order for the copying machinery to “ select ” the proper region , some transcription factors ( TF ) must bind to DNA in a region immediately before the TSS , called the promoter of that gene . DNA is compressed into a structure called chromatin , whose folding structure plays an important role in cell differentiation : a liver cell and a neuron share the same DNA , but thanks to different chromatin profiles they appear and behave in completely different ways . These mechanisms are fundamental for life and their malfunction is linked with cancer ( Morgan & Shilatifard , 2015 ) . RNA absolves many functions inside the cell ( e.g. , gene regulation , signaling , etc . ) and it is also correlated to cancer ( Reddy , 2015 ) and infections ( Fani et al. , 2021 ) . Three main types play universal roles across every living organism : ribosomal RNA ( rRNA ) and transfer RNA ( tRNA ) are the gears in the machinery for protein synthesis , while messenger RNA ( mRNA ) constitutes the “ recipe ” for proteins . Freshly transcribed mRNA filaments migrate each to a specific position inside the cell , known as their subcellular localization ( Holt & Bullock , 2009 ) , and then ribosomes translate them to proteins . The analysis of mRNA localization may help to better understand subcellular compartments and lead to a more detailed and nuanced understanding of cellular architecture ( Martin & Ephrussi , 2009 ) . Ribosomes translate mRNA into proteins and are made of two subunits : small ( SSU ) and large ( LSU ) . The bacterial SSU is called 16S and , since it evolves with a very slow mutation rate , it can be used to reliably classify bacteria , also for the so-called “ microbial dark matter ” , those bacteria which can not be classified using traditional methods ( Kalmbach et al. , 1997 ) , and which can have an effect , for example , on health ( Shreiner et al. , 2015 ) or climate change ( Jansson & Hofmockel , 2020 ) . Eukaryotes can be classified by SSU markers ( 18S ) as well , however popular alternatives are the Internal Transcribed Spacer 2 ( ITS2 ) , a region between the SSU and the LSU which is transcribed , but then destroyed instead of being part of ribosomes , and the mitochondrial SSU ( 12S ) . Sequencing is the process of reading a sequence of DNA , RNA , or protein , starting from a sample . Genomics , transcriptomics , and proteomics are the studies of data ( DNA , RNA , and proteins , respectively ) sequenced from a single organism . When this is done on data sequenced from multiple organisms at the same time , the studies are called metagenomics , metatranscriptomics , and metaproteomics . For example , extracting all the DNA present in an environmental sample in order to determine which bacteria live in that environment is a classical example of metagenomics . Sequencing technologies are characterized by various kinds of reading errors , an important one arising during meta-omics sequencing is the production of chimera artifacts . Chimeras are digital sequences that do not exist in the real world , instead they are composed of portions of existing sequences that have been hybridized . They are produced when sequencing with highly parallel technologies is stalled while it is reading sequences that share a similar region . For example , if 1Since base pairs are complementary , from a linguistic perspective , a single DNA strand contains the entire information , so both bp , for DNA , and nt , for RNA , can be considered the “ characters ” of nucleic acids languages . AACTCTGGA and GGGTCTTTT are both stopped and resumed when reading TCT , the sequencer may mistakenly read the chimeras AACTCTTTT and GGGTCTGGA . More complex chimeras may also arise as hybrids of more than two sequences , palindromic sequences , etc . Chimeras are a serious problem in metagenomics studies , especially on 16S genes , constituting even a large portion of sequencer outputs ( Wang & Wang , 1997 ) , and can cause misclassification of a population of organisms . Moreover , detection has been a problem and some public databases have been historically plagued by chimeras mistaken for legitimate sequences ( Hugenholtz & Huber , 2003 ) .

Summary: This paper presents a benchmark consisting of six classification tasks that make predictions based on DNA or RNA sequences. The benchmark is an attempt to standardize existing datasets in terms of featurization, train/val/test splits, and performance metrics. It also reduces the amount of domain expertise required to benchmark a model on these tasks. The paper provides two neural network architecture baselines as well as different pre-training strategies and finds that no single model/strategy dominates all of the tasks.