BANANA: a Benchmark for the Assessment of Neural Architectures for Nucleic Acids

1 INTRODUCTION . Since the advent of sequencing techniques ( Sanger et al. , 1977 ) , there has been an exponentially increasing availability of data in the form of sequences of DNA , RNA , and proteins . Analysis of these sequences has marked many milestones in every field of biology , from the Human Genome Project ( Watson , 1990 ) to the engineering of vaccines for the SARS-COV-2 virus ( Chen et al. , 2020 ) , from climate-change studies ( Jansson & Hofmockel , 2020 ) to forensics ( Bianchi & Liò , 2007 ) . Due to the huge amount of data , these advances could have not been achieved without the support of bioinformatics algorithms . Many tasks in bioinformatics are traditionally solved by search algorithms , such as BLAST ( Altschul et al. , 1990 ) , which matches a sequence against a reference database containing annotated entries ( and so tasks are solved in terms of similarity ) . Databases are usually very large and search can be very slow , impacting fields in which a timely response is important , such as cancer diagnostics . Indexing strategies or heuristic searches can speed up the querying process , at the expense of a space or accuracy trade-off . The application of machine learning methods has provided a great contribution to this field , allowing to solve tasks in a computationally inexpensive way , with a greater robustness to noise , and to discover new associations ( Vinga , 2014 ; Larranaga et al. , 2006 ) . In particular , the recent development of Natural Language Processing ( NLP ) techniques has lead researchers to use advanced neural models for modeling biological sequences , such as proteins , DNA , and RNA ( Ji et al. , 2021 ; Zaheer et al. , 2020 ; Wahab et al. , 2021 ) . However , the heterogeneity in the databases where biological data are stored and the lack of a general consensus on the data processing makes it difficult to define machine learning tasks , and more in general to approach this domain without having a deep understanding of it . At the same time , there is a consensus in the NLP community that to properly evaluate a language model there is the necessity to test in on multiple tasks to measure whether it can be broadly applied . For this reason , multi-tasks benchmarks such as SUPERGLUE ( Wang et al. , 2019a ) have been developed . To meet these needs , Rao et al . ( 2019 ) have developed TAPE , a benchmark designed to rigorously evaluate the performance of learned protein embeddings through a multitude of tasks , encompassing the domains of structure prediction , evolutionary understanding , and protein engineering , which have similarities with NLP tasks such as named entity recognition and natural language classification . A different class of languages , not covered by TAPE , ranges over the DNA and RNA domains . These domains are crucial in bioinformatics because they enable researching tumors , subcellular structures and the environment . Accordingly , we present BANANA ( Benchmark for the Assessment of Neural Architectures for Nucleic Acids ) , a benchmark consisting of six classification tasks assessing language understanding performance in the DNA and RNA domains . The tasks are defined over three genomics ( human DNA , bacterial 16S gene , nematoda ITS2 gene ) and one transcriptomics ( human mRNA ) language and measure an architecture ’ s capability of performing whole-sequence classification in a variety of setups . We curate existing data providing datasets that do not impose any specific constraint regarding the approach , creating a benchmark that is versatile and easy to use even for those who do not have knowledge of the domain . Finally , we perform few experiments with advanced neural architecture , so as to provide baselines for future works . In Section 2 we describe existing benchmarks and draw similarities and differences with our work . Section 3 provides background for our domain of application and highlights the importance of our tasks , which are described in Section 4 . We evaluate our baselines in Section 5 , while Section 6 list related works for each task . Finally , Section 7 concludes . 2 NATURAL LANGUAGE UNDERSTANDING AND PREVIOUS BENCHMARKS . In the context of unstructured data , such as textual documents , their representation plays a key role in the performance of any machine learning algorithm . Indeed , representations capable of encoding information not only about the lexicon but also about the syntax and the semantics of a sentence , provide the models with additional knowledge that can be exploited in higher-level tasks . The task of learning effective representations is called Natural Language Modelling ( NLM ) , and it is typically addressed through semi-supervised tasks ( Devlin et al. , 2019 ) . Once learned , the quality of a language model can be evaluated by using it to address more advanced NLP tasks that concern the comprehension of the text , in a set of tasks that falls under the broad definition of Natural Language Understanding ( NLU ) . Our proposal draws inspiration from the GLUE benchmark ( Wang et al. , 2019b ) , a set of 9 supervised sentence understanding tasks in the English language based on existing datasets . Its authors proposed it with the purpose of stimulating the development of a more unified language model , capable of solving a range of different linguistic tasks in different domains . Their experimental results showed how models trained in multi-task learning setting outperformed models trained on a single task , and therefore the benefits of sharing knowledge across tasks . But the interest in most of these tasks has quickly decreased with the advent of advanced language models such as BERT , which was capable of easily solving most of them . Therefore , a new benchmark called SUPERGLUE ( Wang et al. , 2019a ) has been later proposed , including new supervised learning tasks that are solvable by humans but are difficult for machines . All the tasks proposed in this benchmark regard the classification of one or multiple sequences of text , and a final score is given based on the average score obtained on every single task . Similarly , our tasks regard the classification of a single sequence , and we compute a final score in a similar fashion . Another analogy between these works and our proposal concerns the fact that they are agnostic regarding the pre-train of the architectures and do not include the language modeling task as part of the benchmark . The TAPE benchmark ( Rao et al. , 2019 ) fills the need for datasets and standardized evaluation methods for machine learning techniques applied to protein sequences , and it is another source of inspiration for our work . The authors propose to use NLP techniques in the domains of proteins , presenting the self-supervised learning task of modeling the “ languages of proteins ” using a dataset containing proteins of different organisms and absolving different functions , along with 5 supervised learning tasks concerning 3 major subfields of protein sciences . All these tasks are based on existing datasets , but they are curated and formalized by Rao et al . The experimental part shows that the selfsupervised task is beneficial for all the neural models and that they can successfully address three of the five tasks . It also highlights how the performances of each model vary across the different sub-fields . Our work differs from TAPE since we address tasks that span across four different “ biological languages ” ( human DNA , human mRNA , bacterial and archaebacterial DNA , nematoda DNA ) and specifically address each of them . Instead , TAPE ’ s tasks focus on multi-lingual settings , which requires the ability to generalize over a broad spectrum of “ protein languages ” . 3 BIOLOGY BACKGROUND . The central dogma of molecular biology ( Crick , 1970 ) states that , for every living organism , the information is stored in DNA ( deoxyribonucleic acid ) , then it flows to RNA ( ribonucleic acid ) and finally to proteins , which are the building blocks of life but can not be transferred from one generation to the next . Both DNA and RNA are composed of a sugar-phosphate scaffold and nucleotides ( nt ) , which contain the genetic information . DNA is a double helix composed of two filaments in which complementary nucleotides ( A , T , C , G ) bind to each other forming base pairs ( bp ) , while RNA is a single strand of nucleotides ( A , U , C , G ) , capable of folding in different ways1 . RNA is synthesized ( transcribed ) by making a complementary copy ( using a different alphabet , but semantically a 1:1 copy ) of a region of DNA ( a gene ) , starting from the transcription start site ( TSS ) up to a termination point . In order for the copying machinery to “ select ” the proper region , some transcription factors ( TF ) must bind to DNA in a region immediately before the TSS , called the promoter of that gene . DNA is compressed into a structure called chromatin , whose folding structure plays an important role in cell differentiation : a liver cell and a neuron share the same DNA , but thanks to different chromatin profiles they appear and behave in completely different ways . These mechanisms are fundamental for life and their malfunction is linked with cancer ( Morgan & Shilatifard , 2015 ) . RNA absolves many functions inside the cell ( e.g. , gene regulation , signaling , etc . ) and it is also correlated to cancer ( Reddy , 2015 ) and infections ( Fani et al. , 2021 ) . Three main types play universal roles across every living organism : ribosomal RNA ( rRNA ) and transfer RNA ( tRNA ) are the gears in the machinery for protein synthesis , while messenger RNA ( mRNA ) constitutes the “ recipe ” for proteins . Freshly transcribed mRNA filaments migrate each to a specific position inside the cell , known as their subcellular localization ( Holt & Bullock , 2009 ) , and then ribosomes translate them to proteins . The analysis of mRNA localization may help to better understand subcellular compartments and lead to a more detailed and nuanced understanding of cellular architecture ( Martin & Ephrussi , 2009 ) . Ribosomes translate mRNA into proteins and are made of two subunits : small ( SSU ) and large ( LSU ) . The bacterial SSU is called 16S and , since it evolves with a very slow mutation rate , it can be used to reliably classify bacteria , also for the so-called “ microbial dark matter ” , those bacteria which can not be classified using traditional methods ( Kalmbach et al. , 1997 ) , and which can have an effect , for example , on health ( Shreiner et al. , 2015 ) or climate change ( Jansson & Hofmockel , 2020 ) . Eukaryotes can be classified by SSU markers ( 18S ) as well , however popular alternatives are the Internal Transcribed Spacer 2 ( ITS2 ) , a region between the SSU and the LSU which is transcribed , but then destroyed instead of being part of ribosomes , and the mitochondrial SSU ( 12S ) . Sequencing is the process of reading a sequence of DNA , RNA , or protein , starting from a sample . Genomics , transcriptomics , and proteomics are the studies of data ( DNA , RNA , and proteins , respectively ) sequenced from a single organism . When this is done on data sequenced from multiple organisms at the same time , the studies are called metagenomics , metatranscriptomics , and metaproteomics . For example , extracting all the DNA present in an environmental sample in order to determine which bacteria live in that environment is a classical example of metagenomics . Sequencing technologies are characterized by various kinds of reading errors , an important one arising during meta-omics sequencing is the production of chimera artifacts . Chimeras are digital sequences that do not exist in the real world , instead they are composed of portions of existing sequences that have been hybridized . They are produced when sequencing with highly parallel technologies is stalled while it is reading sequences that share a similar region . For example , if 1Since base pairs are complementary , from a linguistic perspective , a single DNA strand contains the entire information , so both bp , for DNA , and nt , for RNA , can be considered the “ characters ” of nucleic acids languages . AACTCTGGA and GGGTCTTTT are both stopped and resumed when reading TCT , the sequencer may mistakenly read the chimeras AACTCTTTT and GGGTCTGGA . More complex chimeras may also arise as hybrids of more than two sequences , palindromic sequences , etc . Chimeras are a serious problem in metagenomics studies , especially on 16S genes , constituting even a large portion of sequencer outputs ( Wang & Wang , 1997 ) , and can cause misclassification of a population of organisms . Moreover , detection has been a problem and some public databases have been historically plagued by chimeras mistaken for legitimate sequences ( Hugenholtz & Huber , 2003 ) .

Summary: This paper proposes a set of 6 benchmark tasks to assess the prediction performance of different models in genomics. The motivation is that there lacks standardized datasets and evaluation methods to seamlessly compare different models across different datasets due to the heterogeneity in the databases and the pre-processing required to get it in a form that is suitable for ML analysis. Therefore the goal is to introduce a set of diverse tasks for DNA/RNA to benchmark pre-trained models, similar to what TAPE has done for proteins. There are 3 tasks for regulatory genomics, 2 tasks based on phylogeny, and 1 task based on technical biases. The authors explore a few baseline models, including a Longformer and bi-directional LSTM, either trained directly on the task, pretrained on the task and then fine tuned on the same task, or pretrained on sequences across all tasks and fine tuned on each task individually. The prediction performance was compared using a metric that the authors thought was appropriate. This paper attempts to bridge the NLP community and the comp bio community by framing numerous comp bio tasks with analogies to NLP.