Self-GenomeNet: Self-supervised Learning with Reverse-Complement Context Prediction for Nucleotide-level Genomics Data

In this paper , we introduce a novel contrastive self-supervised method , namely Self-GenomeNet , for nucleotide genome representation learning . To the best of our knowledge , Self-GenomeNet is the first self-supervised framework that learns a representation of nucleotide genomic data , using domain-specific characteristics . Our proposed method learns and parameterizes the latent space by leveraging the reverse-complement of genomic sequences . During the training procedure , we force our framework to capture semantic representations with a novel context network on top of features extracted by an encoder network . The network is trained with an unsupervised contrastive loss . Extensive experiments with different datasets show that our method with self-supervised and semi-supervised settings outperforms state-of-the-art deep learning methods . Furthermore , we show that the learned representations generalize well and can be transferred to new datasets and tasks . The source code of Self-GenomeNet and all experiments are provided as supplementary material . 1 INTRODUCTION . In bioinformatics , learning from unlabeled data can reduce development costs in many deep learning applications , since it reduces the number of required annotated samples that often require manual experimental validation , such as the functional annotation of genes ( Gligorijević et al. , 2021 ) or chromatin effects of single nucleotide polymorphisms ( SNPs ) ( Zhou and Troyanskaya , 2015 ) . Selfsupervised learning is amongst the most promising approaches for learning from limited amounts of labeled data and is an attractive approach in bioinformatics due to the availability of large quantities of unlabeled sequence data . In contrast to supervised methods , these techniques learn a representation of the data without relying on human annotation . Early self-supervised methods focused on solving a pretext task ( Devlin et al. , 2018 ) that allows utilizing the data itself to generate labels , and use supervised methods to solve unsupervised problems . The representations learned by performing this task can be used as a starting point for downstream supervised tasks such as taxonomic prediction or gene annotation . Several different self-supervised representation learning methods were proposed e.g . in the fields of natural language processing ( NLP ) and computer vision ( CV ) . However , self-supervised learning has not yet seen such widespread adoption in bioinformatics and remains an important and challenging endeavor for the representation of genome data . Existing methods for representation learning for omics-data tends to use methods from NLP or CV with slight architecture adjustments and additional preprocessing ( e.g . Rives et al. , 2021 ; Lu et al. , 2020 ; Ciortan and Defrance , 2021 ) . However , these methods so far do not take many biological characteristics into account . In particular , the genome sequence embodied by one strand on the DNA molecule is always accompanied by another strand going in the opposite direction with complementary nucleotides . This means that both the forward and the so-called reverse-complement ( RC ) sequences are valid DNA sequences and equally correspond to several properties of the genome , such as regulatory and taxonomical characteristics . Therefore , machine learning models developed to distinguish different aspects of genomic sequences ideally take the RC into account , for example by being equivariant with respect to it ( Mallet and Vert , 2021 ; Zhou et al. , 2021 ) . Our proposed Self-GenomeNet uses a context network that learns representations that also take advantage of the RC . We propose a simple contrastive self-supervised algorithm for learning representations of nucleotide-level genomic sequences ( Figure 1 ) . Our method divides a sequence into two nonintersecting subsequences , where one of them is transformed to its RC . Then , the contextual representations of these subsequences are required to distinguish each other from the representations of other , random sequences . We exploit the inherent symmetry in the genome sequence implied by the RC to achieve a natural simplification over possible methods that do not take it into account , all while encoding RC-awareness in our architecture . Moreover , we train our network with varyinglength subsequences both for the input and the target subsequence . Because , genome sequences can have semantic structure at varying ranges of length , representations that respect both local and global structures are preferable . For this , the context network makes use of a recurrent neural network layer , which makes the computationally efficient training based on all used length scales within a single step possible . Our self-supervised method utilizes contrastive loss to maximize the similarity between representations of subsequence and its RC while minimizing the similarity between representations derived from unrelated sequences . Contributions : We present the first self-supervised framework tailored specifically to DNA sequence data . Our main findings and contributions can be summarized as follows : ( 1 ) We introduce a contrastive self-supervised method that is specialized for nucleotide-level genomics data . ( 2 ) We propose a context-aware network that makes use of inherent symmetries in genomics data in the form of the reverse-complement and improves the quality of learning representation . ( 3 ) SelfGenomeNet is evaluated in self-supervised and semi-supervised settings on bioinformatic benchmarks and datasets and achieved superior performance in several tasks . ( 4 ) We show the learned representations generalize well when transferred to a new dataset and task . 2 RELATED WORK AND BACKGROUND . Self-supervised learning Deep learning models based on self-supervised learning have become a standard method of choice in natural language processing ( NLP ) . Powerful models such as GPT3 ( Brown et al. , 2020 ) and BERT ( Devlin et al. , 2018 ) are pre-training in a self-supervised manner on a large text corpus . They are then often fine-tuned for various tasks using smaller task-specific datasets , where they show performance that is superior to purely supervised methods . Recently , selfsupervised visual representation learning ( Chen et al. , 2020 ) outperformed supervised methods and achieved state-of-the-art performance on ImageNet ( Deng et al. , 2009 ) . Self-supervised contrastive learning is one of the most successful methods where it maximizes the agreement of learned representations between differently augmented views of the same example ( Chen et al. , 2020 ; Oord et al. , 2018 ) . Motivated by this , we propose a novel contrastive self-supervised model similar to ( Oord et al. , 2018 ) but differing in network architecture , representation learning strategy , and with applications specifically in bioinformatics . Self-supervised learning in bioinformatics Existing self-supervised learning techniques applied on sequential RNA and protein data are adopted from other application fields of deep learning . ( Rives et al. , 2021 ) adopted BERT ( Devlin et al. , 2018 ) as a pre-trained model for predicting masked amino acids and achieved superior performance on sequence prediction task . Contrastive-sc ( Ciortan and Defrance , 2021 ) is a method adapted from computer vision for cell clustering based on single-cell RNA sequencing data . Contrastive-sc creates two copies of sequences with randomly masked nucleotides and trained the network using a contrastive loss . As a downstream task , self-supervised embeddings are used for clustering with K-Means algorithm ( Hartigan and Wong , 1979 ) . All these methods for self-supervision described so far have in common that they rely on randomly masking part of their input to create a pretext task . This differs from Contrastive Predictive Coding ( CPC ) ( Oord et al. , 2018 ) , which splits the input into parts such as image patches or subsequences , and predicts the representation of some of the parts , contrastively , from the representation of other parts . CPCProt ( Lu et al. , 2020 ) adapted the CPC model for protein data.Similarly , Self-GenomeNet is trained by the contrastive loss in which the positive pairs include the input sequence and a reversecomplement of the sequence . Unlike previous works , our network is trained with varying-length targets to explore both local and global structures of sequences . Deep learning methods for genome sequences In genomics , many deep learning architectures use convolutional layers to capture local patterns in data , whereas recurrent layers learn long-range dependencies in the sequences ( Eraslan et al. , 2019 ) . Zhou and Troyanskaya ( 2015 ) created the DeepSEA dataset for predicting chromatin effects of sequence alterations , and used it to train a model consisting of multiple convolutional layers . Later , DanQ ( Quang and Xie , 2016 ) improved the deep learning framework of DeepSEA using a bidirectional LSTM ( Hochreiter and Schmidhuber , 1997 ) unit on top of a convolutional layer . iDeepS ( Pan et al. , 2018 ) , DeepBind ( Alipanahi et al. , 2015 ) , and DeeperBind ( Hassanzadeh and Wang , 2016 ) use a sequence CNNs and RNNs to predict RNA binding proteins sites . DeepDNA ( Wang et al. , 2019 ) uses an LSTM layer on top of a CNN layer to compress human mitochondrial genome data . Instead of recurrent layers , it is also possible to use dilated convolutional layers to model long-range dependencies in genome data , such as done in Basenji ( Kelley et al. , 2018 ) , which is used to predict cell-type–specific epigenetic and transcriptional profiles in large mammalian genomes . Trabelsi et al . ( 2019 ) performed a comprehensive study on network architecture where the ECBLSTM network , composed of an embedding layer , CNNs , and LSTM layer outperformed other networks on several genomic tasks . Here , we also used the DeepSEA dataset to compared the performance of our method with several other self-supervision techniques . The architecture of Self-GenomeNet requires a recurrent layer for it to create representations of varying-length subsequences , and is therefore based off the DanQ-architecture ( Quang and Xie , 2016 ) , which has shown good performance on the DeepSEA task . Reverse Complement DNA molecules in living organisms always occur as a double strand , where both strands have opposite directions and consist of nucleotides that are complementary to each other . The sequence on one strand is therefore accompanied by its reverse complement ( RC ) on the other : The sequence in reverse , and with the A , C , G , T nucleotides replaced by T , G , C , A respectively . Since both forward and reverse-complement sequences are found in the structure , they are equally responsible for several properties of the genome , such as regulatory and taxonomical characteristics . However , next-generation sequencing techniques ( NGS ) output sub-sequences randomly drawn from both strands , since it usually involves breaking the genome into a collection of small DNA fragments that are sequenced individually . Therefore , both the original sequence data and the RC of this sequence are valid inputs to the machine learning models developed to work on NGS datasets ( Shrikumar et al. , 2017 ; Zhou et al. , 2021 ) . In this paper , we exploit this genome-specific feature in contrastive self-supervised learning framework helps to improve the learning representation in several genomic tasks . 3 METHOD . We first describe the motivation of our proposed method before explaining its details in Section 3.2 . 3.1 MOTIVATION . Most successful self-supervised learning approaches are built for computer vision or NLP and are therefore not optimal for bioinformatics data . For example , the self-supervision task of CPC ( Oord et al. , 2018 ) or generative language models ( Dai and Le , 2015 ) are based on predicting the representation of a short , constant length subsequence or a single token , which could be an N-gram or a single nucleotide . We hypothesize single-length short sequence targets hinder the model from learn- ing global semantics effectively . We therefore propose training representations that are based on target subsequences whose lengths are varying . While predicting future observations of constant sequence length from input sequences which are changing in length is common for methods like CPC and single character models , predicting representations of varying length sequences is a novelty in this area . Similar to Oord et al . ( 2018 ) , we use a recurrent context network on top of an encoding network that encodes to constant-size patches . However , we use the output of the context network to predict the output of the same context network applied to other patches , as opposed to predicting only the output of the encoding network . The fact that the RC of an input sequence is a valid genome sequence as well simplifies this approach : instead of having to fit both one forward and one reverse context network that predict each other , only a single context-network is trained that is both applied to the forward as well as to the RC sequence .

Summary: This submission introduces self-genomenet, a self-supervised training method for learning from DNA sequences. The self-supervision is done by predicting the end of a sequence from its start (both broken into smaller subsequences), through a contrastive loss against other random sequences. The predicted part is also reverse-complemented (RC), making the network learn the expected reverse-complement invariance of the prediction function. The method is extensively tested on several learning tasks, where it shows good performances.