Reverse-Complement Equivariant Networks for DNA Sequences

1 Introduction . Incorporating prior knowledge about the structure of data in the architecture of neural networks is a promising approach to design expressive models with good generalization properties . In particular , exploiting natural symmetries in the data can lead to models with fewer parameters to estimate than agnostic approaches . This is especially beneficial when the amount of available data is limited . A famous example of such an architecture is the convolutional neural network ( CNN ) for 1D sequences or 2D images , which is well adapted to problems which are invariant to translations in the data , while exploiting multiscale and local information in the signals . Motivated by the success of CNNs , there has been a fast-growing body of research in recent years to build the theoretical underpinnings and design architectures and efficient algorithms to systematically exploit symmetries and structures in the data [ 3 ] . A central idea that has emerged is to formalize the symmetries in data by a particular group action ( e.g. , the group of translations or rotations on images ) , and to create multilayer neural networks which , by design , “ behave well ” under the action of the group . This is captured formally by the concept of equivariance , which states that each equivariant layer should be designed to be subject to the group 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . action ( e.g. , we should be able to `` translate '' or `` rotate '' the signal in each layer ) , and that when an input data is transformed by a particular group element , then its representation in an equivariant layer should also be transformed according to the same group element . While it is easy to see that convolutional layers in CNNs are equivariant to translations , Cohen and Welling [ 7 ] formalized the concept of group equivariance CNN ( G-CNN ) for more general groups and showed in particular how to design convolutional layers equivariant not only to translations but also to reflections and to a discrete set of rotations . Following this seminal work , the theoretical foundations of group equivariant neural networks were then expanded , going beyond regular representations [ 9 ] , for more groups [ 2 , 18 , 37 , 40 ] , in less regular spaces [ 8 , 10 ] or with more general results on their generality and universality [ 11 , 13 , 14 , 22 ] . The main applications were developed with the groups of rotations in 2D and 3D , mostly to computer vision problems , but also in biology with histopathology [ 17 , 23 ] , medicine [ 41 ] and quantum chemistry [ 33 ] . In this paper , we explore and study the potential benefits of equivariant architectures for an important class of data , namely deoxyribonucleic acid ( DNA ) sequences . DNA is the major form of genetic material in most organisms , from bacteria to mammals , which encodes in particular all proteins that a cell can produce and which is transmitted from generation to generation . The study of DNA in humans and various organisms has led to tremendous progress in biology and medicine since the 1970s , when the first DNA sequencing technologies were invented , and the collapsing cost of sequencing in the last twenty years has accelerated the production of DNA sequences : there are for example about 2.8 billion sequences for a total length of ∼ 1013 nucleotides publicly available at the European Nucleotide Archive ( ENA1 ) . Unsurprisingly in such a data-rich field , machine learning-based approaches are increasingly used to analyze DNA sequences , e.g. , in metagenomics to automatically predict the species present in an environment from randomly sequenced DNA fragments [ 26 , 28 , 36 , 38 ] and to detect the presence of viral DNA in human samples [ 36 ] , in functional genomics to predict the presence of protein binding sites or other regulatory elements in DNA sequences of interest [ 16 , 24 , 30 , 35 , 43 , 44 ] , to predict epigenetic modifications [ 25 ] , or to predict the effect of variations in the DNA sequence on a phenotype of interest [ 1 , 46 ] . Due to the sequential nature of DNA and the translation-equivariant nature of the questions addressed , many of these works are based on 1D CNN architectures , although recently transformer-based language models have also shown promising results on various tasks [ 6 , 20 , 42 ] . However , besides translation , DNA has an additional fundamental symmetry that has been largely ignored so far : the so-called reverse complement ( RC ) symmetry , due to the fact that DNA is made of two strands oriented in opposite direction and encoding complementary nucleotides . In other words , a given DNA segment can be sequenced as two RC DNA sequences , depending on which strand is sequenced ; any predictive model for , e.g. , DNA sequence classification should therefore be RC-invariant , which calls for RC-equivariant architectures . While strategies based on data augmentation and prediction averaging has been commonly used to handle the need for RC invariance [ 1 , 32 ] , one translationand RC-equivariant CNN architecture has been proposed and led to promising results [ 4 , 29 , 34 ] . However , it remains unclear whether that architecture is the only one that can encode translationand RC-equivariance , or if alternative models exist to complement the toolbox of users wishing to develop deep learning models for DNA sequences . Using the general theory of equivariant representations , in particular steerable CNNs [ 9 ] , we answer that question by characterizing the set of all linear translation- and RC-equivariant layers . We show in particular that new architectures exist beyond the ones already explored by [ 4 , 29 , 34 ] , which in the language of equivariant CNNs only make use of the regular representation [ 7 ] while more general representations lead to different layers . We further discuss RC-equivariant pointwise nonlinearities adapted to different representations , as well as RC-equivariant embeddings of k-mers as an alternative to one-hot encoding of nucleotides . We test the new architecture on several protein binding prediction problems , and show experimentally that the new models can outperform existing ones , confirming the potential benefit of exploring the full set of RC-equivariant layers when manipulating DNA sequences with deep neural networks . 2 Methods . 2.1 Group action of translation and reverse complementarity on DNA sequence . DNA is a long polymer made of two intertwined strands , forming the well-known double-helical structure . Each strand is a non-symmetric polymer that can be described as an oriented chain of four possible monomers called nucleotides and denoted respectively { A , C , G , T } . The two strands are oriented in opposite directions , and their nucleotides face each other to form hydrogen bonds . They interact at each position in a deterministic way because only two nucleotides pairings can happen : ( A , T ) and ( G , C ) . Thus , given a nucleotide sequence on one strand , we can deduce the so-called RC sequence of its corresponding strand by complementing each nucleotide and reversing the order ( Figure 1 ) . When a double-stranded DNA fragment is sequenced , the two strands are first separated and , typically , only one of them is randomly selected and is decrypted by the machine . This implies that any given DNA fragment can be equivalently described by two RC sequences of nucleotides . Moreover , several genomic learning tasks amount to a sequence annotation that does not depend on the strand . For example , a protein can bind a double-stranded DNA fragment , and both strands of the bound part can get sequenced . This motivates the search for equivariance to this RC-action for the prediction functions . Moreover , the sequencing often results in long sequences where the relevant parts of the sequence do not correlate with their position . The task of prediction over genomic sequences is thus largely translation equivariant , which explains why the community settled on the use of CNNs to train and predict on arbitrary length segments . To formalize mathematically the translation and RC operations on DNA sequences , we first encode the raw genetic sequence as a signal function in F0 = { f : Z −→ { 0 , 1 } 4 } , as the one hot encoding of the nucleotide content for each integer position . Because of the finite length of this polymer , we assume that beyond a compact support this function takes a constant value of zero . The group ( Z , + ) of translations acts naturally on this encoding by Tu ( f ) ( x ) = f ( x− u ) , for a translation u ∈ Z , and the RC operations amounts to the following : RC ( f ) ( x ) = σ ( −1 ) [ f ( −x ) ] , where σ ( −1 ) is the 4× 4 permutation matrix that exchanges complementary bases A/T and C/G ( while we denote by σ ( 1 ) the 4 × 4 identity matrix ) . We notice that RC is a linear operation on F0 that satisfies RC2 = I , and thus that the RC operation is a group representation on F0 for the group Z2 = { 1 , −1 } endowed with multiplication . To jointly consider translations and RC actions , we naturally consider the semi-direct product group G = Z o Z2 . Elements g ∈ G can be written as g = ts with t ∈ Z , s ∈ Z2 and the group G acts on F0 by the action π0 defined by : ∀ts ∈ G , ∀ ( f , x ) ∈ F0 × Z , ( π0 ( ts ) f ) ( x ) = σ ( s ) [ f ( s ( x− t ) ) ] . In other words , π0 is the representation of G on F0 induced by the representation σ of RC on R4 [ 9 ] . 2.2 Features spaces of equivariant layers . Let us now describe the structure of intermediate layers of a neural network equivariant to translations and RC . Following the theory of steerable CNNs [ 9 ] , we consider successive representations of the input DNA sequence in the following way : Definition 1 . Given ρ a representation of Z2 on RD for some D ∈ N∗ , a ρ-feature space is the set of signals F = { f : Z −→ RD } endowed with the G group action π , known as the representation 1As of May , 2021 : https : //www.ebi.ac.uk/ena induced by ρ : ∀ts ∈ G , ∀ ( f , x ) ∈ F × Z , ( π ( ts ) f ) ( x ) = ρ ( s ) [ f ( s ( x− t ) ) ] . ( 1 ) With this definition , we see in particular that the one-hot encoding input layer maps the input DNA sequence to a σ-feature space , and that the dimension ( i.e. , number of channels in the language of deep learning ) and group action of ρ-feature space are fully characterized by the representation ρ. Interestingly , the theory of linear group representations allows us to characterize more precisely all such representations : Theorem 1 . For any representation ρ of Z2 on RD , there exist a , b ∈ N such that a+ b = D and an invertible matrix P ∈ GL ( RD ) such that ∀s ∈ Z2 , ρ ( s ) = P Diag ( Ia , sIb ) P−1 . In other words , combining Definition 1 and Theorem 1 , we see that any ρ-feature space that we will use to build translation- and RC-equivariant layers is fully characterized by a triplet ( P , a , b ) , which we call its type , and which characterizes both its dimension D = a+ b and the action of the group G by ( 1 ) . By slight abuse of language , we also refer to ( P , a , b ) as the type of ρ. Theorem 1 is a standard result of group theory , which explicits the decomposition of any representation ρ in terms of so-called irreducible representation , or irreps . In the case of Z2 , there are exactly two irreps which act on R , namely , ρ1 ( s ) = 1 and ρ−1 ( s ) = s. If ρ has type ( P , a , b ) , then it means that it can be decomposed as a times ρ1 ( s ) and b times ρ−1 ( s ) . In the particular case where P is the identity matrix , i.e. , when we consider a type ( I , a , b ) , then ρ ( s ) is a diagonal matrix for any s ∈ Z2 , and each channel of F is acted upon by a single irrep . In that case , we will call the channels of type `` 1 '' ( resp . `` -1 '' ) if they are acted upon by ρ1 ( resp . ρ−1 ) , and we will say that F is an `` irrep feature space '' . Now , let us introduce another special case . Since Z2 is finite of cardinality 2 , let us consider the regular representation ρreg of Z2 on R2 defined by : ρreg ( 1 ) = ( 1 0 0 1 ) , ρreg ( −1 ) = ( 0 1 1 0 ) . One can easily check that ρreg is of type ( Preg , 1 , 1 ) , where Preg = ( 1 1 1 −1 ) . It corresponds to a ρ-feature space with two channels , where the RC operations flips the two channels ( and of course the sequence coordinates ) . Let us now consider feature spaces of interest . In the input layer , nucleotides are one-hot encoded in a certain order , let us say ( A , T , G , C ) . As stated above , this input space is acted upon by σ , a 2−cycle that swaps bases A/T and C/G . We see that we can rewrite σ = ( ρreg ⊕ ρreg ) : = ( ρ⊕2reg ) , where ⊕ is the bloc-diagonal operation . Because ρreg is of type ( Preg , 1 , 1 ) , we can diagonalize σ with ( P⊕2reg ) and the diagonal would be alternated +1 and -1 values . Thus , there exists a permutation Π such that σ is of type ( P , 2 , 2 ) , with P = Π ( P⊕2reg ) Π −1 . These concepts are illustrated in Supplementary Section A.1 Interestingly , all RC-equivariant layers proposed so far in [ 4 , 29 , 34 ] follow a similar pattern : the channels go by pair , and the RC action amounts to flipping the channel values within a pair and reversing the sequence coordinates . In our formalism , this corresponds to channels of type ( P , a , a ) , where a ∈ N∗ is the number of pairs of channels , and where up to a permutation of channels the matrix P satisfies P = Π ( P⊕areg ) Π −1 . Following [ 34 ] , we will refer to these layers as Reverse Complement Parameter Sharing ( RCPS ) layers below . This highlights the fact that translation- and RC-equivariant layers explored so far are equivariant according to Definition 1 , but that there exists potentially many other equivariant layers , obtained in particular by allowing ρ-feature spaces of types ( P , a , b ) where a 6= b , on the one hand , and where P is not a direct sum of Preg , on the other hand . We investigate such variants below .

Summary: In this paper, the authors delve deeper into the theoretical foundations of Reverse-complement equivariant convolutions for application to DNA sequences. Prior work has been mainly empirical and this helps to establish a general theoretical framework, thereby encompassing existing RC-equivariant-based models. By identifying this general theory, they were able to propose additional classes of RC equivariant models. They then compare the performance of standard models, existing models, and those proposed as a result from the theory on two regulatory genomics prediction tasks -- binary classification and profile prediction of TF ChIP-seq data. They show that for binary classification, their RC-models perform better than standard networks, but were largely similar for profile prediction.