Reverse-Complement Equivariant Networks for DNA Sequences
Authors: Vincent Mallet
Jean-Philippe Vert
Venue: NeurIPS 2021 Poster (2021)
Keywords: Equivariance
neural network
DNA
reverse-complement

Abstract: As DNA sequencing technologies keep improving in scale and cost, there is a growing need to develop machine learning models to analyze DNA sequences, e.g., to decipher regulatory signals from DNA fragments bound by a particular protein of interest.  As a double helix made of two complementary strands, a DNA fragment can be sequenced as two equivalent, so-called reverse complement (RC) sequences of nucleotides. To take into account this inherent symmetry of the data in machine learning models can facilitate learning. In this sense, several authors have recently proposed particular RC-equivariant convolutional neural networks (CNNs). However, it remains unknown whether other RC-equivariant architecture exist, which could potentially increase the set of basic models adapted to DNA sequences for practitioners. Here, we close this gap by characterizing the set of all linear RC-equivariant layers, and show in particular that new architectures exist beyond the ones already explored. We further discuss RC-equivariant pointwise nonlinearities adapted to different architectures, as well as RC-equivariant embeddings of $k$-mers as an alternative to one-hot encoding of nucleotides. We show experimentally that the new architectures can outperform existing ones.

Reverse-Complement Equivariant Networks for DNA Sequences
Vincent Mallet
Structural Bioinformatics Unit, Department of Structural Biology and Chemistry, Institut Pasteur, CNRS UMR3528, C3BI, USR3756
Mines ParisTech, PSL University, Center for Computational Biology
vincent www.mme960mmmml.ccmmmmmmmmmmm
Jean-Philippe Vert
Google Research, Brain team, Paris jpvert@google.c com
Abstract
As DNA sequencing technologies keep improving in scale and cost, there is a growing need to develop machine learning models to analyze DNA sequences, e.g., to decipher regulatory signals from DNA fragments bound by particular protein of interest. As double helix made of two complementary strands, DNA fragment can be sequenced two equivalent, so-called Reverse Complement (RC) sequences of nucleotides. To take into account this inherent symmetry of the data in machine learning models can facilitate learning. In this sense, several authors have recently proposed particular RC-equivariant convolutional neural networks (CNNs). However, remains unknown whether other RC-equivariant architectures exist, which could potentially increase the set of basic models adapted to DNA sequences for practitioners. Here, we close this gap by characterizing the set of linear RC- equivariant layers, and show in particular that new architectures exist beyond the ones already explored. We further discuss RC-equivariant pointwise nonlinearities adapted to different architectures, well as RC-equivariant embeddings of k-mers as an alternative to one-hot encoding of nucleotides. We show experimentally that the new architectures can outperform existing ones.
Introduction
Incorporating prior knowledge about the structure of data in the architecture of neural networks is promising approach to design expressive models with good generalization properties. particular, exploiting natural symmetries in the data can lead models with fewer parameters estimate than agnostic approaches. This especially beneficial when the amount of available data is limited. A famous example of such architecture the convolutional neural network (CNN) for ID sequences or 2D images, which is well adapted to problems which are invariant to translations in the data, while exploiting multiscale and local information in the signals. Motivated by the success of CNNs, there has been a fast-growing body of research in recent years to build the theoretical underpinnings and design architectures and efficient algorithms to systematically exploit symmetries and structures in the data (3). A central idea that has emerged is formalize the symmetries in data by a particular group action (e.g., the group of translations or rotations on images), and to create multilayer neural networks which, by design, "behave well" under the action of the group. This is captured formally by the concept of equivariance, which states that each equivariant layer should be designed to subject to the group
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

action (e.g., we should be able to "translate" or "rotate" the signal in each layer), and that when an input data is transformed by a particular group element, then its representation in an equivariant layer should also be transformed according to the same group element. While it is easy to see that convolutional layers in CNNs are equivariant to translations, Cohen and Welling 7 formalized the concept of group equivariance CNN (G-CNN) for more general groups and showed in particular how to design convolutional layers equivariant not only to translations but also to reflections and to a discrete set of rotations. Following this seminal work, the theoretical foundations of group equivariant neural networks were then expanded, going beyond regular representations [9], for more groups 12 18, 37 40), in less regular spaces 8 10 with more general results on their generality and universality [3 14 22]. The main applications were developed with the groups of rotations in 2D and 3D, mostly to computer vision problems, but also in biology with histopathology [7 23
medicine [1] and quantum chemistry (33]
In this paper, we explore and study the potential benefits of equivariant architectures for an important class of data, namely deoxyribonucleic acid (DNA) sequences. DNA is the major form of genetic material in most organisms, from bacteria to mammals, which encodes in particular all proteins that a cell can produce and which is transmitted from generation to generation. The study of DNA in humans and various organisms has led to tremendous progress in biology and medicine since the 1970s, when the first DNA sequencing technologies were invented, and the collapsing cost of sequencing in the last twenty years has accelerated the production of DNA sequences: there for example about 2.8 billion sequences for a total length of ~ 10 nucleotides publicly available at the European Nucleotide Archive (ENA")) Unsurprisingly in such data-rich field, machine learning-based approaches are increasingly used to analyze DNA sequences, e.g., in metagenomics to automatically predict the species present in an environment from randomly sequenced DNA 36 38) and to detect the presence of viral DNA in human samples [36]. in
fragments [26] 28
functional genomics to predict the presence of protein binding sites or other regulatory elements in
DNA sequences of interest [16, (24] 30 35 43 44). to predict epigenetic modifications (25) or to
predict the effect of variations in the DNA sequence on a phenotype of interest 46).
Due to the sequential nature of DNA and the translation translation-cquivariann nature of the questions addressed, many of these works are based on ID CNN architectures, although recently transformer-bast based language models have also shown promising results on various tasks 66 20 42 However, besides translation, DNA has an additional fundamental symmetry that has been largely ignored so far: the so-called reverse complement (RC) symmetry, due to the fact that DNA is made of two strands oriented in opposite direction and encoding complementary nucleotides. In other words, a given DNA segment can be sequenced as two RC DNA sequences, depending on which strand is sequenced; any predictive model for, e.g., DNA sequence classification should therefore be RC-invariant, which calls for RC-equivariant architectures. While strategies based on data augmentation and prediction averaging has been commonly used handle the need for RC invariance [ 32, one translation- and RC-equivariant CNN architecture has been proposed and led to promising results [4, 22, 334.. However, it remains unclear whether that architecture is the only one that can encode translation- and RC-equivariance, or if alternative models exist to complement the toolbox of users wishing to develop deep learning models for DNA sequences.
Using the general theory of equivariant representations, in particular steerable CNNs 9.. we answer that question by characterizing the set of all linear translation- and RC-equivariant layers. We show in particular that new architectures exist beyond the ones already explored by 4 29 34 which in
the language of equivariant CNNs only make use of the regular representation 7 while more general representations lead to different layers. We further discuss RC-equivariant pointwise nonlinearities
adapted to different representations, as well as RC-equivariant embeddings of k-mers as an alternative to one-hot encoding of nucleotides. We test the new architecture on several protein binding prediction problems, and show experimentally that the new models can outperform existing ones, confirming the potential benefit of exploring the full set of RC-equivariant layers when manipulating DNA sequences with deep neural networks.
2

---

Sequencing order
Sequencing results ACAATG CATTGT
5' ACAATG
Sequencing
Reverse Complement sequences
I RITI
3' TGTTAC 5'
Sequencing order
Figure 1: Illustration of the reverse-complement symmetry. Both DNA strands get sequenced in opposite directions resulting in redundant information.
2 Methods
2.1 Group action of translation and reverse complementarity on DNA sequence
DNA a long polymer made of two intertwined strands, forming the well-known double-helical structure. Each strand is non-symmetric polymer that can be described an oriented chain of four possible monomers called nucleotides and denoted respectively {A, C, G. T}. The two strands are oriented in opposite directions, and their nucleotides face each other to form hydrogen bonds. They interact at each position deterministic way because only two nucleotides pairings can happen: (A,T) and (G, Thus, given nucleotide sequence one strand, we can deduce the so-called RC sequence of its corresponding strand by complementing each nucleotide and reversing the order (Figure D.. When double-stranded DNA fragment is sequenced, the two strands are first separated and, typically, only one of them is randomly selected and is decrypted by the machine. This implies that any given DNA fragment can be equivalently described by two RC sequences of nucleotides. Moreover, several genomic learning tasks amount to a sequence annotation that does not depend the strand. For example, a protein can bind houble-strandee DNA fragment, and both strands of the bound part can get sequenced. This motivates the search for equivariance to this RC-action for the prediction functions. Moreover, the sequencing often results in long sequences where the relevant parts of the sequence do not correlate with their position. The task of prediction over genomic sequences is thus largely translation equivariant, which explains why the community settled on the use of CNNs to train and predict on arbitrary length segments.
To formalize mathematically the translation and RC operations on DNA sequences, first encode the raw genetic sequence as a signal function in Fo = {f E - {0. 1)*), }. as the one hot encoding of the nucleotide content for each integer position. Because of the finite length of this polymer, assume that beyond a compact support this function takes a constant value of zero. The group (Z, +) of translations acts naturally this encoding by (z) f(x u)), for translation E Z, and the RC operations amounts to the following: permutation matrix that exchanges complementary bases A/T and C/G (while we denote by (11) the 4 x 4 identity matrix). We notice that RC is linear operation on Fo that satisfies RC = 7, and thus that the RC operation is group representation Fg for the group Zo {1, -1] endowed with multiplication.
where (-1) 1) is the x 4
To jointly consider translations and RC actions, we naturally consider the semi-direct product group G = Σz. Elements g E G can be written as g = ts with E Z, E Eo and the group G acts on Fo by the action To defined by:
Vts G, e
(To(ts)f) = [ss)[f(s(r-=t
In other words, To is the representation of G F, induced by the representation of RC on R 99.
2.2 Features spaces of equivariant layers
Let us now describe the structure of intermediate layers of a neural network equivariant to translations and RC. Following the theory of steerable CNNs (9, we consider successive representations of the input DNA sequence in the following way:
Definition 1. Given p a representation of Le on D for some D N p-feature space is the set of signals F = {f Z + }) endowed with the G group action n, known as the representation
As of May. 2021:
https: //www. ebi. uk/ena

---

induced by p:
Yts E G, ((f.)) (z(ts) p(s]ff(s((>>> t])]
(1)
With this definition, we see in particular that the one-hot encoding input layer maps the input DNA sequence to a o-feature space, and that the dimension (i.e., number of channels in the language of deep learning) and group action of p-feature space are fully characterized by the representation p. Interestingly, the theory of linear group representations allows us to characterize more precisely all such representations:
Theorem 1. For any representation p of 22 on RDD,, there exist a, b N such that a + b = D and an invertible matrix P E GL(RP) such that
vs E 22. p(s) = P Diag( la, P-1
In other words, combining Definition D and Theorem 1 we see that any p-feature space that we will to build translation- and RC-equivariant layers is fully characterized by triplet (P, a, b). which we call its type, and which characterizes both its dimension D = a + and the action of the group G by (1. By slight abuse of language, also refer to (P, a, b) the type of p.
Theorem I is a standard result of group theory, which explicits the decomposition of any representation irreps. In the case of Σz, there are exactly two
in terms of so-called irreducible representation, irreps which act on I., namely, P1 (s) = 1 and p-1(s) = s. If has type (P, a, b), then it means that be decomposed identity matrix, i.e., when we consider a type (I, a, b). then p(s) is a diagonal matrix for any s E Zz, and each channel of F is acted upon by single irrep. In that case, will call the channels of type "1" (resp. "-1") if they are acted upon by (resp. p-1), and we will say that F is an "irrep feature space".
times #1 (s) and b times (s)(s). In the particular case where P is the
Now, let us introduce another special case. Since Ze is finite cardinality 2, let us consider the regular representation Preg of 2g on I2² defined by:
Preg = 1 11) 11))
Preg(-1) -
One can easily check that Preg is of type (Pregs 1, where Preg ( -1 ) It corresponds
1
p-feature space with two channels, where the RC operations flips the two channels (and of course the sequence coordinates).
Let us now consider feature spaces of interest. In the input layer, nucleotides are one-hot encoded in certain order, let say (A, T, C). As stated above, this input space is acted upon by d, a 2 -2cyclle that swaps bases A/T and C/G. We see that we can rewrite o = (Preg @ Prrg) == @2 where @ is the bloc-diagonal operation. Because Prey is of type (Preg- 1, 1), we can diagonalize a with (PS9g) and the diagonal would be alternated +1 and -1 values. Thus, there exists a permutation such that is of type (P, 2,2,2). 2), with P = ([PP2)]T= These concepts are illustrated in Supplementary Section A.
follow a similar pattern: the channels go by pair, and the RC action amounts flipping the channel values within a pair and reversing the sequence coordinates. In our formalism, this corresponds to channels of type (P, a, a). where a E N is the number of pairs of channels, and where up to a permutation of channels the matrix P satisfies P = II( [P III- Following [34], we will refer to these layers as Reverse Complement Parameter Sharing (RCPS) layers below.
Interestingly, all RC-equivariant layers proposed so far in 29 34
This highlights the fact that translation- and RC-equivariant layers explored so far are equivariant according to Definition [ but that there exists potentially many other equivariant layers, obtained in particular by allowing p-feature spaces of types (P, a, b) where a of b, on the one hand, and where P not a direct sum of Preg. on the other hand. We investigate such variants below.
2.3 Equivariant linear layers
While Definition D
ccaracterizzs p-feature space in terms of structure and group action, an equivariant multilayer neural network is built by stacking p-feature spaces top of each other and connecting

---

them with equivariant layers. Cohen et al. [[] Theorem 2] gives us general result about such equivariant mappings. Here, we apply this result our specific data and group, and characterize the class of equivariant linear layers, i.e., the linear functions o Fn Fa+1 that satisfy Tn+10 doT,. where nn and #n+1 are respectively the group action on Fn and Fn+1-
Theorem 2. Given two representations Pn and pu+1 of Z2. of respective types (Pns On bn) and (Pn+1, an+1, Dn+1) with an + b, = Dn and an+1 + bn+1 Dn+1, and respective per and Pn+1- feature spaces Fn and FutI, linear map : Fn + Fn+1 1 is equivariant if and only if it can be written as a convolution:
Vfxx) (()()= = Σ
xly - (()))))(w)) x))
(2)
PEZ
where the kernel k: 2 Da+txxDs xn satisfies:
V B, x(-x) = Pn+1l (ntt(-1)x(e)pn(-1)
(3)
or equivalently:
3(x) (xx) 6(z)
((x))
Yx Z k(x) Pat1
P.1
(4)
where O: Z + 1xxin and y: Z Rba+1 Xan are odd functions.
+ Rbu 1xbn are even, while B Z + Ron 1xxb 11xbe and
As stated in Cohen et al. [.]. "Convolution is all you need" to define linear layers which are equivariant our group. In addition, Theorem characterizes all the convolution kernels that ensure equivariance through the two equivalent constraints (3) and
To illustrate this result, let consider two RCPS feature spaces F, and Fu+1 of respective types (II, (Prgg II,"' ag. an) and (In++1 (Pigg )I Ox+1, an+1). Then, the channels in Fn and Fa+1 go by pair, and if we consider slice R : Z + R2xx the convolution kernel y describing how pair of channels in F, maps to pair of channels in FR+1, (3) gives the constraint:
61 2) R21 (-2) R22
Rie(_2)
( ) (1))
(12((r) R12(1)
(2)(2))
R21 (3r)
R(-x):
R(x)
R11(z)
We recover exactly the constraints of the RCPS filters first proposed by 34 proving as a consequence of Theorem 2 that RCPS convolution filters describe exactly all equivariant linear mappings between
RCPS feature spaces.
Moreover, if we now consider any two feature spaces F, and Fa+1 respective types (Pa, ans bn) and (Pa+1, An+1,, bu+1), then Equation (4) tells us that up multiplications by matrices P and P-1, the kernel expressed in terms of even and odd functions, which can be trivially implemented with parameter sharing. For example, to represent the even function a, one just need to parameterize the values of (xr) for x 2 O, and complete the negative values by parameter sharing (--x) = a(x). Hence, the parameter sharing idea used in RCPS [34] extends to any equivariant linear map.
parameterize equivariant convolution kernels, one may also directly write the constraints (3) for specific representations, and potentially save the need of multiplication by Px+1 and P.- in (4.. This is for example the case in RCPS layers [34], and more generally for channels acted upon by the regular representation; for the sake of completeness, we derive in Appendix A.4 the constraints to go from and to the regular representation or the irreps, and use them in our implementation.
Instead of using
2.4 Equivariant nonlinear layers
Besides equivariant linear layers, a crucial component needed for multilayer neural networks the possibility to have equivariant nonlinear layers, such as nonlinear pointwise activation functions or batch normalization [19]. In this section, we discuss particular nonlinearities that are adapted to various equivariant layers.
Pointwise activations. Let us begin with pointwise transformations, that encompass most activation functions used in deep learning. Pointwise transformations formally defined follows: given
5

---

function 0: R + R and vector space V = RA for some index set A, the pointwise extension of f to V is the mapping By V - V defined by (v) = 0 (f(a),, for any (f. ) E V x A. For D-dimensional representation p of Z2 and a p-feature space F with G-group action we say that a pointwise extension 0p : F -> F is equivariant it commutes with n, i.e., n0p 0pn. By definition of the group action (I). this is equivalent to saying that the pointwise extension 8g of f to RDd commutes with p. The following theorem gives an exhaustive characterization of a large class of equivariant pointwise extensions for any p-feature space:
Theorem 3. Let be a representation on and B I - TR. be a continuous function with left and right derivatives at O. Let F be a p-layer and OF F F be the point-wise extension of 0 on this layer. Then Op is equivariant if and only if at least one of the following cases holds:
1. 0 is a linear function.
2. 0 is an affine function, and p(-1)1=
3. 9 not an affine function, and there exists a permutation matrix II, integers a, and scalars (>1,.... Xo) E (R,,), )", such that decomposes as
0
A
0
-1)
@b
+
1
0
@ (1) Bc @ (-1)8d
(5)
i=1
In that case,
Either b = d= 0 and Vi, X; 1 and is any function,
Or = 0 and Bi, A * 1 and 0 is a leaky ReLu function! F
Or d > 0 and Vi, A and is an odd function,
The first case in Theorem is of little interest, since pointwise linear functions are always equivariant to linear group actions. The second case essentially says that adding a constant to a pointwise linear function is only equivariant for representations P such that the sum of all rows of p(-1) is equal 1. This holds for example for the regular representation and the RCPS layers, but not for an irrep feature space of type (I, a, b) with b 0, since in that case, some rows have single entry. The most interesting case is the third one, since it describes what pointwise nonlincarities one can apply. The condition (3) on the decomposition of p essentially excludes all representations that have more than one nonzero value in at least one row of p(-1). Among valid o's that decompose as (5), we see that the regular representation (corresponding to the first block in 5 with X; = 1)), used in RCPS, stands out as the only that allows any nonlinearity, besides of course invariant channels of type "+1" (third block in (5)). Replacing a "1" in the regular representation by a scalar X; # 1 (in the first block of (5). with b d 0) creates valid representation p. however only leaky ReLu pointwise nonlinearities can be applied that case. Another case of practical interest is the irrep feature space type (I, e, d) for some > O and d > 0. By Theorem 3 only odd nonlinearities are allowed in that case, such as the hyperbolic tangent function. Finally, one should keep in mind that other representations, which do not satisfy the conditions listed in Theorem 3
do not allow any equivariant nonlinear pointwise 11/2) which is a valid representation of Zg but does neither meet the condition accept affine activations (case 2), nor to accept nonlinear activations (case 3) because p(- does not decompose according to (5)
transform; this is for example the case of p(-1)
Other activation functions Besides pointwise transformations from a p-feature space itself characterized in Theorem 3 the set of nonlinear equivariant layer is tremendous and the design choices are endless. A first extension is to keep pointwise activation, but to allow different nonlinearities on different channels, e.g., by using any function on the "+1" channels and an odd function on the "-1" channels of an irrep feature space. Another relaxation is use different input and output representations. While odd functions will not affect the field type, even functions will turn field of type 1-1" into "+1" type. is well known that any function decomposes into a sum of an odd and even function. Therefore, given p. representation decomposed as in (5), any point-wise non- mon-linearity can be used in a p-eeauure space by first decomposing it into its odd and even components and applying each component separately for the second and fourth blocks.
A leaky ReLu function is 0(x) asign(s) for some (a+, a-) E R2.

---

Other possibilities exist and include creating new representations by tensorization, which amounts to taking pointwise products between different channels [3] 21 37 or using non point-wist activation
layers, that act several coupled dimensions, such as the ones used in [37] For instance, we could apply the max function to paired channels. These possibilities are discussed in [39
Batch normalization An equivariant batch normalization introduced by [4 It considers feature map and its reverse complement as two instances, which is easy to do because the reverse complement feature map is already computed when using regular representation. We propose another batch normalization for irrep feature spaces that also gives the result we would have had if the hatch contained all the reverse complement of its sequences. For the "+1" dimensions, it amounts scaling as we would have the same values twice. For the "-1" dimensions, we enforce a zero mean and compute a variance estimate based on this constraint.
K-mers. Instead of the standard one-hot encoding of individual nucleotides as input layer, we propose to one-hot encode k-mers for 2 1, i.e. overlapping blocks of consecutive nucleotides. This technique is known to improve performance in several tasks [27 28 In order to implement it into an equivariant network, we need to know how the group acts on the k-mers space, made of 4k elements. The simplest idea is to pair the index of the channels of two RC k-mers. Because some k-mers are their own reverse complement, the canonical way to do so is to have a representation that is a blend of "+1" irrep and regular representation. An alternative is to make the regular representation act on the k-mers instead by redundantly encoding these k-mers into paired dimensions. This the strategy we follow in implementation, to be more coherent with the usual input group action.
3 Experiments
We assess the performance of various equivariant architectures set of three binary prediction and four sequence prediction problems used by Zhou et al. [45] to assess the performance of RCPS networks. The binary classification problems aim to predict if DNA sequence binds to three transcription factors (TFs), based on genome-wide binarized TF-ChIP-seq data for Max, Ctef and Spil in the GM12878 lymphoblastoid cell-line (34]. The sequence prediction problems aim to predict TF binding at the base-pair resolution, using genome-wide ChIP-nexus profiles of four TFs- Oct4, Sox2, Nanog and KIf4 in mouse embryonic stem cells. For more detailed explanation of the experimental setup. please refer to Zhou et al. [45). We report "significant" differences in performance below when the P-value of Wilcoxon signed rank test is smaller than 0.05.
Models. We build over the work of Zhou et al. 45 for both the binary and the sequence prediction problems. They benchmarked an equivariant RCPS architecture and a corresponding non-equivariant model, with the same number of filters and trained with data augmentation, which we respectively refer to as "RCPS" and "Standard" models below. The data augmentation scheme for the "Standard" model consists in adding to the training set the reverse complement sequences of all training sequences, which is a natural procedure to let the model "learn" the equivariance without encoding it explicitly in the architecture of the network. We checked empirically that data augmentation significantly improves the performance of non-equivariae models (Appendix A...I). In addition, we extend the RCPS architecture with one-hot encoding of k-mers input layers, which we refer to "Regular" below. Finally, we add to the comparison a new equivariant network where each RCPS layer is replaced by an (I, a, b) layer with the same number of filters, which we call "Irrep" below. We also use k-mers and vary the ratio a/ (a + b) in this model. We combine the regular and "+1" dimensions with ReLu activations and the "-1" dimensions with a tanh activation.
Influence of hyperparameters in equivariant models To assess the impact of different hyperpa- rameters in the family of equivariant models we propose (k-mer length for Irrep and Regular, a/ (a+b) ratio for Irrep), we train equivariant models with different combinations of hyperparameters on the training set and assess their performance on the validation set, repeating the process ten times with different random seeds. We assess the performance of each run in terms of Area under the Receiver Operator Characteristic (AuROC), and show in Figure the average performance reached by runs with a given ratio a/(a + b) E (0, 1/4, 1/2, 3/4 /, }} (Teft) and with given & E 21, 3, 4] (right). We see clear asymmetry in the performance as a function of a/(a + b). with poor performance when 0 and optimal performance for 0.75, significantly better than all other ratios tested.
7

---

This confirms that exploring different irreps may be valuable. As for the k-mer length, setting k = 3 gives the best performance and significantly outperforms all other values tested. This confirms that going beyond one-hot encoding of nucleotides in equivariant architectures can be beneficial.
0.9700
0.968
0.9600
Autoc
0.9625
0.966
0.9600
0.905
0.9575
0.963
0.9525
0.942
Longth used
Figure 2: Average AuROC performance across four TFs and 10 random seeds for the Irrep model as a function of a/ a/ + b) (left, also averaged over values) and for the Irrep and Regular models as function of (right, also averaged over a/(a + b) values for Irrep).
Binary task. We then compare the test set performance of three different models for the binary classification task: 1) Standard, 2) RCPS, and 3) the best Irrep or Regular equivariant model, where hyperparameters are selected based on the AuROC on the validation set, which we denote as "Best Equivariant". Figure 3 (left) shows the performance of each model on each TTF task and overall. As already observed by M4 the equivariant RCPS architecture has a strong lead over the Standard, non-equivariant model in spite of data augmentation. Interestingly, we see that Best Equivariant is significantly better than RCPS on all tasks, and that the performance gain from RCPS to Best equivariant is of the same order the performance gain from Standard to RCPS. This demonstrates that the family of equivariant architectures we introduce in this paper can lead to significant improvement existing architectures.
Full Dataset
Reduced Dataset
CTCF
MAX
SPI1
Overall
CTCF
MAX
SP11
Overall
0.901
00033
69773
0.961
0.906
0,980
1940
0.20
0.99
0007
0.0322
0.9706
0.954
0,899
0.976
AUROI
0004
0.0295
0.9888
0.947
0.801
0372
580
0,0000
0.969
0.040
088
0.32
Figure 3: AuROC performance of the three different models (Standard, RCPS and Best equivariant after hyperparameter selection on the validation set) on the three binary classification problems CTCF, MAX and SPI1, as well as their average. Error bars correspond to an estimate of the standard error on 10 repeats with different random seeds. The left plot the performance the right plot shows the performance where models are trained on a subset of 1,000 sequences only (notice the differences of AuROC values on the vertical axis in both plots).
the full datasets, while
Reduced models. Since equivariant architectures are meant to be particularly beneficial in the low-data regime [15]. we further assess the performance of the three models on the same binary

---

classification problems but with only 1,000 sequences used to train the models, and show the results on (right), Overall, the performances are worse than in the full-data regime (Figure [ left), which confirms that this is regime where more data helps. We also see that the relative order of the three different methods remains overall the same, with Best Equivariant outperforming RCPS, which itself outperforms Standard. Interestingly, the gaps between the best and worse models widens in the low-data regime, showing that the prior is more useful this setting. More precisely, there is a large gap of about 1% between Best Equivariant and Standard in the low data regime, compared to gap of about 0.3% on the full dataset. We also investigated whether equivariant models converge faster to their solutions, but found not noticeable difference (Appendix A.662
A.6622
On post-hoc models. Zhou et al. 45 introduced the so-called post-hoc model, another equivariant method obtained by averaging the predictions of a Standard model over a sequence and its reverse- complement, and showed that it is competitive with and often outperforms RCPS. The post-hoc model only requires training and storing one network, but aggregates two predictions for each sequence at inference time. Because of that, the good performance of post-hoc may be due in part to the aggregation step common to all ensemble models [2]. To decipher the respective contributions of the network architecture, on the one hand, and of the aggregation of predictions, on the other hand, we add to the comparison an ensemble of two Standard models trained with different random seeds (Ensemble Standard) and an ensemble of two equivariant Irrep models (Ensemble Irrep) and present the results in Figure We see that Ensemble frrep strongly outperforms Best Equivariant, and both post-hoc and Ensemble Standard widely outperform the Standard architecture. This confirms that ensembling equivariant or non-equivariant models through post-hoc of ensemble aggregation is always useful (at the cost of increased computational time). We see that Ensemble Standard is not significantly different from post-hoe Standard on CTCF and SPI1, but that post-hoe Standard is better MAX, suggesting that most of the benefits of post-hoc Standard indeed comes from the ensembling effect. Regarding the impact of the architecture for a given budget of predictions, we saw earlier than Best equivariant significantly outperforms Standard when a single prediction per test sequence is allowed, and see now that Ensemble Irrep strongly outperforms both post-hoc and Ensemble Standard when two predictions are allowed, thus confirming the benefit of equivariant architectures in all settings. We also see that a single Best equivariant models outperforms post-hoc and Ensemble Standard, indicating that enforcing equivariance throughout the network is not only faster but also more more accurate than averaging non-equivariant model over group transformed inputs.
CTCF
MAX
SPI1
Overall
0.9912
0,977
09941
0.9886
09933
AURDC
0.3860
099926
0.770
0.9834
0,290
0.999
0,9896
Best Eoulvariant
Ensemble Standard
Hoc
Ensemble Imep
Figure 4: AuROC performance on the three binary classification problems, for the Best Equivariant model, the post-hoc Standard model, and an ensemble of two Standard or Irrep models. Error bars correspond to an estimate of the standard error on 10 repeats with different random seeds.
Profile task. We now compare the performance of different models on the profile prediction tasks. To limit the carbon footprint of this study, and based on the influence of hyperparameters on the binary task (Figure 2), we only test two equivariant models in addition to Standard and RCPS: Regular model with 3, and an Trrep model with the performance of post-hoc Standard (the best model in (45)), and ensemble of two models of
and a/(a + b) = 75%. We also assess

---

the best performing equivariant model. Figure shows the performance of all models in terms of Spearman correlation between the target profile and the predicted ones, on the full dataset (left) or reduced experiment with only 1,000 training sequences (right).
Full Dataset
Reduced Dataset
SOX2
OCT4
KLF4
NANOG
Overall
SOX2
OCT4
KLF4
NANOG
Overall
0.303
0.22
0.365
0419
1
5620
0297
0620
02
Spcarman Corretation
0,30
0.33
001
0.35
0.255
EdC
0222
0
cou
0
0.89
0338
0.324
1227
0.33
0,20
0,00
0
0.258
0,281
12E'0
6.20
0.2
Figure 5: Spearman Correlation between true and predicted profiles by different methods for four data sets.
First of all, we observe as before that the low-data regime, the gap between standard and equivariant networks grows in favor of equivariant ones. We also observe, surprisingly. that Irrep, which outperformed RCPS on the binary task, now underperforms it. A possible explanation could be that since this task aims annotate individual nucleotide, encoding the nucleotide level information using k-mers makes the signal blurry and decreases performance. However, in the reduced setting, Irrep performs better again. These results indicate that for the best model should be chosen empirically on a validation set. Finally, despite good performance of post-hoc Standard, the ensemble equivariant model once again performs better for the same computational cost inference.
Experiment settings and computational cost. All experiments were run on a single GPU (either GTX1080 RTX6000), with 20 CPU cores. The binary classification experiments to train. To limit our carbon footprint, we chose to run more experiments on this task, c.g., for hyperparameter tuning and reduce the number of replicates for the profile task. The total runtimes of each of those tasks were approximately of a week.
shorter
4 Conclusion
In this paper, we addressed the problem of including the RC symmetry prior in neural networks. Leveraging the framework of equivariant networks, in particular steerable CNNs, we deepened existing methods by unraveling the whole space linear layers and pointwise nonlinearities that are translation and RC-equivariant. We also investigated the links between the linear representations and non-linear layers of neural networks, exposing the special role the regular representation in equivariant networks. Finally, implemented all these equivariant layers available in Keras (5) and Pytorch [1.. We then explored empirically how this larger equivariant functional space behaves in terms of learnin. Our best results improve the state the art performance of equivariant networks, showing that new equivariant architectures can have practical benefits. In the future plan to test more deeply the newly proposed architectures prediction tasks involving double-stranded DNA, such as DNA DNA-protein binding prediction, epigenetics or metagenomics. On the theoretical side, characterized equivariant pointwise nonlinearities that preserve the layer type, but more general nonlinear transforms (e.g., not pointwise, or changing the layer type) remain to be fully characterized.
lincar and nonlinear equivariant layers and make
code available at https: htttps://githuu.com//// http:///hhu...co/////eeetttt//a/
10

---

Acknowledgments and Disclosure of Funding
V.M. recipient of doctoral fellowship from the INCEPTION project [PLA/ANR-I6-CONVEE 16-CONV-0005] and benefits from support from the CRI through Ecole Doctorale FIRE Programme Bettencourt. We thank Marie Dechelle, Jacques Boitreaud, Carlos G. Oliver and Guillaume Bouvier for reviewing the manuscript. We thank Avanti Shrikumar, Hannah Zhou and Anshul Kundaje for helpful discussions and sharing their code.
Conflict of Interest: None declared.
References
[1] B. Alipanahi, A. Delong, M. T. Weirauch, and B. J. Frey. Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning. Nature biotechnology, 33(8):831-838 2015.
[2] B. Anderson, T. S. Hy, and R. Kondor. Cormorant: Covariant molecular neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[3] M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković. Geometric deep learning: Grids, groups, graphs geodesics, and gauges. Technical Report 2104. arXiv, 2021.
[4] R. C. Brown and G. Lunter. An equivariant Bayesian convolutional network predicts recom- bination hotspots and accurately resolves binding motifs. Bioinformatics, 35(13):217722184, 2019.
[5] Chollet. Keras.
2015.
[6] J. Clauwaert and W. Waegeman. Novel transformer networks for improved sequence labeling in genomics. JEEE/ACM Trans. Comput. Biol. Bioinform., 2021.
[7] T. Cohen and M. Welling. Group equivariant convolutional networks. In M. F. Balcan and K. Q- Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2990-2999, New York, New York, USA, 20-22 Jun 2016. PMLR.
[8] Cohen, M. Weiler, Kicanaoglu, and M. Welling. Gauge equivariant convolutional networks and the icosahedral CNN. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2019.
[9] T. S. Cohen and M. Welling. Steerable CNNs. In International Conference Learning Representations (ICLR), 2017.
[10] S. Cohen, M. Geiger, Köhler, and M. Welling. Spherical CNNs. Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.
[11] S. Cohen, M. Geiger, and M. Weiler. General Theory of Equivariant CNNs on Homo- geneous Spaces. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume Curran Associates, Inc., 2019.
[12] T. G. Dietterich. Ensemble methods in machine learning. In Proceedings of the First Interna- lional Workshop Multiple Classifier Systems, MCS '00, page 1-15, Berlin, Heidelberg, 2000. Springer-Verlas Verlag.
[13] N. Dym and H. Maron. On the Universality of Rotation Equivariant Point Cloud Networks. arXiv preprint arXiv:2010.02449, 2020.
[14] C. Esteves. Theoretical aspects of group equivariant neural networks. arXiv preprint axXiv:2004.05754. 2020.

---

[15] F. Fuchs. D. Worrall, V. Fischer, and M. Welling. Equivariant Attention Networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1970-1981, Curran Associates, Inc., 2020.
3D Roto-Translalee Translation
[16] M. Ghandi, D. Lee, M. Nohammad-Noorf and M. A. Beer. Enhanced regulatory sequence prediction using gapped k-mer features. PLoS Comput Biol, 10(7):e100371) 2014.
[17] S. Graham, D. Epstein, and N. Rajpoot. Dense steerable filter cnns for exploiting rotational symmetry in histology images. IEEE Transactions Medical Imaging, 39(12):4124 4136, 2020.
[18) E. Hoogeboom, W. T. Peters, T. Cohen, and M. Welling. Hexaconv. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[19] S. Ioffe and Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on muchine learning, pages 448-456. PMLR, 2015.
[20] Y. Ji. Z. Zhou, H. Liu, and V. Davuluri. DNABERT: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome. Bioinformatics, Feb. 2021.
[21] R. Kondor. N-hody networks: covariant hierarchical neural network architecture for learning atomic potentials. arXiv preprint arXiv:/803.00 2018.
[22] R. Kondor and S. Trivedi. On the generalization of equivariance and convolution in neural networks the action of compact groups. In International Conference Machine Learning, pages 2747-2755. PMLR, 2018.
[23] M. W. Lafarge, E. Bekkers, Pluim, R. Duits, and M. Veta. tato-translation equivariant convolutional networks: Application to histopathology image analysis. Medical Image Analysis, 68:101849, 2021.
[24] D. Lee, D. U. Gorkin, M. Baker, Strober, A. Asoni, A. S. McCallion, and M. A. Beer. A method to predict the impact of regulatory variants from DNA sequence. Nature genetics, 47 (8):955, 2015.
[25] J. Levy, J. Titus, C. L. Petersen, Y. Chen, L. A. Salas, and B. C. Christensen. MethyINet: an automated and modular deep learning approach for DNA methylation analysis. BMC bioinformatics, 21:108, 2020.
[26] Q. Liang, P. W. Bible, Y. Liu, B. Zou, and L. Wei. DeepMicrobes: taxonomic classification for metagenomics with deep learning. NAR genom. bioinform., 2:1qaa009, Mar. 2020.
[27] W. Liang. Segmenting DNA sequence into words based on statistical language model. Nature Precedings, pages 1-1, 2012.
[28] R. Menegaux and J.-P. Vert. Continuous embeddings of DNA sequencing reads and application to metagenomics. Journal of Computational Biology, 96(6):509-511 2019.
[29] K. Onimaru, O. Nishimura, and S. Kuraku. Predicting gene regulatory regions with convolu- tional neural network for processing double-strand genome sequence information. PloS one, 15 (7)xe0235748, 2020.
[30] M. Oubounyt, Z. Louadi, H. Tayara, and K. T. Chong. DecPromoter: Robust promoter predictor using deep learning. Frontiers in genetics, 10:286, 2019.
[31] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. DeVito, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, perforpeffrmance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Auu, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
12

---

[32] D. Quang and X. Xie. Factornet: deep learning framework for predicting cell type specific transcription factor binding from ressouti sequential data. Methods, 166:40-47, 2019.
[33] K. T. Schüit, O. T. Unke, and M. Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. arXiv preprint arXiv:2102.03/50, 2021.
[34] A. Shrikumar, P. Greenside, and A. Kundaje. Reverse-complement parameter sharing improves deep learning models for genomies. bioRxiv, page 103663, 2017.
[35] G. D. Stormo. DNA binding sites: representation and discovery. Bioinformatics, 16(1):16-23, 2000.
[36] A. Tampuu, Z. Bzhalava, J. Dillner, and R. Vicente. ViraMiner: Deep learning on raw DNA sequences for identifying viral genomes in human samples. PloS one, 14:e0222271, 2019.
[37] N. Thomas, Smidt, S. Kearnes, Yang, L. Li, K. Kohlhoff, and Riley. Tensor field networks: Rotation-an and translation- equivariaat neural networks for 3d point clouds. arXiv preprint arXiv: 2018.
[38] K. Vervier, P. Mahé, M. Tournoud, J.-B. Veyrieras, and J.-P. Vert. Large-scale machine learning for metagenomics sequence classification. Bioinformatics, 32:1023-1032. 2016.
[39] M. Weiler and G. Cesa. General e e22)-cquivariant steerable cnns. arXiv:1911.08251, 2019.
arXiv preprint
[40] M. Weiler, M. Geiger, M. Welling, W. Boomsma, and T. S. Cohen. 3d steerable cnns: Learn- ing rotationally equivariant features in volumetric data. In Advances Neural Information Processing Systems, pages 10381-10392, 2018.
[41] M. Winkels and T. S. Cohen. Pulmonary nodule detection in ct scans with equivariant cnns. Medical image analysis, 55:15-26, 2019.
[42] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontañón, P. Pham, A. Ravula, Q Wang, L. Yang, and A. Ahmed. Big Bird: Transformers for Longer Sequences. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neu- ral Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-, 2020, virtual, 2020.
[43] H. Zeng, M. D. Edwards, G. Liu, and D. K. Gifford. Convolutional neural network architectures for predicting DNA-protein binding. Bioinformatiic, 32(12):121-i127, 2016.
[44] H. Zhang, C.-L. Hung, M. Liu, X. Hu, and Y.-Y. Lin. NCNet: Deep learning network models for predicting function of non-coding DNA. Frontiers in genetics, 10:432, 2019.
[45] H. Zhou, A. Shrikumar, and A. Kundaje. Towards better understanding of reverse-complement equivariance for deep learning models in regulatory genomics. bioRxiv, 2020. 1.00.....4,,,803, 2020.
[46] Zhou and O. G. Troyanskaya. Predicting effects of noncoding variants with deep learning- based sequence model. Nature methods, 12(10):93 1-944, 2015.
13


Reviews:
[{"review_id": "yAl-rT39ZSJ", "reviewer": "Reviewer_ny7k", "summary": "The paper provides a mathematical formalization of published equivariant layers (translation and reverse-complement) for DNA sequences and explores other architecture variants of these layers (linear, nonlinear and k-mers).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written and well structured.\n\nThe main contribution of the work is the theoretical characterization of translation and reverse complement (RC) equivarariant CNN layers for DNA sequences. It is based mainly on the work of Cohen and Welling on Group equivariant CNNs applied in vision tasks [1, 2, 3]. Regarding the RC equivariance for DNA sequences, it has been used and empirically studied previously in other papers [4,  5, 6]. Moreover, the authors investigate, define theoretically and implement new architecture variants of, linear and nonlinear, RC equivariant layers.\n\n### Comments:\n\n1) In this comprehensive work, the theoretical results are thoroughly presented and explained. However, the used math is too advanced and not always easy to follow. The authors could make use of graphic figures to support the different concepts as it was done for image processing in [1, 2].  \n\n2) The paper does not clearly discuss the importance of using the equivariant nonlinear layers for DNA sequence processing. Characterizing only the equivariant linear layers would be sufficient for this work if their empirical limitations were thoroughly analyzed and discussed; and nonlinear layers could be addressed in a separate work. \n\n3) Although the authors implemented the studied layers, the empirical study is quite minimal compared to the previous work of [4]. Only two equivariant networks are empirically evaluated from the set of new investigated networks. Moreover, the evaluation of the models is based simply on their performance on two supervised tasks in regulatory genomics. Further investigations are needed such as the assessment of the quality of the internal representation of the equivariant networks [7]. \n\n4) The hyperparameters search should be done separately for each task.  The hyperparameters of models used in the profile task were set based on a search performed with binary tasks. A $k > 2$ or a ratio $a/(a+b) \\neq 0.75$ could lead to better performance on the profile task. \n\n5) The related work is correctly cited throughout the paper, but there is no dedicated section to clearly position the current work from other related machine learning contributions.\n\n### Minor comments:\n\n1) Set the same y-axis limit for both panels in figures 3 and 4.\n3) Binary task, lines 313 and 314: provide AuROC with standard error. \n2) A.1, lines 521, 524 and 525: the second $\\rho_1(s)$ should be $\\rho_{-1}(s)$\n\n### References:\n* [1] T.S. Cohen and M. Welling. Group equivariant convolutional networks. 2016.\n* [2] T. S. Cohen and M. Welling. Steerable CNNs. 2017.\n* [3] T. S. Cohen, M. Geiger, and M. Weiler. A General Theory of Equivariant CNNs on Homo- geneous Spaces. 2019.\n* [4] H. Zhou, A. Shrikumar, and A. Kundaje. Towards a better understanding of reverse-complement equivariance for deep learning models in regulatory genomics. 2020.\n* [5] R. C. Brown and G. Lunter. An equivariant Bayesian convolutional network predicts recom- bination hotspots and accurately resolves binding motifs. 2019.\n* [6] Bartoszewicz, Jakub M., et al. DeePaC: predicting pathogenic potential of novel DNA with reverse-complement neural networks. 2020.\n* [7] A. Shrikumar, P. Greenside, and A. Kundaje. Reverse-complement parameter sharing improves deep learning models for genomics. 2017.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable.", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}, {"review_id": "v9XFNhm0t-", "reviewer": "Reviewer_YzSz", "summary": "The authors note that neural network models operating on DNA sequences desire shift equivariance and reverse complement (RC) equivariance. Existing proposed techniques have been somewhat ad hoc. The authors derive RC equivariant models from first principles, and expose a class of techniques that haven’t yet been studied. They benchmark several equivariant alternatives on ChIP-seq peak classification and nucleotide-resolution coverage prediction and demonstrate interesting results.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Introductory note\n---------\n\nI’m unable to evaluate the correctness of the Methods section due to unfamiliarity with group theory. \n\nMajor comments\n---------\n\nIn describing their experiments, the authors note, “For a more detailed explanation of the experimental setup, please refer to Zhou et al. [44].” For some of the details (e.g. the ChIP-seq data processing pipeline), I think that’s fine. But there are several important choices that I believe need to be described here. First, it is standard when performing deep learning on these data to include both the sequence and its reverse complement in the training set, akin to data augmentation. If this has not been done, then the “Standard” model would be at an unfair disadvantage.\n\nSimilarly, it is typical when making predictions to compute a forward pass for both the forward and reverse complement sequences and take the average as an ensemble. An equivariant model wouldn’t benefit from this option. However, you might argue that it would be fair to benchmark an ensemble of two distinctly trained equivariant models to match the prediction computation to that mode. The authors should offer more details about how predictions are made on their test sets, and I’d encourage them to explore these different modes.\n\nMinor comments\n---------\nThe authors might have more success reaching potential users of this technique in computational biology if they could make their Methods section more accessible to non-experts in group theory. If the authors could convey some intuition for what these operations aim to accomplish and how they do it, I believe a larger audience would be available. I’ll note that the code availability is great and a dedicated reader can likely reach an understanding.\n\nIn line 107, the authors write that “the RC operations amounts to the following : RC(f)(x) = σ(−1)[f(−x)], where σ(−1) is the 4×4 permutation matrix that exchanges complementary bases A/T and C/G”. Is the negative sign in front of x to be explicitly interpreted as changing the one hot encoded vectors like [0,1,0,0] to [0,-1,0,0]. I expected some mathematical treatment of reversing the sequence, but I don’t understand how this negating would accomplish that.\n\nNeurIPS dimensions\n---------\n\nOriginality: The method here represents a novel derivation of equivariant reverse complement DNA deep learning layers. It generalizes prior formulations and suggests alternatives, which are benchmarked.\n\nQuality: The submission is technically sound, and the claims are supported by experiments, although these experiments need to be clarified according to my comments above. The authors do not carefully and honestly evaluate the weaknesses of their work.\n\nClarity: The submission is challenging to follow for readers without group theory expertise. For those who have such expertise, it seems to be reasonably well-organized.\n\nSignificance: Deep learning on DNA sequences has many important applications, and improvements to how it’s done can be quite significant. Exploring the reverse complement equivariance with this mathematical approach is highly informative, but the results suggest that the value is not tremendous and large data applications may choose to forgo this complicated technique.\n\nUpdate after rebuttal\n---------\nThe authors have greatly clarified my points of concern, and their experiments now clearly demonstrate interesting performance versus alternatives.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not describe potential negative societal impacts, but I cannot imagine any.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "dE5rt-apOA4", "reviewer": "Reviewer_9RaF", "summary": "In this paper, the authors delve deeper into the theoretical foundations of Reverse-complement equivariant convolutions for application to DNA sequences. Prior work has been mainly empirical and this helps to establish a general theoretical framework, thereby encompassing existing RC-equivariant-based models. By identifying this general theory, they were able to propose additional classes of RC equivariant models. They then compare the performance of standard models, existing models, and those proposed as a result from the theory on two regulatory genomics prediction tasks -- binary classification and profile prediction of TF ChIP-seq data. They show that for binary classification, their RC-models perform better than standard networks, but were largely similar for profile prediction. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is very interesting and timely. It could open the doors to new types of architectures that incorporate biological priors for genomic data analysis. Such models may lead to better generalization properties, though the evidence here was not as compelling. The main results are theoretical. Although this isn't the first time RC-equivariance was framed using group theory, this paper formalizes it in general terms, which enables them to identify new classes of RC-equivariant models. This is why I believe this work is original and significant. There are a few drawbacks that I outline below. They are mainly about clarity and the very low number of experiments to showcase performance gains and the benefits of RC-equivariant models.  \n\n- Disclaimer: I don't have a strong handle on the theoretical aspects of this work, so my evaluation will be mainly on the experiments and results. So, there could be some aspects of it that could be a little more clear. Perhaps a visualization of the theoretical results with a toy example could help understand the different situations (P,a,a) vs (P,a,b) for instance. \n\n- In the text, \"DNA sequence classification should therefore be RC-invariant, which calls for RC-equivariant architectures.\"  RC parameter sharing creates equivariance across reverse-complement transformations in the representations . It does not necessarily mean that if the input sequence were to undergo a RC transformation, then the network predictions will be the same, which is RC-invariance. A strategy for merging the representations is necessary. However, it's not crystal clear to me what is being done because the details of the architecture are in another paper Ref [44], which itself has the information spread between appendix and a few figures. It would be much clearer to present the architectures in detail within the paper. \n\ta. It seems like the merging mechanisms employed here is to take the sum/mean across the RC feature maps. Doing so indeed ensures RC-invariance. Stating this more clearly could help resolve some of my confusion. \n\tb. Another  key question I have about RCPS is how are the convolution layers employed? Is it 3 conv layers for each strand (but using reverse complement filters at each layer)? or is it doing the conv with two sets of tied filters (regular and RC) and merging with a sum of the feature maps within each RC conv layer? \n\tc. What is the difference between standard and RCPS? does standard contain twice the number of filters? For the same number of filters, RCPS effectively has twice as many filters and hence is more expressive. Could the performance gains (to some extent) be coming from this? If not, a better test would be to use twice the number of filters in standard. \n\n- Comment about RC-invariance:  Taking the sum/mean to merge RC feature maps has the effect of ensuring RC-invariance, but it may not necessarily lead to a desirable outcome for DNA sequence analysis. For instance, if a motif were present in one strand, then it will have a high activation for a given filter scan. The RC filter will have a completely different activation structure on the backwards strand, may just be noise. Indeed, if these two feature maps are merged via a mean, then the result would be invariant to whether the forward or reverse strand are fed to this layer. But the act of merging these feature maps averages down meaningful signal and also introduces noise from the wrong RC strand. Ideally, the results should be taken from one strand that the protein binds to and not the average of both, although information about which strand is not known a priori. On the other hand, RC data augmentation and RC conv layers without merging does not necessarily suffer from this issue -- it learns to detect RC motifs efficiently compared to standard convs. It doesn't enforce model will make the same prediction for different strands (i.e. not RC invariant), but it also doesn't average down information in the \"correct\" but unknown strand and introduce noise into the merged feature map. \n\n- Comment on rigor: There were only 3 binary tasks (CTCF, MAX, and SPI1) for which most claims are drawn from. This is a very small N considering the ease of accessing more TF ChIP-seq datasets for GM12878 cell lines in ENCODE. CTCF and MAX are notoriously simple datasets, based on the performance, I would suspect SPi1 is also. It would be more compelling if the results hold across more experiments.  \n\n- besides performance, is there any other benefit to using RC-equivariant models? It's supposedly more efficient, but that wasn't really shown. Perhaps a comparison of standard models with different number of parameters can be compared to the RC-equivariant models. This way you can show that standard models can achieve similar performance, but require X times the number of parameters? \n\nMinor comments: \n- The reduced dataset size experiment is a bit odd. It's unclear what the original dataset size was so how much it got reduced to is questionable. The performance of all models remains quite high. Could the dataset get reduced further? perhaps 500 sequences would provide a better separation? Making a systematic plot with varying dataset sizes (a few different choices) would certainly be more informative here.\n\n- k-mer representation was found to be beneficial in the hyperparameter search. Was it explored and found to be beneficial for standard model also? \n\n- In Sec 3, under binary tasks, it was stated that \"MAX TF was a failure because of relatively poor performance of RCPS, but using other equivariant networks...\" It doesn't seem to be a failure; it performs better than standard? Indeed, Regular and Irrep perform better for this but failure is a poor description. \n\n- For the profile task, the spearman correlation was used. Could/should pearson r, which is a metric that others have used for profile prediciton tasks, also be shown -- it isn't perfect, but the combination of different metrics could help understand the performance better. \n\n- There wasn't a discussion of RC pooling strategies.  I believe a sum or mean is used here -- it wasn't clearly stated and was referred to in another paper. Previously, an RC max-pooling was also suggested in Ref [4]. \n\n- Comment on the comparisons: In a previous paper [44], RC data augmentation worked well compared to standard and RCPS. How does it fare here?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed some limitations when discussing the \"disappointing performance\" on profile prediction tasks. But they could do a better job of discussing the limitations of their approach to achieve reverse-complement invariance (a stated desired property), but merging methods have limitations at the moment (see commends in main review). There is no mention of negative societal impact of their work as it is unclear how this could be used in a negative way -- so I don't think forcing an arbitrary statement in there would make sense. ", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses a very interesting problem motivated by genomics data analysis. All the 3 reviewers unanimously suggest to accept this paper and the meta reviewer agrees with the reviewers. Thus an acceptance is recommended.
Decision: Accept (Poster)