Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper provides a mathematical formalization of published equivariant layers (translation and reverse-complement) for DNA sequences and explores other architecture variants of these layers (linear, nonlinear and k-mers).
main_review: The paper is clearly written and well structured.

The main contribution of the work is the theoretical characterization of translation and reverse complement (RC) equivarariant CNN layers for DNA sequences. It is based mainly on the work of Cohen and Welling on Group equivariant CNNs applied in vision tasks [1, 2, 3]. Regarding the RC equivariance for DNA sequences, it has been used and empirically studied previously in other papers [4,  5, 6]. Moreover, the authors investigate, define theoretically and implement new architecture variants of, linear and nonlinear, RC equivariant layers.

### Comments:

1) In this comprehensive work, the theoretical results are thoroughly presented and explained. However, the used math is too advanced and not always easy to follow. The authors could make use of graphic figures to support the different concepts as it was done for image processing in [1, 2].  

2) The paper does not clearly discuss the importance of using the equivariant nonlinear layers for DNA sequence processing. Characterizing only the equivariant linear layers would be sufficient for this work if their empirical limitations were thoroughly analyzed and discussed; and nonlinear layers could be addressed in a separate work. 

3) Although the authors implemented the studied layers, the empirical study is quite minimal compared to the previous work of [4]. Only two equivariant networks are empirically evaluated from the set of new investigated networks. Moreover, the evaluation of the models is based simply on their performance on two supervised tasks in regulatory genomics. Further investigations are needed such as the assessment of the quality of the internal representation of the equivariant networks [7]. 

4) The hyperparameters search should be done separately for each task.  The hyperparameters of models used in the profile task were set based on a search performed with binary tasks. A $k > 2$ or a ratio $a/(a+b) \neq 0.75$ could lead to better performance on the profile task. 

5) The related work is correctly cited throughout the paper, but there is no dedicated section to clearly position the current work from other related machine learning contributions.

### Minor comments:

1) Set the same y-axis limit for both panels in figures 3 and 4.
3) Binary task, lines 313 and 314: provide AuROC with standard error. 
2) A.1, lines 521, 524 and 525: the second $\rho_1(s)$ should be $\rho_{-1}(s)$

### References:
* [1] T.S. Cohen and M. Welling. Group equivariant convolutional networks. 2016.
* [2] T. S. Cohen and M. Welling. Steerable CNNs. 2017.
* [3] T. S. Cohen, M. Geiger, and M. Weiler. A General Theory of Equivariant CNNs on Homo- geneous Spaces. 2019.
* [4] H. Zhou, A. Shrikumar, and A. Kundaje. Towards a better understanding of reverse-complement equivariance for deep learning models in regulatory genomics. 2020.
* [5] R. C. Brown and G. Lunter. An equivariant Bayesian convolutional network predicts recom- bination hotspots and accurately resolves binding motifs. 2019.
* [6] Bartoszewicz, Jakub M., et al. DeePaC: predicting pathogenic potential of novel DNA with reverse-complement neural networks. 2020.
* [7] A. Shrikumar, P. Greenside, and A. Kundaje. Reverse-complement parameter sharing improves deep learning models for genomics. 2017.
limitations_and_societal_impact: Not applicable.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors delve deeper into the theoretical foundations of Reverse-complement equivariant convolutions for application to DNA sequences. Prior work has been mainly empirical and this helps to establish a general theoretical framework, thereby encompassing existing RC-equivariant-based models. By identifying this general theory, they were able to propose additional classes of RC equivariant models. They then compare the performance of standard models, existing models, and those proposed as a result from the theory on two regulatory genomics prediction tasks -- binary classification and profile prediction of TF ChIP-seq data. They show that for binary classification, their RC-models perform better than standard networks, but were largely similar for profile prediction. 

main_review: The work is very interesting and timely. It could open the doors to new types of architectures that incorporate biological priors for genomic data analysis. Such models may lead to better generalization properties, though the evidence here was not as compelling. The main results are theoretical. Although this isn't the first time RC-equivariance was framed using group theory, this paper formalizes it in general terms, which enables them to identify new classes of RC-equivariant models. This is why I believe this work is original and significant. There are a few drawbacks that I outline below. They are mainly about clarity and the very low number of experiments to showcase performance gains and the benefits of RC-equivariant models.  

- Disclaimer: I don't have a strong handle on the theoretical aspects of this work, so my evaluation will be mainly on the experiments and results. So, there could be some aspects of it that could be a little more clear. Perhaps a visualization of the theoretical results with a toy example could help understand the different situations (P,a,a) vs (P,a,b) for instance. 

- In the text, "DNA sequence classification should therefore be RC-invariant, which calls for RC-equivariant architectures."  RC parameter sharing creates equivariance across reverse-complement transformations in the representations . It does not necessarily mean that if the input sequence were to undergo a RC transformation, then the network predictions will be the same, which is RC-invariance. A strategy for merging the representations is necessary. However, it's not crystal clear to me what is being done because the details of the architecture are in another paper Ref [44], which itself has the information spread between appendix and a few figures. It would be much clearer to present the architectures in detail within the paper. 
	a. It seems like the merging mechanisms employed here is to take the sum/mean across the RC feature maps. Doing so indeed ensures RC-invariance. Stating this more clearly could help resolve some of my confusion. 
	b. Another  key question I have about RCPS is how are the convolution layers employed? Is it 3 conv layers for each strand (but using reverse complement filters at each layer)? or is it doing the conv with two sets of tied filters (regular and RC) and merging with a sum of the feature maps within each RC conv layer? 
	c. What is the difference between standard and RCPS? does standard contain twice the number of filters? For the same number of filters, RCPS effectively has twice as many filters and hence is more expressive. Could the performance gains (to some extent) be coming from this? If not, a better test would be to use twice the number of filters in standard. 

- Comment about RC-invariance:  Taking the sum/mean to merge RC feature maps has the effect of ensuring RC-invariance, but it may not necessarily lead to a desirable outcome for DNA sequence analysis. For instance, if a motif were present in one strand, then it will have a high activation for a given filter scan. The RC filter will have a completely different activation structure on the backwards strand, may just be noise. Indeed, if these two feature maps are merged via a mean, then the result would be invariant to whether the forward or reverse strand are fed to this layer. But the act of merging these feature maps averages down meaningful signal and also introduces noise from the wrong RC strand. Ideally, the results should be taken from one strand that the protein binds to and not the average of both, although information about which strand is not known a priori. On the other hand, RC data augmentation and RC conv layers without merging does not necessarily suffer from this issue -- it learns to detect RC motifs efficiently compared to standard convs. It doesn't enforce model will make the same prediction for different strands (i.e. not RC invariant), but it also doesn't average down information in the "correct" but unknown strand and introduce noise into the merged feature map. 

- Comment on rigor: There were only 3 binary tasks (CTCF, MAX, and SPI1) for which most claims are drawn from. This is a very small N considering the ease of accessing more TF ChIP-seq datasets for GM12878 cell lines in ENCODE. CTCF and MAX are notoriously simple datasets, based on the performance, I would suspect SPi1 is also. It would be more compelling if the results hold across more experiments.  

- besides performance, is there any other benefit to using RC-equivariant models? It's supposedly more efficient, but that wasn't really shown. Perhaps a comparison of standard models with different number of parameters can be compared to the RC-equivariant models. This way you can show that standard models can achieve similar performance, but require X times the number of parameters? 

Minor comments: 
- The reduced dataset size experiment is a bit odd. It's unclear what the original dataset size was so how much it got reduced to is questionable. The performance of all models remains quite high. Could the dataset get reduced further? perhaps 500 sequences would provide a better separation? Making a systematic plot with varying dataset sizes (a few different choices) would certainly be more informative here.

- k-mer representation was found to be beneficial in the hyperparameter search. Was it explored and found to be beneficial for standard model also? 

- In Sec 3, under binary tasks, it was stated that "MAX TF was a failure because of relatively poor performance of RCPS, but using other equivariant networks..." It doesn't seem to be a failure; it performs better than standard? Indeed, Regular and Irrep perform better for this but failure is a poor description. 

- For the profile task, the spearman correlation was used. Could/should pearson r, which is a metric that others have used for profile prediciton tasks, also be shown -- it isn't perfect, but the combination of different metrics could help understand the performance better. 

- There wasn't a discussion of RC pooling strategies.  I believe a sum or mean is used here -- it wasn't clearly stated and was referred to in another paper. Previously, an RC max-pooling was also suggested in Ref [4]. 

- Comment on the comparisons: In a previous paper [44], RC data augmentation worked well compared to standard and RCPS. How does it fare here?
limitations_and_societal_impact: The authors have addressed some limitations when discussing the "disappointing performance" on profile prediction tasks. But they could do a better job of discussing the limitations of their approach to achieve reverse-complement invariance (a stated desired property), but merging methods have limitations at the moment (see commends in main review). There is no mention of negative societal impact of their work as it is unclear how this could be used in a negative way -- so I don't think forcing an arbitrary statement in there would make sense.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors note that neural network models operating on DNA sequences desire shift equivariance and reverse complement (RC) equivariance. Existing proposed techniques have been somewhat ad hoc. The authors derive RC equivariant models from first principles, and expose a class of techniques that haven’t yet been studied. They benchmark several equivariant alternatives on ChIP-seq peak classification and nucleotide-resolution coverage prediction and demonstrate interesting results.
main_review: Introductory note
---------

I’m unable to evaluate the correctness of the Methods section due to unfamiliarity with group theory. 

Major comments
---------

In describing their experiments, the authors note, “For a more detailed explanation of the experimental setup, please refer to Zhou et al. [44].” For some of the details (e.g. the ChIP-seq data processing pipeline), I think that’s fine. But there are several important choices that I believe need to be described here. First, it is standard when performing deep learning on these data to include both the sequence and its reverse complement in the training set, akin to data augmentation. If this has not been done, then the “Standard” model would be at an unfair disadvantage.

Similarly, it is typical when making predictions to compute a forward pass for both the forward and reverse complement sequences and take the average as an ensemble. An equivariant model wouldn’t benefit from this option. However, you might argue that it would be fair to benchmark an ensemble of two distinctly trained equivariant models to match the prediction computation to that mode. The authors should offer more details about how predictions are made on their test sets, and I’d encourage them to explore these different modes.

Minor comments
---------
The authors might have more success reaching potential users of this technique in computational biology if they could make their Methods section more accessible to non-experts in group theory. If the authors could convey some intuition for what these operations aim to accomplish and how they do it, I believe a larger audience would be available. I’ll note that the code availability is great and a dedicated reader can likely reach an understanding.

In line 107, the authors write that “the RC operations amounts to the following : RC(f)(x) = σ(−1)[f(−x)], where σ(−1) is the 4×4 permutation matrix that exchanges complementary bases A/T and C/G”. Is the negative sign in front of x to be explicitly interpreted as changing the one hot encoded vectors like [0,1,0,0] to [0,-1,0,0]. I expected some mathematical treatment of reversing the sequence, but I don’t understand how this negating would accomplish that.

NeurIPS dimensions
---------

Originality: The method here represents a novel derivation of equivariant reverse complement DNA deep learning layers. It generalizes prior formulations and suggests alternatives, which are benchmarked.

Quality: The submission is technically sound, and the claims are supported by experiments, although these experiments need to be clarified according to my comments above. The authors do not carefully and honestly evaluate the weaknesses of their work.

Clarity: The submission is challenging to follow for readers without group theory expertise. For those who have such expertise, it seems to be reasonably well-organized.

Significance: Deep learning on DNA sequences has many important applications, and improvements to how it’s done can be quite significant. Exploring the reverse complement equivariance with this mathematical approach is highly informative, but the results suggest that the value is not tremendous and large data applications may choose to forgo this complicated technique.

Update after rebuttal
---------
The authors have greatly clarified my points of concern, and their experiments now clearly demonstrate interesting performance versus alternatives.
limitations_and_societal_impact: The authors do not describe potential negative societal impacts, but I cannot imagine any.
review_title: null

Meta-review: This paper addresses a very interesting problem motivated by genomics data analysis. All the 3 reviewers unanimously suggest to accept this paper and the meta reviewer agrees with the reviewers. Thus an acceptance is recommended.
Decision: Accept (Poster)