A mechanistically interpretable neural network for regulatory genomics

Authors: Alex M Tseng, Gökcen Eraslan, Tommaso Biancalani, Gabriele Scalia
Primary area: interpretability and explainable AI
Keywords: interpretability, mechanistic interpretability, attention, convolution, regulatory genomics

Abstract:
Deep neural networks excel in mapping genomic DNA sequences to associated readouts (e.g., protein–DNA binding). Beyond prediction, the goal of these networks is to reveal to scientists the underlying motifs (and their syntax) which drive genome regulation. Traditional methods that extract motifs from convolutional filters suffer from the uninterpretable dispersion of information across filters and layers. Other methods which rely on importance scores can be unstable and unreliable. Instead, we designed a novel mechanistically interpretable architecture for regulatory genomics, where motifs and their syntax are directly encoded and readable from the learned weights and activations. We provide theoretical and empirical evidence of our architecture's full expressivity, while still being highly interpretable. Through several experiments, we show that our architecture excels in de novo motif discovery and motif instance calling, is robust to variable sequence contexts, and enables fully interpretable generation of novel functional sequences.

Decision: reject
Reviewer scores: 6, 5, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1730112146283

thread_type:
official_review

<thread_object>
creator: reviewer_dlzw

<content>
summary:
The authors consider the problem of understanding the transcription factor—DNA binding code in terms of actual binding motifs in the genome. Through a mechanistically interpretable neural network design for the prediction of protein binding events, they ensure that biologically meaningful binding motifs can be learned, showing this ability on synthetic and real data.

strengths:
- The proposed architecture seems novel and thought-through.
- The methodological part is well written and easy to follow.

weaknesses:
- The assumptions made in this work are prohibitively restrictive: 
1) recent efforts, including large-scale consortium-based benchmarking in ENCODE-DREAM challenge [1], have shown that Epigenetics (e.g., openness of chromatin) play a decisive role in the binding of transcription factors and need to be taken into account for accurate prediction [2], a versatile set of tools has hence been designed including [3] to appropriately consider such data;
2) Genomes are unique to individuals and variants in individual genomes play a major driver in diseases such as cancer, often interrupting TF binding due to their location in motifs. Hence, recent tools consider such variants for accurate and actual genome-specific binding [4].
For both, open chromatin and variant data, how would the model incorporate this essential knowledge? How does it perform compared to existing work?
- There is a plethora of deep learning-based tools for predicting TF-binding [5,6,7,8], how does the presented approach compare to them quantitatively? With generic attribution methods, such as layer-wise-relevance propagation or SmoothGrad, extraction of motif sequences that the models respond to can be similarly achieved for these deep architectures – are your motifs qualitatively and quantitatively (e.g. similarity between discovered and known motif) better?
- It is unclear why motifs need to be learned, as there exist extensive motif databases (TRANSFAC, JASPAR, ...) that were generated through rigorous (wet-lab) experiments. Why is it better to use an (inaccurate) inferred motif than detecting motif occurrence and then building a classifier based on that? The paper would benefit from a use-case where the existing databases can not help (e.g., TFs where motifs are not available, or relevant species were no database is available).
- The quality of prediction is not properly evaluated against existing methods (including non-DL models), it is hence unclear whether the interpretability is traded off for prediction quality that would harm downstream applications. A proper comparison and evaluation in that regard would improve the paper.

[1] https://www.synapse.org/Synapse:syn6131484/wiki/402037, and associated conference tracks at RECOMB/ISCB 2016

[2] J Keilwagen, S Posch, J Grau, Accurate prediction of cell type-specific transcription factor binding Genome Biology 2019

[3] F Schmidt, F Kern, MH Schulz, Integrative prediction of gene expression with chromatin accessibility and conformation data Epigenetics & Chromatin 2020

[4] R Steinhaus, PN Robinson, D Seelow, FABIAN-variant: predicting the effects of DNA variants on transcription factor binding Nucleic Acids Research 2022

[5] L Cao et al., Prediction of Transcription Factor Binding Sites Using a Combined Deep Learning Approach Frontiers in Oncology 2022

[6] N Ghosh et al., Predicting Transcription Factor Binding Sites with Deep Learning Int J Mol Sci 2024

[7] C Chen et al., DeepGRN: prediction of transcription factor binding site across cell-types using attention-based deep neural networks BMC Bioinformatics 2021

[8] GB Kim et al., DeepTFactor: A deep learning-based tool for the prediction of transcription factors PNAS 2020

questions:
- How would open chromatin and variant data be incorporated into the model? How does it perform compared to existing work?
- How does the presented approach compare to existing DL-based TF binding prediction methods and established attribution methods?
- Why is it better to use an (inaccurate) inferred motif than detecting motif occurrence and then building a classifier based on that?
- How is prediction performance affected by the architecture compared to existing work (including non-DL based methods)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730554590264

thread_type:
official_review

<thread_object>
creator: reviewer_fkb3

<content>
summary:
This paper presents a neural network model aimed at providing mechanistic interpretability within the field of regulatory genomics. While the topic is relevant and important, the work suffers from several significant shortcomings that undermine its contributions and overall impact.

strengths:
- Relevant Research Area: The focus on regulatory genomics is timely and addresses a critical need for interpretable models in biological applications.
- Potential Applications: The proposed methods may have applications in understanding gene regulation mechanisms, which is valuable for both computational biology and medicine.

weaknesses:
- The approach lacks sufficient novelty compared to existing methods in the field, e.g., DNABERT, NT, HyenaDNA, EVO, et al. The authors do not convincingly demonstrate how their work significantly advances the state of the art in mechanistic interpretability or regulatory genomics.

- The experimental validation is inadequate. The authors fail to perform comprehensive comparisons with baseline methods or other relevant approaches. Without robust benchmarks, it isn't easy to ascertain the practical utility of the proposed model.

-  The conclusions drawn from the results are often overstated and not sufficiently supported by the data. The authors claim significant interpretability and performance improvements without adequate evidence to back these assertions.

questions:
- What specific advancements does this work provide over existing models in mechanistic interpretability and regulatory genomics?
- Can the authors elaborate on their validation methodology and why they chose not to compare with established baseline methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730556501040

thread_type:
official_review

<thread_object>
creator: reviewer_rgno

<content>
summary:
This paper designed a novel mechanistically interpretable architecture for regulatory genomics, where motifs and their syntax are directly encoded and readable from the learned weights and activations. Through several experiments, the authors show that ARGMINN excels in
de novo motif discovery and motif instance calling, is robust to variable sequence contexts, and enables fully interpretable generation of novel functional sequences

strengths:
1. This paper successfully applied the MI methods to the regulatory genomics domain
2. This paper provided thorough assessment for the usage of ARGMINN
3. The authors provided detailed related works and theoretical results based on first-order logic

weaknesses:
1. I find this work may be not interesting to the most readers of ICLR. This paper should fall into a biological journal (e.g. journals for stastical geomics)
2. The technologies introduced in this paper are actually not based on biological mechanics but common mechanical Interpretation tricks including regularizations, separate convolutions and attention for interpretability.

questions:
1. Could you please explain more about the connection between ARGMINN and CBMs?
2. Could you please explain more about the difference between ARGMINN and ExplaiNN?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660269466

thread_type:
official_review

<thread_object>
creator: reviewer_ebey

<content>
summary:
This paper proposes a mechanistically interpretable neural network to solve several genome understanding tasks, including regulatory function prediction, DNA motif discovery and motif interaction prediction. The proposed architecture consists of a motif-scanner, a regularizer and a syntax builder to respectively recognize motifs, filter out wrong patterns and predict interactions between motifs. The paper did many analyses to show the discovered motifs, motif instance calling, QTL prioritization and the architecture robustness.

strengths:
The paper is targeting at an important task, and shows many cases for each evaluated task, which is straightforward.

weaknesses:
1. The paper is hard to follow. After reading the manuscript, I know the authors develop a new architecture consisting of several components, and each of them achieves a specific function. However, I cannot figure out the detailed contributions of this paper. That is to say, what are the differences of the proposed architecture compared to previous method?

2. The experimental settings and results are not complete.:

2.1 The paper didn't show any specific number on the evaluated tasks. All results are shown in Figure. Though we could see some improvements compared to baselines, we still don't know what specific performance the proposed model can achieve. 

2.2 The paper lacks ablation study.

2.3 The paper lacks specific experimental settings. For example, what are the specific values for n_f and n_L?  Do the standard CNN have the same layer as ARGMINN?

2.4 What species genome sequences are used for each task?

questions:
The most uncertain parts for me are the contributions and the detailed experimental settings of the proposed method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699635078

thread_type:
official_review

<thread_object>
creator: reviewer_zz46

<content>
summary:
The paper introduces ARGMINN, a novel DNN architecture designed for recovering motifs with mechanistic interpretability. The model is designed to directly encode biologically meaningful motifs in its weights and activations. Exhaustive experiments show ARGMINN's strength in motif discovery, instance calling, robustness, and interpretable sequence generation. This work successfully narrows the gap between the high predictive performance of DNNs and the need for interpretability in genomics prediction.

strengths:
- The proposed ARGMINN model brings the mechanistically interpretable architecture into a classical DNN architecture, trying to address the Interpretability challenge in applying deep learning to genomics.
- The authors provide theoretical proofs showing that their model can recognize all motif configurations.
- ARGMINN shows superior performance  across all tasks, compared to baselines (e.g. CNNs and ExplaiNN). And the authors also showcase the model's ability to generate novel functional sequences based on the learned motifs, which is a valuable contribution.

weaknesses:
- The evaluation strategy used in the paper involves reserving chr8 and chr10 for validation and chr1 for testing, while training on all other chromosomes. However, because chromosomes can vary in genomic features, using a specific chromosome split might introduce biases in performance evaluation.
- Since the paper introduces a memory stream into the attention mechanism, it would be helpful to include a discussion on the training times and memory usage of the model.

questions:
- For the evaluation strategy, why do you choose chr8 and chr10 for validation and chr1 for testing? Have you considered to use leave-one-out strategy to assess the model's generalization across different genomic regions?
- Could you provide details on the computational resources required to train ARGMINN?
</content>
</thread_object>

</thread>


Metareview:
metareview:
**a) Scientific Claims and Findings:**
The paper introduces a novel neural network architecture, ARGMINN, designed for analyzing DNA sequences in regulatory genomics. The primary objective is to develop a model that not only predicts genomic readouts, such as protein–DNA binding, but also provides direct interpretability by revealing underlying motifs and their syntactical relationships from the learned weights and activations. Traditional methods often struggle with interpretability due to the dispersion of information across filters and layers, and reliance on unstable importance scores. In contrast, this architecture encodes motifs and their syntax in a manner that is directly readable post-training. The authors provide both theoretical and empirical evidence supporting the model's full expressivity while maintaining high interpretability. Through various experiments, the architecture demonstrates excellence in de novo motif discovery, motif instance identification, robustness to variable sequence contexts, and the ability to generate novel functional sequences with full interpretability.

**(b) Strengths:**
* Enhanced Interpretability: The architecture allows for direct reading of motifs and their syntactical rules from learned weights and activations, addressing a significant challenge in regulatory genomics.
* In contrast to CBMs, ARGMINN does not require concept/motif labels. 
* Theoretical and Empirical Validation: The model's expressivity and interpretability are supported by both theoretical analysis and empirical experiments. The theory states that in comparison to the competitor explaiNN, the proposed ARGMINN architecture is more expressive.
* Robust Performance: The architecture excels in tasks such as de novo motif discovery and motif instance identification, and shows robustness to variable sequence contexts, indicating its practical applicability.

**(c) Weaknesses:**
* Scalability Concerns: The paper does not provide a detailed analysis of the model's scalability to large genomic datasets, which is crucial for real-world applications in regulatory genomics. Additional experiments during the rebuttal suggest that ARGMINN requires a relatively high number of parameters (relative to the studied CNN) and increased training time in comparison with competitors. However, runtimes were generally not significant in the reported table.
* Comparative Analysis: While the model demonstrates promising results, a more comprehensive comparison with existing state-of-the-art models would help in understanding its relative performance and potential advantages. Reviewer fKB3 requested comparisons with DNABERT, NT, HyenaDNA, EVO, for instance. As the authors argue, such models have been trained for a different purpose, but they list other models ChromDragoNN, DeepSea, DanQ, BPNe with which they still do not compare. Also Reviewer DLZW has suggested a couple of methods for comparison.
* Scope of the Approach: Reviewer DLZW has raised concerns about the impact of the task and whether novel discovery is possible. The paper would benefit from a use-case where the existing databases cannot help (e.g., TFs where motifs are not available, or relevant species were no database is available).
* Ablations: Additional requested ablations suggest that the filter-overlap regularization has only a marginal effect. It’s main purpose is to provide interpretability in combination with the modified attention layer (in the sense that motifs can be identified).

**(d) Reasons for Rejection:**
After careful consideration, the decision to reject the paper is based on the following reasons:
1. Insufficient Scalability Analysis: The absence of a thorough examination of the model's scalability to large genomic datasets raises concerns about its applicability in practical, large-scale regulatory genomics studies.
2. Limited Comparative Evaluation: The paper does not provide an extensive comparison with existing models, making it challenging to assess the proposed architecture's relative performance and contributions to the field.
3. Limited Impact (Methodological Innovation or Application Relevance):  Reviewer RgNo has criticised a limited methodological innovation, as ARGMINN combines known components and relies on established intuition. This would not be a limiting factor if the empirical evidence for application relevance was very high. Yet, novel discovery is not supported by wet lab experiments, for instance, (see Reviewer DLZW).
Addressing these concerns would enhance the paper's contribution and its potential for acceptance in future submissions.