MARS: A neurosymbolic approach for interpretable drug discovery

Authors: Lauren Nicole DeLong, Yojana Gadiya, Paola Galdi, Jacques D. Fleuriot, Daniel Domingo-Fernández
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords: neurosymbolic, drug discovery, interpretability, knowledge graphs, reinforcement learning

Abstract:
Neurosymbolic (NeSy) artificial intelligence describes the combination of logic or rule-based techniques with neural networks. Compared to neural approaches, NeSy methods often possess enhanced interpretability, which is particularly promising for biomedical applications like drug discovery. However, since interpretability is broadly defined, there are no clear guidelines for assessing the biological plausibility of model interpretations. To assess interpretability in the context of drug discovery, we devise a novel prediction task, called drug mechanism-of-action (MoA) deconvolution, with an associated, tailored knowledge graph (KG), MoA-net. We then develop the MoA Retrieval System (MARS), a NeSy approach for drug discovery which leverages logical rules with learned rule weights. Using this interpretable feature alongside domain knowledge, we find that MARS and other NeSy approaches on KGs are susceptible to reasoning shortcuts, in which the prediction of true labels is driven by "degree-bias" rather than the domain-based rules. Subsequently, we demonstrate ways to identify and mitigate this. Thereafter, MARS achieves performance on par with current state-of-the-art models while producing model interpretations aligned with known MoAs.

Decision: reject
Reviewer scores: 5, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730483629958

thread_type:
official_review

<thread_object>
creator: reviewer_fcaq

<content>
summary:
The paper proposes MARS, a neurosymbolic model designed to improve interpretability in drug discovery through a task called Mechanism of Action (MoA) deconvolution, using a specialized knowledge graph named MoA-net. MARS intends to offer insights into the interactions among drug, protein, and biological process via learned rule weights.

strengths:
* The integration of neurosymbolic approaches provides a better understanding of the evidence chain in drug discovery.
* The dynamically learned rule weights may offer an interpretable explanation for the potential impacts of reasoning shortcuts.
* The experimental results are relatively comprehensive and reasonable.

weaknesses:
* The paper is poorly structured and written, and quite difficult to follow.
* Baseline methods compared in the experiments appear to be outdated, with the latest only dating back to 2020.
* Although the authors claim that interpretable symbols can enhance the learning process, there is insufficient experimental support for this assertion, especially regarding the discovery of unseen reasoning paths.
* While this work is presented as a reasonable exploration of biologically meaningful evidence chain reasoning shortcuts, the authors only apply it to three types of entity relationships. Existing study [1] has researched more comprehensive biological pathway evidence chain mining using symbolic reasoning and reinforcement learning. A detailed comparison of the similarities and differences between the two approaches, along with a more substantial discussion of the advantages of your work, is needed.

**References**

[1] Sudhahar, S., Ozer, B., Chang, J. et al. An experimentally validated approach to automated biological evidence generation in drug discovery using knowledge graphs. Nat Commun 15, 5703 (2024).

questions:
* In Lines 235-239, why do the authors state, "Although other metapaths are possible, we exclude them from our set of metapath-based rules"? What's the reason for excluding possible metapaths?
* Following the previous question, what is the rationale for limiting the length of metapaths to 4? Could this lead to insufficient exploration of reasoning paths? If possible, could a comparative experiment on different lengths be conducted?
* In Lines 277-282, the comparison baselines seem to be the latest from 2020. Are there any recent works related to knowledge graphs?
* Why do a significant portion of the results in the experimental tables lack standard deviation? For example, the top 9 baseline models in Table 1.
* Please provide visualizations of the learned rule weights and corresponding analyses.

If the authors could adequately address my concerns, I will consider raising the score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620321583

thread_type:
official_review

<thread_object>
creator: reviewer_vfxb

<content>
summary:
This study presents the MARS (Mechanism-of-Action Retrieval System), a neurosymbolic (NeSy) approach designed for interpretable drug discovery. MARS aims to improve understanding of drug mechanisms of action (MoAs) using knowledge graphs (KGs) combined with logical rule-based inference. The model introduces a tailored KG called MoA-net, enabling MoA deconvolution. MARS enhances interpretability by assigning weights to logical rules and employing a two-hop joint probability (P2H) metric to improve model calibration and address potential reasoning shortcuts.

strengths:
- MARS applies NeSy methods to drug discovery, a field with a strong demand for interpretability. The study’s introduction of MoA deconvolution as a KG-based task and MoA-net as a novel KG for biomedical applications broadens the use of KGs in drug discovery.
- The authors conduct extensive testing of MARS on both synthetic and real data and provide detailed comparisons to existing NeSy methods, particularly regarding susceptibility to reasoning shortcuts.
- By enhancing interpretability in drug discovery, MARS could help researchers gain insights into drug mechanisms, potentially leading to safer and more effective treatments.

weaknesses:
- The model is primarily tested on synthetic KG data and lacks real-world validation on clinical datasets or pharmacological records, which limits the practical assessment of its interpretability and generalization for drug discovery tasks.
- While the P2H metric is introduced to make MARS shortcut-aware, its reliance on rule weighting and complex computations might hinder scalability, especially in large, densely connected KGs that are typical in biomedical data.
- AMARS mainly addresses computational rather than biological significance.
- MARS is highly susceptible to node degree bias, leading to unintended reasoning shortcuts.

questions:
- Testing the model on real-world datasets, such as drug-protein interactions or clinical outcome data, would strengthen the claims regarding its applicability in drug discovery.
- How does MARS handle drug discovery tasks beyond MoA deconvolution?
- How could biological interpretability be further improved? Considering biological plausibility, could the authors incorporate features like protein binding affinities or pharmacokinetic properties to add biological context to MARS’s interpretability?
- Given the computational cost of two-hop joint probability calculations, what specific modifications or future optimizations are planned to enhance scalability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730727211658

thread_type:
official_review

<thread_object>
creator: reviewer_hv6o

<content>
summary:
The paper proposed a systematic framework for interpretable drug discovery. The model comprises of drug mechanism-of-action (MoA) deconvolution; MoA-net and its variants, a specifically designed knowledge graph based on public biomedical data; MARS, a Neural Symbolic approach with through dynamic rule-weight updates. The work provides enhanced interpretability and strong predictive performances compared to other baselines.

strengths:
The paper is generally well-written and easy to follow. The objectives of the proposed framework clearly outlined and the model effectively handles several specific problems that other comparable methods hold. The proposed method shows competitive performances and provides a reasonable form of interpretability using knowledge graphs.

weaknesses:
- The overview of MARS is unclear, and a figure delineating process would be helpful for reader's understanding. This should include the input-output pairs for the proposed model.
- It is difficult to interpret Figure 3, as the main body does not explain anything about the confidences.
- While the paper proposes the term "drug-discovery", the it unclear in how to find  "novel drugs or chemical compounds". Given that the model is trained on the full data (no test) it could provide some meaningful results.
- The computation cost in training the model could provide valuable insights.
- The term "enhanced interpretability" is difficult to understand. An example for the term (possibly from the data that the authors used) would help better understanding of the term, and the strength of the model.

questions:
- What is the computation time of training the network? How susceptible is the model with inclusion of new data?
- What is the "confidence" in figure 3?
- What is the exact neural network architecture for MARS? What is a simple example of input and output pairs?
- How would we approach in finding drug discovery? Does the model provide any candidate paths or of any sort? If so, how would we approach justification of the result?
- What is the "enhanced interpretability"? Would there be an example of this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730749882959

thread_type:
official_review

<thread_object>
creator: reviewer_efxs

<content>
summary:
*November 26: after first response, the "Presentation" score has been raised from 2 to 3*

This work proposes a new neurosymbolic (NeSy) model for drug discovery as well as an associated dataset, where the problem is cast as a prediction task for drug-biological process (BP) pairs. More specifically, biochemical knowledge is arranged into a knowledge graph (KG) where entities (vertices) such as drugs, proteins and BP are connected with each other via directed edges. By using a new update rule of edge weights, the new model was able to avoid reasoning shortcuts inherent in previous methods.

strengths:
- Using the $ P_{2H} $ update rule, the new method has been shown, via a series of ablation studies, to avoid reasoning shortcuts due to degree bias.

weaknesses:
- The novelty of the manuscript appears to be limited:
    - The new dataset has fewer kinds of entities than PoLo, and in particular does not appears to include those relevant for therapeutics such as "disease" or "side effect".
    - Similarly, one would expect more kinds of entities beyond a generic "protein" (e.g. receptor, transcription factor, enzyme etc.).
    - Another issue is that all metapaths are chains, like $ \mathrm{interacts}(P_{k}, P_{k + 1}) $, whereas in real-world applications there are many instance of multi-protein complexes for biological functions.
    - The model reward function is almost the same as PoLo (Equation 4): $$ R(S_{L + 1}) = 1_{ e_{L + 1} = e_d  }  + b \lambda  \sum\limits_{i = 1}^{m} s(M_i) 1_{\tilde{P} = M_i} $$ In fact, that paper states that the hyparameter $ b $ can be "set to $ 1_{e_{L + 1} = e_d} $" (p. 382).
- The presentation of the manuscript is not as efficacious as one would hope:
    - Key concepts such as "deconvolution" or $ \land $ (presumably conjunction?) were not defined.
    - The 2-hop probability update ($ \S $A.4 and Alg. 1), which is the major novelty in the manuscript, should be in the main text.
    - The main predictive task was not explicitly formulated: from Figure 1A it appears to be link prediction.
    - Moreover, it would be useful to have the catalogue of different entities--entities interaction types (Figure 1B) in a table.
    - The background colour of Figure 1B makes it harder to read, plus the font sizes for the labels are a bit too small.
    - The citation for PoLo should use the published version (https://link.springer.com/chapter/10.1007/978-3-030-77385-4_22)

questions:
- Could you include more training details e.g. hardware, training time, validation loss plots etc.?
- On what basis were the "feasible" MoA's chosen in Table A1?
- When comparing with PoLo, which rewards function(s) were used?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732726916847

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks to the reviewers

comment:
Dear reviewers, 

Thank you for your constructive feedback. We greatly appreciate all of the points and suggestions made, and we believe your reviews have helped us to improve our manuscript. 

Please let us know if you have any further questions during this discussion period.  

Thank you!

Sincerely,
the authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a neurosymbolic system for drug discovery, which consists of drug mechanism-of-action (MoA) deconvolution; MoA-net and its variants, a specifically designed knowledge graph based on public biomedical data; MARS, a Neural Symbolic approach through dynamic rule-weight updates. The paper is about AI for science (biology and drug discovery), neurosymbolic approaches have the potential for interpretable drug discovery, and the paper is clearly written. However, reviewers raised concerns that the experiments conducted on a dataset don't sufficiently capture the biochemical complexity. For example, there is no differentiation between kinds of proteins (e.g. receptors, enzymes, transporters), drug-protein interactions should be more than just upregulates and downregulates, and drug acts by binding to multiple target proteins that are jointly responsible for a biological process. Thus, there lack of concrete case studies to demonstrate its real-world value in drug discovery.