Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a computational drug repurposing framework that not only predicts the "treatment probabilities" between drugs and diseases but also predicts the path-based, "testable" mechanisms of action as their biomedical explanations, all based on a "massive" biomedical knowledge graph. Specifically, the paper uses an existing GraphSAGE model (NIPS 2017) with RF for the predictions and uses an adversarial actor-critic RL (SIGIR 2020) to predict MOA paths. The paper uses existing methods for an AI for Science problem.
strength_and_weaknesses: Strength
The paper uses existing ML methods for a real-world AI for Science problem.

Weaknesses
1. The paper is neither novel from ML (only uses existing, well-known methods) nor from biomedicine (the formulation of drug repurposing as link prediction is not rigorous and convincing; the biomedical knowledge graph comes from other bioinformatics papers).

2. While the paper has been claimed as a drug repurposing study, it lacks some necessary background information for this field.
   a. There are many AI for drug repurposing methods, such as chemical structure, drug-target interactions, drug perturbations of gene expression, cell and animal phenotypes, and clinical information. But the related work ignores all of them.
   b. Drug repurposing is one of the hardest problems in biomedicine, but it seems that (from Table 1) the paper reports an extremely high accuracy (0.934) for this task. It is necessary to compare the proposed method with the existing drug repurposing methods in the field, chemical structure, drug-target interactions, gene expression etc., and fully evaluate the method.

3. While the paper all depends on a "massive" biomedical knowledge graph, it doesn't provide enough information for evaluating its quality as the training/testing data.
   a. How to generate "true negative" data in the knowledge graph? In other words, how to prove the drug is NOT AND NEVER possible to treat a disease? In this sense, only RepoDB data is possible (negative is defined as failure or withdrawal in RCT). For each dataset, the paper needs to discuss how to get the true negative without any RCT (it seems a mission impossible). And if there exists such a pos/neg, any binary classification methods will do the job.
   b. The paper doesn't provide raw data for the review (those ids are hard to interpret). But from Table 8 and Table 9, it seems that the quality of the biomedical knowledge graph is poor. e.g., "therapeutic agent" and "antipsychotic agent" as a drug? The authors will need significant efforts to curate the knowledge graph.
   c. To avoid GIGO, the authors need to provide the full knowledge graph with all detailed information and reliable references.

4. There are some major problems in the experimental design.
   a. The paper uses only cross-validation (8:1:1) for a link prediction problem, which leaks information. Since RepoDB is the ONLY reliable TP/TN resource, I suggest the authors to use the other 3 datasets to predict links in RepoDB - it avoids dataset information leak.
   b. Also, I suggest the authors to leave-drug-out (instead of leave-link-out) when reporting all the scores - as the abstract mentions "identify new indications... of drugs/compounds". The paper needs to report the performance of predicting indications for each drug.
   c. Moreover, it would be great to leave-drug-class-out or leave-similar-drug-out to rule out the me-too drugs (I will not be surprised if pravastatin, atorvastatin, fluvastatin or oxprenolol, penbutolol, pindolol treat the same group of diseases) in the evaluations. Only doing this will demonstrate the true contribution of this paper.
   d. What does the "probability" mean in the table 8 and table 9? For example, 0.882 for amifampridine (a muscle drug): does it mean if a drug company launches amifampridine in RCT, there is 88.2% chance to be successful or if amifampridine be used for 1000 Huntington patients there are 882 patients will be saved? Some discussions and justifications are needed.

5. (minor) The references are not sorted based on ICLR template.
clarity,_quality,_novelty_and_reproducibility: 1. There are some minor grammar errors or typos but the paper is readable.
2. There are some major concerns in the data and experiments.
3. The paper is neither novel from ML nor from biomedicine.
4. The paper provides code and training data. But the training data contains only ids, which are hard for a reviewer to understand the names and their underlying science.
summary_of_the_review: The paper uses some existing, well-known methods for a biomedical, high-profile drug repurposing problem. The paper is neither novel from ML nor from biomedicine. There are also some major concerns in the data and experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed an approach for drug repurposing based on a biomedical knowledge graph. It not only predicts drug efficacy against a disease, but it also finds paths that could provide a biological explanation for the predictions. They compared their method against competitors and showed their performance in predicting drugs and in recapitulating curated drug mechanism 
strength_and_weaknesses: The approach combines existing methods: PubMedBERT for defining the node attributes in the knowledge graph, GraphSAGE for learning node embeddings, random forest for predicting drug efficacy, and ADversarial Actor-Critic (ADAC) Reinforcement learning model for predicting drug mechanism of action paths. 

 

The novelty of the work relies on the combination of these methods. In addition, it formulates the ADAC model properly for the specific problem. 

 

The proposed approach makes sense, and seems to work well for both problems addressed by the paper: prediction of drug efficacy and prediction of drug mechanism-of-action paths.  

 

It is important to notice that some of the competitors rely on related ideas (reinforcement learning for predicting drug mechanism of action pathways), but they use different models. 

  

The manuscript is well written, and the authors provided the code used for their approach. 

 

I reviewed this paper for NeurIPS 2022. The authors already answered many of my original concerns and improved the paper. However, I still have a concern on the evaluation and some minor comments that I detail below. 

 

1) Evaluation 

3.1. In Section 5.2 and Figure 4, the authors say that they evaluated the methods in a dataset composed of the validation and test sets. Could the authors provide the evaluation only on the test set? 

 

2) Minor comments 

- The authors refer to a bioRxiv version of the RTX-KG2 paper. I saw that the paper was recently published on BMC Bioinformatics: 

Wood, E.C., Glen, A.K., Kvarfordt, L.G. et al. RTX-KG2: a system for building a semantically standardized knowledge graph for translational biomedicine. BMC Bioinformatics 23, 400 (2022). https://doi.org/10.1186/s12859-022-04932-3 

 Could the authors update the reference? 

- It looks like references are not ordered by the last names of the authors. It was difficult to look up for the references. 
clarity,_quality,_novelty_and_reproducibility: *Clarity* 

The paper is well-written and the ideas are clearly explained 

*Novelty* 

All techniques described in the paper already exist. The main contribution relies on the combination of the methods and the formulation of ADAC model for drug repurposing. 

*Reproducibility* 

The authors provided their code for reproducing the experiments and also described the analyses in details. 
summary_of_the_review: The paper combines existing approaches on a non-trivial way for solving an interesting and difficult problem.   

For accepting the paper, I think the evaluation should be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses an important issue in drug repurposing methods, which is how to find biologically reasonable paths between drugs and potentially targeted diseases. The authors present a computational framework that can not only predict the treatment probabilities between drugs and diseases but also produce path-based biomedical explanations. To perform the first task (i.e., treatment probability prediction), the proposed framework uses GraphSAGE to integrate neighborhood information and apply Random Forest to predict treatment probabilities on top of the outputs of GraphSAGE. For the second task, an adversarial actor-critic reinforcement learning approach with demonstration paths are used to mine explainable paths between drugs and diseases. The experimental results are promising, indicating the proposed method is able to discover possible drug-disease pairs with much lower false positives.

strength_and_weaknesses: $Strengths$:

1. The proposed approach is sound and well clarified. The whole manuscript is friendly to readers who are not familiar with drug repurposing. Enough pre-processing and implementation details are provided for enhancing the reproducibility. The visualization results of explored paths are appreciated.

2. The performance improvements are obvious when using MRR and Hit@K as evaluation metrics, showing that the number of false positive predictions is greatly reduces.

$Weaknesses$:

1. The novelty is limited. Knowledge graph (KG) based drug repurposing is not a new problem (see the second paragraph in sec. Related Work).  The use of graph neural networks (GraphSAGE in this paper) and Random Forest for treatment probability prediction is also not novel. The biomedical explanation exploration methodology, i.e., ADversarial Actor-Critic (ADAC) reinforcement learning with demonstration paths, was initially proposed by (Zhao et al., 2020) for path exploration. I agree that how to appropriately integrate existing machine learning techniques is an important issue for interdisciplinary research. Thus, the authors should clarify the novelty of ideas or techniques in the rebuttal.

2. The involved baselines are too simple and not optimized for the drug repurposing problem. The authors have mentioned that there exist a number of computational approached designed for drug repurposing in sec. Introduction and sec. Related Work. However, none of these methods are included for comparison. Since KG-based drug repurposing is not a new problem, the authors should include some recent methodologies as baselines, which could better demonstrate the strengths of proposed method.

3. Experimental results in Table 1 are not explained. In fact, I am surprised that a simple combination of GraphSAGE and Random Forest can outperform those baselines by such large margins (nearly 10% on MRR and Hit@3, and over 10% on Hit@5). The authors should at least give some explanations about where these improvements come from.

Minor points:

1. When the authors listed three groups of existing drug repurposing methods (in the second paragraph of sec. Introduction), some references should be added to help readers get an overview of the current research.

2. The name mechanism of action (MOA) is kind of confusing. If this is a previously used phrase, please add a citation. If not, the authors should replacing it with another more intuitive name.

3. The related work section should include some discussions about the differences between the proposed framework and previous related methods.

4. In equation 1, the authors claimed "$v$ is a node that co-occurs with $u$ in fixed-length random walks." But how to choose $z_v$ is not clarified.
clarity,_quality,_novelty_and_reproducibility: $Clarity\ and\ Quality$:

Ths paper is well written. The proposed computational drug repurposing approach is well clarified. Important baselines are missing, and some performance advantages are not explained.

$Novelty$:

The novelty is limited. As far as I'm concerned, the authors neither propose a new problem (idea) nor a new techique (technical). 

$Reproducibility$:

The proposed method is reproducible based on textual descriptions.
summary_of_the_review: I lean upon a borderline reject due to the limited novelty and a lack of domain-related baselines. However, if the authors could clarify these problems in the rebuttal, I would be happy to raise my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors proposed an approach for drug repurposing based on a biomedical knowledge graph. It not only predicts drug efficacy against a disease, but it also finds paths that could provide a biological explanation for the predictions. They compared their method against competitors and showed their performance in predicting drugs and in recapitulating curated drug mechanism. The approach combines existing methods: PubMedBERT for defining the node attributes in the knowledge graph, GraphSAGE for learning node embeddings, random forest for predicting drug efficacy, and ADversarial Actor-Critic (ADAC) Reinforcement learning model for predicting drug mechanism of action paths.

Strength of the paper: 
1. The paper is written clear with introduction to drug repurposing. 
2. Experiments show that the number of false positive predictions is reduced.

Weakness of the paper: 
1. The paper lacks comparison with important drug repurposing baselines. 
2. Evaluation setup is questionable. The authors fix some of them (validation/testing split) in response. 
3. The technical novelty is limited while the main novelty is the combination of building blocks. 

In summary, the paper is not yet ready to appear in ICLR. 
justification_for_why_not_higher_score: Inadequate comparison with baseline methods, validation in experiments, and the limited technical novelty.
Decision: Reject