Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method to generate molecular graphs with optimized properties. Molecular graphs are constructed by the iterative addition of molecular fragments in a deep reinforcement learning framework. The method is benchmarked against a set of baselines in a task to generate molecules that maximizes the docking score to a protein (NSP15) from the SARS-CoV-2 virus, and shows good performance.
main_review: Strengths:

Relevant application of designing small molecules with desired properties, eg inhibitors of SARS-CoV-2  

Paper is overall well written  

Proposed method compared with quite a large range of relevant baselines  

Some informative ablation studies done  

Code has been provided   

Other comments/questions:

Although the proposed method is apparently the first to utilize a fragment by fragment graph construction in a deep reinforcement learning framework, I think the idea to extend the atom by atom graph construction in previous RL works with a fragment based action space (explored in more recent works Jin 2018, Jin 2020, Xie 2021) is interesting but not particularly novel.  

Parts of figure 4, such as the molecular structures, are too small and unreadable   

It would be useful to see the distribution of the properties in Table 1 (dock score, QED, SA, FCD). Also, it would be interesting to see the QED, SA, and FCD that correspond to the top 3 dock score molecules  

summary_of_the_review: Overall, I vote for acceptance. The well-structured paper proposes a useful molecule graph generation approach, accompanied by quite a comprehensive set of experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new fragment-based method for molecule generation. The model, called DGAPN, uses chemical fragments extracted from a public compound library with their chemical context (atom neighborhoods to which these fragments are attached). This way, all modifications made by the model should produce synthetically accessible compounds with high probability. Additionally, the authors introduce a new class of graph neural networks to predict chemical properties that employ an attention mechanism over atoms and chemical bonds. These models are used to guide the generative MDP trained with reinforcement learning. Inhibition of a SARS-CoV-2 antiviral target, NSP15, is used as an example task for the proposed model. The experimental section shows the results of single- and multi-objective compound generation, in which DGAPN obtains better docking scores of the generated molecules than other methods.
main_review: Pros:
- The authors extend graph attention networks to edge features and notice that spatial information is crucial in activity prediction. The introduced sGAT model is a novel contribution. It would be interesting to see an ablation study comparing this model to simpler graph neural networks, e.g. to the closely related EGAT architecture [1].
- Chemically reasonable mutations (CReM) are sampled by the RL algorithm to ensure that the generated structures are realistic.
- An innovation reward is added to encourage exploration. The reward is based on random network distillation.
- The ablation study shows the usefulness of spatial convolution for the supervised learning task.
- The results of the pretraining are shown in the ablation study even though they do not improve the generated compounds in terms of the main objective (an interesting negative result).

Cons:
- The authors state that "fragment-by-fragment sequential generation [...] has not been utilized in a deep graph RL framework", but I am not sure it is true. For example, Ståhl et al. [2] use deep reinforcement learning on fragments to optimize molecules.
- Recently, more and more publications focus on optimizing molecular docking scores using de novo drug design models, and I think this topic should be briefly summarized in the related work. For example, see [3].
- In Figure 1, the meaning of lines, symbols, and colors is unclear. Even after reading Equations 3-6, the drawing is difficult to process.
- The mathematical notation is difficult to follow in some parts of the methods section. For example, what is the purpose of the OHC operator? Is it there only to sample from the softmax distribution and map the decision onto the set of valid molecules?
- The QED score is very low for DGAPN. Can it be caused by generating big compounds that exceed a typical molecular weight of drug-like compounds?  The compounds are generated by adding whole chemical fragments, which can cause fast growth of the molecules. If that is the case, then these structures will be difficult to optimize (cannot be easily used as hits, which was suggested by the authors). This defeats the purpose of the generative model. The generated structures should be shown at least in the supplementary material.
- In this paper, only one antiviral target is considered. The results would be more convincing if there were more proteins in the experiments.
- The rewards are based on docking scores, so it would be interesting to see the docking poses of generated compounds. Especially, large compounds like those shown in Figure 4, which probably barely fit in the binding pocket.

Other comments:
- Why is the second column for DGAPN bolded in Figure 1?
- In Section 5, there is a citation that failed to compile: "MARS, xie2021mars"
- In Appendix D.3, the notation $p_w$ is used two times - should it not be just $p$ for the generated molecules?

[1] Chen, Jun, and Haopeng Chen. "Edge-Featured Graph Attention Network." arXiv preprint arXiv:2101.07671 (2021).

[2] Ståhl, Niclas, et al. "Deep reinforcement learning for multiparameter optimization in de novo drug design." Journal of chemical information and modeling 59.7 (2019): 3166-3176.

[3] Thomas, Morgan, et al. "Comparison of structure-and ligand-based scoring functions for deep generative models: a GPCR case study." Journal of Cheminformatics 13.1 (2021): 1-20.
summary_of_the_review: For now, I recommend the rejection of this paper (weak reject). The novelty of the proposed method is significant, but there are some unclear parts in the description of the method, and the results do not sufficiently prove the superiority of the proposed approach (see the comments above).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: With the goal of generating molecules that bind to functional sites of SARS-Cov-2 protein, the paper proposed a reinforcement learning model with a fragment-based framework for chemical structure modification. In the network part, spatial graph attention and spatial convolution are utilized to extract more structural information from the input graph into the representation of nodes or graphs. Based on the actor-critic algorithm, the reinforcement learning part is designed to find the state with the best docking score computed by docking programs and some novel approaches like PPO and RND are used to train the model more effectively. In the experiments, the model shows great performance while reducing the complexity of chemical synthesis meanwhile.
main_review: Strengths：
1.	The model in the paper is designed to resolve the up-to-date challenge of discovering novel inhibitors to target the SARS-CoV-2 non-structural protein endoribonuclease, and the experimental results show that molecules generated by the proposed model enjoy better synthetic accessibility according to SA value, which is a metric generally used to measure the synthetic difficulty of drugs. Meanwhile, the molecules show higher docking scores compared to other existing algorithms.
2.	The reinforcement learning part of the model is based on a fragment-based chemical environment for chemical synthesis and utilizes many novel approaches like Proximal Policy Optimization and Random Network Distillation as an effective attempt. This may give inspirations of the model architecture to future researches in this field.

Weaknesses:
1.	Spatial graph attention and spatial convolution indeed improve the model’s ability to extract structural information from input graphs, but these modules are not innovative enough. Actually, spatial graph attention is just a multi-head attention mechanism applied to graphs and spatial convolution is a Euclidean distance version of GNN’s convolution operation. Some new modules are expected to be employed by the model to make it more powerful.
2.	As is seen from the experimental results, the docking scores and SA value are better than other existing algorithms indeed. But the model’s performance on other metrics is unsatisfactory and particularly, Diversity is even the worst among algorithms in Table 1. Although this has been explained in the paper as a general drawback of fragment-based algorithms, more optimizations may need to be realized.
3.	Some statements in the paper may make readers confused, which are listed below. Equation 3 and 4 don’t share a consistent representation of the element in the adjacency matrix. Table 1 lacks some necessary annotations like what is the definition of Diversity and what is the meaning of the values in bold. The right plot in Figure 4 also lacks some necessary explanations and this plot seems to be redundant now.

summary_of_the_review: The paper utilized many novel approaches in graph representation and reinforcement learning to resolve the timely challenge about SARS-CoV-2, but the experimental results of the proposed model fail to show sufficiently powerful performance compared to existing algorithms except for better synthetic accessibility and docking scores. More innovative designs may need to be introduced to the present model to get a persuasive improvement.
review_title: null

Meta-review: After discussion, all reviewers are convinced about the novelty of the proposed method, and adjusted scores to recommend acceptance. They all appreciate the attempt to attack COVID-19 using machine learning.
Decision: Accept (Poster)