Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a new way of molecular optimization that is based on chemical fragments. The authors employ VQ-VAE to encode a fragment library in a coherent latent space. They argue that this method produces more diverse rationales than a Gaussian VAE. The proposed generative method uses reinforcement learning (PPO) to select modifications of the chemical structure: a fragment decoded by VQ-VAE is added at a chosen attachment point, or a part of the compound is removed. The model retains only modifications that fulfill initial constraints regarding the novelty, diversity, and chemical properties of a generated molecule measured by formerly trained predictive models. The experiments show significantly better performance of the proposed FaST model compared to a diverse set of baseline models.
main_review: Pros:
- VQ-VAE is a natural model for encoding discrete objects and produces more diverse structures, which is confirmed by t-SNE plots.
- The authors use Upper Confidence Bound (UCB) to sample compounds effectively and accelerate the search of compounds with optimized properties.
- There is an interesting discussion on the usability of logP optimization and the flaws of measuring compound similarity using molecular fingerprints.

Cons:
- The text structure is very similar to the MARS paper, and there is an apparent similarity between these two methods, although there are some significant changes.
- One of the similarities is the main results table, which includes metrics that do not correctly emphasize the superiority of one method above others. The success rate (SR), which should be the main objective, is equal (almost) 1 for many models, so it is difficult to say if one method is significantly better than others.
- Besides SR, other metrics describe the novelty and diversity of the generated compounds. These metrics are directly optimized by the proposed model. Are other methods in the comparison trained so that these metrics are maximized? If not, then it is not surprising that FaST wins in this competition. Also, the last metric, PM, is the product of all the other metrics, so suboptimal diversity or novelty scores of other models negatively impact the scoring.
- To confirm the usefulness of the model, a different comparison should be proposed, or more biological targets should be added. One idea would be to create a benchmark that uses uncertainty prediction models. This way, the novel compounds that cannot be reliably predicted by the activity models would be discarded.
- In Section 4, some claims are not supported experimentally in the paper: 
  - "we find that using a VQ-VAE with a categorical prior instead of the typical Gaussian prior makes RL training stable and provides good performance gains" - the FaST variant with the Gaussian prior should be included in the results table,
  - "we find the SELFIES decoder works well empirically" - this could be further discussed in the ablation studies.
- More examples of generated compounds would be welcome.

Other comments:
- In the paragraph about VQ-VAE in Section 3, should it not be $k^d$ instead of $d^k$ distinct fragments?
- typos: "specify the the molecule to keep"
summary_of_the_review: Based on the above comments, I currently recommend rejecting this paper. The novelty of the work lies in incremental changes to existing methods, and I think the modifications are substantial. However, the experiments, in my opinion, do not demonstrate well the impact of these changes.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new reinforcement learning method for molecular optimization. They train VQ-VAE to learn the vocabulary of molecular fragments which enables the optimizer to efficiently search the chemical space. The optimizer consisted of general reinforcement learning and MPNN, and devised sampling schemes to improve performance. By integrating evaluation criteria as a reward function, the proposed model can additionally improve the performance in terms of those criteria. The experiments show not only its superior performance but shorter optimization time.

main_review: Strength:
1. The proposed model outperforms other baselines in the multi-objective molecules optimization benchmark. 
2. Applying VQ-VAE into molecular fragment vocab learning is a properly motivated hypothesis and empirically well-validated. 
3. MolDQN[1] and CMG[2] are other multi-objective optimization models. Better to include them at least in the related work.

[1] Zhou, Zhenpeng, et al. "Optimization of molecules via deep reinforcement learning." Scientific Reports 9.1 (2019): 1-10.
[2] Shin, Bonggun, et al. "Controlled molecule generator for optimizing multiple chemical properties." Proceedings of the Conference on Health, Inference, and Learning. 2021.


Weakness:
1. In the Encoder/Decoder section on page 4, the authors claimed as "we ﬁnd the SELFIES decoder works well empirically". It's not convincing why they choose to use the SELFIES decoder. It'd be good if it's well-motivated and empirically shown. 
2. Minor typo: In eq (6), the outer parenthesis of the diversity may be before the "<" sign?
3. The overall framework is novel, however, it relies on many known methods.
summary_of_the_review:  Overall it is a well-written paper with good motivation, appropriate method, and supportive experiments. The overall framework is novel, however, it relies on many existing methods.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to perform molecular optimization by sequential translation. A reinforcement learning policy is employed to increase the diversity of the generated molecules. The proposed method is shown to be able to generate high-quality molecules on single-property and multi-property optimization tasks. The paper is well written and easy to follow.
main_review: Combing variational auto-encoders and RL policy for fragment-based molecular optimization gives a good practical performance, but the novelty contribution is marginal. Their method of encoding fragments into latent space is not particularly novel and has been applied in other works for similar tasks (e.g. [1][2][3]). Fragment-based drug design is also a very common approach that has been widely explored for a long while from the cheminformatics community, and there is nothing unique about the fragmentation or the definition of molecular editing action in this work (For example, there are three editing actions defined in [4] and seven in [5] ). The diversity-aware reinforcement learning methods are also common in NLP community [6]. FaST introduces a similarity regularization in the reward function to increase the diversity, which makes a little sense but makes it easy for the model to fool your defined metrics (the difference between AP-based and Morgan-based similarity calculations isn't that great). Such a high level of Nov. and SR. is not proof that your model really works, but rather more like some leakage has occurred, as you did with Table 2 in Appendix B (this experiment actually makes a lot of sense, as the logP task has always been considered less suitable for evaluating molecular optimization). 

Minor
1.	It would be better if the authors provide a more intuitive and clear illustration of the motivation of introducing each module in the model (e.g. the VQ-VAE).
2.	Figure 1 is confusing, why does the arrow at the bottom point to a blank space?



[1] Bradshaw, John, et al. "Barking up the right tree: an approach to search over molecule synthesis dags." arXiv preprint arXiv:2012.11522 (2020).

[2] Bradshaw, John, et al. "A model to search for synthesizable molecules." arXiv preprint arXiv:1906.05221 (2019).

[3] Fu, Tianfan, Cao Xiao, and Jimeng Sun. "Core: Automatic molecule optimization using copy & refine strategy." Proceedings of the AAAI Conference on Artificial Intelligence. 2020.

[4] Fu, Tianfan, et al. "MIMOSA: Multi-constraint Molecule Sampling for Molecule Optimization." arXiv preprint arXiv:2010.02318 (2020).

[5] Leguy, Jules, et al. "EvoMol: a flexible and interpretable evolutionary algorithm for unbiased de novo molecular generation." Journal of cheminformatics 12.1 (2020): 1-19.

[6] Shi, Zhan, et al. "Toward diverse text generation with inverse reinforcement learning." arXiv preprint arXiv:1804.11258 (2018).

summary_of_the_review: In general, I do not see too much theoretical or technical contribution. I would request the authors to carefully position their work with the related work mentioned above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The present paper aims to discover novel molecules with desirable properties. One of the main contributions is to compose a molecule fragment-by-fragment, where each fragment is drawn not from a fixed fragment dictionary but from a NN-based pre-trained fragment generator. Other contributions are related to the optimization module using RL techniques; introducing fragment deletion as an action, sequential translation, novelty- and similarity-aware reward design, and initial state distribution.
The proposed method is evaluated by ligand generation tasks, where generated molecules are evaluated in terms of success rate, novelty, and diversity. 
main_review: ## Strengths

### Fragment generation model
 Fragment-based generation with a fragment generation model is novel and seems to be a reasonable generative model of molecules. It successfully inherits the benefit of the existing fragment-based generative model while avoiding from the limited generative capability due to the fixed fragment dictionary. I also like the authors validate this approach empirically in Section 6.

### Realistic benchmark task
 I like the authors working on a realistic benchmark task, in addition to the traditional penalized logP optimization task.

## Weaknesses
I have concerns on the reinforcement learning module and experiments.

### Non-stationary environment
As the authors state in the "implementation details" paragraph, the environment is non-stationary, i.e., both the initial state distribution and the reward function change during training. This non-stationarity is generally not favorable, and should be avoided if possible, because I don't think RL on non-stationary environments is not well understood. I would avoid the non-stationarity by devising the definition of the environment.

### No theoretical justification
The authors employ the UCB score to define the initial state distribution, but I don't see any theoretical background to justify it.

### Reward design
The reward is designed so that the agent can discover molecules with desirable properties, while being novel and diverse enough. On the other hand, the agent aims to maximize the cumulative reward. I don't see a reasonable interpretation of the cumulative reward, and I would like the authors to discuss it.

### Potentially unfair experimental design?
As the authors state in "optimize for different novelty/diversity metrics" paragraph, the baseline models are not novelty/diversity-aware during training. I am not very sure this is a fair comparison, because the existing RL-based method can be novelty/diversity-aware by introducing the proposed reward design (Eq. 7). In my opinion, the main contribution of this paper is the generative model of fragments, and it would be more valuable (to me) if the experiment is designed so that it highlights the benefit of the generative model of fragments by for example using the same RL component across different methods.
summary_of_the_review: My recommendation is weak accept.

I like the idea of a generative model of fragments, and the analysis on its benefit in Section 6. However, I am not very convinced by the RL component, because of three concerns raised above. Therefore, I can't aggressively recommend to accept this paper.

My humble suggestion is to set the generative model of fragments as the main contribution and the RL components as minor ones (or even re-using the existing RL component), because the generative model of fragments itself seems to be novel enough (at least to me) to be presented to the community.
review_title: null

Meta-review: Reviewers agree that the paper is well-motivated and the proposed method is somewhat interesting and well-experimented. However, reviewers feel that the paper relies on many existing methods and does not appear to be novel enough.
Decision: Reject