Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies molecule generation, proposing a new model using a retrieval mechanism, which has recently shown to be successful in NLP.

The method is evaluated on several established benchmarks and reaches performance comparable to state of the art
strength_and_weaknesses: strengths
- comprehensive evaluation
- new, reasonably motivated model

weaknesses
- ML novelty somewhat limited
- docking as oracle is not ideal
clarity,_quality,_novelty_and_reproducibility: - description is reasonable clear
- novelty is given but limited
- quality good enough for ICLR
- overall, results seems to be reproducible
summary_of_the_review: good evaluation, decent results, limited novelty
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a retrieval-based method for optimizing molecular properties under similarity constraints. The proposed method leverages a pre-trained encoder-decoder generative model and trains an information fusion module to steer the pre-trained model toward generating more similar molecules to the retrieved ones. During training time, the model is asked to predict the nearest molecule in the retrieved ones. And then, during inference time, an iterative refinement process is employed to perform multi-step optimization. The proposed method performed competitively on several molecular optimization tasks using only a small set of exemplar molecules.
strength_and_weaknesses: Strength:
- Data efficiency is a practical issue in the optimization of molecular properties. The proposed method could be used to optimize molecules given a small number of positive examples.
- To my knowledge, the self-supervised training strategy of predicting the nearest neighbors given the remaining neighbors is novel. The task-agnostic training makes it depend less on the number of task-specific samples.

Weaknesses:
- The paper did not justify why such self-supervised training strategy would work since the inference time objective is not aligned with the training time objective.
  - During the training time, the information fusion is trained to predict the nearest molecule given the K-1 remaining nearest neighbors. An information fusion model that adds a small noise to the current molecule would yield a high objective score, which means the information fusion model learns to predict a similar molecule to the current one.
  - However, during inference time, the model is asked to predict a similar molecule but in the direction of the positive examples. The inference time objective is quite different from the training time objective. This is a mismatch that needs further explanation and/or empirical analysis in the paper.
- The design of the iterative refinement approach is ad-hoc, and the description in Section 2.3 seems to miss important details about random perturbing.
  - From my understanding, the model is expected to interpolate between the current molecule and the positive examples in the axis of both structural similarity and property scores. It’s unrealistic to expect the model to extrapolate beyond the database, even if we do iterative refinement. The key mechanism to allow local exploration is random perturbing, mentioned briefly in Section 2.3. However, details are missing here.
- The abstract and intro section gives the readers the wrong impression that the proposed method only requires a small number of samples for a property optimization task. However, the method still requires a large number of samples (number of calls to the property prediction), even in the SARS-COV-2 experiment where there are only 23 positive examples given. Since this paper focuses on sample efficiency, it would be more convincing if the authors could provide the exact numbers of samples used in the proposed method and all baselines for a full comparison. For example, in the multi-property optimization experiment in Section 3.2, the number of samples required by each methods are (copied from the Appendix): 
  - 296, 000 for the proposed method.
  - 2, 750, 000 for MARS [1].
  - 200, 000 for MolEvol [2] (missing in the baselines).
  - xxx for RationaleRL[3].

[1] Xie, Yutong, et al. "MARS: Markov Molecular Sampling for Multi-objective Drug Discovery." International Conference on Learning Representations. 2021.
[2] Chen, Binghong, et al. "Molecule optimization by explainable evolution." International Conference on Learning Representation (ICLR). 2021.
[3] Jin, Wengong, Regina Barzilay, and Tommi Jaakkola. "Multi-objective molecule generation using interpretable substructures." International conference on machine learning. PMLR, 2020.

clarity,_quality,_novelty_and_reproducibility: This paper is mostly clear. The authors did not release the code but promised to release it in the future.
summary_of_the_review: The paper proposed a retrieval-based method for molecular property optimization. Although the proposed method achieved competitive results, the methodology is less principled and needs more work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Motivated by the need for sample-efficient molecule generation, this paper introduces a simple retrieval mechanism to retrieve and fuse the exemplar molecules for controllable molecule generation. Evaluated on a variety of synthetic molecule generation tasks, the proposed method outperforms other baselines. 
strength_and_weaknesses: # Strength

- The experiments are very extensive with lots of details, including both qualitative and quantitative materials.
- It evaluates the effectiveness of simple retrieval modules for controllable molecule generation. 

# Weakness


- Inference via iterative refinement introduced in Section 2.3 is quite heuristic, yet the heuristic is not designed carefully. The authors mention that iterative update is common in approaches such as GA methods, and there are many off-the-shelf GA tools. It seems to me that it’s a better choice to use those tools instead of relying on a design with much simpler heuristics.

- On most benchmark tasks, RetMol does not significantly outperform other baselines, and those baselines methods are usually proposed in 2019. The authors should include recent stronger baselines to justify the value of retrieval modules.
clarity,_quality,_novelty_and_reproducibility: # Clarity
This paper is well-written and easy to follow

# Novelty
The authors fail to provide a comprehensive literature survey on general retrieval-based training methods, especially those in NLP and CV. For example, in “Instance-Conditioned GAN, NeurIPS 2021”, the authors show that a retriever can improve the generation capabilities with very few data points. The authors claim briefly the way the retrieval modules retrieve and integrate the information is slightly different from MSA, but it is not clear to me where are those differences: The retrieval pipeline looks quite standard and the fusion module is just based on the standard attention mechanism. It is true that the authors in this submission focus on applications to bio-related tasks, but the methods used here are not modified accordingly, and thus I think there are very limited technical contributions.

# Reproducibility
The authors do not provide the source code in the supplement., and they do not mention in the paper if they will release their code, making it difficult for others to reproduce this work.
summary_of_the_review: In summary, this paper proposes a retrieval-based method for controllable molecule generation and extensively evaluates its effectiveness on benchmark datasets. The introduction of a retriever for molecule generation is new as far as I know, but it’s not new in other ML communities. However, the design of the retrieval components does not bring in any technical contributions to the community and is mostly based on ungrounded heuristics.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work presents a retrieval-based framework for controllable molecule generation. The framework consists of:

- A mechanism to retrieve a set of molecules that (partially) satisfy a set of desired properties the generated molecule should have (e.g. similarity to the input molecule, enhanced drug-like properties);
- A fusion module that combines the retrieved molecules with the input molecule through cross-attention;
- A novel self-supervised objective to train the fusion module, consisting in reconstructing the nearest neighbor molecule of the input molecule (as opposed to the classical self-reconstruction objective);
- An iterative refinement process to update the retrieval database with the generated molecules.

The fusion module can be plugged-in into any pre-trained encoder-decoder architecture (in this work, they use the SMILES representation of molecules and the ChemFormer model), which is frozen (i.e. only the fusion model is trained for efficiency). 

The proposed approach is evaluated in three controllable generation scenarios, one real-world drug design scenario (find novel inhibitors for SARS-COVID protease), and the GuacaMol benchmark. In every case, it obtains an improvement over a range of different metrics such as success rate or docking score.
strength_and_weaknesses: **Strenghts**

- Strong performances in what appears to be a well-designed set of experiments;
- Training is very efficient (just the retrieval module); inference is not affected too much since we are adding a small number of parameters with respect to the large size of the encoder-decoder backbone;
- Works with a small set of retrieved molecules (up to 23 in the experiments), which is ideal when one wants to generate alternative molecules from limited data;
- The nearest neighbor training objective is a simple but very clever idea that works experimentally.

**Weaknesses**

- Not sure it would be applicable in cases where no exemplar molecules to be retrieved are unavailable (as stated in Conclusions); this might limit its widespread applicability;
- It poorly exploits the chemical space, i.e. it will most likely generate molecules that are very similar to the input molecule.
- Even though it is claimed to work with any encoder-decoder architecture, performances are shown considering only a single backbone model (ChemFormer). I would have expected that its cross-applicability would be one of the main focuses of this paper, but unfortunately it is not.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

This work is extremely well written. It was a pleasure to read from top to bottom.

**Quality**

The quality of this work is high. The choice of experiments is sensible, several ablations and analyses are presented to understand the impact of the modeling choices, all the claims made are substantiated.

**Novelty**

The paper applies well-known ideas from retrieval-augmented language models to the space of molecules, which does qualify as novel to me. Plus, the nearest-neighbor training objective is a novel idea (to my knowledge). 

**Reproducibility**

Not reproducible. The authors state that the code will be released in the future (upon approval, I suspect).
summary_of_the_review: This appears to me as a strong submission. I am giving a score of 6 for the time being. Will be happy to raise my score when the authors answer the following set of questions/comments: 

- In section 2, you acknowledge the existence of weighting coefficients $w_{\ell}$. However, they don't seem to be present in the rest of the work. What are they (or, are they just construction to explain the problem)?
- What is the justification for having used ChemFormer instead of any other encoder-decoder architecture as the backbone model? It would be great to see RetMol applied to another backbone, to see if it works across backbones or if it is just a perfect match with ChemFormer.
- In which order are the constraints removed to relax the feasible set (Sec. 2.1 - Molecule retriever)? It would be interesting to know the impact of using different ordering strategies.
- In (Sec. 2.1 - Information fusion), I see that the set of retrieved exemplars is a matrix $\in \mathbb{R}^{(\sum_{k=1}^K L_k) \times D}$, which means that you concatenate all the embeddings of the retrieved molecules. Considering that they might have different dimensions, how do you handle the mismatch? Padding? Also, doesn't this introduce an unnecessary dependence on the order in which the molecules are retrieved (which could have been avoided, e.g. by a permutation invariant aggregation)?
- I understand the model is very efficient to train. But what about preprocessing (i.e. computing all the embeddings and the pairwise similarities)? What is its impact?
- This is for my understanding: do you confirm that the iterative refinement is applied only at inference, but not during training?
- Due to the large standard deviation in Table 1b, I think it's better to also report the mode of the penalized logP values in addition to the mean.
- How come uniqueness is sometimes not reported in the generation experiments (e.g. Table 2, Table 4-right)? 


**EDIT**

After a successful rebuttal, I am raising my score to 8, since the method is sufficiently new, has good experimental results, and has wide applicability across different molecular generators.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a retrieval-based approach for generating molecules. The method is similar to retrieval-based language modelling. The pre-trained generation model is used to predict the nearest molecules satisfying desired properties (drug-likeness etc). The retrieved molecules are then fused to iteratively update the generation. The proposed approach is evaluated in three controllable generation scenarios, one real-world drug design scenario (for SARS-COVID protease), and the GuacaMol benchmark. The method achieves improvement over a range of different metrics such as success rate or docking score.

Strengths of the paper:
1. Novel application of retrieval-based generation in molecule domain
2. Iterative refinement with retrieved nearest molecules. 
3. comprehensive experiments on a variety of targets. 

Weakness of the paper: 
1. Justification of self-supervised training
2. the experiment shows it improves the diversity and novelty of generated molecules, which is a bit counter-intuitive to retrieval-based methods. From the intuition, the method should generate molecules that are very similar to the input molecules.
3. design of iterative refinement can be improved. 

Overall, the paper has descent technical contribution, experimental support, and is well-written. 
justification_for_why_not_higher_score: The paper may be only interesting to a particular domain.
justification_for_why_not_lower_score: All reviewers appreciate the paper's contribution and discussions with the authors.
Decision: Accept: notable-top-25%