Can LLMs Generate Diverse Molecules? Towards Alignment with Structural Diversity

Authors: Hyosoon Jang, Yunhui Jang, Jaehyung Kim, Sungsoo Ahn
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Large language model, molecular generative model, drug discovery

Abstract:
Recent advancements in large language models (LLMs) have demonstrated impressive performance in generating molecular structures as drug candidates, which offers significant potential to accelerate drug discovery. However, the current LLMs overlook a critical requirement for drug discovery: proposing a diverse set of molecules. This diversity is essential for improving the chances of finding a viable drug, as it provides alternative molecules that may succeed where others fail in wet-lab or clinical validations. Despite such a need for diversity, the LLMs often output structurally similar molecules from a given prompt. While decoding schemes like beam search may enhance textual diversity, this often does not align with molecular structural diversity. In response, we propose a new method for fine-tuning molecular generative LLMs to autoregressively generate a set of structurally diverse molecules, where each molecule is generated by conditioning on the previously generated molecules. Our approach consists of two stages: (1) supervised fine-tuning to adapt LLMs to autoregressively generate molecules in a sequence and (2) reinforcement learning to maximize structural diversity within the generated molecules. Our experiments show that (1) our fine-tuning approach enables the LLMs to better discover diverse molecules compared to existing decoding schemes and (2) our fine-tuned model outperforms other representative LLMs in generating diverse molecules, including the ones fine-tuned on chemical domains.

Decision: reject
Reviewer scores: 3, 3, 8, 3

Reviews and discussion:
<thread>
creation date:
1730374034754

thread_type:
official_review

<thread_object>
creator: reviewer_xfwi

<content>
summary:
This paper tackles an essential problem in AI-driven drug discovery: generating structurally diverse molecules using LLMs. Addressing the challenge that traditional LLMs often produce structurally similar molecules from a single prompt, the authors propose a two-stage approach: (1) supervised fine-tuning to enable autoregressive mutiple molecule generation, and (2) RL to enhance structural diversity. Through experiments with several LLMs and comparison against various decoding schemes, the proposed approach, Div-SFT+RL, demonstrates an improved ability to generate diverse molecular structures, which is crucial for discovering viable drug candidates. The results indicate that Div-SFT+RL outperforms existing models on diversity-related metrics, supporting its potential for applications in drug discovery.

strengths:
1. **Focus on an Essential Problem**: The paper addresses a critical requirement in drug discovery—molecular diversity in generated candidates. By providing a more diverse set of molecules, the proposed method aligns with real-world drug discovery processes, enhancing the chance of identifying successful compounds.
2. **Clear Presentation**: The paper is well-structured, with a clear exposition of the methodology and experimental procedures.
3. **Detailed Experiments**: The authors perform thorough evaluations across multiple metrics and baselines, including the comparison of Div-SFT+RL with various decoding schemes. The use of IntDiv, and NCircles as diversity-related metrics offers a comprehensive view of the method’s performance.

weaknesses:
1. **Limited Discussion of Related Works on RL for Diversity in Molecular Generation**:
   The paper lacks an in-depth discussion of prior reinforcement learning (RL) approaches that also aim to improve molecular diversity, particularly those using RL for targeted or diverse molecule generation. Works such as Blaschke et al. (2020), Pereira et al. (2021), Hu et al. (2023), and He et al. (2024) provide diverse RL strategies in molecular design, which may not all involve LLMs but offer relevant insights on RL methods and should be discussed to position this work better.

   [1] Blaschke, Thomas, et al. "Memory-assisted reinforcement learning for diverse molecular de novo design." Journal of cheminformatics 12.1 (2020): 68. 

   [2] Pereira, Tiago, et al. "Diversity oriented deep reinforcement learning for targeted molecule generation." Journal of cheminformatics 13.1 (2021): 21. 

   [3] Hu, Xiuyuan, et al. "De novo drug design using reinforcement learning with multiple gpt agents." Advances in Neural Information Processing Systems 36 (2023). 

   [4] He, Jiazhen, et al. "Evaluation of reinforcement learning in transformer-based molecular design." Journal of Cheminformatics 16.1 (2024): 95.

2. **Applicability of the ChEBI-20 Dataset for Diversity Evaluation**:
   The use of the ChEBI-20 dataset raises concerns, as each molecular description in this dataset corresponds to a single molecule. Baseline approaches typically focus on generating one molecule to match the target rather than generating a diverse set for a single description. Furthermore, some descriptions may specify molecular details to the extent that producing diverse structures could be contradictory, making ChEBI-20 potentially unsuitable for assessing molecular diversity.

3. **Reward Function’s Balancing of Terms**:
   The reward function includes two terms: one for structural diversity and another for description matching. It would be beneficial to introduce a coefficient to balance these terms, enabling tuning based on the importance of each aspect (diversity vs. adherence to the description).

4. **NCircles Values in Figure 5**:
   In Figure 5, NCircles values at threshold $h=0.75$ are higher than those at $h=0.65$. This seems inconsistent with the metric’s definition, where values at a higher threshold should logically be lower. The unexpected result suggests potential misinterpretation or misuse of the metric, which requires clarification.

questions:
1. **Cluster Representation in Figure 6**:
   Is it reasonable to represent each cluster as a circle in Figure 6? Given that a 2D projection of a circle in a high-dimensional space may not accurately be circular, I wonder if this visualization accurately reflects clustering in chemical space.
2. **Effect of SMILES Randomness on BLEU Scores**:
   Since a single molecule can be represented by multiple SMILES strings, the randomness in SMILES representations could impact BLEU scores in the experiments. Has this potential source of variability been considered, and what effect does it have on the method’s stability or the evaluation of generated molecules?

In conclusion, this paper offers a valuable approach to improving molecular diversity in AI-driven drug discovery, and I will raise my score if my concerns (in Weaknesses and Questions) are well-addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730481896649

thread_type:
official_review

<thread_object>
creator: reviewer_oku9

<content>
summary:
The paper proposes a fine-tuning method to generate diverse molecules using LLMs. The method involves supervised fine-tuning (SFT) and reinforcement learning (RL).

For SFT, the LLM is prompted to generate a molecule with a given property. The prompt is sampled many times, the generated molecules are filtered and then concatenated to create the SFT training set. The SFT step uses a prompt describing the property and requesting a set of molecules. The set of molecules generated previously is appended to this prompt.

For RL, the SFT prompt is used to generate a sequence of molecules. Every time a molecule is generated, the LLM policy is updated using Proximal Policy Optimization (PPO). The reward is based on how well the molecule matches a property and how different it is from the previous molecules.

strengths:
**Originality**
To my knowledge, the method is novel and the problem is under-researched.

**Quality**
The SFT and RL methods are clear and intuitive. The algorithms are well described in the text and code is provided in Supplementary. Figure 4 helps the understanding of the method. The gap in existing LLMs at generating structurally diverse molecules is well explained in the text and in figure 2. The baselines and metrics are mostly well chosen. The results are convincing, with the authors' method producing significant increase in the chosen metrics.

**Clarity** The methods are explained well. The figures are easy to read. Including different texture in Figure 5 helps the readability. The Appendix with all the experimental details and the code in Supplementary is appreciated and helps reproducibility. I also appreciate the title and claims being straightforward and to the point. 

**Significance** Generating diverse molecules is important in discovery workflows, thus the problem here is significant. The proposed method is easy to understand and implement. The paper also inspires a few interesting follow-ups and thus I think is an important addition to the community.

weaknesses:
1) In the experiments with BioT5+ and MolT5, the authors used the BLEU score on prompts and molecules from ChEBI-20. I think BLEU is a misleading score in the text to molecule task. For example, in BLEU, the order of words in a sentence matters, while there are many ways of writing a molecules as SMILES strings. Also, small changes in a molecule SMILES in the right place can have high changes in a property, such as in the case of hydrogen bond donors and acceptors. I recommend the authors discuss this in their manuscript and compare it with scores generated by RDKit (which they used in the other experiments). 

2) There are a few minor things that could improve the paper: 
    - Figure 7 could have different symbols. It was hard to read in a black-and-white version of the paper.   
    - Worth considering giving the method a name, this will help wider adoption.

questions:
1) Can you improve the explanation of NCircles? I think I get it, but given it's a relatively new method I yet don't have an intuition for it like I do for things like T-SNE or UMAP. What kinds of hyperparameters does it have? What are the conditions for two nodes to be close to each other? Do the overlap of the circles mean anything or are they a consequence of the force-directed algorithm? 

2) In Table 1, you show the outputs of your method for the description "The molecule is a primary aliphatic ammonium ion which is obtained from streptothricin F by protonation of the guanidino and amino groups. It has a role as an antimicrobial agent. It is a guanidinium ion and a primary aliphatic ammonium ion. It is a conjugate acid of a streptothricin F". Are all the generated molecules a conjugate acid of  streptothricin F? Do they have the correct scaffold? This would be important to understand and is related to limitations of the BLEU score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730589203090

thread_type:
official_review

<thread_object>
creator: reviewer_fzz8

<content>
summary:
This paper investigates the limitations of current large language models (LLMs) in generating structurally diverse molecules and proposes a novel two-stage fine-tuning approach to address this challenge. Specifically, the authors apply supervised fine-tuning to enable LLMs to generate a sequence of molecules and subsequently leverage reinforcement learning to enhance structural diversity. Experimental results demonstrate that the proposed method improves molecular diversity compared to existing decoding strategies.

strengths:
This paper identifies a limitation in the molecular generation capabilities of LLMs and proposes a diversity enhancement strategy based on autoregressive generation and reinforcement learning.

weaknesses:
1. The method directly adopts LLMs with reinforcement learning. Thus, the technical novelty of the method is limited.
2. To help readers understand the impact of the approach, the authors could provide additional explanations for the diversity metrics or introduce more similarity evaluation methods.

questions:
1. In Table 3, it is unclear why the comparison is made when the number of generated molecules is different. Additionally, all relevant metrics should be included in the comparison to provide a comprehensive assessment.

2. In Table 4, the authors should specify which large language model (LLM) was used as the basis for fine-tuning and reinforcement learning in the compared methods. Furthermore, when comparing Tables 2 and 4, the performance improvement of the Div-SFT approach appears to be minimal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710293380

thread_type:
official_review

<thread_object>
creator: reviewer_aphu

<content>
summary:
This paper proposes a two-stage fine-tuning approach to address the challenge of generating diverse molecules: (1) supervised fine-tuning to enable LLMs to autoregressively generate molecules in a sequence, and (2) reinforcement learning to enhance structural diversity among the generated molecules.
The contributions of this work include introducing an innovative method that combines supervised learning and reinforcement learning to generate structurally diverse molecules. Additionally, the paper demonstrates that this approach surpasses existing LLMs and traditional decoding methods in producing high-quality, diverse molecular structures.

strengths:
1.This paper addresses an important need in molecular generation by developing a method that enhances structural diversity, a key factor in drug discovery. 
2. The two-stage fine-tuning approach, which combines supervised learning with reinforcement learning (RL) for diversity maximization well-designed solution. 
3. Through empirical testing, the paper shows that its approach outperforms both current LLM-based molecular generators and advanced decoding schemes, highlighting the practical advantages of its method for achieving high-quality, diverse outputs.
4. The proposed method is adaptable and could be extended to other domains requiring diversity in generated outputs, such as protein design or materials discovery. This versatility increases the impact of the work beyond just molecular generation.

weaknesses:
1. The two-stage fine-tuning approach which is highlighted in this paper is not that novel.
2. Although the model is tested on standard datasets and metrics for structural diversity, it lacks experimental or real-world validation to demonstrate the generated molecules’ practical utility in drug discovery.
3. The proposed method’s multi-stage fine-tuning approach may require extensive hyperparameter tuning and setup, which can hinder reproducibility and adoption among researchers.
4. The paper claim "the first to explore the use of LLMs for generating diverse molecule" and "the first propose a fine-tuning approach for LLMs to generate diverse solution", which is overstate.

questions:
See weakness
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this work, authors propose a two-stage fine-tuning approach for large language models (LLMs) to generate diverse molecular structures for drug discovery applications. The method combines supervised fine-tuning to enable autoregressive molecule generation followed by reinforcement learning optimization to maximize structural diversity while maintaining adherence to target properties. The authors evaluate their approach against existing LLMs and decoding schemes using various diversity metrics.

There are several strengths for the work as follows. The paper addresses an important need in computational drug discovery by focusing on molecular diversity generation. The methodology is clearly presented with detailed experimental protocols and comprehensive evaluations across multiple metrics (Reviewer oKU9). The authors provide thorough empirical validation showing improvements over baseline approaches in generating structurally diverse molecules that match target properties. The supplementary materials include code and detailed experimental settings that support reproducibility.

However, there are several limitations for the work as evident from the discussion during the review period. The two-stage fine-tuning approach largely adopts existing methods without substantial innovation (Reviewer APHU, FZz8). While the authors argue their contribution lies in applying these techniques to molecular diversity generation, the core technical components are standard. Reviewer xfWi raises significant issues about the appropriateness of using ChEBI-20 for diversity evaluation, since it maps descriptions to single target molecules. While the authors argue diversity remains valuable even with single targets, this creates a fundamental mismatch between the baseline models' objectives and the proposed evaluation metrics. Multiple reviewers (APHU, FZz8) note that the paper's claims of being ``first'' to explore LLMs for diverse molecule generation are overstated, given existing work in this space. Further, the reliance on BLEU scores for molecular similarity is problematic given the multiple valid SMILES representations for identical molecules (Reviewer oKU9). While the authors added Tanimoto and Dice scores in revision, this highlights initial gaps in the evaluation approach.

Based on these comments, the manuscript cannot be recommended for acceptance at this point. To strengthen future submissions, the authors should: (1) develop a more appropriate evaluation framework that fairly compares diversity-oriented approaches, (2) more precisely position their technical contributions relative to existing work, and (3) strengthen the novelty of their methodological approach beyond standard fine-tuning techniques. More comments below.