Can LLMs Generate Diverse Molecules? Towards Alignment with Structural Diversity

## Abstract

Abstract Recent advancements in large language models (LLMs) have demonstrated impressive performance in generating molecular structures as drug candidates, which offers significant potential to accelerate drug discovery. However, the current LLMs overlook a critical requirement for drug discovery: proposing a diverse set of molecules. This diversity is essential for improving the chances of finding a viable drug, as it provides alternative molecules that may succeed where others fail in wet-lab or clinical validations. Despite such a need for diversity, the LLMs often output structurally similar molecules from a given prompt. While decoding schemes like beam search may enhance textual diversity, this often does not align with molecular structural diversity. In response, we propose a new method for fine-tuning molecular generative LLMs to autoregressively generate a set of structurally diverse molecules , where each molecule is generated by conditioning on the previously generated molecules. Our approach consists of two stages: (1) supervised fine-tuning to adapt LLMs to autoregressively generate molecules in a sequence and (2) reinforcement learning to maximize structural diversity within the generated molecules. Our experiments show that (1) our fine-tuning approach enables the LLMs to better discover diverse molecules compared to existing decoding schemes and (2) our fine-tuned model outperforms other representative LLMs in generating diverse molecules, including the ones fine-tuned on chemical domains.

## 1 Introduction

Figure: Figure 1: LLMs generate a molecule with SMILES representation.
Refer to caption: x1.png

Recent advances in large language models (LLMs) have demonstrated remarkable potential to accelerate scientific discovery by leveraging their language processing capabilities. This progress has been particularly impactful for candidate design problems such as drug discovery (Pei et al., ), protein sequence design (Zhuo et al., ), and material design (Gruver et al., ). In particular, with expansive biomolecular datasets and molecular string representations, e.g., SMILES (Weininger, ) or SELFIES (Krenn et al., ), LLMs have demonstrated impressive abilities to generate molecules from textual descriptions, e.g., molecular properties (Edwards et al., ; Ye et al., ; Pei et al., ), as illustrated in Figure 1.

However, current LLM-based molecular generation approaches (Edwards et al., ; Ye et al., ; Pei et al., ) often overlook a critical requirement for drug discovery: proposing a diverse set of molecules. In computer-aided drug discovery pipelines, identifying a single molecule with a desired property does not guarantee success in real-world pipelines that require additional cell-based studies and clinical trials (Vamathevan et al., ). Therefore, drug discovery requires a collection of structurally diverse molecules, as illustrated in Figure 2. The generation of structurally diverse molecules increases the chances of finding a viable drug candidate (Xie et al., ), as different molecules may succeed where others fail. This diversity is essential to enhance the robustness and success of the drug discovery process (Krantz, ; Hong et al., ; Sadybekov & Katritch, ).

Figure: Figure 2: The compute-aided drug design. A collection of structurally diverse molecules is required to increase the chance of identifying viable drug candidates in the real world.
Refer to caption: x2.png

In response, we explore the use of LLMs for diverse molecular generation. We begin by identifying the limitations of recent LLMs (Ye et al., ; OpenAI, ) and decoding schemes (Vijayakumar et al., ; Su et al., ) in generating diverse molecules. Then, we present a new method for fine-tuning LLMs to generate diverse molecules. Our approach can be broadly applied to other LLM-based candidate design problems, e.g., computer-aided design (Wu et al., ).

Existing LLMs have limitations in generating diverse molecules. To obtain diverse molecules, one may consider querying the recently developed generalist LLMs, e.g., Llama (Touvron et al., ) or ChatGPT (OpenAI, ). However, our empirical observation in Figure 3(a) reveals that even the most recent models produce structurally identical or highly similar molecules from the given prompt.(^1^11ChatGPT-4o (OpenAI, ) generates different SMILES strings that map to an identical molecule.) This observation aligns with previous observations that have shown LLMs may fail to generate diverse outputs (Kirk et al., ) for general text-based domains.

Decoding schemes for diversified generation do not align with molecular diversity. We also acknowledge the existence of decoding schemes, e.g., diverse beam search (Vijayakumar et al., ) or contrastive beam search (Su et al., ), which have been proposed to improve the diversity of output sequences generated by LLMs. However, these decoding schemes are limited to improving the textual diversity which often does not correspond to molecular structural diversity, e.g., there exist many SMILES or SELFIES strings that correspond to the same molecule, as illustrated in Figure 3(b).

Figure: (a) Existing LLMs (Ye et al., ; OpenAI, ) lack the ability to generate a diverse set of molecules.
Refer to caption: x3.png

Our approach. We repurpose existing molecular generative LLMs to autoregressively generate a diverse set of molecules from a single prompt. By enabling the LLMs to generate a new molecule conditioned on previously generated molecules, we expect the LLMs to learn to enhance the structural diversity between the generated molecules. To this end, we propose a two-stage approach to fine-tune LLMs: (a) a supervised fine-tuning stage to repurpose LLMs to autoregressively generate a sequence of multiple molecules and (b) a reinforcement learning stage to maximize the molecular structural diversity. Note that both stages do not require external diverse molecular datasets and are purely based on self-improvement procedures, where the LLMs train on the samples generated by themselves.

In the supervised training stage, we train the LLMs to autoregressively generate a set of molecules in a single sequence. The training dataset, i.e., a set of molecules, is collected from the LLMs themselves through iterative sampling, and then filtered to enhance the quality, e.g., removing invalid molecules. However, this stage does not necessarily incorporate molecular diversity, as the training may not involve sufficiently distinct molecules (e.g., limitations in Figure 3(b)). To tackle this, we subsequently apply reinforcement learning with exploration towards discovering diverse molecules.

Next, in the reinforcement learning stage, we train LLMs to maximize the diversity of molecules within a generated sequence. However, for our task, conventional sequence-wise reinforcement learning (Ouyang et al., ) suffers from the credit assignment problem (Zhou et al., ): the challenges in identifying and promoting the generation of molecules responsible for increasing diversity, among a larger set of molecules in the sequence. To resolve this issue, we solve multi-stage molecule generation problems for a sequence of molecules, where the generation of each molecule aims to maximize the diversity with respect to the previously generated molecules. We train LLMs to maximize the associated rewards using proximal policy optimization (Schulman et al., ).

We compare our method with the decoding schemes for diversified generation (Vijayakumar et al., ; Su et al., ) and other representative LLMs, including chemical-task specialists (Edwards et al., ; Christofidellis et al., ; Pei et al., ; ), fine-tuned generalists on chemical domains (Fang et al., ; Yu et al., ), and the ChatGPT series (OpenAI, ; ). We observe that (1) our fine-tuning approach enables LLMs to better discover diverse molecules compared to existing decoding schemes and (2) our fine-tuned LLM outperforms other recent LLMs.

To conclude, our contributions can be summarized as follows:

- •
We are the first to explore the use of LLMs for generating diverse molecules.
- •
We first propose a fine-tuning approach for LLMs to generate diverse solutions, which presents a new direction distinct from existing approaches focused on the decoding scheme.
- •
Experimentally, our method outperforms the baselines in generating diverse molecular structures.

## 2 Related work

Large language models (LLMs) for molecular generation. Recent advancements in LLMs have shown increasing promise in scientific applications, especially for molecular generation (Edwards et al., ; Pei et al., ; Fang et al., ; Pei et al., ). First, Edwards et al. () proposed MolT5 which translates between SMILES (Weininger, ) and molecular text descriptions. Text+Chem T5 (Christofidellis et al., ) is a model pre-trained on both the chemical and natural language domains. Next, BioT5 (Pei et al., ) considers T5 models pre-trained on datasets including bio-text, protein sequences, and molecules. Additionally, Ye et al. (), Fang et al. (), and Yu et al. () fine-tuned generalist LLMs, e.g., Llama (Touvron et al., ), through biological instructions, molecular modifications, and large-scale molecular datasets, respectively.

Decoding schemes for generating diverse output sequences. To generate diverse and high-quality solution candidates from LLMs, existing literature on LLMs has studied improving decoding schemes. To acquire multiple distinct sequences with high likelihoods, one can consider employing beam search, which jointly decodes multiple distinct outputs (Och, ). To enhance diversity between sequences, Vijayakumar et al. (; ) incorporated token-wise differences between generated sequences in the beam search. Furthermore, Su et al. () considered the contrast between the candidate sequences. In addition, Holtzman et al. () proposed nucleus sampling, which enhances random sampling by balancing the quality and the diversity.

Reinforcement learning (RL) for fine-tuning LLMs. RL has been effectively applied to fine-tune LLMs, aligning them with desired behaviors expressed through reward signals. One notable example is RL from human feedback to align LLMs with human preference (Ouyang et al., ). In addition, there has been a surge in research on devising RL for LLMs as well, such as addressing multi-turn settings (Shani et al., ) and incorporating multiple fine-grained reward signals (Wu et al., ). For molecular generation, Ghugare et al. () proposed RL-based fine-tuning to generate a molecule satisfying target properties. However, to the best of our knowledge, there exist no prior RL-based approaches that aim to increase the diversity of LLM-generated outputs.

Figure: (a) Supervised fine-tuning: enables generating a sequence of multiple molecules.
Refer to caption: x5.png

## 3 Method

In this section, we present our method for fine-tuning LLMs to generate diverse molecules. Specifically, we consider applying fine-tuning to existing molecular generative LLMs that produce molecular representations such as SMILES or SELFIES. Importantly, our approach is versatile and can be broadly applied to other domains, e.g., protein sequence (Zhuo et al., ) or computer-aided design (Wu et al., ). Furthermore, it leverages self-improvement techniques and does not require additional datasets containing diverse molecules.

Overview. Our goal is to generate a sequence of structurally diverse molecules from a given prompt by producing them in a single concatenated output. To achieve this, we fine-tune the LLMs in two stages: (a) a supervised fine-tuning phase that repurposes the LLMs to generate a sequence of molecules rather than a single one, and (b) a reinforcement learning phase aimed at further enhancing the structural diversity among the generated molecules.

Task details. In detail, we consider generating molecules from a prompt $p_{\text{desc}}$, where the prompt describes a molecular property that the generated molecules should possess. Here, we aim to generate diverse molecules that satisfy the given description $p_{\text{desc}}$. We let $\mathcal{P}$ denote the prompts used for training, which does not intersect with the ones used for evaluation.

Figure: Algorithm 1 Supervised fine-tuning

### 3.1 Supervised fine-tuning

We first describe our supervised fine-tuning process for repurposing the pre-trained LLMs to autoregressively generate multiple molecules in a sequence. This involves collecting a dataset of molecules from a pre-trained LLM $\pi_{\text{pre}}$, and then fine-tuning the LLM $\pi_{\text{SFT}}$ on the collected dataset. We describe the process in Figure 4(a) and Algorithm 1.

Dataset collection. The dataset collection and supervised training processes are conducted with a set of training prompts $\mathcal{P}$. Initially, the pre-trained LLM $\pi_{\text{pre}}$ produces a set of molecules by iterative sampling molecules for a given prompt $p_{\text{desc}}\in\mathcal{P}$ as follows:

$$ $m_{i}\sim\pi_{\text{pre}}\left(m_{i}\middle|p_{\text{desc}}\right)\quad\text{ for}\;i=1,\ldots,T,$ $$

where $m_{i}$ denotes the string representation of the molecule. In practice, we employ beam search to collect the set of molecules $\{m_{i}\}^{T}_{i=1}$. Then, we filter out the invalid string representations, duplicate molecules, and molecules that do not satisfy the given prompt $p_{\text{desc}}$. This results in reducing the set of molecules from $\{m_{i}\}^{T}_{i=1}$ to $\{m_{i}\}^{K}_{i=1}$. The details are described in Appendix A.

Supervised training. Given the filtered set of molecules $\{m_{i}\}^{K}_{i=1}$, we train the LLM $\pi_{\text{SFT}}$, which is initialized with $\pi_{\text{pre}}$, to generate them as a single concatenated sequence. We denote this sequence by $\mathcal{M}_{1:K}=m_{1}||\cdots||m_{K}$, where $||$ denotes the concatenation of the molecular string representations. Specifically, given a modified prompt $p_{\text{desc+div}}$ describing the target property with a request for generating diverse molecules, we train the LLM to maximize the log-likelihood:

$$ $\log\pi_{\text{SFT}}\left(\mathcal{M}_{1:K}\;\middle|\;p_{\text{desc+div}} \right).$ (1) $$

However, the policy $\pi_{\text{SFT}}$ does not necessarily incorporate a molecular structural diversity, as the set of molecules $\{m_{i}\}^{K}_{i=1}$ collected from $\pi_{\text{pre}}$ may insufficiently involve diverse molecular structures (e.g., due to limitations in Figure 3(b)). To tackle this, we next introduce an online reinforcement learning stage with exploration towards discovering diverse molecules.

### 3.2 Reinforcement learning

Figure: Algorithm 2 Multi-stage RL fine-tuning

We apply reinforcement learning to maximize the diversity of the generated molecules within a sequence. However, when applied to a sequence of molecules $\mathcal{M}_{1:K}$, conventional sequence-wise reinforcement learning (Ouyang et al., ) suffers from the credit assignment problem (Zhou et al., ): the challenge in identifying and promoting the generation of molecules responsible for increasing diversity, among a set of molecules $\{m_{i}\}^{K}_{i=1}$. To circumvent this, we introduce a molecule-wise reinforcement learning.(^2^22We compare both approaches in Table 4 of Section 4.4.)

Specifically, we consider reinforcement learning on a sequence of molecules $\mathcal{M}_{1:K}$ as learning in $K$ individual stages. Each stage corresponds to generating a molecule $m_{k}$ conditioned on a sequence of previously generated molecules $\mathcal{M}_{1:k-1}$. Then, the LLM $\pi_{\text{RL}}$ is trained to maximize the return of each stage, which is defined by the reward of the generated molecule $m_{k}$.
The reward is defined as the diversity between the previously generated molecules $\{m_{i}\}_{i=1}^{k-1}$ and the new molecule $m_{k}$. We also incorporate an auxiliary reward to ensure that the molecule $m_{k}$ satisfies the given description $p_{\text{desc}}$. We illustrate our approach in Figure 4(b) and provide the detailed algorithm in Algorithm 2.

Reward. The reward evaluates the generated molecule $m_{k}$ with a diversity reward $r_{\text{div}}(m_{k},\{m_{i}\}^{k-1}_{i=1})$ and a description-matching reward $r_{\text{match}}(m_{k},p_{\text{desc}})$, as follows:

$$ $r(m_{k})=r_{\text{div}}(m_{k},\{m_{i}\}^{k-1}_{i=1})+r_{\text{match}}(m_{k},p_ {\text{desc}}),$ $$

where the diversity reward $r_{\text{div}}$ evaluates structural differences between the molecule $m_{k}$ and the previously generated molecules $\{m_{i}\}^{k-1}_{i=1}$ by assessing their true molecular structures. Note that $r_{\text{div}}(m_{1})$ is zero. The description-matching reward $r_{\text{match}}$ evaluates whether the molecule $m_{k}$ satisfies the given description $p_{\text{desc}}$. In a practical implementation, the final action that completes the molecule $m_{k}$ yields the reward $r(m_{k})$. The detailed reward implementation is described in Appendix A.

Policy optimization. We optimize the LLM $\pi_{\text{RL}}$ to maximize the reward using proximal policy optimization (PPO; Schulman et al., ; Ouyang et al., ). Note that $\pi_{\text{RL}}$ is initialized with $\pi_{\text{SFT}}$. In addition, we combine per-token KL penalty from the supervised fine-tuned model at each token following prior studies (Ouyang et al., ).

## 4 Experiment

In this section, we validate our supervised fine-tuning and reinforcement learning methods for generating diverse molecules, coined Div-SFT and Div-SFT+RL, respectively. In these experiments, one can observe that (1) our fine-tuning approach enables LLMs to better discover diverse molecules compared to decoding schemes for diversified generation and (2) our fine-tuned LLM outperforms other representative LLMs, including generalist and specialist models for chemical tasks.

### 4.1 Comparison with decoding schemes

In this experiment, we show that our method enables LLMs to better generate diverse molecules compared to the existing decoding schemes for diverse sequence generation. To validate the consistent improvement, we implement our fine-tuning method and decoding schemes on two models specialized in the chemical domain: BioT5^+ (Pei et al., ) and MolT5 (Edwards et al., ).

Figure: (a) Comparison with decoding schemes on BioT5^+
Refer to caption: x8.png

Tasks. We consider description-guided molecule generation using the ChEBI-20 dataset, which includes training and test sets (Edwards et al., ). Note that the training dataset has also been used to pre-train the base LLMs, BioT5^+ and MolT5, i.e., not an external dataset. The test dataset is unobserved during both the pre-training and fine-tuning phases. Each data point consists of a pair of a description and a ground-truth example molecule that satisfies the description.

In our experiments, we consider generating $50$ molecules for each description in the ChEBI-20 test dataset involving $3300$ molecular descriptions. We then evaluate both the structural diversity of the generated molecules and how well they satisfy the given descriptions.

Metrics. To measure the similarity between two molecules, we use Tanimoto similarity (Bajusz et al., ), which based on distance of molecular structures from fingerprints. To be specific, we consider the following metrics for generated molecules:

- •
The number of accepted and unique molecules (Accepted & unique): This metric measures the number of valid and unique molecules satisfying the given description.(^3^33Following the evaluation metric of prior studies (Edwards et al., ; Pei et al., ), we evaluate the molecule as satisfying the given description when its BLEU score between a ground-truth sample is high ($>0.7$).)
- •
The number of circles (NCircles; Xie et al., ): This metric considers both the quality and diversity of generated molecules. For the accepted molecules satisfying the given description, this metric counts the number of maximum distinct clusters with Tanimoto similarity (Bajusz et al., ) below a threshold $h$, i.e., indicating chemical space coverage (illustrated in Figure 6).
- •
Internal diversity (IntDiv.; Polykovskiy et al., ): This metric is the complement of the average of pair-wise Tanimoto similarities between the accepted molecules satisfying the given description.
- •
Average of top 10 scores (Top 10): This metric measures the quality of the generated molecules by averaging the BLEU scores (Papineni et al., ) of the top unique $10$ molecules yielding high BLEU scores between the ground-truth example.

Figure: (a) BS (22,11)
Refer to caption: x10.png

**Table 1: Visualization of the generated molecules with their diversity. The eight molecules are generated with the description: “The molecule is a primary aliphatic ammonium ion which is obtained from streptothricin F by protonation of the guanidino and amino groups. It has a role as an antimicrobial agent. It is a guanidinium ion and a primary aliphatic ammonium ion. It is a conjugate acid of a streptothricin F’.” The generated molecule with blue line indicates the accepted molecule.**
| Method<br>(IntDiv.) | Example of outputs |
| --- | --- |
| BS<br>(0.51) |  |
| Diverse BS<br>(0.28) |  |
| Contrastive BS<br>(0.53) |  |
| Div-SFT+RL<br>(0.69) |  |

Implementations. For supervised fine-tuning, we collect a hundred molecules for each molecular description $p_{\text{desc}}$ in ChEBI-20 training sets. Then, they are filtered to remove invalid molecules, duplicated molecules, and unaccepted molecules. The description-matching reward $r_{\text{match}}(m_{k})$ is defined as a BLEU score between a ground-truth example. The diversity reward $r_{\text{div}}(m_{k},\{m_{i}\}^{k-1}_{i=1})$ is defined as the complement of the maximum Tanimoto similarity between generated molecules $\{m_{i}\}^{k-1}_{i=1}$. The detailed implementations and hyper-parameters follow Appendix A.

Baselines. We compare our fine-tuning approach with various decoding schemes. We consider the random sampling with different temperatures $\{0.7,1.0,1.5\}$ and nucleus sampling (Holtzman et al., ). We also consider beam search (BS) and the variants of BS to promote sequence-level diversity between the generated samples: diverse BS (Vijayakumar et al., ) and contrastive BS (Su et al., ). The detailed settings are described in Appendix B.

Results. We present the main results on BioT5^+ and MolT5 in Figure 5(a) and Figure 5(b), respectively. One can see that our approach, i.e., Div-SFT+RL, shows superior performance compared to the considered baselines. Especially, it is worth noting that our approach makes significant improvements in NCircles metrics that require generating diverse molecules satisfying the given description.

We also provide the qualitative results in Figure 6 and Table 13 with sampled molecules. One can see that variants of beam search for diverse sentence generation, i.e., diverse BS and contrastive BS, do not enhance molecular structural diversity. In contrast, our approach demonstrates significantly higher molecular structural diversity. In Appendix C, we provide additional examples of molecules generated by the considered baselines and our method.

**Table 2: Comparison with existing LLMs on description-based molecule generation. Our fine-tuned model shows superior performance compared to the considered baselines.**
| Method | Accepted & Unique | NCirclesh=0.75 | NCirclesh=0.65 | IntDiv. | Top 10 |
| --- | --- | --- | --- | --- | --- |
| Chemical-task Specialist LLMs |  |  |  |  |  |
| MolT5 | 3.12 | 1.85 | 1.59 | 0.29 | 0.41 |
| Text+Chem T5 | 13.78 | 4.82 | 3.29 | 0.23 | 0.63 |
| BioT5 | 15.58 | 8.45 | 6.07 | 0.33 | 0.69 |
| BioT5^+ | 16.03 | 7.93 | 6.16 | 0.31 | 0.63 |
| Generalist LLMs |  |  |  |  |  |
| Mol-instructions | 0.06 | 0.03 | 0.03 | 0.02 | 0.01 |
| LlaSMol | 17.68 | 6.73 | 4.94 | 0.26 | 0.68 |
| GPT-3.5-turbo | 3.87 | 2.83 | 2.49 | 0.20 | 0.41 |
| GPT-4o | 5.74 | 4.09 | 3.53 | 0.22 | 0.48 |
| o1-preview | 6.92 | 3.46 | 2.36 | 0.13 | 0.31 |
| BioT5^++ours | 21.98 | 16.98 | 14.35 | 0.45 | 0.74 |

### 4.2 Comparison with existing LLMs

Here, we compare our fine-tuned BioT5^+, presented in Figure 5, with various recent LLMs, including chemical-task specialists, generalists, and fine-tuned generalist LLMs on chemical domains. The purpose of this experiment is to highlight the limitations of existing LLMs in capturing molecular diversity, whereas our fine-tuned model successfully captures molecular diversity during generation. The tasks and metrics are the same as settings in Section 4.1. We use the first $500$ molecular descriptions in the ChEBI-20 test dataset for evaluation.

Baselines. We compare our fine-tuned model with various existing LLMs. We consider four LLMs specialized for chemical tasks: Text+Chem T5 (Christofidellis et al., ), MolT5, BioT5 (Pei et al., ), and BioT5^+. Next, we consider two generalist LLMs: Mol-instructions (Fang et al., ) fine-tuned from Llama-7B (Touvron et al., ), and LlaSMol (Yu et al., ) fine-tuned from Mistral-7B (AI, ), based on description-based molecular generation tasks. For each baseline, we report the best results (highest NCirclesh=0.65) obtained using either random sampling, beam search, diverse beam search, or contrastive beam search. We also consider three strong API-based generalist LLMs: GPT-3.5, GPT-4o, and o1-preview (OpenAI, ; ).(^4^44We use gpt-3.5-turbo-0125 and gpt-4o-2024-08-06.) The detailed experimental settings and prompts are described in Appendix B.

Results. We present the results in Table 2. One can see that our fine-tuned model shows superior performance compared to the considered baselines in discovering diverse and high-quality molecules. Furthermore, it is noteworthy that most existing LLMs yield low NCircles with respect to the number of generated accepted and unique molecules, while our method yields relatively high NCircles.

### 4.3 Fine-tuning generalist LLMs

We further validate whether our fine-tuning method improves the generalist LLMs as well in terms of the diversity of generated molecules. Here, as a base LLM for implementing our method, we consider DrugAssist (Ye et al., ) which is fine-tuned from the Llama-7B. As baselines, we apply random sampling and contrastive beam search to DrugAssist.

Tasks. In this experiment, we consider prompts based on the four quantitative molecule properties: hydrogen bond (HB) donors, HB acceptors, Bertz complexity (Bertz, ), and quantitative estimate of drug-likeness (QED) (Bickerton et al., ). The goal of this task is to generate diverse molecules that satisfy the properties described in the given prompt.

Implementations. We apply our supervised fine-tuning and reinforcement learning to enhance DrugAssist. For supervised fine-tuning, we collect multiple molecules for prompts about three properties: HB donors, HB acceptors, and Bertz complexity. The prompts about QED are excluded from training but included in the evaluation to assess generalization to unseen properties. The sampled molecules are filtered to remove invalid molecules, duplicated molecules, and molecules that do not satisfy the given properties. The property-match reward $r_{\text{match}}(m_{k})$ yields a non-zero value when satisfying the given property, and the diversity reward $r_{\text{div}}(m_{k},\{m_{i}\}^{k-1}_{i=1})$ is defined as same as Section 4.1. The detailed implementations and hyper-parameters follow Appendix A.

Figure: (a) HB donors
Refer to caption: x18.png

Results. We present the results in Figure 7. One can see that our approach consistently improves the performance in generating diverse and high-quality molecules when implemented on generalist LLMs. Note that our approach consistently demonstrates superior performance for the unseen prompt, i.e., the prompt about QED in Figure 7(d).

**Table 3: Experiments with the large number of samples. Our method discovers more diverse molecules with respect to the (1) the number of generations and (2) time costs.**
| Method${}_{\text{num. of generations}}$ | ${\text{BS}_{300}}$ | ${\text{BS}_{400}}$ | ${\text{BS}_{500}}$ | ${\text{Ours}_{85}}$ | ${\text{Ours}_{120}}$ | ${\text{Ours}_{155}}$ |
| --- | --- | --- | --- | --- | --- | --- |
| NCirclesh=0.65 | 18.6 | 20.4 | 21.3 | 20.4 | 23.6 | 25.6 |
| Time (sec.) | 323 | 452 | 585 | 65 | 86 | 107 |

### 4.4 Ablation studies

**Table 4: Comparison with variants of implementations. Applying multi-stage reinforcement learning shows superior performance compared to (1) supervised fine-tuning with hard filtering and (2) single-stage reinforcement learning.**
| Method | Accepted & Unique | NCirclesh=0.75 | NCirclesh=0.65 | IntDiv. | Top 10 |
| --- | --- | --- | --- | --- | --- |
| Div-SFT${}_{\text{hard}}$ | 9.23 | 7.46 | 6.15 | 0.33 | 0.59 |
| Div-SFT+RL${}_{\text{single}}$ | 14.16 | 8.50 | 6.49 | 0.29 | 0.68 |
| Div-SFT+RL | 21.98 | 16.98 | 14.35 | 0.45 | 0.74 |

Large number of samples vs. performance. We also analyze how well our method discovers diverse molecules with respect to the number of generations. In this experiment, we extend beyond the settings in Section 4.1. We consider our method to generate $85$, $120$, and $155$ molecules with a single NVIDIA A100 SXM4 40GB GPU. Next, we consider beam search with beam sizes of $300$, $400$, and $500$, respectively. The beam search is implemented with four NVIDIA A100 SXM4 40GB GPUs due to the memory limitation of a single GPU. Note that this experiment uses $250$ molecular descriptions in the ChEBI-20 test set. We present the results in Table 3. One can see that our method (1) discovers more diverse molecules with respect to the total number of generations and (2) exhibits further performance improvements as the number of generations increases.

Time costs vs. performance. In addition, we also analyze how well our method discovers diverse molecules with respect to the time costs. In Table 3, we present the time costs for each method. One can see that our method discovers more diverse molecules in a practical time compared to the decoding schemes for diverse sequence generation.

SFT with hard filtering vs. RL. To train LLMs to generate diverse molecules, one may also consider supervised fine-tuning on hard-filtered datasets, e.g., using a set of molecules filtered by similarity, as an alternative to reinforcement learning. In this experiment, we additionally perform supervised fine-tuning with distinct molecules, where each pair of molecules yields a Tanimoto similarity below $0.65$. The results are presented in Table 4. One can see that while it yields relatively high NCircles with respect to the number of accepted unique molecules, the overall performance is worse compared to the performance of reinforcement learning.

Single-stage vs. multi-stage RL. As mentioned in Section 3.2, we consider a multi-stage setting for generating multiple molecules. However, one may also consider a single-stage setting, where the return of a generated sequence is defined as the sum of the rewards from multiple generated molecules. In Table 4, we compare both approaches. One can see that the multi-stage setting significantly outperforms the single-stage setting. We hypothesize that this result stems from credit assignment issues in the single-stage setting. Namely, the single-stage setting lacks signals to capture molecule-wise impacts on diversity among a large set of molecules and fails to promote the generation of molecules responsible for increasing diversity.

## 5 Conclusion

In this paper, we identify the limitations of large language models (LLMs) for generating diverse molecules. Then, we present new supervised fine-tuning and reinforcement learning methods to adapt existing molecular generative LLMs to generate a diverse set of molecules. Experiments show that our approach enables LLMs to better discover diverse molecules compared to the existing approaches. An interesting avenue for future work is to reduce the space and time complexity in generating the sequence of molecules, for example, based on set encoding techniques (Zaheer et al., ).

## Appendix A Detailed implementations and training

### A.1 Supervised fine-tuning

Dataset Collection. To fine-tune BioT5^+ and MolT5 (Sections 4.1 and 4.2), we collect $T=100$ molecules using beam search for each training molecular description in the ChEBI-20 training dataset. Note that this dataset has been considered in the original BioT5^+ and MolT5.

For the fine-tuning of DrugAssist (Section 4.3), we collect $T=300$ molecules using beam search for each training prompt. Note that the collected molecules were filtered to remove invalid string representations, duplicated molecules, and molecules that do not satisfy the given description. The invalid string representations are evaluated with RDKit package (Landrum, ). Additionally, the collected molecules are concatenated into a single sequence $m_{1}||\cdots||m_{K}$. Note that two molecules are separated by introducing a newline character (‘\n’).

Supervised learning. We consider four NVIDIA A100 GPUs for supervised fine-tuning.

- •
For the supervised fine-tuning of BioT5^+ and MolT5 (Sections 4.1 and 4.2), we consider $80$ epochs, $8$-batch size, $5\textrm{e}-4$ learning rate, $0.05$ warm-up ratio, and apply a cosine learning scheduler. The maximum sequence length in supervised training is limited to $2560$ due to memory limitations.
- •
For the supervised fine-tuning of DrugAssist (Section 4.3), we consider $80$ epochs, $4$-batch size, $3\textrm{e}-5$ learning rate, $0.05$ warm-up ratio, and apply a cosine learning scheduler. The maximum sequence length in supervised training is limited to $1024$ due to memory limitations. We also apply LoRA (Hu et al., ), where the rank and alpha are $64$ and $128$, respectively.

### A.2 Reinforcement learning

Reward Design. In experiments with BioT5^+ and MolT5 on the ChEBI-20 dataset (Sections 4.1 and 4.2), we define the description-matching reward using the BLEU score as follows:

$$ $r_{\text{match}}(m_{k},p_{\text{desc}})=\text{BLEU}(m_{p_{\text{mol}}},m_{k})^ {\alpha},$ (2) $$

where $m_{p_{\text{mol}}}$ is a ground-truth molecule satisfying the given description. The ChEBI-20 dataset involves a set of pairs $(p_{\text{desc}},m_{p_{\text{mol}}})$. Note that $\alpha$ is a hyper-parameter.

For experiments with DrugAssist (Section 4.3), the property-matching reward yields $1$ if the molecule satisfies the quantitative properties described in $p_{\text{desc}}$, e.g., HB donors, HB acceptors, and Bertz complexity, and 0 0 otherwise. The properties are evaluated with the RDKit package (Landrum, ).

Next, the diversity reward, $r_{\text{div}}$, is defined to consider molecular structural diversity as follows:

$$ $r_{\text{div}}(m_{k},\{m_{i}\}^{k-1}_{i=1})=1-\text{max}_{m\in\{m_{i}\}^{k-1}_ {i=1}}T(m_{k},m)^{\beta},$ (3) $$

where $T(m_{i},m_{j})$ measures the Tanimoto similarity between $m_{i}$ and $m_{j}$ by assessing their true molecular structures, i.e., fingerprints. This metric yields a value between 0 0 and $1$ and can be obtained using the RDKit package. Note that $\beta$ is a hyper-parameter.

Policy optimization. We consider four NVIDIA A100 SXM4 40GB for reinforcement learning implemented with proximal policy optimization.

- •
For the reinforcement learning of BioT5^+ and MolT5 (Sections 4.1 and 4.2), we consider $200$ PPO iterations, $8$ mini-batch size, $128$ batch size, and $5\textrm{e}-5$ learning rate. We also consider $0.01$ KL penalty. Note that $\alpha$ and $\beta$ in Equations 2 and 3 are $0.5$ and $2.0$, respectively. The reward signal is amplified by multiplying by a value of $8.0$. The maximum sequence length in reinforcement learning is limited to $2560$ due to memory limitations. Here, we also apply LoRA (Hu et al., ) where the rank and alpha are $16$ and $32$, respectively. We save the model every $25$ PPO iteration and select the model yielding the highest rewards for the training prompts.
- •
For the reinforcement learning of DrugAssist (Section 4.3), we consider $200$ PPO iterations, $4$ mini-batch size, $64$ batch size, and $3\textrm{e}-6$ learning rate. We also consider $0.1$ KL penalty. Note that $\beta$ in Equation 3 is $2.0$. The reward signal is amplified by multiplying by a value of $4.0$. The maximum sequence length in reinforcement learning is limited to $1280$. We also apply LoRA (Hu et al., ) where the rank and alpha are $64$ and $128$, respectively. We save the model every $25$ PPO iteration and select the model yielding the highest rewards for the training prompts.

## Appendix B Detailed experimental settings

Comparison with decoding schemes (Section 4.1). In this experiment, we first consider random sampling with different temperatures $\{0.7,1.0,1.5\}$. For the other decoding schemes, we consider conventional configurations: nucleus sampling with top-p $0.8$, beam search, diverse beam search with a diversity penalty of $0.5$, and contrastive beam search with a penalty alpha of $0.5$. We apply greedy decoding for our approach. The prompts for BioT5^+ are described in Table 5. The prompts for MolT5 are considered as a molecular description without any additional comments.

**Table 5: Prompts for BioT5^+ (Pei et al., ).**
| Prompt | Contents |
| --- | --- |
| $p_{\text{desc}}$ | “Definition: You are given a molecule description in English. Your job is to generate the molecule SELFIES that fits the description.<br>Now complete the following example -<br>Input: <molecular description><br>Output: ” |
| $p_{\text{desc+div}}$ (fine-tuning) | “Definition: You are given a molecule description in English. Your job is to generate the molecule SELFIES that fits the description.<br>Now provide a set of molecules -<br>Input: <molecular description><br>Output: ” |

Comparison with LLMs (Section 4.2). In this experiment, we consider generalist LLMs. For Mol-instructions, and LlasMol, we apply random sampling, beam search, diverse beam search with a diversity penalty of $0.5$, and contrastive beam search with a penalty alpha of $0.5$. The prompts are described in Table 6. For ChatGPT, we apply either $50$ random sampling with $p_{\text{desc}}$ or greedy decoding with $p_{\text{desc+div}}$, where both prompts are described in Table 7. The prompt design is based on (Li et al., ). For o1, we use the prompt $p_{\text{desc+div}}$ described in Table 7. We report the test score using the decoding scheme or the prompt that yields the highest NCirclesh=0.65 for $500$ evaluation molecular description.

**Table 6: Prompts for generalist LLMs (Fang et al., ; Taylor et al., ).**
| Method | $p_{\text{desc}}$ |
| --- | --- |
| Mol-instructions | “Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.<br>### Instruction:<br>Create a molecule with the structure as the one describe.<br>### Input:<br><molecular description><br>### Response: ” |
| LlasMol | “Give me a molecule that satisfies the conditions outlined in the description: <molecular description>” |

**Table 7: Prompts for ChatGPT and o1 (Li et al., ).**
| Input | Contents |
| --- | --- |
| System $p_{\text{desc}}$ | “You are now working as an excellent expert in chemistry and drug discovery. Given the caption of a molecule, your job is to predict the SMILES representation of the molecule. The molecule caption is a sentence that describes the molecule, which mainly describes the molecule’s structures, properties, and production. You can generate the molecule SMILES representation from the caption.<br>Example:<br>Instruction: Given the caption of a molecule, predict the SMILES representation of the molecule.<br>Input: <molecular description>.<br>Your output should be:<br>{"molecule": "<SMILES>"}<br>Your response should only be in the JSON format above; THERE SHOULD BE NO OTHER CONTENT INCLUDED IN YOUR RESPONSE.” |
| User $p_{\text{desc}}$ | “Input: <molecular description>” |
| System $p_{\text{desc+div}}$ | “You are now working as an excellent expert in chemistry and drug discovery. Given the caption of a molecule, your job is to generate diverse molecules. The molecule caption is a sentence that describes the molecule, which mainly describes the molecule’s structures, properties, and production. You can generate the molecule SMILES representation from the caption.<br>Example:<br>Instruction: Given the caption of a molecule, generate fifty structurally diverse molecules with SMILES representations.<br>Input: <molecular description>.<br>Your output should be:<br>{"molecule1": "<SMILES>"}<br>{"molecule2": "<SMILES>"}<br>{"molecule3": "<SMILES>"}<br>…<br>{"molecule50": "<SMILES>"}<br>Your response should only be in the JSON format above; THERE SHOULD BE NO OTHER CONTENT INCLUDED IN YOUR RESPONSE.” |
| User $p_{\text{desc+div}}$ | “Input: <molecular description>” |

Fine-tuning generalist LLMs. We synthesize $600$ pairs of training prompts and corresponding sets of molecules. The prompts specify hydrogen bond donors and acceptors ranging from one to four, and a Bertz complexity ranging from 0 0 to $300$. The sets of molecules are collected by applying beam search on DrugAssist and then perturbed by shuffling their order. We use the prompts described in Table 8. The system prompt follows the default settings from (Ye et al., ). For evaluation, we use four prompts specifying three hydrogen bond donors, three hydrogen bond acceptors, a Bertz complexity between $100$ and $200$, and a QED value between $0.4$ and $0.6$. Additionally, as shown in Figure 3, we try to generate diverse molecules by designing prompts without fine-tuning (Table 9). However, these prompts show lower performance compared to applying beam search.

**Table 8: Prompts for DrugAssist (Ye et al., ).**
| Prompt | Contents |
| --- | --- |
| $p_{\text{desc}}$ | Hydrogen bond donors and acceptors: “Can you generate a molecule with <value> <property>? Print it in SMILES format.”<br>QED and Bertz complexity: “Can you generate a molecule with <property> below <value1> but at least <value2>? Print it in SMILES format.” |
| $p_{\text{desc+div}}$ (fine-tuning) | Hydrogen bond donors and acceptors: “Can you generate a set of molecules that have <value> <property>? Print each of them in SMILES format.”<br>QED and Bertz complexity: “Can you generate a set of molecules that have <property> below <value1> but at least <value2>? Print each of them in SMILES format.” |

**Table 9: Prompts for DrugAssist (without fine-tuning, Figure 3).**
| $p_{\text{desc+div}}$ |
| --- |
| “Can you generate a set of molecules? Each molecule has <value> <property>. Print each of them in SMILES format.” |
| “Can you generate a diverse set of molecules? Each molecule has <value> <property>. Print each of them in SMILES format.” |
| “Can you generate a structurally diverse set of molecules? Each molecule has <value> <property>. Print each of them in SMILES format.” |
| “Can you generate fifty molecules? Each molecule has <value> <property>. Print each of them in SMILES format.” |
| “Can you generate fifty diverse molecules? Each molecule has <value> <property>. Print each of them in SMILES format.” |
| “Can you generate fifty structurally diverse molecules? Each molecule has <value> <property>. Print each of them in SMILES format.” |
| “Can you generate a set of molecules with <value> <property>? Print each of them in SMILES format.” |
| “Can you generate a diverse set of molecules with <value> <property>? Print each of them in SMILES format.” |
| “Can you generate a structurally diverse set of molecules with <value> <property>? Print each of them in SMILES format.” |
| “Can you generate fifty molecules with <value> <property>? Print each of them in SMILES format.” |
| “Can you generate fifty diverse molecules with <value> <property>? Print each of them in SMILES format.” |
| “Can you generate fifty structurally diverse molecules with <value> <property>? Print each of them in SMILES format.” |

## Appendix C Additional results

**Table 10: Visualization of the generated molecules from beam search. The $48$ molecules are generated with the description: “The molecule is a primary aliphatic ammonium ion which is obtained from streptothricin F by protonation of the guanidino and amino groups. It has a role as an antimicrobial agent. It is a guanidinium ion and a primary aliphatic ammonium ion. It is a conjugate acid of a streptothricin F’.”**
| Example of outputs (IntDiv. is $0.53$) |
| --- |
|  |

**Table 11: Visualization of the generated molecules from diverse beam search. The $48$ molecules are generated with the description: “The molecule is a primary aliphatic ammonium ion which is obtained from streptothricin F by protonation of the guanidino and amino groups. It has a role as an antimicrobial agent. It is a guanidinium ion and a primary aliphatic ammonium ion. It is a conjugate acid of a streptothricin F’.”**
| Example of outputs (IntDiv. is $0.67$) |
| --- |
|  |

**Table 12: Visualization of the generated molecules from constrastive beam search. The $48$ molecules are generated with the description: “The molecule is a primary aliphatic ammonium ion which is obtained from streptothricin F by protonation of the guanidino and amino groups. It has a role as an antimicrobial agent. It is a guanidinium ion and a primary aliphatic ammonium ion. It is a conjugate acid of a streptothricin F’.”**
| Example of outputs (IntDiv. is $0.58$) |
| --- |
|  |

**Table 13: Visualization of the generated molecules from our method. The $48$ molecules are generated with the description: “The molecule is a primary aliphatic ammonium ion which is obtained from streptothricin F by protonation of the guanidino and amino groups. It has a role as an antimicrobial agent. It is a guanidinium ion and a primary aliphatic ammonium ion. It is a conjugate acid of a streptothricin F.”**
| Example of outputs (IntDiv. is $0.77$) |
| --- |
|  |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: See weakness
soundness: 2
strengths: 1.This paper addresses an important need in molecular generation by developing a method that enhances structural diversity, a key factor in drug discovery. 
2. The two-stage fine-tuning approach, which combines supervised learning with reinforcement learning (RL) for diversity maximization well-designed solution. 
3. Through empirical testing, the paper shows that its approach outperforms both current LLM-based molecular generators and advanced decoding schemes, highlighting the practical advantages of its method for achieving high-quality, diverse outputs.
4. The proposed method is adaptable and could be extended to other domains requiring diversity in generated outputs, such as protein design or materials discovery. This versatility increases the impact of the work beyond just molecular generation.
summary: This paper proposes a two-stage fine-tuning approach to address the challenge of generating diverse molecules: (1) supervised fine-tuning to enable LLMs to autoregressively generate molecules in a sequence, and (2) reinforcement learning to enhance structural diversity among the generated molecules.
The contributions of this work include introducing an innovative method that combines supervised learning and reinforcement learning to generate structurally diverse molecules. Additionally, the paper demonstrates that this approach surpasses existing LLMs and traditional decoding methods in producing high-quality, diverse molecular structures.
weaknesses: 1. The two-stage fine-tuning approach which is highlighted in this paper is not that novel.
2. Although the model is tested on standard datasets and metrics for structural diversity, it lacks experimental or real-world validation to demonstrate the generated molecules’ practical utility in drug discovery.
3. The proposed method’s multi-stage fine-tuning approach may require extensive hyperparameter tuning and setup, which can hinder reproducibility and adoption among researchers.
4. The paper claim "the first to explore the use of LLMs for generating diverse molecule" and "the first propose a fine-tuning approach for LLMs to generate diverse solution", which is overstate.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In Table 3, it is unclear why the comparison is made when the number of generated molecules is different. Additionally, all relevant metrics should be included in the comparison to provide a comprehensive assessment.

2. In Table 4, the authors should specify which large language model (LLM) was used as the basis for fine-tuning and reinforcement learning in the compared methods. Furthermore, when comparing Tables 2 and 4, the performance improvement of the Div-SFT approach appears to be minimal.
soundness: 2
strengths: This paper identifies a limitation in the molecular generation capabilities of LLMs and proposes a diversity enhancement strategy based on autoregressive generation and reinforcement learning.
summary: This paper investigates the limitations of current large language models (LLMs) in generating structurally diverse molecules and proposes a novel two-stage fine-tuning approach to address this challenge. Specifically, the authors apply supervised fine-tuning to enable LLMs to generate a sequence of molecules and subsequently leverage reinforcement learning to enhance structural diversity. Experimental results demonstrate that the proposed method improves molecular diversity compared to existing decoding strategies.
weaknesses: 1. The method directly adopts LLMs with reinforcement learning. Thus, the technical novelty of the method is limited.
2. To help readers understand the impact of the approach, the authors could provide additional explanations for the diversity metrics or introduce more similarity evaluation methods.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1) Can you improve the explanation of NCircles? I think I get it, but given it's a relatively new method I yet don't have an intuition for it like I do for things like T-SNE or UMAP. What kinds of hyperparameters does it have? What are the conditions for two nodes to be close to each other? Do the overlap of the circles mean anything or are they a consequence of the force-directed algorithm? 

2) In Table 1, you show the outputs of your method for the description "The molecule is a primary aliphatic ammonium ion which is obtained from streptothricin F by protonation of the guanidino and amino groups. It has a role as an antimicrobial agent. It is a guanidinium ion and a primary aliphatic ammonium ion. It is a conjugate acid of a streptothricin F". Are all the generated molecules a conjugate acid of  streptothricin F? Do they have the correct scaffold? This would be important to understand and is related to limitations of the BLEU score.
soundness: 4
strengths: **Originality**
To my knowledge, the method is novel and the problem is under-researched.

**Quality**
The SFT and RL methods are clear and intuitive. The algorithms are well described in the text and code is provided in Supplementary. Figure 4 helps the understanding of the method. The gap in existing LLMs at generating structurally diverse molecules is well explained in the text and in figure 2. The baselines and metrics are mostly well chosen. The results are convincing, with the authors' method producing significant increase in the chosen metrics.

**Clarity** The methods are explained well. The figures are easy to read. Including different texture in Figure 5 helps the readability. The Appendix with all the experimental details and the code in Supplementary is appreciated and helps reproducibility. I also appreciate the title and claims being straightforward and to the point. 

**Significance** Generating diverse molecules is important in discovery workflows, thus the problem here is significant. The proposed method is easy to understand and implement. The paper also inspires a few interesting follow-ups and thus I think is an important addition to the community.
summary: The paper proposes a fine-tuning method to generate diverse molecules using LLMs. The method involves supervised fine-tuning (SFT) and reinforcement learning (RL).

For SFT, the LLM is prompted to generate a molecule with a given property. The prompt is sampled many times, the generated molecules are filtered and then concatenated to create the SFT training set. The SFT step uses a prompt describing the property and requesting a set of molecules. The set of molecules generated previously is appended to this prompt.

For RL, the SFT prompt is used to generate a sequence of molecules. Every time a molecule is generated, the LLM policy is updated using Proximal Policy Optimization (PPO). The reward is based on how well the molecule matches a property and how different it is from the previous molecules.
weaknesses: 1) In the experiments with BioT5+ and MolT5, the authors used the BLEU score on prompts and molecules from ChEBI-20. I think BLEU is a misleading score in the text to molecule task. For example, in BLEU, the order of words in a sentence matters, while there are many ways of writing a molecules as SMILES strings. Also, small changes in a molecule SMILES in the right place can have high changes in a property, such as in the case of hydrogen bond donors and acceptors. I recommend the authors discuss this in their manuscript and compare it with scores generated by RDKit (which they used in the other experiments). 

2) There are a few minor things that could improve the paper: 
    - Figure 7 could have different symbols. It was hard to read in a black-and-white version of the paper.   
    - Worth considering giving the method a name, this will help wider adoption.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. **Cluster Representation in Figure 6**:
   Is it reasonable to represent each cluster as a circle in Figure 6? Given that a 2D projection of a circle in a high-dimensional space may not accurately be circular, I wonder if this visualization accurately reflects clustering in chemical space.
2. **Effect of SMILES Randomness on BLEU Scores**:
   Since a single molecule can be represented by multiple SMILES strings, the randomness in SMILES representations could impact BLEU scores in the experiments. Has this potential source of variability been considered, and what effect does it have on the method’s stability or the evaluation of generated molecules?

In conclusion, this paper offers a valuable approach to improving molecular diversity in AI-driven drug discovery, and I will raise my score if my concerns (in Weaknesses and Questions) are well-addressed.
soundness: 2
strengths: 1. **Focus on an Essential Problem**: The paper addresses a critical requirement in drug discovery—molecular diversity in generated candidates. By providing a more diverse set of molecules, the proposed method aligns with real-world drug discovery processes, enhancing the chance of identifying successful compounds.
2. **Clear Presentation**: The paper is well-structured, with a clear exposition of the methodology and experimental procedures.
3. **Detailed Experiments**: The authors perform thorough evaluations across multiple metrics and baselines, including the comparison of Div-SFT+RL with various decoding schemes. The use of IntDiv, and NCircles as diversity-related metrics offers a comprehensive view of the method’s performance.
summary: This paper tackles an essential problem in AI-driven drug discovery: generating structurally diverse molecules using LLMs. Addressing the challenge that traditional LLMs often produce structurally similar molecules from a single prompt, the authors propose a two-stage approach: (1) supervised fine-tuning to enable autoregressive mutiple molecule generation, and (2) RL to enhance structural diversity. Through experiments with several LLMs and comparison against various decoding schemes, the proposed approach, Div-SFT+RL, demonstrates an improved ability to generate diverse molecular structures, which is crucial for discovering viable drug candidates. The results indicate that Div-SFT+RL outperforms existing models on diversity-related metrics, supporting its potential for applications in drug discovery.
weaknesses: 1. **Limited Discussion of Related Works on RL for Diversity in Molecular Generation**:
   The paper lacks an in-depth discussion of prior reinforcement learning (RL) approaches that also aim to improve molecular diversity, particularly those using RL for targeted or diverse molecule generation. Works such as Blaschke et al. (2020), Pereira et al. (2021), Hu et al. (2023), and He et al. (2024) provide diverse RL strategies in molecular design, which may not all involve LLMs but offer relevant insights on RL methods and should be discussed to position this work better.

   [1] Blaschke, Thomas, et al. "Memory-assisted reinforcement learning for diverse molecular de novo design." Journal of cheminformatics 12.1 (2020): 68. 

   [2] Pereira, Tiago, et al. "Diversity oriented deep reinforcement learning for targeted molecule generation." Journal of cheminformatics 13.1 (2021): 21. 

   [3] Hu, Xiuyuan, et al. "De novo drug design using reinforcement learning with multiple gpt agents." Advances in Neural Information Processing Systems 36 (2023). 

   [4] He, Jiazhen, et al. "Evaluation of reinforcement learning in transformer-based molecular design." Journal of Cheminformatics 16.1 (2024): 95.

2. **Applicability of the ChEBI-20 Dataset for Diversity Evaluation**:
   The use of the ChEBI-20 dataset raises concerns, as each molecular description in this dataset corresponds to a single molecule. Baseline approaches typically focus on generating one molecule to match the target rather than generating a diverse set for a single description. Furthermore, some descriptions may specify molecular details to the extent that producing diverse structures could be contradictory, making ChEBI-20 potentially unsuitable for assessing molecular diversity.

3. **Reward Function’s Balancing of Terms**:
   The reward function includes two terms: one for structural diversity and another for description matching. It would be beneficial to introduce a coefficient to balance these terms, enabling tuning based on the importance of each aspect (diversity vs. adherence to the description).

4. **NCircles Values in Figure 5**:
   In Figure 5, NCircles values at threshold $h=0.75$ are higher than those at $h=0.65$. This seems inconsistent with the metric’s definition, where values at a higher threshold should logically be lower. The unexpected result suggests potential misinterpretation or misuse of the metric, which requires clarification.

Meta-review: In this work, authors propose a two-stage fine-tuning approach for large language models (LLMs) to generate diverse molecular structures for drug discovery applications. The method combines supervised fine-tuning to enable autoregressive molecule generation followed by reinforcement learning optimization to maximize structural diversity while maintaining adherence to target properties. The authors evaluate their approach against existing LLMs and decoding schemes using various diversity metrics.

There are several strengths for the work as follows. The paper addresses an important need in computational drug discovery by focusing on molecular diversity generation. The methodology is clearly presented with detailed experimental protocols and comprehensive evaluations across multiple metrics (Reviewer oKU9). The authors provide thorough empirical validation showing improvements over baseline approaches in generating structurally diverse molecules that match target properties. The supplementary materials include code and detailed experimental settings that support reproducibility.

However, there are several limitations for the work as evident from the discussion during the review period. The two-stage fine-tuning approach largely adopts existing methods without substantial innovation (Reviewer APHU, FZz8). While the authors argue their contribution lies in applying these techniques to molecular diversity generation, the core technical components are standard. Reviewer xfWi raises significant issues about the appropriateness of using ChEBI-20 for diversity evaluation, since it maps descriptions to single target molecules. While the authors argue diversity remains valuable even with single targets, this creates a fundamental mismatch between the baseline models' objectives and the proposed evaluation metrics. Multiple reviewers (APHU, FZz8) note that the paper's claims of being ``first'' to explore LLMs for diverse molecule generation are overstated, given existing work in this space. Further, the reliance on BLEU scores for molecular similarity is problematic given the multiple valid SMILES representations for identical molecules (Reviewer oKU9). While the authors added Tanimoto and Dice scores in revision, this highlights initial gaps in the evaluation approach.

Based on these comments, the manuscript cannot be recommended for acceptance at this point. To strengthen future submissions, the authors should: (1) develop a more appropriate evaluation framework that fairly compares diversity-oriented approaches, (2) more precisely position their technical contributions relative to existing work, and (3) strengthen the novelty of their methodological approach beyond standard fine-tuning techniques. More comments below.

Decision: Reject