Conversational Drug Editing Using Retrieval and Domain Feedback
Authors: Shengchao Liu
Jiongxiao Wang
Yijin Yang
Chengpeng Wang
Ling Liu
Hongyu Guo
Chaowei Xiao
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
prompt
retrieval
domain feedback
conversation
drug editing
drug optimization
controllable generation
small molecule
peptide
protein
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose ChatDrug, a framework that utilizing Large Language Models for conversation drug editing with a retrieval and domain feedback module.

Abstract: Recent advancements in conversational large language models (LLMs), such as ChatGPT, have demonstrated remarkable promise in various domains, including drug discovery. However, existing works mainly focus on investigating the capabilities of conversational LLMs on chemical reactions and retrosynthesis. While drug editing, a critical task in the drug discovery pipeline, remains largely unexplored. To bridge this gap, we propose ChatDrug, a framework to facilitate the systematic investigation of drug editing using LLMs. ChatDrug jointly leverages a prompt module, a retrieval and domain feedback module, and a conversation module to streamline effective drug editing. We empirically show that ChatDrug reaches the best performance on all 39 drug editing tasks, encompassing small molecules, peptides, and proteins. We further demonstrate, through 10 case studies, that ChatDrug can successfully identify the key substructures for manipulation, generating diverse and valid suggestions for drug editing. Promisingly, we also show that ChatDrug can offer insightful explanations from a domain-specific perspective, enhancing interpretability and enabling informed decision-making.

Published as a conference paper at ICLR 2024
CONVERSATIONAL DRUG EDITING USING RETRIEVAL AND DOMAIN FEEDBACK
Shengchao Liu'', Jiongxiao Wang",
Yijin Yang', Chengpeng Wang',
Ling Liu
Hongyu Guo Chaowei Xiao²*
'University of Califoria, Berkeley
UUniversity of Wisconsin-Madison Arizona State University
*University of Illinois Urbana- Champ-Ca 'Princeton University
"National Research Council Canada "University of Ottawa
Equal Contribution
Joint Advising
ABSTRACT
Recent advancements in conversational large language models (LLMs), such as ChatGPT, have demonstrated remarkable promise in various domains, including drug discovery. However, existing works mainly focus on investigating the capabil- ities of conversational LLMs on chemical reactions and retrosynthesis. While drug editing, critical task in the drug discovery pipeline, remains largely unexplored. To bridge this gap. we propose ChatDrug, a framework facilitate the systematic investigation of drug editing using LLMs. ChatDrug jointly leverages a prompt module, a retrieval and domain feedback module, and conversation module streamline effective drug editing. We empirically show that ChatDrug reaches the best performance on all 39 drug editing tasks, encompassing small molecules, pep- tides, and proteins. We further demonstrate, through 10 case studies, that ChatDrug can successfully identify the key substructures for manipulation. generating diverse and valid suggestions for drug editing. Promisingly, we also show that ChatDrug can offer insightful explanations from a domain-specific perspective, enhancing interpretability and enabling informed decision-making.
INTRODUCTION
Recently, artificial intelligence (AI) tools have made remarkable strides in revolutionizing the field of drug discovery, offering tremendous potential for accelerating and enhancing various stages the process (Sullivan (019), including but not limited to virtual screening (Rohrer & Baumann 2009 Liu
et 20188, lead optimization (Tin et al. [2020] Irwin et al. [022] Wang et al. 2022] et al,222022b) reaction and retrosynthesis (Gottipati al. 2020), and protein folding and inverse folding (Jumper ct al. 2021 Hsu et al. 2022). However, existing research has predominantly focused on the drug struc- ture information, solely considering the inherent chemical structure of the drugs
single modality. In contrast, the drug discovery pipeline involves iterative refining processes that entail conversations with domain experts incorporate their feedback, ultimately achieving the desired outcome. On the other hand, significant advancements have been made in large language models (LLMs)) (Brown et al., 2020 Devlin al., 20188 Yang et al.. 2019), showcasing exceptional capabilities in understanding human knowledge and exhibiting promising reasoning abilities (Huang et al. 2022 Zhou et al. 2022 Kojima et al., 2022). Such observations inspire us investigate the potential of leveraging LLMS conversation and reasoning abilities for Al-assisted drug discovery in a multi-modalit fashion.
Potential of Conversational LLMs for Drug Discovery and Editing. Conversational LLMs exhibit three compelling factors that make them highly promising for drug discovery. Firstly, these model TTaloor et al. 2022 Touvron al., 2023) are pretrained on a comprehensive knowledge base, enabling their application across various fields, including drug discovery. This extensive "world-level" knowledge serves as robust foundation for drug-related tasks. Second, conversational LLMS possess outstanding abilities in fast adaptation and generalization. For example, by leveraging few-shot demonstrations, these models can effectively activate the relevant concepts learned during pretraining, enabling them to deliver more accurate and desired answers (Xie et al. (021). Such adaptability and generalization capacity holds immense potential for addressing complex drug discovery challenges and generating valuable insights. Lastly, interactive communication is vital characteristic of

---

Published as a conference paper at ICLR 2024
CAR DOU A (SE FArE CARIIN A3777
ar 1e-3
cerNe (a1
Converatiion Round
Conversation Round
Converaation Round
PDDS
X
RaDF
ReDF
by belurry
ReaF
Three Types Drugs
Retrivval DB
Domain Poodback
Modioade Diyeed
Padaa 901 SIPE
Prstas piza MAYY
User
Conversational LLM
Prompt Design for Domain-Specfttc Module
Retriacal and Domain Faedhack Module
Conversation Module
Figure 1: The pipeline for ChatDrug with 3 modules. PDDS generates drug editing prompts. ReDF updates the prompts using retrieved information and domain feedback. Finally, ChatDrug adopts the conversational module for interactive refinement. Further, we demonstrate 3 drug types: small molecules, peptides, and proteins.
conversational LLMs. This feature allows for a dynamic exchange of information, enabling users to incorporate feedback from prior knowledge or domain experts into the model. This bidirectional flow of information facilitates self-calibration of the answers, leading to improved accuracy and relevance in drug discovery tasks. To sum up, these factors collectively highlight the untapped potential of conversational LLMs for drug discovery tasks. Noticeably, drug editing (AKA lead optimization or protein design) is an important and challenging task in drug discovery. This is a routine task in pharmaceutical companies, and it aims updating the drug's ' substructures (Mihaliē & Trinajstic 1992), related to certain key tactics in drug discovery like functional group change (Ertl et al. 2020) and scaffold hopping (Böhm et al.,, 22044] Hu et al. (017). Traditional solutions relying on domain experts for manual editing can be subjective or biased (Drews) 2000 Gomez, 2018). Recent works (Liu et al. 2023b 2022a) have started explore text-guided drug editing in multi-modal manner. However, they do not utilize the conversational potentials in LLMs (Peng et al. 2023).
ChatDrug. Motivated by the aforementioned factors and challenges, we propose ChatDrug, a framework aiming to unlock new possibilities and enhance drug editing by using LLMs. Such framework can applied to various L.LMs backbones, including ChatGPT and open-source LLMS model GALACTICA and Llama2, and the pipeline is shown in Figure In ChatDrug, users can activate the conversation with LLMs involving domain knowledge and inject the retrieved information into the conversation. Specifically, ChatDrug includes the following modules for conversational drug editing. First, ChatDrug adopts PDDS (prompt design for domain-specific) module, enabling strong prompt engineering capability from LLMs. Second, ChatDrug integrates a ReDF (retrieval and domain feedback) module. By leveraging the vast domain knowledge available, such a ReDF module serves guidance for prompt updates and augments the model's performance in generating accurate approach, aligning with the iterative refinement
Our Approach:
outputs. Third, ChatDrug adopts a conversation-basee nature of the drug discovery pipeline. Such interactive schema enables a dynamic and collaborative process, effectively incorporating feedback from domain experts to achieve desired outcomes.
Through our design, ChatDrug demonstrates two appealing properties for drug editing tasks: (1) ChatDrug exhibits an open vocabulary property, allowing for exploring novel drug concepts beyond fixed set of pre-defined annotations. The model can generalize to new drug-relate concepts due to the unrestricted nature of natural language. (2) ChatDrug possesses the compositional property. It can decompose complex concepts, such as multi-ohjective lead optimization, into simpler attributes like binding to a new protein and high permeability, enabling handling complicated drug editing tasks. [
Then, to fully verify the effectiveness of ChatDrug, we need to design a benchmark for a wide range of drug editing tasks. Before going into details, we would like to claim two criteria for the task design: (1) The tasks should involve indeterministic answers, as they can serve as a source of inspiration for domain experts. (2) The tasks should be able evaluate computationally since the lab experiment can be quite laborious and is beyond the discussion of this paper. Following these criteria, we introduce 39 editing tasks over three common drugs: 28 for small molecules, 9 for peptides, and 2 for proteins.
Note that ChatDrug aims to inspire domain experts rather than replace them. While ChatDrug can propose optimized drugs novel attrihutes, its primary role is to serve as tool for knowledge exploration. The generated outputs can provide valuable insights and spark inspiration for domain experts in the drug discovery process.
2

---

Published as a conference paper at ICLR 2024
Last but not least, we offer empirical evidence substantiating the capability of ChatDrug for wide range of drug editing tasks on three LLM backbones: Turbo (a.k.a. ChatGPT), GALACTICA and Llama2. Quantitatively, ChatDrug reaches the best performance all 39 drug editing tasks compared to seven baselines, among which, ChatDrug- Turbo reaches generally better performance and higher stability 32 of them. Thus, further qualitatively provide 10 case studies illustrating can successfully identify the important substructures for each type of drug, as follows. (1) For small molecules, ChatDrug-Turbo detects the key scaffold for molecule editing, such as changing polar or hydrophobic functional groups for tuning properties like solubility in water and permeability. (2) For peptides, ChatDrug- Turbo accurately identifies the pooten--specffe binding motifs of the peptide sequences. (3) For proteins. ChatDrug- Turbo modifies sequences with more o-helix or 3-strand structures after folding (Jumper et al. 2021: Lin et al., 2022). We additionally illustrate that ChatDrug provides insightful explanations, serving as a knowledw ledge extraction tool.
that ChatDrug-Ture
2 PRELIMINARIES
Data Structure of Drugs, Drugs (Wishart al., 2008: dru) refer to certain specific substances that can be adopted to prevent, diagnose, treat, relieve symptoms of disease abnormal condition. In this paper, we would like to explore the three most common drugs: small molecules (Jayatunga et al. 2022), proteins (Frokjaer & Otzen| 2005), and peptides (Craik et al. 2013). Small molecules are sets of atoms connected together through the covalent bonds. Commonly-used data structures include SMILES (simplified molecular-input line-entry system) strings (Weininger 1988) and molecular graphs (Duvenaud et al. 2015 Kearnes et al. 2016 Liu et al. 2019). In ChatDrug, we consider using complex macromolecules, and they are composed of 20 amino
the SMILES strings. Proteins acids, where each amino acid is a small molecule. Regarding the protein data structure, we adopt the amino acid sequence (i.e., amino acid string), and the mapping between 20 alphabets and amino acids can be found in Appendix [ Peptides are short chains of amino acids and can be viewed as special type of protein. The demonstration of three data structures can be found in Figure [
Drug Editing and Problem Formulation. In this paper, we focus on the drug editing task. Drug editing is also known as lead optimization or protein design, an important drug discovery task. From the machine learning perspective, drug editing is conditional generation problem and can be formulated as follows. Suppose the input drug (SMILES string or amino acid sequence) is Pin- and target or desired property in the textual description is also known as the text prompt æ; in literature (Raffel et al. 2020; Liu et al. 2023a). Then condition on such text prompt, the goal is to obtain an optimized drug Tout
P(x Tins Zt). Under the ChatDrug framework, the formulation is:
Cou ChatDrug(Zmnn 22))
(1)
Then an evaluation metric E(xin. Tout; xx.) E (True, False} is applied to check if the edited drugs can satisfy the desired properties compared to the input drugs, and we will average this over each corre- sponding task to get the hit ratio. Note that :(: is task-specific, as will be discussed in Section 4
3 METHOD: THE CHATDRUG FRAMEWORK
Overview. Our ChatDrug framework is illustrated in Figure I consists of three components: (1) Prompt Design for Domain-Specifie (PDDS) module, (2) Retrieval and Domain Feedback (ReDF) module, and (3) conversation module. Given a task prompt and input drug. PDDS aims to generate the domain-specific prompt and concatenate it with the input drug to request LLMs for answers. One problem for current LLMs is that it does not fully utilize the prior domain knowledge. Thus, we design the ReDF module aiming to (1) guide the I.IMs to solve this task by retrieving structurally similar examples from the database and adding examples into the prompt as demonstrations and (2) verify the correctness of the output by using domain feedback function. If the output drug is incorrect after REDF, we then adopt the conversation module to ask LLMS to generate a new drug iteratively. We highlight that ChatDrug is parameter-free scheme and does not require any learning procedure.
3.1 PDDS MODULE
ChatDrug is proposed to solve challenging problem: generalization of universally (w.r.. data type and data source) well-trained LLM to solving scientific tasks. In natural language processing (NLP), prompt design or prompt engineering (Liu et al. 2023a) has proven to be effective paradigm for
3

---

Published as a conference paper at ICLR 2024
generalizing well-trained LLMs to various NLP downstream tasks, including but not limited sen-
timent classification (Han et al. 2022 Hu et al. 2021). textual entailment (Webson & Pavlick 2021
Shin et al. 2020), text summanization (He et al. 2020 Schick & Schütze 2020
Dou et al. 2020).
However, the explorations of adapting LLMs for drug editing tasks have been lagging behind. In this paper, we investigate this problem in the three most common types of drugs: small molecules, protein-binding peptides, and proteins. Recall that the goal of ChatDrug is (as in Equation (T): Lout - ChatDrug (iin wt). Here, the text prompts 2; are specifically designed enable the gener- alization for domain-specifie tasks with computationally feasible metrics. Additionally, we want to highlight that the objectives for drug editing (in æ,) should be related the high-level property instead of exact substructure replacement for two main reasons. (1) As discussed in Appendix C ChatDrug suits better for fuzzy matching like edited drugs with desired properties. In contrast, exact substructure replacement can be easily and precisely performed by domain experts, and such replacement may lack the creative inspiration for humans. (2) Property-related questions have an ambiguous nature, leading to dispersed answers that spark inspiration for domain experts in the drug discovery process.
Then concretely on the prompt design, for small molecules, we consider properties like solubility, drug-likeness, permeability, and the number of acceptors/donors. For peptides, we consider the properties of peptide-MEt MHH binding. For proteins, we consider the secondary structure. The text prompts explicitly depict the desired properties to be either higher or lower, and corresponding task prompts will be briefly explained in Section One concrete example for molecule editing is " "Can you make molecule [Xin] more soluble in water.", "," and more details can be found in Appendix
3.2 REDF MODULE
To better utilize the domain knowledge, we propose an important module: the ReDF (retrieval and domain feedback) module. The intuition is that there exists rich domain knowledge in the form of a retrieval database (DB), and ReDF will retrieve the useful information and inject it into the text prompt, adopting the fascinating language understanding ability of conversational LLMs.
Specifically, for each input drug Vin and prompt 2s, we have a candidate drug ă, which does not satisfy the desired property change in æ;, The candidate drug has diverse data resources based on the problem sctup, and in ChatDrug, it is the output drug with the negative result at each conversation round (will be introduced in Section 3.3) Then, the ReDF module returns a drug æn satisfying:
RR ReDF (i x.)) st's Retriieal
arg max
x'k) D(win x'k;
(2)
where D(-.::) E (True, False) is the domain feedback function, and z'R) is the similarity function. We use Tanimoto similarity (Bajusz et al., 2013) for small molecules and Levenshtein distance for peptides and proteins. Notice that here we take D(.,.:) .) the same as evaluation metric :). while there is some critical difference on the task-specific thresholds, as will be discussed in the ablation study in Section 4.5) Then the ReDF module injects æR into new prompt, e.g., the updated prompt for a molecule task is "Your provided sequence [ã] J is not correct. We find a sequence [a xe] which is correct and similar to the molecule you provided. Can you give me new molecule?"
We also want to highlight that the domain feedback injection in ReDF is relevant to the in-context learning (ICL) (Dong et al. (022). Such knowledge injection can result in performance gain (Min) et al. 2022) not only because of the mapping between ground truth data-label pairs but also due to the demonstration of the in-distribution data and label space. An ablation study on this is in Section 4
3.3 CONVERSATION MODULE
Another appealing attribute of conversational LLMS (like ChatGPT) is their interactive capability. This enables the LLMs to iteratively update the results by injecting prior knowledge. Inspired by this, we also consider adapting the conversational strategy for ChatDrug, which can naturally fit the ReDF module as described in in Section 3.2 Then, concretely on this conversational strategy in ChatDrug, first suppose there are C' conversation rounds, and we have an edited drug Ie for the conversation round c. If x satisfies our condition in the task prompt, then ChatDrug will exit. Otherwise, users will tell ChatDrug that T, is wrong, and we need to retrieve another similar and correct drug from the retrieval DB using ReDF: æ ReDF( T..., with ã a. in Equation (2).
4

---

Published as a conference paper at ICLR 2024
Table 1: Results on 16 single-objective small molecule editing, and the evaluation is ratio of the property change. For ChatDrug, we report the mean and std of five random seeds. The best results are marked in bold.
Single Tages Progenty
Randon
PCA
High Varience
GS-Matne
MoloorkNTMM
Ouelbbng
SMILES
Graph
GALACTICA
Llans2
Turbo
101 mome sofuble water
05
=1
35.3
$3.00
1000
5.24
10.80
3.5
61.87
49/02 4:
67
54.44
78,20
4: 1.91
1.83
102 laze rolabie uater
0.5
63.95
10.75
= 1.56
1,06
15.12
18.66 2.55
2.99
42.8 1:36
18.22
0.33
42.50-000 0:41
1 0.82
30.47
$3.91 +: 1.67
+ 3.26
56 66-80-1 7.96
47.09
s 3.42
61.47
91.41 4:44
3.55
34.70
41.99
0: 2.45
2.05
7008::3.44
96.6-
103 sore ding
0.1
5.7
1:5
33:99 a 3.72
3.7 0.10
16.20
20:00
6.33
0:71
2,00
36.53
4:
14:05 0:
19 4:33
21.23
41.19
3.66
0.96
32.88-4
11:
3.56
0KAF-3319
19.37:5.54
104 Si dosg
0.1
16.96
6.16 = 1.97
35.17
5:26 a 0.95
7.56
0.29
27,67:3333
$8.59 6.55
37.56 3: 1.76
$4.22 8 3.12
77.62
85.83
1.29
1.76
20.13
6: 2.17
1.26
70:75:292
50 30.99::.66
105 liokee persassiilly
10
25.33
17.41 = 1.45
2.1%
21.36 a 0.79
14.52 = 0.50
31.98
14.66
2.13
5.25
22.00 0.82
6.17 0.62
47.51 +: 1.89
57.34 1 0.60
50.42
59.84 0.76
2.73
14.21
79.91
1.25
4.87
28.47
25.00 +: 1.20
2:76
43 43:00:22
16.56.11 1884
106 permsobiilty
10 11,02
10.22
=5.17
17.38
66.9:22.74
107 sone tystngga hond arepprs
12.4 = 1.64
0.89 = 0.61
10.85 + 2.24
0.90 :: 0.84
0.67 0.11
11:76:00:11
21.17 3.09
1.83 0.47
51.01 + 5.26
27.33
4: 2
37:35 = 0.79
16.13 - 2.87
55.18 9:667
30
1.84
10:70 013
7.18 2: 1.76
95.35-
72.00:2.51
byrnogos Awnors
=
6.23 0:66
19.50=
25.55 +
32.55
3.32
36.43:3.32
Table 2: Results on 12 objeet-b small molecule editing, and the evaluation is the hit ratio of the property change. For ChatDrug. report the mean and std of five random seeds. The best results are marked in bold.
Two Target Property
Random
PCA
High Varianas
GB-Matze
Mr
ChaiDrag Lluana
SKILES
GALACTICA
201 more solable is water and
D-00
0.88 1: 1.M
864 1: 2.06
9.09 8: 1.25
14,00 2.48
27.87 1: 3.86
21.43 1.6.1.35 3.41
39.51 1. 3.41
26.05
2.55
79.62 0.64
wove hpuinges bosd accepeos
115-1
0.23 0.33
0:45 5: 0.644
0.22 0,31
067 0.62
5,50 :1 0.04
11:10 +: 1.50
26.44
1.07
13.24
1.11
lazz rebable in water and sove hysloges
0-0
2:99 0.58
2.00 0.8.9
2.45 0.67
5:17 0.95
1.75
38:00:1
3:11
891 1.06
acceptocs
0.5
0.45 0.32
0.00
0.33
0.17 0.24
0.30
0.00
12.06
1.00
8.27
24
203 sorr salahir water and nov inaroges lund donies
D-0
2:28 1:15
2:33 0-34111 1616
4:44 0.55
11:35 7.84
4.58
49.35 4- 1.71
47.91 5:533
10.00
7.90
19.34-1196
D5-1
0.00 2: 0.00
0.00 0.00
0.00
0:00
9.98
2: 1.03
23.94 à: 1.09
26.49 4.3.7
8.17
3:34
53.44:511
234 lery innolabir uster and nuee indroges hond devoer
D-0
0:00 0.55
1.96 a: 830
1.79 0.06
5AT 0.2
17.08
+ 2.75
14:42 + 3.43
25:00-4:2:0
16.30
4:43
39.90-3.66
0
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
3.59
1.14
3.84 à
9.83
0.0.5
9.04
1.45
24.19:2.19
silablc is
0-00
5,0 1:21
3,53 0:34
4,88
8,17 1,65
3,19
39:74 2:36
56.00
4.15
5:50
12:5=2208
Alple pensesbiltt
055 1
1 a: 0.68
0.07 à 0.55
0.66 e 0.54
0.00 0:00
19.15
s: 0.73
22.06 6: 1.90
3M.32 8: 0.33
15.24
2:
1.63
10,4485775
206 more solabic is rater and
D- 0
12 103
10.43 2
13.08 5
19:33 2.46
44.35
4 0.68
30.87
#: 0062
$4.87 0.96
41.97
0:87
65.33:2116
feeer screcabiity
0.5 10
6.30
0.64
6.67 0.53
4.83
0.85
1.22
20006 1:
1.77
52.90:2223
To sum up, for each conversation round, we request drug nn similar to Te, which will be updated at each conversation round. The Te and R serve as two in-context pairs to feed into ChatDrug, i.e.
"The output drug at round [c] is [I,ll, which is wrong. We find a sequence [œ which is correct and similar. Can you help improve the edited results?" An illustration of this conversation is in Figure]
4 EXPERIMENT
Specifications for ChatDrug. In this section, we verify the effectiveness of ChatDrug for drug editing on three types of drugs: small molecules, peptides, and proteins. Since ChatDrug is agnostic LLMs, we select GALACTICA, Llama2 and ChatGPT as our backbone LLMs in our experiments, named ChaiDrug-GALLCCTIC
ChatDrug-LJama2, and ChatDrug-Turre introduce three types of drugs and five categories of tasks accordingly: tasks Ixx and 2xx are single- and multi-objective tasks for small molecules (each task further includes two subtasks w.r.t. two thresholds as will be discussed next), tasks 3xx and 4xx are single- and multi-objective editing tasks for peptides, and task 5xx is for single-objective protein editing. Due to the space limitation, please check Appendix [ for the full list. Details of implementation and hyperparameters are in Appendix G
respectively. We
4.1 TEXT-GUIDED MOLECULE PROPERTY EDITING
The first experiment is text-guided molecule editing or molecule optimization. We adopt 16 single- objective tasks and 12 multi-objective editing tasks from MoleculeSTM (Liu et al.; 2022a These tasks are about the high-level properties of small molecules, like solubility in water and permeability.
Data: Both the input molecules and retrieval DB are sampled from ZINC (Irwin et al. 2012): : we sample 200 and 10K molecules (with SMILES strings) from ZINC as input molecules and retrieval DB, respectively. Prompt: The text prompt "Can you make molecule [SMILES) placeholder] [task requirement]? The output molecule should be similar to the input molecule". The [task requirement] is the textual description for each specific task, e.g., more soluble in water and with higher permeability. Evaluation. We take the hit ratio to measure the success ratio of edited molecules, i.e., the percentage of edited molecules that can reach the desired properties compared to the input molecules. All the properties for small molecules considered here can be calculated deterministically using RDKit (Landrum et al., 2013). Another important argument is the threshold A: it is a successful hit if the difference between input and output properties is above the threshold.
5

---

Published as a conference paper at ICLR 2024
Table 3: Visualization of six small molecule editing tasks, The blue regions red regions and green regions correspond to the edited substructures in the input molecule Cin, intermediate molecule æi for the 1st conversation round, and the output molecule Wour, respectively.
(ul Proceet for 10I: more atuble wate:
(b) Prompt 102: los solable water Broonediate Melecale in Outor Molectie foe
Ingat Midecalle
Inouseditte Molectle in
Ouron Melocle fue
Tnpal Melccall my.
Lagh 1.45
Lagh
Laght 0
Ligh:120 1.24
Logh
4M
Prosept for higher gertecabiity
Pronut 106; Joner penseabiliy
Midiccall
Inverediine Molecole
Melsosle
Milecall
brooreedatee Melecale
Ouror Milootee
IPSA. 68
JPSA 42
IPSA 55
IPSA:
(PSA 79
203, sclatie ligher penrealilty
Pienae 206,
loveer pecnesbilty
Ingut Maticzati si..
Inoeraoorr Moiresle in
Ourpur Melaorie Fe
Inpar Malecaat is.
Invormatorr Malocale in
Ohepue Malioriee
LogP (PSA:
LogP: 3,59, (PSAL
LogP: 2.3, (PSA
Light -188, (PSA:
.037, 125
Loght
Baselines: The baselines (Liu et al.] (022a) are based on MegaMolBART (Irwin et al., 2022), a pretrained auto-regressive model. Baselines include Random, PCA, High-Varianee GS-Mutate, and MoleculeSTM with SMILES or Graph as the molecule representation.
Observation. We illustrate the descriptions and the single- and multi-objective editing results in Tables and 2] respectively. The threshold A for each specific task is specified in Table for multi-objective editing tasks in T the threshold A has two values corresponding tasks. We further conduct an ablation study on the thresholds of ReDF in Section 4.3] We can observe that ChatDrug-Ture
the two
can reach the best performance on 21 out of 28 tasks, 20 of which possess over 20% hit ratio than the second-best baseline method. We also note that -aaaurgGGALACCICA performs well, showing consistently higher hit ratios than baseline methods and reaching the best performance on 7 out of 28 tasks. However, ShatDrug-Llamal is not outstanding the other ChatDrug models, outperforming on 7 out of 28 tasks. The reason may be that the corresponding backbone model Llama2 is not well pretrained 6 examples of molecule editing where ChatDrug- Turbo fails to generate molecules æ with desirable property change in the first conversation round, while successfully generates molecules oon ultimately. For example, in Table Bh, x converts methyl group to a propyl which incorrectly yields a less soluble molecule. Through conversational guidance, ChatDrug-Turbo changes its output Vour to an aminoethyl group, successfully fulfilling the task. In Table æ2 installs a phenyl urea to the molecule, which brings lower permeability as requested but makes the molecule less soluble. In contrast, ChatDrug- Turbo is able to replace the hydrophobic aromatic substituent with a hydrophilic amide Vaut, consistent with the requirement of higher solubility in water.
small molecule domain datasets. Table 3 visualizes
4.2 TEXT-GUIDED IMMUNOGENIC BINDING PEPTIDE EDITING
The second task is text-guided immunogenic binding peptide editing. Immunogenic peptides are promising therapeutic targets for the personalized vaccine, which triggers a person's 's immune system, e.g., CD8+ T cells, to fight diseases (Craiu et al. 1997, Hennecke & Wiley 2001). Immunogenic peptides are typically degraded from intracellular antigens. To activate CD8+ T cell immune responses, these peptides must first bind to Major Histocompatibility Complex (MHC) proteins, forming peptide-MHC complexes which are then presented on the surface of infected or malignant cells to interact with the T cells. Although the peptide-MHC binding process is critical for immune response, is highly specific, making editing known peptides to improve their binding affinity to specific MHC proteins challenging yet crucial task for peptide engineering and discovery. Recall that peptides are typically short protein chains, with most peptides having less than 16 amino acids.
Data: In this experiment,
the experimental dataset of PeP binding affini- ties (O'Donnell et al. 2020). This dataset contains 149 human MHC Class proteins (alleles) and 309K peptides. We follow existing works (Chen et al. 2023 on using the 30 common MHC

---

Published as a conference paper at ICLR 2024
Table Results on single-objective and three multi-objective peptide editing tasks. Random Mutation- for At mutated positions. The evaluation is the hit ratio of the increased binding affinity score. The best results are marked in bold. Due to the space limitation, please check Appendix F for the text prompt of each task.
single-objective editing
multi-objective editing
301
302
303
304
305
306
401
402
403
Random Mutation-1
1.80
14.40
1.80
1.80
12.00
5.60
3.20
0.80
0.40
Random Mutation-2
1.80
13.40
2.80
3.00
8.40
4.40
2.20
0.60
1.20
Random Mulation-3
1.80
9.40
2.40
4.20
9.00
3.80
3.00
0.60
0.80
ChatDrug-Gi BoaDrrug-GALCTIIN GALACTICA
11.55
12.78
13.47
9.28
8.40
14.85
5.51
4.50
2.48
ChatDrug-Llama
27.64
14.89
21.18
13.79
19,52
26.33
8.33
4.95
3.80
Chairug-Turi Turbo
56.60
69.80
64.33
59.04
65.00
44.69
34.54
41.77
(a) Motifs of input peptides for 301.
(b) Motifs edited peptides for 301.
(c) Motifs of experimental peptides 301.
L
=
(f) Motifs of experimental peptides 302. Figure 2: Visualization of two peptide editing tasks using PWM. The x-axis corresoonds to the position index,
(d) Motifs input peptides for 302.
Motifs of edited peptides for 302.
while the y-axis corresponds to the distribution of each amino acid (in alphabets) at each position.
proteins (alleles) and we randomly pick one as the source allele and one or more alleles as the target alleles. Notice that for single-allel allele tasks, 30 MHC proteins be further divided into categories: HI.A-A, HLAAB, and HIAAC;; we make sure that the sampled source and target alleles from different categories. Then we sample 500 peptides from the source allele types. For the retrieval DB, the experimental data of the target allele(s) are adopted. The sampled MHC types are further specified in Appendix Prompt: We expect the edited peptides can bind to the target MHC protein(s), so the prompt template is We want peptide that binds to [target allele]. We have peptide [peptide sequence] that binds to [source allele), can you help modify it? The output peptide should be similar to the input peptide. "Evaluation; The actual bindings require wet-lab experiments, which are expensive and prohibited for large scaled evaluation. Following existing works (Chen) et al. 2021 2023), we leverage the MHCffury2.0 (O'Donnell et al. 2020) as a pseudo-oracle to predict the peptide-MHC binding affinity. MHCflurr2.00 is the of-teooftheear method enabling accurate estimating of the binding affinity of peptides with MHC proteins. The success of the peptide editing needs to satisfy two conditions: (1) The output peptide should have a higher binding affinity with the target allele compared to the input peptide; (2) The binding affinity of the output peptide and target allele should be above a certain threshold. Here we take the threshold as one-half of the average binding affinity of experimental data the target allele. Baselines: Since there is no existing approach for text-guided binding peptide editing, we use random mutation as the baseline, i.e., conducting random mutation on the amino acid sequence of the input peptides.
Observation. We illustrate the single- and multi-objective editing results in Table We can observe that ChatDrug reaches the best performance over all 9 tasks compared the random mutation baselines for all three LLMs backbones. ChatDrug-Turbe reaches the best performance with significantly higher hit ratios than ChatDrug GALACTICA and ThatDrug-Llama2. We further visualize peptides generated by ChatDrug-Turn Turbo using position weight matrices (PWMs) in Figure 2 PWM has been widely used for the visualization of protein motifs (patterns), and plots the distribution of each amino acid at the corresponding position. Thus, more important motifs with higher probabilities will be marked in higher alphabets. According to F the edited or optimized peptides follow similar patterns to the experimental data presented. For instance. for task 301, the edited peptides can successfully upweight the alphabet E (glutamic acid) at position 2. Results suggest strong correlation between edited peptide binding motifs and those derived from experimental data.
4.3 TEXT-GUIDED PROTEIN SECONDARY STRUCTURE EDITING
Last but not least, we consider text-guided protein secondary structure editing (PSSE) Klausen et al. 2019). For protein ID sequence, it can fold into the 3D structure, as shown in Figure Specifically, proteins possess four levels of structures, and secondary structures are fundamental building blocks, which are local folding patterns stabilized by hydrogen bonds. Typical secondary structures include
7

---

Published as a conference paper at ICLR 2024
Vggut Preten
Ouiped Protes
Vouut Preten
Ouipad Prutee
Protein editing with more helix structures. Figure 3: Visualization of two protein editing tasks. For the protein secondary structures, the a-helix is marked red, and B-shect marked in yellow. The edited regions hefore and after ChatDrug are marked in blue circles.
(b) Protein editing with more strand structures.
a-helix and ß-sheet, consisting of ß-strands. Here we interested in two PSSE tasks, i.e. using ChatDrug to edit protein sequènces with more helix or strand structures after folding.
Data: TAPE (Rao al., 2019) is benchmark for protein sequence property prediction, including the secondary structure prediction task. We take the test dataset and training dataset as the input proteins and retrieval DB, respectively. Prompt: For an input protein sequence (protein sequence), can you modify it with more helix/strand structures? Baselines: Same with peptide editing, we adopt random mutation as baselines. Evaluation. For evaluation, we adopt the state-of-the pretrained secondary structure prediction model, i..., ProteinCIAAP AA--CCLA------ETTT from ProteinDT (Liu et al., 2023b). The hit condition if the output protein sequences have more secondary structures than the input sequences.
Observation. As shown in Table 5 we can tell the large performance gain by ChatDrug, where ChatDrug-Llama2 and ChatDrug-Turbe achieve the highest hit ratio on task 501 and 502, respectively. We fur- ther visualize cases on how ChatDrug- Turbo successfully edits the proteins with more he- lix/strand structures. We adopt pretrained ESMFold (Lin et al. (2022) for protein folding (protein sequence to protein structure prediction) and then plot the protein structures using PyMOL (Schrödinger & DeLano). We show two examples in Figure As circled in the blue regions in Figures and 3bb the edited proteins possess more helix structures and strand structures, respectively. More visualization can be found in Appendix H
Table 5: Results on two protein editing tasks. Random
Mutation- R for R mutated positions. The evaluation is the hit ratio of increased secondary structures accordingly.
501 more helix
502 more strand
Random Mutation-3
26.90
21.44
ChaiDrug GALACTICA ChatDrug-Laam ChaiDrug- Turbo
11.75
5.99
34.79
34.79
33.18
59.68
4.4 WHY CHATDRUG WORKS? KNOWLEDGE EXTRACTION
We are also interested in understanding how ChatDrug can work. As shown in Figure we illustrate a case study on small molecule editing. It can observed that ChatDrug can do knowledge extraction: for specific task on editing molecules to be more soluble in water, ChatDrug can extract the reasonings and summarize them into five rules. This gives us the confidence that the success ChatDrug is its ability of domain interpretation. We conduct further ablation studies like knowledge extraction without the context as a control experiment in Appendix Although ChatDrug can extract Nomain-specifie information for the editing tasks, we notice a minor issue: the redundancy among knowledge. As shown in Figure 4 the extracted rules 1, 3, and 5 are all centered on introducing polar functional groups for solubility in water, despite from slightly different perspectives. In Appendix [
Figure 4: Knowledge extraction of ChatDrug.
Hoplee decicciio toa eith aloyt chaire tntegenn
prantccing oystl cessuss stmadity
perten fomer
ummea on adouad ar Them foo
bnlly resuaiee
we continue explore how ChatDrug can play positive role in knowledge summarization, aiming to alleviate this knowledge redundancy issue.
4.5 ABLATION STUDIES
Ablation Study on Comparison with Zero-shot and In-context Learning. There are two important modules for ChatDrug: conversation for result refinement and the ReDF for knowledge retrieval. Thus in this ablation study, we would like to explore the effect of such two modules. The first case is zero-shot prediction. It is indeed ChatDrug with 0, i.e., without conversation ReDF. On the other hand, in-context learning (ICL) can treated as ChatDrug equipped with the ReDF module but without any conversational round. Concretely, the retrieved drug is æ ReDF (in, Pmn) with ă
8

---

Published as a conference paper at ICLR 2024
Table 6: Ablation studies on comparison with in-context learning (ICL) and conversation rounds on molecule editing. The threshold is the loose threshold with A 0, and the random seed is O.
101
102
106
107
203
204
205
ICL (few shot)
52.11
75.45
37.76
46.23
30.61
42.86
54,97
09.31
59.88
39,86
53.45
49.36
37.42
42.77
C 0 (acro-shot)
78,3
71.35
16.15
32.12
16.01
8.33
59.41
63. 63.16
43.09
0.52
54.49
0.53
2.11
22.32
1
89.56
93.64
48.35
61.62
47.93
56.97
90.00
93.08
72.29
36.26
86. 14
50.00
9.44
54.14
-hurDng-Turho
C
93.37
97.11
52.81
67.93
55.76
78.40
95.57
98. 10
80.57
48.52
39.88
12.72
67.23
96.11
97.69
55.11
75.54
98.
83.75
76.74
4
96.67
97. 69
59.20
78.14
63.35
95,41
98.08
98. 73
86.77
68.32
91.41
57,42
22.36
80.00
5
91.22
97.09
59.77
83.06
65.84
95.03
99.36
98.73
89.17
70.19
94.41
63.40
25.32
81.55
Table 7: Ablation studies on thresholds in domain feedback function D with two conversational rounds. The
evaluation function E uses the strict threshold. Wc report the mcan of five sceds, and stds are in Appendix !
100
103
104
105
106
107
108
201
202
203
204
205
206
loose chreshold
80.73
41.00
11.23
16.94
33,16
53.59
14.96
21.93
20. 14.14
7.96
17.93
5.79
3,66
41.04
stract thresbold
N8.67
70.08
19.37
30.99
43.08
66.09
72.00
76.43
49.64
24.92
53.64
24.19
10.44
32.9
Iou.
(tr.tt)
0.
03
(a) Task 101 more solable in water
(b) Task 102 less soluble water
(c) Task 107 more hydrogen bond acceptors
Figure 5: Similarity distribution between input molecules Ti and retrieval æR. intermediate æ1, and output
molecules Fou.. We pick up three tasks on small molccules for visualization, and more results are in Appendix H
Tin in Equation ( The text prompt for zero-shot and ICL are "Can you edit the molecule [æme] to be
more soluble?" and "We know that æ is similar to [xn]] and is more soluble in water. Can you edit the molecule [ to be more soluble?" The results are in Table and we observe that both ChatDrug and ICL are better than the zero-shot, and conversational refinement performs best on all 14 tasks.
Ablation Study the Number of Conversation Rounds in ChatDrug. In ChatDrug, the number of conversation rounds is an important hyperparameter. Here we conduct an ablation study on small molecules test its effectiveness. The results are in Table 6 For molecule editing tasks tested here, Turbo tends to converge after C = 2 conversation rounds.
the performance of ChatDrug- -hurrro
In ChatDrug, another important factor is the domain feedback function D). For molecule editing, we discuss two thresholds when evaluating with -). One is A 0 (loose condition). and the other is A 0 (strict condition), where the A value is different for each task. Here we conduct ablation studies on two conditions for feedback function D. The results are in Table 7 and the observation that ChatDrug- Turbo with the stricter threshold in feedback condition can lead to higher accuracy by a large margin. Note that for each task in Tables and 2
Ablation Study on the Thresholds in Feedback Condition Function.
we keep the same threshold for D and F.
Ablation Study on the Similarity Between Input and Output Drugs. We plot the distribution of similarities between input molecules Fin and retrieval æg. intermediate 1 and output molecules oou using ChatDrug-Turte
The similarity distributions of three tasks are in Figure and more results are in Appendix H One interesting observation is that the similarities between Din and intermediate molecules are quite I1, but the hit ratio is the lowest among the three. Then we plot the similarity Vm and x RR where the similarities are comparatively low, yet the hit ratio is the highest. This reveals that the ReDF module can explore the chemical space to search for more optimal solutions. Then, by utilizing such retrieved information and repeating C' conversational rounds, ChatDrug will do a trade-off between the similarity with input molecules Fin and knowledge explorations.
5 LIMITATION AND CONCLUSION
In this work, we present ChatDrug, a framework that utilizes LLMs for drug editing tasks. We build up a benehmark on 39 tasks over three main types of drugs: small molecules, peptides, and proteins. Empirical results have verified the effectiveness of ChatDrug on these drug editing tasks, and the visual analysis further qualitatively illustrates how ChatDrug can modify the key substructures for the target properties. Meanwhile, ChatDrug also possesses limitations. One limitation is that ChatDrug is bad at understanding the complex structures of drugs, i.e., the 3D geometries. This may require a more profound utilization of geometric modeling. Another limitation is that ChatDrug requires certain conversational rounds to reach strong performance. An ideal solution is to reduce such computational costs using the knowledge summarization ability of L.LMs, and we leave this for future exploration.
9

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
The codes and datasets can be found at this GitHub link
Additionally, we provide a detailed description of ChatDrug in Section 3 All the datasets used in our experiments can be found in each subsection of Section All prompts we used in ChatDrug are attached in Appendix FF] We also present our implementation settings and hyperparameters in Appendix
ACKNOWLEDGEMENT
Jiongxiao Wang and Chaowei Xiao are supported by the Department of Homeland Security under Grant Award I7STQAC00001-00-00.
REFERENCES
Drug definition. https: www. wwncer.ccc....o ccwpec/.coyyublllaaaaaaaoaaloola cancer tew-ttermssdeieee drug Accessed: 2023-04-22.
Dávid Bajusz, Anita Rácz, and Károly Héberger. Why is tanimoto index appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics, 7(1):1-13, 10:1113,,0101 2015.
Hans-Joachim Böhm, Alexander Flohr, and Martin Stahl. Scaffold hopping. Drug discovery today:
Technologies, 133):217-224, 2004.
Daniil A Boiko, Robert MacKnight, and Gabe Gomes. Emergent autonomous scientific research capabilities large language models. arXiv preprint arXiv:2304.05332, 2023.
Andres M Bran, Sam Cox, Andrew D White, and Philippe Schwaller. Chemerow: Augmenting large-language models with chemistry tools. arXiv preprini arXii:2304.05376, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 240.. 2020.
Ziqi Chen, Martin Renqiang Min, and Xia Ning. Ranking Ransing-based convolutional neural network models for peptide-mle mhe binding prediction. Frontiers molecular biosciences, 2021.
Ziqi Chen, Baoyi Zhang, Hongyu Guo, Prashant Emani, Trevor Clancy, Chongming Jiang, Mark Gerstein, Xia Ning, Chao Cheng, and Martin Renqiang Min. Binding peptide generation for mhc class proteins with deep reinforcement learning. Bioinformatics, 39(2):btad055, 2023.
David J Craik, David P Fairlie, Spiros Liras, and David Price. The future of peptide-based drugs. Chemical biology & drug design, 81(1):136-144) 14,, 2013.
Abie Craiu, Tatos Akopian, Alfred Goldberg, and Kenneth L Rock. Two distinct proteolytic processes in the generation of a major histocompatibility complex class i-presented peptide. Proceedings of the National Academy of Sciences, 14020)00855011855 1997.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18Xiv:1810.048805 2018.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00244 2022.
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. Gsum: A general framework for guided neural abstractive summarization. arXiv preprint arXiv:2010.08014 2020.
Jürgen Drews. Drug discovery: historical perspective. Science, 287(5460);1960-11964 2000.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan Adams. Convolutional networks on graphs for learning molecular fingerprints. Advances in neural information processing systems, 28, 2015.
10

---

Published as a conference paper at ICLR 2024
Carl Edwards, ChengXiang Zhai, and Heng Ji. Text2mol: Cross-modal molecule retrieval with natural language queries. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 595-607, 2021.
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, and Heng Ji. Translation between molecules and natural language. arXiv preprint arXiv:2204. 18817, 2022.
Peter Ertl, Eva Altmann, and Jeffrey M McKenna. The most common functional groups in bioactive molecules and how their popularity has evolved over time. Journal of medicinal chemistry, 63(15): 8408-8418, 2020.
Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint 2rXiv:2206.08853 2022.
Michelle Francl. Chatgpt saves the day. Nature Chemistry, pp. 1-2, 2023.
Simon Frieder, Luca Pinchetti, Ryan-Rhya Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. Mathematical capabilities of chatgpt. arXiv preprint arXiv:2301.1386; 2023.
Sven Frokjaer and Daniel Otzen. Protein drug stability: a formulation challenge. Nature reviews drug discovery, 244):298-306, 2005.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text- annotation tasks. Proceedings of the National Academy of Sciences, 120x30):23350016120, 2023. 2305016120, pnas. 2305016120)
doi: 10.
URL https: //wp::///wwwwww// pna org /oi ottp://////b////.0100//
Laurent Gomez. Decision making in medicinal chemistry: The power of our intuition. ACS Medicinal Chemistry Letters, 9(10)956-955 958,, 2018.
Sai Krishna Gottipati, Boris Sattarov, Sufeng Niu, Yashaswi Pathak, Haoran Wei, Shengchao Liu, Simon Blackburn, Karam Thomas, Connor Coley, Jian Tang, et al. Learning to navigate the synthetically accessible chemical space using reinforcement learning. In International Conference on Machine Learning, pp. 3668-3679, PMLR, 2020.
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. Ptr: Prompt tuning with rules for text classification. Al Open, 3:182-192, 2022.
Junxian He, Wojciech Kryšciński, Bryan McCann, Nazneen Rajani, and Caiming Xiong. Ctrlsum:
Towards generic controllable text summarization. arXiv preprint arXiv:20/2.0428// 2020.
Jens Hennecke and Don C Wiley. T cell receptor- mhe interactions up close. Cell, 104(1): 1:):)::14, 2001. 2001.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. biokxiv, 2022.
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. Knowledgeable prompt- -ouptt-uuning Incorporating knowledge into prompt verbalizer for text classification. arXiv preprint arXiv:2 108. 02035, 2021.
Ye Hu, Dagmar Stumpfe, and Jurgen Bajorath. Recent advances in scaffold hopping: miniperspective. Journal of medicinal chemistry, 60(4):1238-12) 1246, 2017.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models self-improve. arXiv preprint arXiv:2210.11661 2022.
John Ingraham, Max Baranov, Zak Costello, Vincent Frappier, Ahmed Ismail, Shan Tie, Wujic Wang, Vincent Xue, Fritz Obermeyer, Andrew Beam, et al. Illuminating protein space with a programmable generative model. bioRxiv, 2022. doi: 10.1 (0.1101/202.1) 12.06.51862..
John J Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman. Zine: free tool to discover chemistry for biology. Journal of chemical information and modeling, 52(7): 1757-1768, 2012.
11

---

Published as a conference paper at ICLR 2024
Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: a pre-trained transformer for computational chemistry. Machine Learning: Science and Technology, 3(1): 015022, 2022.
Madura KP Jayatunga, Wen Xie, Ludwig Ruder, Ulrik Schulze, and Christoph Meier. Ai in small- molecule drug discovery: A coming wave. Nat. Rev. Drug Discov, 21:175-176, 2022.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hicrarchical generation of molecular graphs using structural motifs. In International conference machine learning, pp. 4839-4848, PMLR, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588) 2021.
Steven Kearnes, Kevin McCloskey, Mare Berndl, Vijay Pande, and Patrick Riley. Molecular graph convolutions: moving beyond fingerprints. Journal of computer-aided molecular design, 30: 595-608, 2016.
Michael Schantz Klausen, Martin Closter Jespersen, Henrik Nielsen, Kamilla Kjaergaard Jensen, Vanessa Isabell Jurtz, Casper Kaae Soenderby, Morten Otto Alexander Sommer, Ole Winther, Morten Nielsen, Bent Petersen, et al. NetSurff-220:: improved prediction of protein structural features by integrated deep learning. Proteins: Structure, Function, and Bioinformatics, 87(6): 520-527, 2019.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint 2rXiv:2255.19916 2022.
Greg Landrum et al. RDKit: A software suite for cheminformatics, computational chemistry, and predictive modeling, 2013,
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan Tom Sercu, Sal Candido, et al. Language models of
dos Santos Costa, Maryam protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023a.
Shengchao Liu, Moayad Alnammi, Spencer S Ericksen, Andrew F Voter, Gene E Ananiev, James L Keck, Michael Hoffmann, Scott A Wildman, and Anthony Gitter. Practical model selection for prospective virtual screening. Journal of chemical information and modeling. 59(1):282-29) 2933, 2018.
Shengchao Liu, Mehmet F Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised repre- sentation for graphs, with applications molecules. Advances in neural information processing systems, 32, 2019.
Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Anima Anandkumar. Multi-modal molecule structure-text model for text-based retrieval and editing. arXiv preprint arXiv:2212.10788: 2022a.
Shengchao Liu, Chengpeng Wang. Weili Nie, Hanchen Wang. Jiarui Lu, Bolei Zhou, and Jian Tang. GraphCG: Unsupervised discovery of stecrable factors in graphs. In NewrIPS 2022 Workshop: New Frontiers in Graph Learning, 2022b. URL httph::t////po..t https:://ppeeeeeeeww.... net forum? fummm idd=
BhR44NzeK I
Shengchao Liu, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Anthony Gitter, Chaowei Xiao, Jian Tang, Hongyu Guo, and Anima Anandkumar. A text-guided protein design framework. arXiv preprint arXiv:2302.04611 046/ 2023b.
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.00 034997, 2020.
12

---

Published as a conference paper at ICLR 2024
Zlatko Mihalić and Nenad Trinajstié. A graph-theoretical approach struttur-properte relationships. Journal of Chemical Education, 69(9),701, 1992.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zeetlemoyer. Rethinking the role demonstrations: What makes in-context learning work? arXiv preprint ar arXiv:2202.128377 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
Timothy J O' O'Donnell, Alex Rubinsteyn, and Uri Laserson. Mheflurry 2.0: improved pan-allele prediction of mhc class i-presented peptides by incorporating antigen processing. Cell systems, 11 (1):42-48 2020.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen Or, and Dami Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings the IEEE/CVF International Confer- ence on Computer Vision, pp. 2085-200000 2094, 2021.
Baolin Peng, Michel Galley, Pengeheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint 2302:::2002//8....3 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 87688 PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, (1(1):4485-555 -5551, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with TAPE. Advances in neural information processing systems, 32, 2019.
Alexander Rau, Stephan Rau, Daniela Zöller, Anna Fink, Hien Tran, Caroline Wilpert, Johanna Nattenmüller, Jakob Neubauer, Fabian Bamberg, Mareo Reisert, and Maximilian F. Russe. A context-based chatbot surpasses radiologists and generic chatgpt in following the acr appro- priateness guidelines. Radiology, 308(1):e230970, 2023. doi: 10.1 10.1148/adiioo.230977.. URL
https:///o1: http:///ooo..r////.. http:////....1//./a..ooooo...... httpppps//oo..oor//.....aaaaoo..
Sebastian G. Rohrer and Knut Baumann. Maximum unbiased validation (muv) data sets for virtual screening based on pubchem bioactivity data. Journal of Chemical Information and Model- ing, 1992))1691188) 84, 2009. doi: 10. 1010//80000,,,, URL htttp:://doot http///i...oo....1//
118002649 PMID: 19161251.
Timo Schick and Hinrich Schütze. Few-shot text generation with pattern-exploiting training. arXiv preprint arXiv:20/2. /926, 2020.
LLC Schrödinger and Warren DeLano. Pymol. URL URLhhtt::///ww.pmm www.pp/ww.........o........
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint 2333 2023.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
13

---

Published as a conference paper at ICLR 2024
Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji-Rong Wen. A molecular multimodal foundation model associating molecule graphs with natural language. arXiv preprint arXiv: MxXiv:2209.05444 05481, 2022.
Thomas Sullivan. A tough road: cost develop one new drug $2.6 billion; approval rate for drugs entering clinical development is less than 12%. Policy & Medicine, 2019.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint www.i::2211.90000 2022.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, pp. 1-11, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Zichao Wang, Weili Nie, Zhuoran Qiao, Chaowei Xiao, Richard Baraniuk, and Anima Anandkumar. Retrieval-based controllable molecule generation. arXiv preprint arXiv:2208. 11126, 2022.
Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? arXiv preprint 1rXiv:2109.0/247 2021.
David Weininger. Smiles, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
David S Wishart, Craig Knox, An Chi Guo, Dean Cheng, Savita Shrivastava, Dan Tzur, Bijaya Gautam, and Murtaza Hassanali. Drugbank: a knowledgebase for drugs, drug actions and drug targets. Nucleic acids research, 36(suppl 1t::111:DDDD00- D906, 2008
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. ICLR, 2021.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv: 08237 2019.
Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature communications, 13(1):862 2022.
Yanjian Zhang, Qin Chen, Yiteng Zhang, Zhongyu Wei, Yixu Gao, Jiajie Peng, Zengfeng Huang, Weijian Sun, and Xuan-Jing Huang. Automatic term name generation for gene ontology: task and dataset. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4705 4710, 2020.
Yongchao Zhou, Andrei Toan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022.
14

---

Published as a conference paper at ICLR 2024
A THE UTILIZATION OF CHATGPT AS BACKBONE LLM
Some concerns have been raised on the utilization of using ChatGPT as the backbone LLM. We want explain this from multiple perspectives.
(I) Our ChatDrug is agnostic to the backbone LLMs, and we test in-total three backbone LLMs: Turbo (ChatGPT), GALACTICA, and Llama2. Among these, ChaiDrug-Tuubo achieves the best performance and highest stability for all three drug types. For ChatDrug on the open-source LLMs, ChatDrug-GAlI GALACTICA only performs better than bascline on the small molecule editing and peptide editing, but cannot deal with protein editing tasks. ShatDrug-LLlama can reach better performance than baseline methods on peptide editing and protein editing but fails to cover small molecule editing tasks, Such unstable performance across different drug types may be attributed to the limited domain- specific training data for open-source LLMs. However, the discussions about training LLMs are beyond the scope of this paper.
(2) In general, ChatGPT has been widely used in both the machine learning and domain (chemistry and biology) community. For example:
scientific papers that use ChatGPT for discovery have been published in top journals such PNAS (ChatGPT Outperforms Tasks (Glardi et al. 2023)), Radiology (A Context-based Chatbot Surpasses Trained Radi- ologists and Generic ChatGPT Following The ACR Appropriateness Guidelines (Rau et al. 2)23)), Nature Medicine (Large Language Models in Medicine (Thirunavukarasu
In domain journals:
for Text-Annotation
et al. 2023 ), and Nature Chemistry (ChatGPT Saves The Day (Francl) 2023)).
In machine learning communities: many machine learning papers with ChatGPT usage have been recently accepted by NeurlPS 2023 including HuggingGPT: Solving AI Tasks with ChatGPT and its Friends Hugging Face (Shen et al., 2023). Mathematical Capabilities of ChatGPT (Frieder et al. 2023) and ChaGGPT-Powered Hierarchical Comparisons for Image Classification.
B EXAMPLES OF CHATDRUG
Here we show three concrete examples of the editing tasks on small molecules, peptides, and proteins.
Can culla mutlacus
placs tak dile andiioum padiages busd the rnckeca Give na thi indeccutee using
explenabie mented
Sars hers
SMBLES
hyshegee band diners
OeO www.On/CCCCOoooooo (wt.o/onnnicco//ooo Tie Necec/MICCOCEHH0
loqparss
cerris and aeraencc
Glec (O
oheer aediodde
is medicale but iscresssd sulabiiity by
Figure 6: Example of small molecule editing (task 203).
15

---

Published as a conference paper at ICLR 2024
pocride AdAGie pepttde lela coally aryut and seporditi di mockee
AMGHITSL
Tour proviled cnique MlAaaaii aud porruct and sin to papele provided.
pestide avquence that may bind to HiAA- 44000 could be
corruct
Figure 7: Example of peptide editing (task 301).
furms pnodia dutrro
IiMaOOO Can spe mariin
ansuctunot The rour and cxltaul probiis snca anccea
You san readl pross cari aesuanac
petoduciing
the skarpp mails belowss
ifeeaaed the umerro anda rt" posiiiee with TL" which has suulcced aelara 1e"
whlcch
formation.
itlelccdd aniire
position whichh
begs properail marka tommatiis
reculing anatee sapuoeen
Tores Lit da perss ANTICIII saras piEiDc
This requanaa should lave incrussad lut dandan larra
Figure 8: Example of protein editing (task 501).
16

---

Published as a conference paper at ICLR 2024
C MORE DISCUSSIONS ON CHATDRUG
In this section, we will discuss three aspects of ChatDrug and the general conversational LLMs for drug discovery: the scope, the main attributes, and the key challenges and guidelines when using ChatDrug.
Scope of ChatDrug. The first natural question raised is What are the feasible drug discovery tasks for ChatDrug? To answer this question, we need to reiterate the conversational LLM's feasibility for drug discovery tasks. There have been a series of works Zeng et al. 2022 [dwards et al., 2022 2021]
Taylor et al. 2022; [Su et al., 2022) exploring the LLMS for small molecule and protein discovery, ranging from molecule representation to text-to- toloeccll generation. These are important machine learning tasks, revealing domain data's (conditional) distribution learning ability. However, there exists another important task in real scenarios: drug editing (a.k.a. lead optimization protein design in domain applications). This is a routine task in pharmaceutical companies, and it aims at updating the molecule's 's substructures (Mihalić & Trinajstii 1992), related to certain key tacties in drug discovery like functional group change (Ertl et al. 2020) and scaffold hopping (Böhm et al. 2004 Hu et al. 2017). Traditional solutions relying on domain experts for manual editing can be subjective or biased (Drews) 2000; Gomez 2018). To alleviate this issue, mulli-moda models with LLMs provide a promising solution, and recent works (Liu et al. 2023b; 2022a) have started explore text-guided drug editing or controllable drug generation. However, these works are domain-specifer (e.g., only for small molecules or proteins) and do not possess conversational potentials like ChatGPT.. In contrast, ChatDrug possesses outstanding generalization abilities to various domain tasks and enables conversational refinement in drug editing tasks.
Attributes of ChatDrug. We conclude three fundamental attributes of ChatDrug: open vocabulary, compositionality, and inspiration. (1) Open vocabulary means ChatDrug is not limited to a fixed set of pre-defined drug-related annotations. Instead, it can generalize to novel drug concepts due to the unbound vocabulary depicted by the natural language. (2) Compositionality means we can express complex concept by decomposing it into simple concepts. (3) Inspiration means the position of ChatDrug is to inspire domain experts with knowledge exploration but not replace them. A concrete example illustrating these three attributes is multi-objective lead optimization. We can use natural language to guide us to generate an entirely new attribute of a molecule (open vocabulary); meanwhile, the new attribute is composed of multiple simple attributes, such as binding to new protein and high permeability (compositionalityy) Finally, such an optimized molecule may not be directly used for real scenarios, but can provide insights for domain experts in drug design (inspiration).
Challenges and Guidelines when Using ChatDrug. Now that we have decided to narrow ChatDrug to the drug editing tasks, we need to scrutinize more details before deploying it. With careful reconsidering, we summarize two main challenges that we need to keep in mind. (1) ChatDrug can do better in fuzzy searching than exact searching in drug editing tasks. Drug editing tasks, or drug controllable generation, can cover various topics. However, one critical difference between ChatDrug and other LLMs in vision tasks is that ChatDrug or drug discovery is a scientific problem while the image and video (Radford et al. 202 Nichol et al.) [022 Ramesh et al., 2022] Patashnik et al. 2021 Fan et al. (002) generation is more of an artistic endcavor. Namely, for text prompts like want add an isobutyl group on the 3-position of the aromatic ring in Aspirin", domain experts can do this precisely, and thus the impact of ChatDrug is limited here. However, for other tasks like "I want to modify this molecule to be more soluble in the water", the results are not deterministic, and this is where LLMs can act as a more useful tool to inspire the domain experts. These two types of text prompts are called exact searching and fuzzy searching, respectively. We conclude that ChatDrug is more beneficial for the fuzzy searching problem. (2) ChatDrug relies on the pretrained LLMs, initially pretrained on a large-scale and universal corpus. Thus, there is a noticeable domain shift when applying them to domain-specifie tasks. However, as will be shown in Sections 3 and 4 the existing LLMs illustrate the interpretation ability of the domain knowledge. Though such interpretation is preliminary, we believe that ChatDrug is an inspiring and promising direction for future usage in both communities.
"e acknowledge that there have been vertain parallel works Brun et al. 2023 Baiko et al. 2023 exploring conversational L.Mss on reaction and synthesis tacks.
17

---

Published as a conference paper at ICLR 2024
D RELATED WORK
D.1 MULTI-MODAL MODELING FOR SMALL MOLECULE DISCOVERY
Small molecules can be roughly categorized into two big modalities Zeng al. 2022 Liu al. 2022ay: the internal chemical structure and external description. The internal chemical structure refers the molecule's 's structure information, e.g., ID sequence (SMILES), 2D molecular graph, and 3D geometric graph. On the other hand, the external description depicts the high-level information of molecules, e,g,, the molecule's binding affinity with potential targets, and the functionalities of molecules.
Recently, research line has been starting to bridge the gap between such two modalities. KV- PLM (Zeng et al.) 2022) first applies the joint masking auto-encoding the SMILES string and biomedical textual description. Text2Mol (Edwards et al. 202T) conducts contrastive learning between molecular graph and text data for retrieval tasks between modalities. MolT5 (Edwards et al. 2022) does the translation between SMILES and textual annotation of molecules a mutual way. MoMu (Su et al., 2022) also conducts contrastive learning, while it considers both the retrieval and molecule captioning and text-to-molecule tasks. MoleculeSTM (Liu et al. 2022a) proposes larger molecule-text dataset and highlights the text-guided molecule editing tasks. Such tasks reveal the potential of LLMs for more realistic drug discovery tasks.
D.2 MULTI-MODAL MODELING FOR PEPTIDE AND PROTEIN DISCOVERY
There have also been several works exploring multi-modal modeling for protein discovery. Pro- Gen (Madani et al. 2020) is rext-to-eqqeene protein design framework, but it is fixed to predefined set of texts, which can treated with indices. Thus it not open-vocabulary and lacks the generaliza- tion ability novel textual deseriptions. Besides, the predefined texts and indices cannot sufficiently describe the protein functions (Zhang et al. 2020). ProteinDT (Liu et al., (023b) a recent work that addresses this issue with the free-fext protein design. parallel work is Chroma (Ingraham et al.) 2022), and it conducts text-guided protein editing on the backbone structure instead of the sequence.
18

---

Published as a conference paper at ICLR 2024
E DATA SPECIFICATION
Drugs like small molecules and proteins can have multiple modalities. Specifically, small molecules can be naturally represented ID sequence, 2D molecular graph, and 3D geometric graph, biological knowledge graph, and textual description. The first three data structures capture the internal chemical structure information, while the last two data structures provide higher-level view of the molecule's functionalities (e.g,, the molecule's s interactions with other proteins or diseases.).
There are 20 amino acids in nature, as listed below:
Table 20 amino acids and corresponding abhreviations.
Amino Acid
Alphabet
Isoleucine
Valine
V
Leucine
Phenylalanine
F
Cysteine
Methionine
M
Alanine
A
Glycine
Threonine
T
Serine
Tryptophan
W
Tyrosine
Proline
P
Histidine
H
Asparagine
N
Asparatic acid
OORRA
Glutamine
Glutamic acid
Lysine
Arginine
R
19

---

Published as a conference paper at ICLR 2024
F
TASK SPECIFICATION
Here we present all the task specifications and prompts used in our experiments.
Llama2, list the template of prompts of two stages For ChatDrug- Turbo and ChatDrug-Llama2, of PDDS and ReDF in Tables and 15 for small molecules, peptides, and proteins, respectively.
Different from ChatDrug-Turbo and ChatDrug-Llama2, the base model of ChatDrug- GALACTICA, GALACTICA-6.77 is not instruction tuned. Besides. GALACTICA can support Question Answering tasks and has additional special tokens for marking the start and the end of molecules and protein annotations. Such differences make us to choose different template of prompts specifically designed for ChalDrug-GALAATICA GALACTICA, which are
shown Tables I0 13 and [6
We list the corresponding task requirement and allele type information in Tables 17
14
We further list the prompts of in-context learning in Table 18 for reference.
Table 9: Prompt for small molecule editing. The task requirement he found in Table 11
Task
Module
Prompt
Can you make molecule [input SMILES) [task requirement 1]? The output molecule should be similar to the input molecule. Give me five molecules in SMILES only and list them using bullet points. No explanation is needed.
PDDS
Ixx
(101-108)
Your provided sequence [output SMILES] is not correct. We find a sequence [retrieved SMILES) which is correct and similar to the molecule you provided. Can you give me new molecule?
ReDF
Can you make molecule [input SMILES] [task requirement I] and [task require- ment 21 The output molecule should be similar the input molecule. Give me five molecules SMILES only and list them using bullet points. No explanation is needed.
PDDS
2xx (201-206)
Your provided sequence [output SMILES] is not correct. We find a sequence [retrieved SMILES] which is correct and similar to the molecule you provided. Can you give mc new molccule?
RcDF
Table 10: Prompt for small molecule editing with BRaaDrug-GLLACTIC The task requirement be found Table
Task
Module
Prompt
Question:
Can you make molccule SYTART_LSMILES///me SMILESIJEND, _ SMILES] [task requirement 1]? The output molccule should be similar to the
PDDS
input molecule.An
Ixx
(101-108)
Question:
Your provided sequence [START] [SMILES][ SMILES]JEND L_MMILSS] is correct. We find sequence [START] [rctriedd SMILESJJEND L_ SMMLLS) which correct and similar to the molecule you
RcDF
provided. Can you give me a new molecule'An
Question: Can you make molecule [START] SMILES SMILESJIEND SMILES] [task requirement 1J and [task requirement 2J? The output molecule should be similar to the input molecule.
PDDS
2xx
Your provided sequence ISTART_LSS LLLLLSLLIS[[Suurr SMILES/JEND [_SMILES] is not correct. We find sequence [START__SMIL J_S_MSMILES [retrieeeddd SMILES]JEND [_SMILES) which is correct and similar to the molecule you provided. Can you give me new molecule?!n
Question:
(201-206)
RcDF
20

---

Published as a conference paper at ICLR 2024
Table 11: Task requirement for small molecule editing, corresponding to Table 9 and Table 10
Task ID
Task Requirement 1
Task Requirement
101
more soluble in water more like drug less like a drug
None
103
None
104
None
105
higher permeability
None
106
lower permeability
None
107
more hydrogen bond acceptors
None
108
more hydrogen bond donors
None
201
more soluble in water less soluble in water more soluble in water
more hydrogen bond acceptors more hydrogen bond acceptors more hydrogen bond donors more hydrogen bond donors higher permeability lower permeability
202
203
204
less soluble in water more soluble in water
205
206
more soluble in water
Table 12: Prompt for peptide editing. The source allele target type and target allele type can be found in Table 14
Task
Stage
Prompt
We want peptide that binds to ftarget allele type 11. We have peptide fimput peptide] that binds to [source allelc type], can you help modify it? The output peptide should be similar to input peptide. Please provide the possible modified peptide sequence only. No explanation is needed.
PDDS
3xx
(301-306)
Your provided sequence [output peptide] is not correct. We find sequenee |retrieved peptide] which is correct and similar the peptide you provided. Can you give mc new peptide?
RcDF
We want peptide that binds [target allele type 1J and [target allele type 2]. We have a peptide linput peptide that binds to [source allele type]. can you help modify it? The output peptide should be similar to input peptide. Please provide the possible modified peptide sequence only. No explanation is needed.
PDDS
4xx
(401-403)
Your provided sequence [output peptide] is not correct. We find a sequence [retrieved peptide] which is correct and similar to the peptide you provided. Can you give new peptide?
RcDF
Table 13: Prompt for peptide editing with ChatDrug-GALACTIIC The source allele target type and target allele type can be found in Table
Task
Stage
Prompt
Question: We want peptide that binds to [target allele type 1]. We have peptide finput peptide] that binds [source allcle typel, can you help modify it? The output peptide should be similar to input peptide,An
PDDS
3xx
Your provided sequence foutput peptide] sequence [retrieved peptide] which is correct and similar to the peptide you provided. Can you give me new peptide?\n
Question:
correct. We find
(301-306)
ReDF
Question: We want a peptide that binds to [target allele type 1] and [target allele type 2). We have a peptide linput peptide] that binds to [source allele type], can you help modify it?Nn
PDDS
4xx
Your provided sequence [output peptide] is not correct. We find a sequence [retrieved peptide] which is correct and similar to the peptide you provided. Can you give me new peptide?<n
Question:
(401-403)
ReDF
21

---

Published as a conference paper at ICLR 2024
Table 14: Target allele type and source allele type for peptide editing, corresponding to Table 12 and Table 13
Task ID
Source Allele Type
Target Allele Type
Target Allele Type 2
301
HLA-C* 16:01
HLA-B*44:02
None
302
HLA-B*08:01
HLA-C*03:03
None
303
HLA-C* 12:02
HLA-B HLA-B+40:01 #44001
None
304
HLA-A 11:00
HLA-B 00:001
None
305
HLA-A*24:02
HIA-BB08:01
None
306
HLA-C*12:02
HLA-B+40:02
None
401
HLA-A #2:002
HLA-B*08:00
HLA-C* 15:02
402
HLA-A*03:01
HLA-B#40:02
HLA-C 14:02
403
HLA-C* 14:02
HLA-B*08:01
ILA-A*1I:00
Table 15: Prompt of Conversation Module for protein editing. The task requirement can be found in Table 17
Task ID
Prompt
We have protein [input protein]. Can you update modify it by [task require- ment]? The input and output protein sequences should be similar but different. No explamation needed.
PDDS
Sxx (501-502)
Your provided sequenee [output protein] is not correct. We find sequenee [retrieved protein] which correct and similar the protein you provided. Can you give mc new protein?
RcDF
Table 16: Prompt of Conversation Module for protein editing with ChatDrug GALACTICC The task require- ment can be found Table
Task ID
Prompt
We have a protein [START AMINOA/iNo protein|[END_ANAI AMINOJ. Can you update modify it by [task requirement]? The input and output protein sequences should be similar but different.An
Question:
PDDS
5xx (501-502)
Question: Your provided sequence ISTART Autpu protein][EN] AMIINJ is not correct. We find a sequence [START AMMiNO/rreetrveeed AMINOJ cein|EEDDAMMINOO
ReDF
which is correct and similar the protein you provided. Can you give me a new protein'An
Table 17:
Task requirement for protein editing, corresponding Table 15 and Table 16
Task ID
Task Requirement
501
making more amino acids into the helix structure (secondary structure) making more amino acids into the strand structure (sccondary structure)
502
22

---

Published as a conference paper at ICLR 2024
Table 18: Prompt of in-context learning.
Task
Prompt
Can you make molecule [input SMILES] [task requirement]? The output molecule should be similar the impul molccule. We have known that similar molecule fretrieved SMILES] is one the correct answers. Give me another five molecules in SMILES only and list them using bullet points. No explanation is needed.
1xx
(101-108)
Can you make molecule [input SMILES] [task requirement 1] and [ask requirement 2J The output molecule should be similar the input molecule. We have known that similar molecule fretrieved SMILES] is one of the correct answers. Give me another five molecules in SMILES only and list them using bullet points. No explanation is needed.
2xx
(201-208)
We want peptide that binds to [target allcle type]. We have peptide [input peptide] that binds to [source allele type], can you help modify it? The output peptide should be similar to imput peptide. We have known that similar peptide [retrieved peptide] is one of the correct answers. Please provide another possible modified peptide sequence only. No explanation is needed.
3xx
(301-306)
We want peptide that binds to [target allele type 1] and [target allele type 2]. We have peptide [input peptide] that binds [source allele type], can you help modify it? The output peptide should be similar to input peptide. We have known that similar peptide [retrieved peptide] is one of the correct answers. Please provide another possible modified peptide sequence only. No explanation needed.
4xx
(401-403)
We have a protein [input protein]. Can you update modify by [task requirement]? The input and output protein sequences should be similar but different. We have known that similar protein [retrieved protein] is one of the correct answers. Please provide another possible modified protein only. No explanation is needed.
5xx
(501-502)
23

---

Published as a conference paper at ICLR 2024
G IMPLEMENTATION AND HYPERPARAMETERS
G.1 CHATGPT-TURBO SETTINGS
We implement experiments with ChatGPT-Turbo through OpenAl API. Specifically, utilize the model 1pn--3.5-turbo-030 under ChalComplelion function, which a snapshot of 9pl-3.5-turbo from March 1st 2023. This model will not receive updates, so can ensure the reproducibility of our results. We also set the temperature to 0 to reduce the potential randomness in our experiments. Additionally, we observe that ChatGPT often generates repeated sequences fails to stop generating sequences for themistry-related questions. To mitigate this issue, we set the frequency Trequency_penalty to 0.2. Moreover, for improved adaptation different domains, advisable to incorporate a system role prompt within ChatGPT. In our case, we utilize the following prompt: "You are an expert in the field of molecular chemistry."
G.2 OPEN SOURCE LLMS USED WITHIN CHATDRUG
For ChatDrug-GALann GALACTICA and ChatDrug-Llamss we used the open source model checkpoint through Hugging Face. Corresponding model names in Hugging Face are facebookggalactico-6 7b for ChatDrug ChatDrug-GAALATT GALACTICA and meta-llamal Llama/L (l-ma-2-7b-chathh 7b-chat-h for SeatDrug-Llaman For implementation, set all parameters default parameters.
G.3 EXPERIMENTS THRESHOLD FOR SMALL MOLECULE EDITING
Following MoleculeSTM (Lu et al. 2022a). in our small molecule editing experiments, utilize two different threshold settings: loose threshold and striet threshold. For the main results in Tables [ a we keep the same threshold for domain feedback function D and evaluation function E. The threshold A used for each small molecule editing task is shown in Table 19 which holds for both functions.
Table 19: Threshold for each small molecule editing task, A₁ and Ag represent threshold task requirement and task requirement 2, respectively.
Loose Threshold
Strict Threshold
Task ID
A
A2
A
A
101
0
0.5
102
0.5
103
0
0.1
104
0.1
105
0
10
106
10
107
1
108
201
0
0
0.5
202
0
0.5
203
0
0
0.5
00000
00000
204
0.5
205
0
0000
0.5
10
206
0.5
10
G.4 EXPERIMENTS THRESHOLD FOR PEPTIDE EDITING
For the peptide editing task, mentioned in Section 4 take the threshold one-half of the average binding affinity of experimental data on the target allele. The original average binding affinity of each experimental data be found in the source code.
24

---

Published as a conference paper at ICLR 2024
G.5 EVALUATION METRIC
We evaluate the performance of ChatDrug by hit ratio, which is computed by the following equation:
Number of Success Sequence Editing
Hit Ratio
Number of Valid Sequence Editing
One point we need to highlight is that if ChatDrug returns an invalid sequence, we would just skip and do not consider it in computing the hit ratio. That is why we use "Number of Valid Sequence Editing" the denominator here.
In small molecule editing tasks, ChatDrug tends to return more than one sequence the PDDS module. Thus, we add prompt "Give me five molecules in SMILES only and list them using bullet points." to unify the numbers and format molecules returned by ChatDrug. In the experiments of the Conversation module, always choose the first valid molecule the beginning of the conversation. We further carry out an ablation study to explore the effect of using more molecules in the PDDS module.
G.6 RANDOMNESS
The experiment results of the PDDS Module are entirely deterministic. Any randomness observed in ReDF Module and Conversation Module due to the utilization of different seeds during the sampling of retrieval database DB from ZINC for molecule editing.
Specifically, for small molecule editing, we adopt seed 0,1,2,3,4 for main results in Tables and and seed O for the other ablation studies.
G.7 COMPUTATIONAL RESOURCES
All of our experiments for ChatDrug- Turbo are conducted on a single NVIDIA RTX A5000 GPU, The GPU is only used for peptide and protein evaluation. The primary cost incurred during our experiments comes from the usage of the OpenAl API for ChatGPT, which amounted to less than $100 in total.
For open source LLMs backbones, NVIDIA RTX A5000 GPUs for small molecule editing and peptide editing. For protein editing tasks, due to extra usage of GPU for protein evaluation, 4 NVIDIA RTX A5000 GPUs are needed in our experiments.
and SaatDrug-Llamad need at least
25

---

Published as a conference paper at ICLR 2024
H QUALITATIVE ANALYSIS
In the main body, we provide 10 case studies and similarity distributions to illustrate the effectiveness of ChatDrug for small molecule editing, peptide editing, and protein editing.
In this section, we provide additional case studies and similarity distributions as follows:
We list case studies functional group change of small molecules in Appendix H.I.1
We list 14 similarity comparisons on small molecules in Appendix [H.1.2
We list motif updates for all 9 peptide editing tasks in Appendix H.2
We list case studies on secondary structure change of proteins in Appendix H3
We want specify that for all the qualitative analyses listed here, rounds. Especially for small molecules, we consider random seed with 0 and the loose threshold,
using C conversation
i.e., A for all tasks.
H.1 SMALL MOLECULES
H.1.1 FUNCTIONAL GROUP CHANGE ON SMALL MOLECULES
Table 20 visualizes examples of molecule editing tasks where ChatDrugTTute -urbo successfully gener- ates output molecules Your with desirable property change, while the output of the first conversation round æ1 fail. In Table 20 and b, Zout successfully adds the desirable fragments alter the drug likeness of Imn, while does so in the wrong direction. In Table 20c. æ1 installs a chloride but maintains same number of hydrogen bond acceptors (HBAs). In contrast, ChatDrug-Turiis adds a salicylamide moiety that brings two more HBAs. Similarly, in Table 20d, the number of hydrogen bond donors (HBDs) remains 2 but successfully increases Fot via insertions of alcohols and amines.
In Table 20c and both cases æ; are able to increase the number of HBAs as indicated in the prompt, but the water solubilities shift oppositely. The output molecules successfully fix the trend. In particular, hydrophibicity is appropriately employed Table 20 to balance the additional polarity from HBAs, generating a less soluble molecule. In Table 20 and h, both cases of æı satisfy the solubility requirement but not through the change of HBDs. In Dot, the problems are solved by having extra HBDs with further enhanced solubility changes.
26

---

Published as a conference paper at ICLR 2024
Table 20: Visualization of additional eight small molecule editing cases. The blue regions red regions and green regions correspond to the edited substructures in the input molecule Zin. intermediate molecule in the 1st conversation round, and the output molecule Cour, respectively.
no Pronge for tot: new like drag
09 Froeng for 1il: los like dreg
Chepst
QED 0:48
QED 0,550
QED: GALS
QED: 6631
QED 0J557
QED: 0,307
byshogon seoepeon
lnpsi Midccale
Insemnodiaee Motecule
Ourpor Melazule
lepai Molccale
Ixsennedatte Melecale
Ouapur Molocsie
HBA: 2
HBX
HBAL4
HBD
HBD
HBD
66) Pionpt for 201, moxe wilille in water and none hydinge boal acceptecs
03 Proceet for 200 less sntable water and more tydloges hond acceptors Ouapor Milecuie Iue
basat Mixiculle at.
Iocusrediitt Motecole in
Oupor Melocale Ar
lapai Melccale av
Brceuediate Melecale in
Light
Ligh:
Lagh
Light
Eagh
(g) Pronge more volille wave and now hyshogre hand Bnon
Proeage fee voliiiee water tyshegre dences
Chepsr
LigP: 1.20, HBD:
Logi: HBD.
LugP: AO7, HBD.
LogP. 2.14, HBD. 0
LagP: 2775 HBD
LogP: 3,22, HBD
27

---

Published as a conference paper at ICLR 2024
H.1.2 SIMILARITY BETWEEN INPUT, INTERMEDIATE, RETRIEVED, AND OUTPUT MOLECULES
In Figure 5 we plot the distribution of similarities between input molecules Zin and retrieval RR. intermediate 21, and output molecules zout using ChatDrug Turbo. Here we provide more results. The distributions of 8 single-objective small molecule editing tasks can be found in Figure 9 and multi-objective small molecule editing tasks can be found in Figure T0
ChatDrug-Turbo.
As shown in Figures and TO the observation of similarity distribution satisfies for all 8 single- objective and 6 multi- oojeittobeeeivee tasks.
X)
Task 101.
(b) Task 102.
(c) Task 103,
Dcp Xp1
Dg. Keal
Io. Xsl
Dix X)
Ie.. Xil
Oa. X)
(d) Task 104.
(e) Task 105.
(I) Task 106.
9md
(g) Task 107.
(h) Task 108.
Figure 9: Similarity distribution between input molecules and retrieval intermediate æ1, and output molecules Don- Here show the distribution of single-objective editing tasks.
Dio Kel
Day Xaxl
Ii. Xsl
Olx X1)
Iru. Xil
Dou
Oax Xt
02
30
60
50
(a) Task 201.
(b) Task 202.
(c) Task 203.
My Kue)
Uox Acl
(d) Task 204.
(e) Task 205.
(f) Task 206.
Figure 10: Similarity distribution between input molecules ai and retrieval æn, intermediate æi, and output molecules Tag. Here we the distribution of multi-ohjective editing tasks.
28

---

Published as a conference paper at ICLR 2024
H.2 PEPTIDE
In the main body, we have illustrated how the motif of peptides changes for two peptide editing tasks. Here show all single-objective editing tasks in Figures 11 to 16
For task 301 in Figure 11 ChatDrug-Turbo can successfully upweight E. (Glutamic acid)
at position 2.
For task 302 in Figure 12
ChatDrug- Turbo can successfully upweight A (Alanine) at
position 2. and L (Leucine) at position 9.
For task 303 in Figure 13
ChatDrug-Turtee can successfully upweight E (Glutamic acid)
at position 2, and L (Leucine) at position
For task 304 in Figure 14
ChatDrug- Turbo can successfully upweight R (Arginine) and (Leucine) at position 9.
K (Lysine) at position 5, and
For task 305 in Figure 15
ChatDrug- Turbo can successfully upweight R (Arginine) and
K (Lysine) at position 5, and L (Leucine) at position 9.
ChatDrug- Turbo can successfully upweight E (Glutamic acid) at position 2, and L (Leucine) at position 9.
For task 306 in Figure T6
E
(a) Motifs of input peptides.
(b) Motifss of edited peptides.
(c) Motifs of experimental peptides.
Figure 11: Visualization for peptide editing for task 301, higher binding affinity HLA-B+44:00
R
(a) Motifs of input peptides.
(b) Motif's of edited peptides.
(c) Motifs of experimental peptides.
Figure 12: Visualization for peptide editing for task 302, higher binding affinity to HLA-C*03:03.
E マ
(a) Motifs of input peptides.
(b) Motifs of edited peptides.
(c) Motifs experimental peptides.
Figure 13: Visualization for peptide editing for task 303, higher binding affinity to HLA-B+40:01.
L
Motifs of input peptides.
(h) Motif's edited peptides.
(c) Motifs experimental peptides.
Figure 14: Visualization for peptide editing for task 304, higher binding affinity HLA-B+08:01
3
(a) Motifs of input peptides.
(b) Motifs of edited peptides.
(c) Motifs of experimental peptides.
Figure 15: Visualization for peptide editing for task 305, higher binding affinity HLA-B+08:01
E b
(a) Motifs of input peptides.
(b) Motifs of edited peptides.
(c) Motifs of experimental peptides.
Figure 16: Visualization for peptide editing for task 306, higher binding affinity to H0:-BB40:02.
29

---

Published as a conference paper at ICLR 2024
Here show all multi-objective editing tasks in Figures 17 to 19 Notice that here there two target allele types, and we mark them as "target allele 1" and "target allele 2".
For task 401 in Figure 17 ChatDrug- Turbo can successfully upweight R (Arginine) and K (Lysine) at position and (Leucine) and F (Phenylalanine) at position for target also upweight I. (Leucine) position 7, and V (Valine) and L (Leucine) at position for target allele type 2.
allele type ChatDrug
For task 402 in Figure [8 ChatDrug-Turbo at position 2, and L (Leucine) at position 9 for target allele type 1. ChatDrug can also upweight F (Phenylalanine) and L (Leucine) at position 9 for target allele type
successfully upweight E (Glutamic acid)
For task 403 in Figure 19 ChatDrug Turbo can successfully upweight R (Arginine) and K ysine) at position 5,
(Leucine) at position 9 for target allele type
Potential issue on conflicts among target allele types. One potential challenge is that for multi- objective editing, the target allele types could have conflicting motifs, like the two target alleles for task 403, We leave this for future exploration.
30

---

Published as a conference paper at ICLR 2024
Y
(a) Motifs of input peptides.
(b) Motifs edited peptides.
(c) Motifs of experimental peptides (target allele 1).
(d) Motif's of experimental peptides (target allele 2).
Figure 17: Visualization for peptide cditing for task 401, higher binding affinity to HLA-B=08:01 and HLA- C#15:02.
(a) Motifs of input peptides.
(b) Motifs edited peptides.
Y
F
(c) Motifs experimental peptides (target allele 1).
(d) Motifs experimental peptides (target allele 2).
Figure 18: Visualization for peptide editing for task 402, higher binding affinity to HLLA-B44:00 and HLA- C414:0.
Y
F
P
S
LADERES
4
(a) Motifs of input peptides.
(b) Motifs of edited peptides.
R
5
6
(c) Morifs of experimental peptides (target allele 1).
(d) Motifs of experimental peptides (target allele 2).
Figure 19: Visualization for peptide editing for task 403, higher binding affinity to HLA-B B=08:01 and HLA-
A A110:
31

---

Published as a conference paper at ICLR 2024
H.3 PROTEIN
Recall that we consider two types of secondary structures for protein editing tasks. Both the inputs and outputs protein sequences. Then folding (protein sequence to protein structure prediction) and then plot the protein structures using PyMOL (Schrödinger & DeLano). For all the protein structure visualizations, we mark a-helix structures and Ø-strand structures. The edited regions are highlighted in the blue circles.
ESMFold (Lin et al. 2022) for protein
Task 501: edit proteins with more helix structures.
mimMm
hows
(a) Protein editing with more helix for data 1.
(b) Protein editing with more helix for data 2.
hout Protein
Ostput Prosen
Iren Procesn
Outpod
(c) Protein editing with more helix for data 3.
(d) Protein editing with more helix for data 4.
Figure 20: Protein editing with more o-helix structures.
Task 502: edit proteins with more strand structures.
(a) Protein editing with more strands for data 1.
(b) Protein editing with more strands for data 2.
Irput Protein
Outpot
Octput
(c) Protein editing with more strands for data 3.
(d) Protein editing with more strands for data 4.
Figure 21: Protein editing with more I-strand structures.
32

---

Published as a conference paper at ICLR 2024
I ABLATION STUDIES
I.1 ZERO-SHOT AND IN-CONTEXT LEARNING FOR PROTEIN AND PEPTIDE
conduct an ablation study on small molecules show the comparison between the zero-shot, in-context learning, and ChatDrug-Turbe Turbo. Here we conduct the same ablation study on peptides and proteins as follows.
In Section 4.5
Table 21:
Results on six singlc-objective and three multi aujti-objective peptide editing tasks. Random Mutation for It mutated positions. The evaluation is the hit ratio of the increased binding affinity score. The best results are marked in bold. Due to the space limitation, please check Appendix F for the text prompt of each task.
single-objective editing
multi-objecttive editing
301
302
303
304
315
306
401
402
403
Random Mutation-1 1
1.80
14.40
1.80
1.80
12.00
5.60
3.20
0.80
0.40
Random Mutation-2
1.80
13.40
2.80
3.00
4.40
2.20
1.20
Random Mutation-3
1.80
9.40
2.40
4.20
9.00
3.80
3.00
0.60
0.80
In-context I Learning (few-shot)
24.05
38.40
27.40
32.00
45.50
32,80
29.20
17.47
14,40
ChatDrug 0, zcro-shot)
1.60
16.80
2.40
8.22
15.00
8.02
5.41
2.00
1.20
ChatDrug 2)
56.60
69.80
64.33
59.04
65.00
64.13
44.69
34.54
41.77
Table 22: Results on two protein editing tasks. Random Mutation-fl for mutated positions. The evaluation the hit ratio of increased secondary structures accordingly.
The best results are marked in bold.
501 more helix
502 more strand
Random Mutation-1
18.32
17.35
Random Mutation-2
24.95
19.69
Random Mutation-3
26.90
21.44
In-context Learning (few-shot)
36.64
44.47
ChaiDrug (C zero-shot)
21.43
23.73
ChatDrug (C' 2)
33.18
59.68
33

---

Published as a conference paper at ICLR 2024
1.2 ABLATION STUDY ON THE THRESHOLDS FEEDBACK CONDITION FUNCTION
In the main body, we conduct an ablation study on the thresholds in the feedback condition function. Due the space limitation, we only list the mean in Table Here list both the mean and standard deviation as follows.
Table 23:
Ablation studies single-objective small molecule editing and feedback condition D with five seeds two conversational rounds. The evaluation metric uses the strict threshold for each task.
101
103
104
105
106
107
108
Jouse lhreshold 80, 1.373:11.32
41 0.
11 21.23:22700
16.94=1.224
33, 13.16::2222
1.55
21.93:1.82 22.60=2.51
14.9621 1.96
striet threshold
88.67=0.99
70.0K+3.44
19. 19.37-5..44
30.99=2.66
43.08+2.9
66 26.6742777
76.4++3.52
Table 24: Ablation studies on multi-objective small molecule editing and feedback condition D with five seeds and conversational rounds. The evaluation metric E uses the strict threshold for each task.
201
202
2013
204
205
206
ihreshold strict lhreshold
20.14::0550 29.64:2.66
7.96112.5
3.66110.24
41,042
24.92:1:4.85
53.64:5.881
24.19:32.0
10:44::5775
52.9:22.25
34

---

Published as a conference paper at ICLR 2024
1.3 ABLATION STUDY ON THE NUMBER OF REQUEST ANSWERS IN ZERO-SHOT CHATDRUG-TUREE
we list five molecules (a.k.a. five trials) for each answer. In this subsection, would like to conduct an ablation study to explore in the zero-shot setting of ChatDrug-Tune with the conversation round C This means that for each input small molecule, we have five edited small molecules; as long as one of them hit, then we say this is successful hit. The results for 14 tasks with the loose threshold are listed below.
Notice that in Table 9
i.e. we can obtain higher performance using more trial numbers.
Table 25: Ablation studies on different trial numbers on single-objective molecule editing, with and seed is O.
loose condition A 0
striet condition A
0
trial
trial 3
trial =
trial
trial 3
trial
101 more soluble water
78.26
88.77
93.05
68.48
80.21
85,03
102 less soluble in water
71.35
89.95
93.12
24.16
74.60
78.84
103 more like drug
16.15
45.64
53.81
2.08
4.62
7.11
104 less like a drug
32.12
68.37
75.00
2.07
17.86
31.12
105 higher permeability
16.04
27.98
33.16
9.63
18.13
22.28
106 lower permeability
8.33
34.04
57.67
5.56
24.47
42.86
107 more hydrogen bond acceptors
59.41
76.57
83.15
1.76
18.29
33.71
108 more hydrogen bond donors
63.16
85.23
89.77
5.85
19.89
32.39
Table 26: Ablation studies different trial numbers multi-objective molecule editing, with C 0 and seed
0.
loose condition A
0
strict condition A >> 0
trial
trial 3
trial 5
trial
trial = trial
201 more soluble water and more hydrogen bond acceptors
43.09
69.89
75.40
6.08
22.04
34.22
202 less soluble in water and more hydrogen bond acceptors
0.52
13.47
31.44
0.00
0.52
2.06
203 more soluble in water and more hydrogen bond donors
54.49
79.67
81.97
6.18
18.13
35.52
204 less insoluble water and more hydrogen bond donors
0.53
5.21
13.54
0.00
0.00
1.04
205 more soluble water and higher permeability
2.11
4.15
5.18
1.05
1.04
1.04
206 more soluble in water and lower permeability
22.22
59.07
69.43
16.40
48.70
55.44
35

---

Published as a conference paper at ICLR 2024
1.4 ABLATION STUDY ON REASONING AND KNOWLEDGE EXTRACTION OF CHATDRUG
In Figure we show the knowledge extraction ability of ChatDrug. As control experiment, here we test the knowledge extraction ability of ChatDrug without the examples illustrated the conversation. As shown in Figure 22 the extracted knowledge is more general and less informative when we remove the examples.
Can the rules that make molecules more soluble water?
On
Cortaintyl
general
Polar molecules tend to be more soluble in water than nonpolar
molecules. This is because water la polar solvent and can
interctt with polar malecules through hydrogen banding.
2.
are
surface volume which
for tham
intersct with water moleculess.
3. Salte and other ionic compounds are unually aoluble watter
bocause water can interact with the lons through electrostatic
attractonn..
(amino) -COON (carboryf) often soluble
because these functional gr form hydrogen bonds with water malecules.
5. incrcasing lemporature can also increase solubility for many
compounds. becouse higher lemperstures
Important note that these nules are not absolute them
are many exceptions, The specific propertics malscule comples depend vartaty
Figure 22: Knowledge extraction directly without any conversations.
Then as discussed previously, the extracted information in ChatDrug contains redundant information. For example, the one listed in Figure 22 extracts three rules with overlap meanings. Then we conduct further ablation study by forcing ChatDrug extract three rules from the five original rules in Figure 4 We observe that ChatDrug successfully merges rule and rule 5, both of which express the also merges rule 3 with rule 4. Though these 2 rules share the idea of ring structure, the essence of rule 3 is introducing polar substituents. Thus, rule 3 should be more relevant rules I and Nevertheless, it clear that ChatDrug understands the domain knowledge and is capable of extracting and summarizing it. Overall, would like give positive feedback ChatDrug.
concept that polar functional groups lead to good solubility. Interestingly, ChatDrug
Now have the following five rulex to make maleculae
more soluble water But there are some redundanciis.
can you
more concise
points?
(-COOR) hydrosy!
(-HH2),
Replecing nonpolur
chains
halegane with polar functional groupe.
Modifying aromatic rings by int introducing polar substituents as hysk onyl
groups.
cyclic
Increaxing number of poter functional the molecule.
you
points:
Introduce polor functional groups (such .NH2 -coon) and increase the number of poler functionsl groups the molecule.
Replace nonpolar groups
allyl choins halogonsi
introducing subetituents -OH kirtt introducing cyclic structunes that enhance stablity and salubility
Modify aromatic
Figure 23: Further knowledge extraction.
36

---

Published as a conference paper at ICLR 2024
To sum up, observe that ChatDrug can understand and extract the knowledge to some extent, though not perfectly. We believe this is a promising direction for future exploration.
37


Reviews:
[{"review_id": "XZc49kmjTh", "reviewer": "Reviewer_pPSW", "summary": "The paper presents an approach to editing small molecules / peptides / proteins by interacting with an out-of-the-box LLM such as ChatGPT Turbo.\n\nThey describe how to prompt the model and then how to give feedback using a small number of supervised examples  using a retrieval and domain feedback module which finds the positive example that is closest to the negative prediction.\n\nThe results are strong across 3 tasks that involve small molecules, peptides and proteins but I have some questions / concerns.", "questions": "I am curious as to what baselines the authors think are fair for each task that use similar amounts of supervised data. This is a bit unclear to me in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-The paper presents an interesting approach to drug editing that leverages pretrained LLMs. It is interesting to know that non-protein LLMs out of the box can reason about small molecules / proteins / peptides through iterative in-context learning.", "weaknesses": "After the author response, the authors answered my concern (1).  \n\nFor (2), I emphasized that I wasn't suggesting comparing with supervised baselines on a supervised task, but rather reframing some of the datasets as a few shot task could increase the impact of the work (but I agree may be a significant challenge since it involves quantitative measurements).\n\nAccordingly I have now increased my score.\n\n-----\n\n(1) I am not sure about the small molecules experiment but atleast for the other two, the only baselines provided are random. However, I don't feel this is fair since the author's approach is seeing a few positive examples. Would be great to have a baseline that uses a similar number of examples.\n\n(2) The tasks focus on some basic properties of molecules like water solubility. However, often what we are really interested in drug design is a quantitative measurement like the binding affinity to a given target (or something similar). I don't see any results along these lines.\n\nIn small molecules there exist benchmarks that measure the binding to specific targets or other more detailed attributes. For example the datasets/baselines used in this paper:\n\nhttps://arxiv.org/pdf/2206.07632.pdf\n\n----\n\n(Maybe less related since the paper's focus seems to be more on small molecules with proteins a secondary experiment) For  proteins there exist benchmarks like Deep Mutational Scanning and FLIP and associated works that try to optimize a protein towards one of these attributes.\n\nhttps://www.biorxiv.org/content/10.1101/2021.11.09.467890v1\n\nhttps://www.biorxiv.org/content/10.1101/2021.07.09.450648v2\n\nhttps://arxiv.org/abs/2303.04562\n\nhttps://arxiv.org/abs/2307.00494", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kzuzTfljc9", "reviewer": "Reviewer_XYSR", "summary": "The paper proposes ChatDrug, a framework for conversational drug editing using large language models (LLMs). ChatDrug leverages prompt design, retrieval and domain feedback, and conversation modules to generate diverse and valid suggestions for drug editing. ChatDrug can handle various types of drugs, such as small molecules, peptides, and proteins. The paper evaluates ChatDrug on 39 drug editing tasks and shows that it outperforms several baselines.", "questions": "1. How do you ensure the quality and reliability of the retrieval and domain feedback module? How do you handle the cases where the retrieved information is inaccurate or outdated?\n2. How do you measure the similarity between the input and output drugs? How do you balance the trade-off between similarity and diversity in drug editing?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses an important and challenging problem of drug editing using LLMs.\n2. The paper introduces a novel and comprehensive framework that incorporates domain knowledge and interactive feedback for drug editing.", "weaknesses": "1. The paper does not provide a clear comparison or analysis of the different LLM backbones used in ChatDrug.\n2. The paper does not provide any user study or evaluation from domain experts to validate the usefulness and usability of ChatDrug.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dgTJrQRp30", "reviewer": "Reviewer_WoGj", "summary": "This paper proposes a framework named ChatDrug that leverages large language models to streamline effective drug editing. The authors demonstrate that ChatDrug outperforms existing methods on all 39 drug editing tasks and can offer insightful explanations from a domain-specific perspective. The authors comprehensively evaluate of ChatDrug's performance on 39 drug editing tasks, and conduct a detailed analysis of ChatDrug's ability to provide domain-specific explanations for its decisions.", "questions": "Refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.  This is the first paper to leverages large language models for effective drug editing. \n2. The authors conduct comprehensive evaluation of ChatDrug's performance on drug editing tasks, as well as the analysis of explainability.", "weaknesses": "1. The domain feedback function in the ReDF is defined as the evaluation metric. I wonder if it may potentially cause 'information leakage', as it is accessing the information of 'success sequence editing'.\n2. While the detailed implementation of the modules are new, the paradigm is lack novelty, as it falls into the 'prompt, retrieval for factuality, evaluate and repeat' paradigm, which is not new.\n3. The paper includes much domain knowledge for demonstration, which causes troublesome in comprehending the main idea. The authors may consider simplify the terms and focus on the main experimental phenomenon only.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)