MolJET: Multimodal Joint Embedding Transformer for Conditional de novo Molecular Design and Multi-Property Optimization
Authors: Orion Walker Dollar
Sameera Horawalavithana
Scott Vasquez
W. James Pfaendtner
Svitlana Volkova
Venue: Submitted to ICLR 2023 (2023)
Keywords: Transformers
Multimodal
Molecules
Generative
Drug-design
LLM

Abstract: Multi-property constrained optimization of molecules using generative de novo design models is vital for the successful application of Artificial Intelligence (AI) towards materials and drug discovery. Yet there remains a gap between the reported performance of such models in the literature and their practical utility in real world design scenarios. Furthermore, existing models are largely inaccessible to chemists without an extensive background in computer science. To address these challenges, we propose a generative foundation model, the Multimodal Joint Embedding Transformer (MolJET), which performs conditional generation of desired molecular distributions based on human-interpretable chemistry prompts in a zero-shot manner. We assess MolJET on the standard benchmarks available in the GuacaMol and MIMOSA evaluation frameworks. These include structure-based sampling tasks as well as a range of multi-property optimization tasks that probe a models ability to design drug-like molecules given realistic property constraints. We demonstrate that with self-supervised pretraining, MolJET outperforms 80% of task-optimized models while using zero-shot inferences and beats all baselines after minimal supervision. Moreover, the performance of MolJET on text-only conditioning tasks improves with the inclusion of property modalities during training, highlighting the importance of a multimodal approach to molecular design. MolJET is the first example of text-based de novo molecular design using large-scale multimodal foundation models and should serve as a building block towards further improvements to accessible AI for chemists.

Under review as a conference paper at ICLR 2023
MOLJET: MULTIMODAL JOINT EMBEDDING TRANSFORMER FOR CONDITIONAL DE NOVO MOLEC- ULAR DESIGN AND MULTI-PROPERTY OPTIMIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Multi-property constrained optimization of molecules using generative de novo design models is vital for the successful application of Artificial Intelligence (AI) towards materials and drug discovery. Yet there remains gap between the re- ported performance of such models in the literature and their practical utility in real world design scenarios. Furthermore, existing models are largely inaccessi- ble to chemists without an extensive background in computer science. To address these challenges, we propose generative foundation model, the Multimodal Joint Embedding Transformer (MoLJET), which performs conditional generation of desired molecular distributions based on human- intrrrn-rtrrretbble chemistry prompts in a zero-shot manner. We assess MOLJET on the standard benchmarks available in the GuacaMol and MIMOSA evaluation frameworks. These include structure- based sampling tasks well range of multi-propen property optimization tasks that probe a models ability to design frug-like molecules given realistic property con- straints. We demonstrate that with self-supervised pretraining, MOLJET outper- forms 80% of task-optimized models while using zero-shot inferences and beats all baselines after minimal supervision. Moreover, the performance of MOLJET on text-only conditioning tasks improves with the inclusion of property modalities during training. highlighting the importance of multimodal approach molec- ular design. MOLJET is the first example of text- boxt-baedd de novo molecular design using sarge-scale multimodal foundation models and should serve as building block towards further improvements to accessible AI for chemists.
INTRODUCTION
Emerging crises in climate, disease and human health threaten to permanently disrupt global stabil- ity and must be actively met with creative solutions. Many such solutions dependent on the rapid discovery of innovative functional materials or novel drug-like molecules with optimal propertics. For instance, the viability of using redox-flow batteries (RFBs) for long-term and large-scale energy storage is contingent on finding stable redox species with fast electrochemical kinetics, a feasible redox potential and high solubility (Zhang et 2018). Due the immense size and complexity of chemical phase space (Polishchuk et al. 2013 the search for suitable materials is far from triv- ial and traditional "direct" design approaches based on iterative modifications existing chemical structures are often far too slow (Kuhn & Beratan 1996).
To address this issue, researchers have increasingly begun to look towards generative de novo design models to efficiently navigate the vast molecular phase space (Meyers et al.) (221]). These models are evaluated on their ability to generate diverse array of novel molecular structures while simul- taneously biasing them towards desired property distribution (Polykovskiy et al. (020). Due to the ubiquity of string-based molecular representations (Weininger, 1988, Krenn et al. 2020), re- cent innovations in natural language modeling have been successfully applied to de novo molecular design. For instance, transformer architectures have achieved state-of-the. results on property prediction tasks that require quantum-level accuracy (Ross et al. 2021) and have also been shown to increase the diversity of candidates sampled from al. 2021)
molecular distributions (Dollar

---

Under review as a conference paper at ICLR 2023
I. Sample From Training Set
II.
Train Model to Reconstruct Molecule
III. Conditional Generation
"pyrazol"
"C,
Text Conditions
Joint
FuncGroups
phare rundllane
Embedding Transformer Layers
"90th percentile LogP"
MoiForua
PLog?
-2.70
QED
0.82
TPSA
46.45
Property Conditions
MOL.JTT Framework, Prompts are (i) stochastically sampled from the available modal- ities in the dataset and (ii) used to condition autoregressive reconstruction of SELFIES strings. Con- ditions are then chosen during inference to (iii) shift the generated molecular distribution towards the desired structural physicochemical properties.
Figure I:
Aside from string-based representations of molecular structures, there are other textual modalities which could provide additional context to generative models and thus improve their performance. important chemical
Such modalities include IUPAC names, molecular formulas, descriptions moieties or functional groups and natural language descriptions of chemical behavior. Yet despite the large overlap between architectures used for natural language modeling and molecular sequence modeling, there have only been few attempts to incorporate more than single modality within a model (Rothchild et al. 2021 Sun et al. 2021 Zeng et al.] 2022) and none have included the capacity for property-driven molecular design. Massive scaling has also been primarily limited to property prediction tasks (Honda et al., 2019 Chithrananda et al. 2020) despite growing evidence
of the performance benefits derived from increasing model sizes, dataset sizes and compute across all downstream tasks (Kaplan et al. 2020 Hoffmann et al. 2022).
In this work introduce MOLJET, large-scale multimodal joint embedding transformer for con- ditional molecular generation and multi-propene optimization. Within this framework, molecular generation is conditioned by text-based prompts that control the structural and physicochemical characteristics of the desired molecular distributions as depicted in Figure We demonstrate condi- tional generation on three modalities textual descriptions of molecular structural features, physic- ochemical properties and ID atomistic molecular graphs and provide a general framework for the inclusion additional modalities during pretraining.
To prove the efficacy of our models in realistic design scenarios, evaluate MOLJET diverse set of tasks including molecular rediscovery, similarity and substructure- based sampling, isomer generation and multi-property optimization (Brown et al. 2019; Fu et al., 2021). With only self- supervised pretraining, MolJET outperforms all task-optimized baseline models on five out of the eight task categories and outperforms the baselines all eight task categories after minimal task- specific supervised optimization. Furthermore, the prompts are designed to be easily interpretable by chemists without any prior knowledge of deep learning and thus accessible We provide access to our pretrained models through an online API and hope to encourage increased participation in AI-driven de novo molecular design among scientific researchers in much the same way that DALL-E and GPT have inspired increased interaction with deep learning models among the general public (Brown et al. 2020
wider audience.
Ramesh et al. 2022).
RELATED WORK
Multi-Properte Optimization. Several strategies for multi culti-property optimization of molecular structures have been explored to date. Some works propose condition the generation of molecular structures with a learnable embedding corresponding to the values of one or more desired properties (Lim et al.
2018 Li et al. 2018: Gebauer et al. 2022 These models jointly learn the conditional
2

---

Under review as a conference paper at ICLR 2023
distributions during training and then allow for the selection of specific conditions during inference. Others treat optimization as a translation task, in which an improved version of the input molecule is reconstructed during training (Jin et al. 2018a 2020). These models learn the desired molecular distribution directly, however they also require the construction of translation pairs which can be time-consuming and without careful control can introduce biases into the model or result in poste- rior collapse (Jin et al., 2018b). Another popular strategy for optimization is by making stepwise modifications to an existing molecular structure through an efficient sampling method like Markov driven policy network (Nigam et al.) 2019 Khem- chandani et al. 2020 Fu et al.) 2021). reward function determines the success of the model and guides further modifications. These models are flexible as they can modify their actions based on any reward, however they often shift the generated distribution too far from the original and can struggle to generate realistic samples (Popova et al. 2018 Brown et al. 2019).
Chain Monte Carlo or a reinforcement-leamit
Foundation Models for Chemistry. Given that the vast majority of de novo molecular design models operate a single molecular representation, there are only few examples of multimodal learning in the field of chemistry. KV-PLM and CHEMET both combine structural representa- tions of molecules with natural language, the former by embedding SMILES strings directly into a biomedical corpus and the latter by performing cross- moosl attention between embeddings of a molecular graph and a description of the molecule (Sun al., 2021 Zeng al. 2022). However, these models are better suited for classification tasks than generation tasks as it is challenging to build a corpus annotated with molecular structures that large enough to train a generative model. Other examples of multimodal chemistry models include GeomGCL (Li et al. 2022) which per- forms contrastive learning on 2D and 3D molecular graphs for property prediction and VJTNN (Tin) et al. 2018b) which combines junction tree and atomic graph representations during the encoding and decoding of the latent vector in VAE.
MODEL FRAMEWORK AND PROMPT DESIGNING
Herein, we describe the Multimodal Joint Embedding Transformer (MOLJET), a large-scale gener- ative foundation model for conditional molecular design and multi-property optimization. The aim of MOLJET is to efficiently navigate the molecular phase space while simultaneously reaching a desired property distribution. This task is non-trivial the molecular landscape is high dimensional
and rugged making optimization within this space difficult (Stumpfe et al.) 2020). We hypothesize that jointly learning across text, molecular structure and properties will enhance the model's ability to learn structrr-propert relationships and thus improve its performance at designing optimized
molecules. We introduce the multimodal fusion with our prompt design framework in Section 3.1
and then present the model architecture and conditional sampling scheme in Sections 3.2 and 3.3 respectively.
3.1 MULTIMODAL FUSION WITH PROMPT DESIGNING
Our goal is to learn inter-modal and cross-modal information with an expressive prompt design that can facilitate both the self-supervised pretraining and zero-shot evaluation. We propose an carly-fusion strategy to jointly reason over the text, molecular structure, and property modalities with a shared multifaccted representation. We represent the textual description and associated physicochemical properties of molecule in the prompt sequence x = ($1, 82, $2,.., .., 8n) of the form (Stext, Sprop: m,ol),
<text type............................ </text typa>> <text>... </coxt>>>>>>>>>> <psoparty>... Sttplv.....//ppppp </proparty> <vai>..</val>> <moi>>... </no1>
We include <text tyre> and <property> tags to differentiate across molecule descriptions (Stent) and properties (Sprop). The <text>> and <v<l> tags designate the search space on the respective data modalities. The <esl> tag designates the SELFIES string describing the molecular structure (smot). The proposed prompt design is flexible so that other textual representations of molecules or associated properties may be easily substituted. We also allow each modality to contain multiple sub tobmprompts. For example, we can represent multiple physicochemical properties separately as sub- prompts in Sprop- We introduce strict ordering of the prompt sequence with the corresponding text, property and molecular structure representations to enable the model to conditionally generate molecular distributions given the other modalities.
3

---

Under review as a conference paper at ICLR 2023
3.2 MODEL ARCHITECTURE
Our objective is to pretrain a large-scale foundation model with the ability to generalize to unseen tasks without requiring any labeled data. This specially relevant molecular design scenarios where we need to generate new molecules that have not been previously seen (out-of--istributtion generalization). However, it is intractable to enumerate across all possibilities due to the unbounded molecular search space. We present the unsupervised distribution estimation p(z) from a set of prompts (01, T2, .,, [n) as the product of conditional multimodal token probabilities,
n II p(sn $1 $1..., 8n-1)
p(z)
Our model design is inspired by the recent success of applying the transformer encoder architecture on shared mulitmodal multifaceted representations (e.g.. UTF-8 bytes in Perceiver-IO (Jaegle al, 2021), vision-language decoding (Aghajanyan et al., 2022)). In this work, we investigate whether transformer architectures are capable of learning over multimodal molccular information and trans- lating it into rich knowledge of the relationship between molecule's s structure and its properties. We seek to analyze whether transformer architectures are suitable to distill and accumulate both inter- and cross modal information from the molecular descriptions, and test whether the pretrained models generalize to novel contexts during de novo molecular design.
To this end, adopt the autoregressive transformer decoder model architecture similar to GPT- 3 (Brown et al. (2200) and apply it on conditional multimodal prompt based molecule generation tasks. We translate the general left-to-right language modeling objective to a joint modeling objec- tive that predicts the next modality token. We minimize the joint loss defined as
(())
-logpo (8:(ag))
Dtraen
ZED
The model learns the conditional multimodal token distribution jointly given the in-context refer- ences to other modality tokens. We do not modality-peciffic encoders in this setup since translate all modalities into the discrete language space. remains as future work to explore how other modalities such vision (continuous), graph (2D) atomic coordinates (3D) could be used in our framework to further enrich the learned multimodal molecular representations.
3.3 CONDITIONAL MOLECULE GENERATION
Given the molecular structure represented as sequence of tokens describing the atoms, their nectivity and their valence states (m,)., the conditional multimodal prompt-based molecule generation is as follows:
m arg max log po(me]stent Sprop, mcz)
We use temperature sampling to autoregressively sample the SELFIES tokens m, conditioned on the multimodal prompt. The sampling takes the molecule textual description Stents physicochemical properties Sprop and <eel> mcc C 8mot as the initial inputs in the joint multimodal embedding space. In addition, the molecule generation is conditional to the property values in Sprop-
m g(- Stents Sprop, meet)
(4)
Smol(1) UmcrEsmute {ma mz) [m?])
We sample molecule tokens until we reach a </mol> tag. The sampled tokens are concatenated with other top scoring molecule tokens generate the molecule structure Smol(t)-
4

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTAL SETUP
4.1 IMPLEMENTATION AND TRAINING DETAILS
Dataset Creation. We gathered over 100M unique molecular structures from the PubChem com- pound records database (Kim et al. 2019) to use for pretraining. Each structure includes a valid SMILES representation, an IUPAC name, and a molecular formula. Functional groups are ex- tracted from the full IUPAC name and SMILES are encoded as SELFIES strings. In accordance with the method outlined in GuacaMol (Brown al., 2019). we calculate the ECFP4 fingerprints (Rogers & Hahn 2010) for every molecule in our dataset and a holdout set of drug-liks like molecules used the benchmarks. Any molecule in the training set with a tanimoto fingerprint similarity of 2 0.343 to any molecule in the holdout set is removed. This ensures the model has not simply mem- orized solutions to the benchmark tasks during pretraining. Similarly, all isomers corresponding to the two isomer generation tasks were also removed from the training set.
Conditional prompts for each molecule are generated stochastically so the model may only see a portion of the available modalities for any given sample. This allows the user to ignore some modal- ities during inference while still allowing the model to jointly learn over all possible modalities. The rules for prompt sampling are outlined in Appendix B
Available Modalities. We provide three modalities on which the models are conditioned tex- tual molecule descriptions, properties and ID atomistic molecular graphs. Table [ shows the sub- modalities available for the text and property modality types. Each text type provides a different level of detail regarding the molecular structure and are all commonly used by chemists when describing molecules. The properties are selected to cover a wide range of chemical behavior important to drug design. Each property is calculated using the cheminformatics package RDKit (Landrum al.) 2013) aside from DRD2 which is predicted by the model published in Olivecrona et al. (2017). We use SELFIES as our 1D atomistic molecular graph to guarantee the validity of all molecules generated during inference (Krenn et al. 22200).
Table 1: Details of the multimodal inputs used in the pretraining and zero-shot evaluation.
Textual Molecule
IUPAC, text that fully specifies atomic
FuneCroups, text that specifics only the stomic connectivity local envirenments within the molecule
MolFormula. text that does not specify any connectivity information but does specify the overall atomic makeup of molecule.
Descriptions
connectivity of the entire molecule
Physicochemical properties
Topological polar surface (TPSA),
LogP/Penalized Logl' (PLogl), a method for estimating the solubility of malecule (Wildman & Crippen)
BertZTT, topological index meant
of the overall surface polarity of the molecule (Prasanna & Doerksen 2009)
quantify the "complexity" of
malecule Bertz 198
PLoPP includes penalites molecules with low syathesizability
QED quantitative meusure
Number of fluorine atoms, Number of aromatic rings, Total number rings
DRDZ the hiological activity of
of the "drug-iikeness"
molecule towards the dopamine rcceptor D₂
molocule (Bickerton et al. 2012)
Tokenization We develop a custom vocabulary that consists of the tokens representing the molecule textual description Steat. physicochemical properties Sprop and molecular structure Smot- IUPAC and FuncGroups share a vocabulary learned from byte-pair encoding of the IUPAC names in the training set. The MolFormulas and SELFIES are tokenized on a per-atom basis. Property values are represented as either scalars or decile ranges labeled 1-10 with each digit tokenized sep- arately. Finally, all tags ( <<.>> <../> ) and property names are encoded as special tokens.
4.2 TASK DESCRIPTIONS
We evaluate MOLJET on 22 tasks split across 8 different categories: molecular rediscovery, similar- ity sampling, substructure sampling, isomer generation, median molecules, multi-property optimiza- tion, diug-likeness and biological activity. Each task is taken from either the GuacaMol evaluation
IUPAC (International Union of Pure and Applied Chemistry) nomenclature provides international stan- dard of naming compounds which can he used create unambiguous structural formula.
5

---

Under review as a conference paper at ICLR 2023
framework (Brown et al. 2019) the MIMOSA multi multi-propentt proppetty optimization framework (Fu al. 2021]. Table 2 provides examples of tasks from a few of the optimization categories and their cor- responding prompts. Detailed descriptions of each task category are provided below.
Table Example of the downstream tasks and prompt designs used in the zero-shot evaluation. We color each prompt with the modality(s) that they are associated with. For the prompts for all 22 tasks, please refer Tables and in Appendix
Task/Example
Prompt
Molecular Rediscovery
<text type IOPAC <ttxt///ox..
Celecoxib
(ooc=>><(((((t-mabbyypepe//// bomerpoeeoe...eeeeeeeeoommmme /tt.a/iddddd///tt//t//// <mo1>>
Similarity Sampling
<tox:.typ: Fannerroups /topttoppppp/tttttpppppp>
Albuterol
<cexx: > botubuyylllnno.oooo hyw.oyyytyyy < (
Isomer Generation
<oext mexx.lype MolFormul </text_yyp=>>
C22
<sex: <ex1:>c11824 2/1011222////ee/// 2-mml
<text pepp IUPACC S/taac///eettteeeeeee
Multi-Property Optimization
<tex: >e (2- p.im 2-opp-e--ee----------eeeeee.. ceane
Osimertinib
<pxepercy>> tpea httppo/cyyppaa/////ppptt>>>>/// //tpa////err//></c11/ee////
<propercy Lage www.ge/////geeeeee wtw.ggertt S</vel p
Molecular Rediscovery. The model must generate an exact match to the target. This task tests the model's ' ability to explore regions of molecular phase space which it has not encountered during training.
Similarity Sampling. The model must generate many samples that are structurally similar to the target but not an exact match. This task tests the model's ability to make small structural modifi- cations to target without diverting too far from the original molecule. This is analogous to how chemist might approach the design of new drug by modifying small chemical motifs of a starting structure to improve specific desired behavior while maintaining other drug-like qualities from the original molecule.
Substructure Sampling. The model must generate many samples that contain a specific structural motif or set of motifs. some tasks, the model may also be penalized for generating molecules with non-desired motifs for diverging too far from the pharmacological properties of the molecule from which the desired motif is drawn. This task tests the model's 's ability to generate functional moicties off scaffold "fill in" the scaffold given set of functional moieties.
Isomer Generation. The model must generate as many structural isomers as it can from given molecular formula. This task tests the model's ability map coarse-grained chemical information to fully connected atomic graph. It also tests if the model can enumerate all possible structures from local region of chemical phase space.
Median Molecules. The model must generate samples that are maximally similar to two different target molecules. This task tests the model's ability to interpolate between two valid chemical struc- tures, a common goal when trying to discover a molecule that maximizes the desired properties of two separate existing molecules.
Multi-Property Optimization (MPO). The model must simultaneously match both structural and property requirements as dictated by the task. For instance, the model might tasked with finding structural analogue to the antihistamine fexofenadine that is "less greasy" by reducing the LogP and increasing the TPSA while maintaining a high structural similarity to the target. These tasks put the model in realistic drug design scenarios and demonstrate its ability to perform structural sampling while also constraining the generated molecules to the desired property ranges.
To demonstrate the versatility of the MOLJET framework, also evaluate the model the multi- property optimization tasks outlined in Fu et al. (202T). These require the model to maintain high structural similarity input drug-like molecule while simultaneously maximizing PLogP and

---

Under review as a conference paper at ICLR 2023
DRD2 (Biological Activity). We report performance these two tasks as success rate which is defined as the proportion of imput molecules that the model is able to improve beyond pre-defined threshold for each property while maintaining high similarity. Further details on the definition of success rate are provided in Jin et al. (0188). Fach GuacaMol task is evaluated based on weighted average of the top 100 scoring molecules for that task. Further details
either QED (Drug-Likenesl
on the definitions of each GuacaMol metric are provided in Brown et al. (0119) and Appendix E
Conditional Language Model Pretraining. We train two independent version of MOLJET, MOLJET-Gur and MOLJET-BIo, MOLJET-GUAC is trained and evaluated with the three text types and TPSA. LogP, BertzCT, number of fluorine atoms and ring counts (total and aromatic). MOLJET-BI0 is trained and evaluated with the three text types and PLogP, QED and DRD2. We train two additional model variants one to study the difference between scalar and decile prop- erty value representations (MoLJET-GUACss) (UCT-GGUUCCCLLL/DEEUE)) (AAAAAGAAAAAALEEEEEECAAAAAAAAAA
and one without property conditioning to study the cumulative effect that additional modalities have on text-only inference tasks (MOLJET- The models pretrained from scratch the filtered PubChem train- ing set. Further details on the training procedure, hyperparameters, baseline models and sampling scheme can be found in Appendices C &
EXPERIMENTAL RESULTS
The performances of MOLJET--BO and MOL MOLJET-GUAC the MIMOSA and GuacaMol evalua- tion frameworks are displayed in Tables and 4 Both models are very competitive during zero-shot inference with MOLJET-Gue GUT- outperforming 78% of all baselines on the GuacaMol benchmarks and MOLJET-BE improving the success rate on the Drug-Likeness and Biological Activity tasks by 18.75% and 13.5% respectively. It should be noted that the baselines are fine-tuned on each task a supervised manner, whereas MOLJET has only undergone self-supervised pretraining and is seeing the task-specific optimization prompts for the first time during inference. Thus, the perfor- mance on these benchmarks demonstrates the efficacy of our multimodal framework in generalizing previously unseen molecular distributions.
Multi-Properrt, Optimization. We first show that MOLJET is able to leverage information from multiple modalities to simultaneously control the structure and properties of generated molecules during zero-shot inference. By conditioning the model on the modalities that are optimal for a given task, can generate molecular distributions that outperform previously state-of-the-art baselines on variety of multi-property optimization benchmarks. It accomplishes this by inferring how the desired structural features must be modified to satisfy the additional property constraints. We use the conditional generation sampling method described in Section 3.3 to efficiently explore the local region of molecular phase space dictated by the multimodal prompt.
For example, MOLJET--Bc JET-BIO outperforms the previous state-of-the-arr, MIMOSA, in both absolute property improvement and success rate on the Drug-Likeness and Biological Activity MPO tasks. It does by exploring the local region of molecular phase space surrounding the target molecule more efficiently by directly sampling from the conditional distribution. Because MIMOSA makes iterative modifications the target molecule, it does not venture far from the original structure during optimization. While this leads to a higher similarity score on both tasks, fails to find as many molecules that satisfy the property optimization constraints and thus has lower success rate.
Table 3: Benchmark results the MIMOSA MPO evaluation framework. PLogP, QED and DRD2 columns refer to the absolute improvement in property values from successful samples.
Drug-Likeness
Biological Activity
Method
Similarity PLogP
QED
Success
Similarity PLogP
DRD2
Success
VJTNN
0.17
0.46
0.02
1.0%
0.18
0.55
0.27
3.4%
DeepGA
0.35
0.93
0.09
24.9%
0.38
0.68
0.20
29.3%
MIMOSA
0.42
0.93
0.10
32.0%
0.54
0.75
0.35
43.7%
-oI.E'T-BIO
0.37
1.19
0.14
38.0%
0.35
3.38
0.48
49.6%
(Zero-shot)

---

Under review as a conference paper at ICLR 2023
Table 4: Benchmark results GuacaMol which contains both MPO and molecular structure gen- eration tasks. Bold values indicate the best performing model and underlined values indicate the second best performing model measured against the baselines.
Benchmark
Best of Data
SMILES
SMILES
Graph GA
MOLJET-GUAC (Zero-shot)
MOLJET-GUAC + Graph GA
Category
Set
LSTM
GA
MPOs
0.698
0.778
0.717
0.868
0.838
0.878
Rediscovery
0.613
1.000
0.523
0.945
1.000
1.000
Similarity
0.546
1.000
0.771
0.977
1.000
1.000
Substructure
0.643
0.973
0.769
0.985
0.817
0.985
Isomers 0.716
0.912
0.745
0.954
1.000
1.000
Median 0.371
0,403
0.362
0.417
0.409
0.447
Total
0.623
0.850
0.671
0.877
0.857
0.900
We observe similar trend from zero-shot MOLJET-GUAA the GuacaMol MPOs. When break- ing the tasks down individually, it outperforms all three baselines on the ranolazine, perindopril, and amlodipine MPOs and is within 1% and 2.5% of the best performing model the fexofenadine and osimertinib MPOs, respectively (Appendix or more property specifications while maintaining high similarity to a target molecule (see Fexofe- nadine and Perindopril MPOs, Figure] 2 In total, MOLJET outperforms or is competitive with the leading baseline on seven out of nine MPOs across both evaluation frameworks demonstrating the versatility and efficacy of our multimodal framework.
F These tasks also require the model to meet one
Conditional Molecular Structure Generation. MOLJET- also performs well the zero- shot molecular structure generation tasks, achieving perfect score on rediscovery, similarity sam- pling and isomer generation (Table 4.. This indicates that the model able to accurately estimate the molecular structural probability manifold of the training set and navigate it based on the condi- tional multimodal prompts. Each of the three text modalities provide a different degree of structural specificity with which the model can be conditioned. For instance, tasks with stringent similarity requirements are better suited for IUPAC conditioning, whereas FuneGroup conditioning yields a more diverse of generated molecules (see Drug-Likenes FuncGroup conditioning is also the most flexible as can be used to combine the structural charac- teristics of multiple input molecules (see Median Molecules, Fig. 2
Fexofenadine MPO in Fig. 2).
Drug-Likeness
Perindopril MPO
Text Conditions
Isp Mokicule plop: 2.37
Text Conditions
Mobecule somalec trgge
verving
FueeGrouph
peoperiet Mraple
PLagg
10"h decla 11
11
Madeccde ploge 0.87
qod. 0.58
Arometic Piaga
Outzut Melecaai momaiic ringee
Proparty Conditions
Proparty Conditions
Fexofenadine MPO
Median Molecules
Text Conditions
Molecale loga: 8.51
Text Conditions
Ioed Meleculos
2014t sudloy.
tnta: $1.00
F.noGroups
Loof
lagpe 5.7
Datge Midecade
functiool groups corpled
Outout Moleouie
bss: 121
Property
Figure 2: Prompts, inputs and high- Ligh-scoring samples for four of the de novo design tasks.
We confirm these observations quantitatively by measuring the performance of each text modality individually on the similarity sampling tasks. We choose similarity as it is the most common struc- tural objective for the MPOs and thus highlights important differences in sampling performance for realistic drug design scenarios. The results of this experiment are shown Figure 3 As expected, explore the largest subset of relevant phase space when conditioning FuneGroups. How-
8

---

Under review as a conference paper at ICLR 2023
there some circumstances where IUPAC conditioning is just effective, namely when the molecule is complex such as the stereoisomer mestranol.
To estimate how amenable MOLJET is further opti- mization, we re-run the Graph GA method but replace the starting population with the top 100 molecules gen- erated by MOLJET. On average, the Graph GA seeded with molecules generated by MOLJET improves upon the zero-shot MOLJET by ~ 5% and the baseline Graph GA by 2.6% (Table 4p. This demonstrates the capac- ity of MOLJET to be further improved by task-specific fine-tuning strategies and we leave further work in this direction as future research.
Aripiprazole
Albutarol
Mestranol
0.50
Millprazte Altubert Vestrand
0.00
Average
o.m
Evaluating Prompt Design. We also run ablations study a) the effect of the choice of numerical property representation on GuacaMol tasks with property con- ditioning and b) the impact of the inclusion of property modalities during training on GuacaMol tasks with text- only conditioning. On the GuacaMol tasks with property conditioning, MOOJEET-GUASSS per- forms slightly hetter than prediction capacity of the scalar model is only slightly greater than the average distance between decile bins. For most properties, this distance is fairly large so this result indicates a potential area in which MOLJET could be improved.
Malformade
Punciroups
Figure 3: Similarity sampling from each text modality.
JELJETT---ADEELLEE (0.881 vs. 0.872). This suggests that the property
Finally, we evaluate NOLJET-GUACTHEEE ence tasks from GuacaMol (Table (5. These tasks do not require any property conditioning during inference and thus the performance of the two models should be expected to be comparable if cross- modal learning does not occur during training. However, we find that SOAJJET-GUACTETT-PPPP performs better on the text-only inference tasks, supporting our hypothesis that our multimodal prompt design framework supports both inter- and cross-modal learning. The property information that is jointly embedded during training enhances the models understanding of molecular structure even when that information not provided during inference.
and MOLJET MOLJJET-GUCCTETN+PPP GUACTEN++RROP on the text-only infer-
To confirm this behavior, we construct two additional text-only inference tasks, IUPAC Reconstruction and FuncGroup Reconstrue- tion. IUPAC Reconstruction tests the mod- els ability accurately reconstruct SELFIES string given its IUPAC from a holdout set of IUPAC-SELFIES pairs that were seen dur- ing training. FuneGroup Reconstruction tests the models ability generate molecules that contain the requested functional group from list of 102 functional groups developed by the au- thors to include a wide range of atom types and complexities. Additional implementation details for each task are outlined in Appendices[A] F.. Again, we find that MOT.JET-GUACTeS SETTT-GAATTET-PP.. out- providing additional evidence that both inter- and cross-modie learning occur during training and that multimodal joint embeddings are capable of enhancing the performance of novo molecular design models
Table 5: Multimodal Model Ablations
Reconstruction
Modality
GuacaMol
IUPAC
FuneGroup
Text
0.827
62.1%
60.2%
Text + Property
0.843
68.7%
63.4% %
performs
CONCLUSION
We introduce MOLJET, a multimodal foundational chemistry model for conditional de novo design of organic molecules. MOLJET demonstrates state-of-the-art performance realistic drug design tasks in zero-shot manner. Our framework is adaptable and easy to interpret, making well-suited for the inclusion of other modalities such scientific text. We make code, models and data publicly available and provide API access to our pretrained models to allow chemistry researchers of all backgrounds participate in the future development of AI-driven de molecular design.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, al. Cm3: A causal masked multi- modal model of the internet. arXiv preprint wwxiv:22010075222 2022.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hal- lahan, Josh Levy-Kramer Connor Leahy, Lucas Nestler, Kip Parker. Michael Pieler, Shivanshu Purohit, Tri Songz, Phil Wang, and Samuel Weinbach. GPT-NeoX: Large scale autoregressive lan- guage modeling in pytorch, 2021. URL patttp://githan www.///wwttttae///ee//
neox
Steven H Bertz. The first general index of molecular complexity. Journal of the American Chemical Society, (03(12)3359--360), 1981.
Richard Bickerton, Gaia Paolini, Jérémy Besnard, Sorel Muresan, and Andrew Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autore- gressive language model. 2022.
Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3): 1096-1108, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: large-scale self- supervised pretraining for molecular property prediction. arXiv preprint arXiv:2010.09888, 2020.
Orion Dollar, Nisarg Joshi, David AC Beck, and Jim Pfaendtner. Attention-based generative models for de novo molecular design. Chemical Science, 12(24):8322-8377, 2021.
Tianfan Fu, Cao Xiao, Xinhao Li, Lucas M Glass, and Jimeng Sun. Mimosa: Multi-constring molecule sampling for molecule optimization. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 35, pp. 125-133, 2021.
Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaum McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, 40(D1):DI 100-D1 107, 2012.
Niklas WA Gebauer, Michael Gastegger, Stefaan SP Hessmann, Klaus-Robert Müller, and Kristof T Schütt. Inverse design of 3d molecular structures with conditional generative neural networks. Nature communications, 13(1):1-11, 2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Shion Honda, Shoi Shi, and Hiroki R Ueda. Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery. arXiv preprint 4rXiv:191 2019.
Andrew Jacgle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning, pp. 4651 PMLR, 2021.
Jan H Jensen. graph-based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space. Chemical science, 10(12):357733772 2019, 2019.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323-2332 PMLR, 2018a.
10

---

Under review as a conference paper at ICLR 2023
Wengong Jin, Kevin Yang. Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to- graph translation for molecular optimization. arXiv preprint arXiv: 18/2:118/2.01000, 2018b.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. International conference on machine learning, pp. 4839 4848. PMLR, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Yash Khemchandani, Stephen O' O'Hagan, Soumitra Samanta, Neil Swainston, Timothy J Roberts, Danushka Bollegala, and Douglas B Kell. Deepgraphmolgen, a multi-objective, computational strategy for generating molecules with desirable properties: graph convolution and reinforce- ment learning approach. Journal of cheminformatics, 12(1))1-17, 2020.
Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Ben- jamin A Shoemaker, Paul A Thiessen, Bo Yu, et al. Pubchem 2019 update: improved access to chemical data. Nucleic acids research, (IID))DD102-DDIIII
2019.
Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik. Self- refereneing embedded strings (selfies): A 100% robust molecular string representation. Machine Learning: Science and Technology, 1(4):045024, 2020.
Christoph Kuhn and David N Beratan. Inverse strategies for molecular design. The Journal of Physical Chemistry, 100525)::0995105599, 1996.
Greg Landrum et al. Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum, 2013.
Shuangli Li, Jingbo Zhou, Tong Xu, Dejing Dou, and Hui Xiong. Geomgel: geometric graph contrastive learning for molecular property prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4541 4541-4549, 2022.
Yibo Li, Liangren Zhang, and Zhenming Liu. oulti-objective de novo drug design with conditional graph generative model. Journal of cheminformatics, 10(1):1-24, 2018.
Jaechang Lim, Seongok Ryu, Jin Woo Kim, and Woo Youn Kim. Molecular generative model based on conditional variational autoencoder for de novo molecular design. Journal of cheminformatics, 10(1):1-9, 2018.
Joshua Meyers, Benedek Fabian, and Nathan Brown De novo molecular design and generative models. Drug Discovery Today, 26(11)2707-2215, 2021.
AkshatKumar Nigam, Pascal Friederich, Mario Krenn, and Alán Aspuru-Guzik. Augmenting ge- netic algorithms with deep neural networks for exploring the chemical space. arXiv preprint VrXiv:1909.11655 2019.
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of cheminformatics, 9(1):1-14, 2017.
Pavel G Polishchuk, Timur I Madzhidov, and Alexandre Vamne.. Estimation of the size of drug- like chemical space based on gdb-17 data. Journal of computer-aided molecular design, 27(8): 675-679, 2013.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al. Molecular sets (moses): a benchmarking platform for molecular generation models. Frontiers in pharmacology, 11:665644, 2020.
Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 4(7)xeaap7885, 2018.

---

Under review as a conference paper at ICLR 2023
S Prasanna and RJ Doerksen. Topological polar surface area: useful descriptor in 2d-qsar. Current medicinal chemistry, 16(1):21 2009.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06177 2022.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 3505-3506, 2020.
David Rogers and Mathew Hahn. -xtended-conneettvity fingerprints. Journal of chemical informa- tion and modeling, 50(5):442-754, 2010.
Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mroueh, and Payel Das. Do large scale molecular language representations capture important structural information? arXiv preprint arXiv:2 (06.0053,, 2021.
Daniel Rothchild, Alex Tamkin, Julie Yu, Ujval Misra, and Joseph Gonzalez. C5t5: Controllable generation of organic molecules with transformers. arXiv preprint arXiv:2108. 10300770 2021.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. arXiv preprint (tti:::9990000000 2019.
Dagmar Stumpfe, Huabin Hu, and Jürgen Bajorath. Advances in exploring activity cliffs. Journal of Computer-Aided Molecular Design, 34(9):929-99) 942, 2020.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864 2021.
Chenkai Sun, Weijiang Li, Jinfeng Xiao, Nikolaus Nova Parulian, ChengXiang Zhai, and Heng Ji.
Fine-grained chemical entity typing with multimodal knowledge representation. In 2021 IEEE In- ternational Conference on Bioinformatics and Biomedicine (BIBM), pp. 1984-1991. IEEE, 2021.
David Weininger. Smiles, a chemical language and information system. introduction to method- ology and encoding rules. Journal of chemical information and computer sciences, 1988.
Scott Wildman and Gordon M Crippen. Prediction of physicochemical parameters by atomic contributions. Journal of chemical information and computer sciences, 3965):868 873, 1999.
Naruki Yoshikawa, Kei Terayama, Masato Sumita, Teruki Homma, Kenta Oono, and Koji Tsuda. Population based de novo molecule generation, using grammatical evolution. Chemistry Letters, 47(11):1431-1434 2018.
Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable human professionals. Nature communications, 13(1):1-11, 2022.
Changkun Zhang, Leyuan Zhang, Yu Ding, Sangshan Peng, Xuelin Guo, Yu Zhao, Gaohong He, and Guihua Yu. Progress and prospects of next-generation redox flow batteries. Energy Storage Materials, 15:324-350, 2018.
12

---

Under review as a conference paper at ICLR 2023
A PROMPT DESIGN
Table Example the multi property optimization tasks and prompt designs used the zero-shot evaluation. We color each prompt with the modality(s) that they are associated with.
Example
Prompt
Osimertinib
<lexi (- (dinmthyy peop- 2-eaanide </LAAL>
<praperty > tpss w/ppppprty> <or. 0</va1>
<prooeenty logp < w/propprryypee.occoo <va)
<mbl>
Stexiulyp-20 > SUPACK </ y:*
Fexofenadine
<towt (=10-=---o---ddroor methoppoopnooo acid]</tam </text>
<property //tp://///porrrr////1aa./1
Stext > 6-dimahhyyyyyyeyyyyy
Ranolazine
<propercy <ha
<property> wttp::/rarrootttteerooccccceeeee //pppppcerraggcc//ppoottee //tt
<property f.count /
<nol>
<lext.yype Puncpppppaapc </exxt.yype>>>
Perindopril
<text ethooy oxopentan, seppentanncctsahyydrrtttooo carboxylic acid<//tett>
Kpropecly wco.tii.. http:///iiig//////tt K</ol>>
<iexi.lype>> BuncGroupa Aypx
Amlodipine
wwn.oethwspeeti.....eeeeee..e..e didyroooyyd.yyyyyyyyyyyrdddddd dicarbong/ate<//ten>
<properly sing /tt.:ppeeppccoeecccce////c/cpccp
Aype>
trieiuorometyyl pytarinc/ttttttt
Sitagliptin
<[ext. Mat.:eMoolllaaaccc //aemaaaaa/////eaaeee
<text >- 01000000000113000///OO
<praperty <ttpo//ss/>
<property Strtt w/ppopertyp Coa <ooll>
<textut wtw. SUPAC wtwpcc/eeet..ypee
Zaleplon
<eoxs> (3 M--thyyacctammdde </text>
Stext.type>MolPonnee c/text.yycc
PLogP/QED
indole, carboxanide de<//eet>>>>>>
(Drug-Likeness)
<propecty wlogp< Seoogp//rroppee htpp///pppyyy///eooooo <val>>
<prsperty'> Roa h//p:pprrty>-wee Cvall </vall C-Coo>>>>>
PLogP/DRD2
<text.type PunoGroups </text.typo>
< <i > exo, triazolo, www.hhhyyyy....... wwnsinidaroll sinetyyyeaaayyyyyyneiiiiiee //Let
(Biological Activity)
<praperty > www.oppc/prpppstyybbaa... <va.: 10//v>>>
<propenty's h/prrpprrtyaae. S <ncl>>
13

---

Under review as a conference paper at ICLR 2023
Table 7: Example of the conditional molecular structure generation tasks and prompt designs used the zero-shot evaluation. We color each prompt with the modality(s) that they are associated with.
Task
Example
Prompt
Celecoxib
<text type> SOPAC<CPP////er type>
Molecular
Rediscovery
(Cax>>//ipp(((aaabyyyaeeee)) de <mol>>
Troglitazone
<text_type>> </optt//ettt.ype
Thiothixene
<coxt_type>> </ox0c//.
<text>> (9X) -M (, N-dinethyl thi.aantaeeeee-eelooooaaaee
Albuterol
</text..ype>>
Similarity
Sampling
<rext Butyylanatyl pttr:///eoo..yrrrryyeyy..ppooo/ <
Aripiprazole
<text. TwneGroups
<Lext www.hloooopaee....ipeaoooo. pipaaaar.,......ppii......i quinolin /taliii////e </Lox:><<nl>>>
<text.type wwwnstroupe//text...oa
Mestranol
<text>> nethor, wtt.eeeeeyyy.eeeoy..eeee....... otta.ryo, hwt.://wwpppoooonnnnnneer/sssees
Generation
Cn
Isomer
<hox=>>>11BB<<<<>a>>>>>>>>> <mol>
CgHHoN2O,PP.CC
Median
Molecules
Camphor/Menthol
<Lexi bype wwncGroups
<tort beptan, methyl, hwtp://...tethhhhapoooooooooooee
Tadalafi/Sildenaail
<text.type
< ext pysaxolo, wwwessteeetwayllopypss..eatte httpa:/ww.......aoooooocccaaaaaa
<iext IOPACCCAACC.. MU0ACC/L=xxt.yy3e.
| < exx methanoyyl phanyl]mmet (hytly wttiiinneeeeeteeeeeeeee
Valsartan
Substructure
Sampling
<pzoparty tew.rrryyttteeeeoo httpa///coooo////<//c1o37777
<propert </property><<cv
Deco Hop
type>>
<<owe>> www.goooooooeeee//ecsr...
Scaffold Hop
<coxt>> propanol, bewoothhaoooo///Los <nol>
14

---

Under review as a conference paper at ICLR 2023
B PROMPT SAMPLING STRATEGY
Prompts are stochastically generated from the available modalities by the following set of rules:
The text modality is sampled uniformly from the list cTuPaC, ForeB-oups, If Rone is selected then no text conditioning is included for that sample. This allows the user to perform property-only conditioning by leaving out the text conditioning during inference.
If FuneGroups chosen, then the number of functional groups, N, used for conditioning is sampled uniformly from (1-M) where M is the total number of functional groups for the given molecule. Then N functional groups are selected from the list and concatenated with commas.
Next, the number of property conditions, K, is sampled uniformly from [0-2] where L is the total number of property modalities available for training. Then K properties are chosen from the list and their property names and values are added to the prompt after the text type and text. The ordering of property sub-modale modalities is also stochastic.
C TRAINING & SAMPLING IMPLEMENTATION DETAILS
We use the GPT-NeoX Python library Andonian et al.
2021 developed with Megatron Shoeybi al.] (2019) and DeepSpeed Rasley et al. (2020). We optimize the autoregressive log- likellhodd
(i.e., cross-entee cross-entropy loss) averaged over 256-token context. We set the global batch size as 2048, and the learning rate to 10-4, and rely on the cosine decay. We use an Adam optimizer with B1 = 0.9, 82 = 0.99, and a = 10- and clip the gradient norm at 1.0. We use the Rotary positional embeddings Su et al. ). parallel attention and feed-forward (FF) Black et al. (2022), and all dense layers in comparison to the original transformer decoder model architecture Radford et al. 2019).
We a q temperature value of 1.0 for sampling for evaluating all 22 tasks. We found that this value gives us the best tradeoff between the validity and diversity of the generated molecules. For each GuacaMol task, generate 128K samples of samples that are generated and evaluated during fine-tuning of the GuacaMol baselines. For the Drug-Likeness and Biological Activity tasks, evaluate on 250 molecules randomly sampled from a subset of the ZINC dataset provided in Jin et al. (2018)) in accordance with the methods outlined in Pu et al. (2021). For each molecule, we generate 1K samples which is on the order of the number of samples that are generated and evaluated during fine-tuning of the MIMOSA baselines.
for evaluation. This is the order of the number
D BASELINE MODELS
We compare MOLJET to two sets of baselines - one for the GuacaMol tasks and another for the LiuggLLieees/BBlooiccaa Activity tasks. The GuacaMol baselines include:
Best of Data Set, the metrics evaluated on the top molecules from the ChEMBL dataset (Gaulton et al. 2012))
SMILES ES LSTM, an LSTM model which is fine-tuned with the hill-climbing method
(Brown al. 2019))
SMILES GA, genetic algorithm that makes mutations to a SMILES string Yoshhikawal al. 2018))
Graph GA, a genetic algorithm that makes mutations directly molecular graph (Jensen 2
The Bhug-Likeness/Billoiicl Activity baselines include:
VJTNN, a graph-to-ggaa translation VAE that utilizes adversarial regularization (Jin et al., (018b))
DeepGA, genetic algorithm enhanced with a discriminator neural network to improve molecular diversity (Nigam al. 2019)
15

---

Under review as a conference paper at ICLR 2023
MIMOSA, Markov chain Monte Carlo sampling strategy augmented by pretrained graph neural networks (Fu et al. 2021))
E MODEL PERFORMANCE ON INVIDIDUAL GUACAMOL TASKS
Table g shows the detailed performance view on the GuacaMol benchmark. Aside from the redis- covery tasks, the final score for each metric is evaluated a weighted average of the top 100 scoring molecules that were generated during sampling. The scores for individual molecules are based on their ECFP4 (Rogers & Hahn 2010) fingerprint similarities to the targets, calculated property values and structural features. These values are passed through set of modifiers and thresholds to scale them between 0 and 1. The score is then calculated the geometric mean of each scaled task- specific value. For further details on the metric definition of each benchmark, please refer to Brown al. (2019).
Table 8: Benchmark results on GuacaMol which contains both MPO and molecular structure gen- eration tasks. Bold values indicate the best performing model and underlined values indicate the second best performing model
Benchmark
Best of Data
SMILES SMILES Graph
MOLJET-GUAC
MOLJET-GUAC + Grapph
Benchmark
Category
GA
(Zero-shot)
Osimertinih
0.781
0.894
0.880
0.937
0.914
0.992
Fexofenadine
0.817
0.926
0.904
1.000
0.997
1,000
Ranolazine
0.836
0.833
0.832
0.913
0.920
0.920
MPOs
Perindopril
0.701
0.764
0.644
0.803
0.804
0.823
Amlodipine
0.696
0.885
0.678
0.888
0.895
0.903
Sitagliptin
0.509
0.536
0.526
0.809
0.758
0.823
Zaleplon
0.547
0.610
0.552
0.728
0.625
0.688
Celecoxib
0.674
1.000
0.570
0.836
1.000
1,000
Rediscovery
Troglitazone
0.558
1,000
0.523
1.000
1.000
1.000
Thiothixene
0.608
1.000
0.476
1.000
1.000
1.000
Albuterol
0.522
1.000
0.871
1.000
1.000
1.000
Similarity
Aripiprazole
0.595
1.000
0.747
0.985
0.999
1.000
Mestranol
0.520
1.000
0.695
0.945
1.000
1.000
Valsartan
0.259
0.931
0.628
0.958
0.930
0.977
Substructures
Deco Hop
0.933
0.996
0.876
0.995
0.893
0.996
Scaffold Hop
0.738
0.993
0.803
1,000
0.632
0.984
Isomers
0.684
0.96.
0.734
0.952
1.000
1.000
0.747
0.860
0.757
0.955
1.000
Median
Camphor/Menthol
0.334
0.398
0.348
0.405
0.386
0.416
Tnddlafi//ildemaaill
0.407
0.408
0.377
0.429
0.434
0.478
Total
0.623
0.850
0.671
0.877
0.857
0.900
F RECONSTRUCTION TASKS
To validate the ablation on the Text + Property vs. the Text-Only models, we construct two addi- tional tasks that evaluate the model's performance on text-only conditioning - IUPAC Reconstruction and FuncGroup Reconstruction. An IUPAC reconstruction is counted as successful if the generated SELFIES string exactly matches the canonical SMILES from the holdout set after being decoded back into SMILES and canonicalized. IUPAC Reconstruction is evaluated on 10000 randomly sampled IUPAC/SMILES pairs from the holdout validation set. A FuncGroup reconstruction is counted successful when the SMILES string decoded from the generated SELFIES string matches the substructure pattern matching the requested functional group (we use SMARTS substructures for matching). We hand select 102 functional groups test the model on its ability recognize simple functional groups, basic nitrogen heterocycles, basic oxygen heterocycles, basic mixed heterocycles, double ring nitrogen heterocycles, double ring oxygen heterocycles, polycyelic aromatic hydrocar- bons, fused rings and phenyls among others. The full dataset will be made available upon request.
16


Reviews:
[{"review_id": "z4kv7IaaWI", "reviewer": "Reviewer_X1nX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multimodal transformer for molecular generation with multiple property optimization against multiple objectives, using language-based inputs. The authors describe prompt engineering based upon combinations of descriptive text and string representations of molecular properties and molecules as SELFIES strings. Improved performance on some standard benchmarks for molecular generation, GuacaMol and MIMOSA, are demonstrated. ", "summary_of_the_review": "A multimodal foundation model for MPO is a good idea, and potentially very impactful. In addition, here a strictly text-based modality here makes for a neat and simple model. The paper is clearly written, and in some cases the results on benchmarks appear quite impressive. However, the experimental results are overall not strong as much of the improvement was gained by using the model to seed a genetic optimization algorithm, and this behaviour needs to be explored more thoroughly. The motivations for the particular choice of prompts (apart from the limitations of publicly available data) are not clear and the paper would benefit from more discussion of the likely generalization ability of such a model, and how well it is likely to have learned chemical rules, rather than just substructure matching. \n\nOverall, while the idea is a good one, this is not yet a suitably good contribution because the execution and experimental results are not sufficiently convincing. ", "strength_and_weaknesses": "Strengths\n* A new multimodal \"foundation\" model for drug-like molecules, with a range of carefully designed prompts. This the first attempt of this specific kind in the area of molecular generation with MPO. \n* This model is to be publicly available via an API. \n* The paper is clearly written and accessible to a range of audiences, which is important in a cross-disciplinary piece of work. \n\nWeaknesses:\n* The experimental results are not strongly convincing. MolJET-GUAC is only outperforming methods not based on MolJET in the isomers task alone. However, Table 4 is ultimately misleading. The best results are in fact obtained by taking MolJET and using it to seed GA. Given that seeding has such a strong impact on the outcome of GA, it is not concrete to claim that the outcome is due to MolJET. To properly substantiate this claim, a more detailed study of the performance of GA under different seeding strategies is a necessary comparison. It is not clear whether the GA-only model was randomly seeded, seeded by similar models or another strategy. It is also notable that MolJET by itself is unable to produce these results, hinting possibly at limited diversity in the molecules sampled from MolJET. \n* While the model has been trained to beat standard benchmarks, it is not clearly motivated and explored in more genuine zero-shot settings; many of the properties optimised for are \"global\" and dependent on a set of substructures being present, with the conditional generation most likely to ensure that the backbone scaffold is in the correct region of chemical space. Therefore, it is not clear that this model has learned any deeper representation of molecules other than one able to recognise functional groups. It would be helpful to include evaluation on more local properties such as binding affinity to specific proteins. \n* It is arguable that this represents true \"zero-shot\" generation. The properties have been included in the training set, and while the holdout set requests molecules optimised for those properties in very different regions of chemical space, the features required to optimize these properties will often be conserved across chemical space. Therefore, performing well on MPO tasks does not represent zero-shot generalization. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "wBzs2bmWAdM", "reviewer": "Reviewer_rKPy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper illustrates a new generative foundation model for multi-property optimization in molecular de novo design. The authors leverages transformer based self-supervised learning to pre-train the model with carefully designed input sequence. The input token sequence contains multiple prompts which are human interpretable. The model is able to conduct conditional generation in a zero-shot manner. The model is evaluated against an exhaustive list to evaluation tasks. The results show the model ourperforms 80% of the task-specific optimization in zero-shot and achieve SOTA when finetuning. Extensive ablation study also show the necessity of the multimodal design where property improved the performance compared to text-only model. The pretrained model was provided as foundation model for finetuning", "summary_of_the_review": "The paper is technically sound, and clearly written. The novelty is sufficient to bring multimodal into the AI driven molecular design. The prompt design is smart to handle multiple modalities in the pretraining. Though I am not an expert in this direction, I recommend an accept since the paper is self-contained for easy reading.", "strength_and_weaknesses": "*Strength*\n- The paper is well written with clear organization.\n- The design is really novel to bridge the cross-area research to bring AI to real-world problem\n- The experiments are thorough\n\n*Weaknesses*\n- minor, see clarity section", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "jv6v2f0K371", "reviewer": "Reviewer_cdBp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to train a \"foundation\" transformer model on inputs featuring molecule names, properties, and their string representations. The transformer can then be used for optimization by prompting it with molecular names and properties, then predicting the most likely molecule to result. They perform several experiments on GuacaMol tasks (toyish tasks) with promising results.", "summary_of_the_review": "The technique proposed by this paper uses interesting prompt engineering, but seems unlikely to be useful for real-world problems and is very similar to other work on transformers for molecules. I also did not find their experimental results convincing. Therefore I recommend rejection of the current manuscript.", "strength_and_weaknesses": "Strengths: reasonable idea (the same sort of thing worked well in language, so why not chemistry?), clear writing, good ideas for prompt generation\n\nWeaknesses:\n\n- Missed a lot of related work. There have been many papers proposing \"foundation models\" for molecules [1-5], but the discussion is minimal. My understanding is that most of these methods have essentially tried something very similar to this work. The key difference between this work and previous works seems to be the structure of the prompt, but it would be good for the authors to acknowledge the previous work and explain more clearly how their contribution expands upon previous work.\n- Missed related work on optimizing molecules: there have been many papers focused on producing molecules with optimized properties, but the authors do not discuss this or cite many of them. I think the authors' focus in the related work section was too narrow by only focusing on a handful of works which explicitly focus on multi-property optimization. A common method of multi-property optimization is to combine all the objectives together into a single scalar objective, then use a single task optimization technique (see [6] which benchmarks many of these methods). I think this needs to be discussed more to explain how this work fits into a large existing literature on property optimization.\n- Method seems geared towards trivial properties and unlikely to work for more complicated real-world properties. The authors train the transformer on prompts constructed from easily-available molecular properties like QED, ring counts, TPSA (things all available from `rdkit`). The benchmarks from GuacaMol all involve producing molecules with specific values of these properties, so it is not surprising that it does well on them. However:\n  1. Producing molecules like this is not a challenging problem since these properties can easily be calculated on a computer. Therefore one-shot generation isn't necessary or desired; instead iterative algorithms like those in GuacaMol could be used.\n  2. The thing which _is_ desired is to produce molecules with non-trivial properties (e.g. binding affinities, toxicities) given few (or no) labelled training examples. However, the authors do not show this, and it seems incredibly unlikely that MolJET would successfully do this without a large quantity of training data. The authors do experiments on DRD2, but as far as I can tell this is the output of a predictive model, not an experimental measurement (although the authors should clarify this). Therefore this method does not seem to be posed to solve practical, real-world problems.\n- The origin of the numbers in Table 3-4 are unclear. Are these averages over multiple tasks? There should be error bars. I think not reporting error bars is bad practice.\n- One critical detail which the authors only mention in the appendix is that the numbers in Tables 3-4 come from the best of _128,000_ samples from the model (i.e. they are the top 0.1% of all outputs). I think it is not appropriate to call this \"zero-shot\": in real life one would need to evaluate these 100k+ proposed molecules to find the best one, which creates a lot of training data in the process. Therefore it would be more appropriate to use an iterative optimization algorithm. I think it would be more honest to plot the distribution of property scores from the molecules outputted by MolJET.\n- The authors only used baselines present in the original GuacaMol paper. Since its publications other methods have beaten these scores, for example [7]. The table from [7] also contains many other baseline methods. I think the authors should use it.\n\n[1] Rong, Yu, et al. \"Self-supervised graph transformer on large-scale molecular data.\" Advances in Neural Information Processing Systems 33 (2020): 12559-12571.\n\n[2] Zhang, Zaixi, et al. \"Motif-based graph self-supervised learning for molecular property prediction.\" Advances in Neural Information Processing Systems 34 (2021): 15870-15882.\n\n[3] Chithrananda, Seyone, Gabriel Grand, and Bharath Ramsundar. \"ChemBERTa: large-scale self-supervised pretraining for molecular property prediction.\" arXiv preprint arXiv:2010.09885 (2020).\n\n[4] Wang, Sheng, et al. \"SMILES-BERT: large scale unsupervised pre-training for molecular property prediction.\" Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics. 2019.\n\n[5] Irwin, Ross, et al. \"Chemformer: a pre-trained transformer for computational chemistry.\" Machine Learning: Science and Technology 3.1 (2022): 015022.\n\n[6] Gao, Wenhao, et al. \"Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization.\" Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.\n\n[7] Ahn, Sungsoo, et al. \"Guiding deep molecular optimization with genetic exploration.\" Advances in neural information processing systems 33 (2020): 12008-12021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "aatbh6ohbS", "reviewer": "Reviewer_TDmG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multimodal joint embedding transformer for molecule generation. Specifically, it leverages both text descriptions, molecular string descriptors, and molecular properties to train a join embedding and introduces prompt building on the success of language models. After training such a model, the user can use prompt for a variety of molecule generation tasks, such as property optimization, isomer generation, etc.", "summary_of_the_review": "Overall, I think this is a good initial attempt to introduce text-based pretraining to molecule generation. However, from its current form, I do not strongly agree that it is a significant contribution yet mainly because of its problem formulation, motivation and experiment results. I would put a high standard on this since it is claimed the first or at least from what I know the first batch of works in this direction, it should be well-motivated and formulated for other people to follow in the future. I am happy to adjust my score if the authors make a good rebuttal. ", "strength_and_weaknesses": "Strength\n* There is certainly merit to adopt a prompt-based method from the language model for flexible molecule generation.\n* The developed method is simple and technically sound.\n* The experiments are conducted on reliable benchmarks (e.g. GuacaMol).\n\nWeaknesses\n* Even though the \" multimodal foundation model\" for molecule generation sounds desirable, I do not see it is well-supported or has empirical merit of it yet. First of all, the proposed model is trained over 100M molecule dataset. Compared to the baseline methods, tons of knowledge, or at least molecular structures and simple properties such as QED and logP, are integrated into the model. It is hard to tell the merit of such a multimodal foundation model over the other compared methods. \n* Is this multimodal foundation model suitable for the predictive task?\n* This paper misses a large number of important literature on molecule generation [2], even though the authors argue another modality such as graph is left for future study, again this lowers the significance to introduce a text-based model if not compared against other approaches. Even for the string-based method, [1] is missing. \n* Related to the previous point, it is not compared against many other molecule optimization methods, even though the authors may argue they focus on single property optimization. I would suggest the authors add a single property optimization task as a comparison to other methods.\n* Even though it is called zero-shot generation, I feel this is a bit misleading since given the pretraining context, it is almost equivalent to train a predictor for properties.\n\n[1] Flam-Shepherd, D., Zhu, K. and Aspuru-Guzik, A., 2022. Language models can learn complex molecular distributions. Nature Communications, 13(1), pp.1-10.\n[2] Du, Y., Fu, T., Sun, J. and Liu, S., 2022. MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design. arXiv preprint arXiv:2203.14500.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WeuQWxLz_G", "reviewer": "Reviewer_KXSs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of this paper investigate a unified model for multiple tasks in molecular science. The model named MolJET is based on Transformer and inspired by unification models in NLP. MolJET takes a prompt sequence and is able to conduct multiple tasks. Experimental results on several tasks show better or competitive performance compared with baselines. ", "summary_of_the_review": "Overall, it is a good paper in the area of molecular design and optimization. This proposed model is able to conduct multiple tasks in zero-shot settings. ", "strength_and_weaknesses": "Strengths:\n1. One major strengths of this work is text-based prompt design for multimodal fusion within the Transformer model, such that the model can do versatial tasks.\n2. The model MolJET can be used for various conditional design and multi-property optimization tasks. Experimental results show that MolJET can achieve excellent tasks. \n\nWeaknesses:\n1. Math expressions can be better formulated. For example, in Eq. 1, x is at the left-hand side, however s appears at eh right-hand side. In Eq. 2, i is not reflected in the sum. log -> \\log, etc. \n2. Minors: transformer -> Transformer, tanimoto -> Tanimoto. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q0S3Qu4ItVT", "reviewer": "Reviewer_qqLn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a large transformer model that generates small molecule SELFIES from input prompts that convey properties of relevance in molecular design. The model shows good performance on GuacaMol and MIMOSA, two previously published benchmarks.\n", "summary_of_the_review": "\nOverall, I believe that this work is more appropriate for a specialized audience and perhaps it could benefit from additional discussions between the authors and those specialists, for example about how to design an improved benchmark to demonstrate the power of such a pretrained model.  The current draft is *not* strong enough and broad enough for publication in ICLR2023.\n", "strength_and_weaknesses": "\nThe high level idea behind this model originates from conditional generation methods in natural language, and has some promise.  The paper demonstrates the method on two benchmarks.\n\nThe paper only includes a limited review of relevant literature; it does not highlight existing promising mixed representations of small molecules that include the 2D graphs or 3D coordinates and that have shown a level of success in mixed models.  Unfortunately the paper does not provide code, explicit data, or descriptions of the framework or hardware used in training, so it is hard to verify the correctness or assess the reproducibility of the method presented in this paper.  It is not clear a priori to me if the ECFP4 similarity of 0.343 for holdout removes  memorized solutions from these benchmarks when genetic algorithms also come into play in Table 4, which could easily recombine the benchmark molecules from prior pieces and predict properties as linear combinations. I like and encourage the author's sentiment for going after big models, however, it would be nice to pick a downstream task that would be impossible to trivially guess from the training (for example finetuning to the prediction of a new property not included in the training set; for example, take a look at the open graph benchmarks, including the large-scale challenge for small molecules.)  I have a pet peeve against the optimization of the property PLogP, which I believe is unlikely to come up in a real drug discovery project (who wants to make their molecules more hydrophobic and more linear?)---do the authors think that being able to optimize this property adds value to molecule generators, and if so, why?  The listed DRD2 success rate in table 3 is small compared to the graph-to-graph applications on the same problem, so I'd recommend that the authors do a more extensive review and update the table with the correct state of the art results. Finally, why is the reconstruction from IUPAC in Table 5 so low?  Wouldn't the pretraining include translation and didn't past translation models from IUPAC to SELFIES basically achieve perfect results?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject