C5T5: Controllable Generation of Organic Molecules with Transformers
Authors: Daniel Rothchild
Alex Tamkin
Julie Yu
Ujval Misra
Joseph E. Gonzalez
Venue: ICLR 2022 Submitted (2022)
Keywords: molecular modeling
sequence modeling
conditional sequence modeling
drug discovery

Abstract: Methods for designing organic materials with desired properties have high potential impact across fields such as medicine, renewable energy, petrochemical engineering, and agriculture. However, using generative models for this task is difficult because candidate compounds must satisfy many constraints, including synthetic accessibility, intellectual property attributes, ``chemical beauty'' (Bickerton et al., 2020), and other considerations that are intuitive to domain experts but can be challenging to quantify. We propose C5T5, a novel self-supervised pretraining method that works in tandem with domain experts by making zero-shot select-and-replace edits, altering organic substances towards desired property values. C5T5 operates on IUPAC names---a standardized molecular representation that intuitively encodes rich structural information for organic chemists but that has been largely ignored by the ML community. Our technique requires no edited molecule pairs to train and only a rough estimate of molecular properties, and it has the potential to model long-range dependencies and symmetric molecular structures more easily than graph-based methods. We demonstrate C5T5's effectiveness on four physical properties relevant for drug discovery, showing that it learns successful and chemically intuitive strategies for altering molecules towards desired property values.


Under review as a conference paper at ICLR 2022
C5T5: CONTROLLABLE GENERATION OF ORGANIC MOLECULES WITH TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
Methods for designing organic materials with desired properties have high po- tential impact across fields such as medicine, renewable energy, petrochemical engineering, and agriculture. However, using generative models for this task is difficult because candidate compounds must satisfy many constraints, including synthetic accessibility, intellectual property attributes, "chemical beauty" (Bicker- ton et al. 2012), and other considerations that intuitive to domain experts but can be challenging to quantify. We propose C5T5, a novel self-supervised pre- training method that works in tandem with domain experts by making zero-shot select-and-replace edits, altering organic substances towards desired property val- C5T5 operates IUPAC names-4 standardized molecular representation that intuitively encodes rich structural information for organic chemists but that has been largely ignored by the ML community. Our technique requires edited molecule pairs to train and only rough estimate of molecular properties, and it has the potential to model long-range dependencies and symmetric molecular structures more easily than graph-based methods. We demonstrate C5T5's ef- fectiveness on four physical properties relevant for drug discovery, showing that it learns successful and chemically intuitive strategies for altering molecules towards desired property values.
(1) Original: <med> 2-acetyloxybenzoic acid
(2) Input: <high> 2-<mask> benzoic acid
(3) Prediction:
decyl
Generation: 2 2-decylbenzoic acid
Figure 1: Increasing molecule's octanol-water partition coefficient with C5T5. (1) A molec- ular fragment (acetylo (xxy)) is identified in a molecule interest. (2) The molecular fragment is replaced with mask token (<mask>) and the property value is set to the desired bucket (<high>). (3) Sampling from C5T5 produces a new fragment (decyl). Substituting in the fragment yields new molecule. The long chain of carbons added to the molecule increases its solubility in octanol while decreasing its solubility in water.
INTRODUCTION
Organic molecules are used in countless applications across human society: medicines, industrial chemicals, fuels, pesticides, plastics, television screens, solar cells, and many others. Traditionally, new molecules are designed for particular tasks by hand, but the space of all possible molecules is so vast (e.g. the total number of drug-like molecules may be as high as 100h) that most useful materials are probably still undiscovered (Reymond et al. (012).. To automate materials discovery, domain experts have turned to high-throughput screening, which large library of potentially useful molecules is generated heuristically. and the most promising molecules are chosen for further study using computational models that estimate how effective each substance will be for the target application (Hughes et al. 201T). Unfortunately. even gightthroughput methods can still only screen tiny fraction of all possible molecules.

---

Under review as a conference paper at ICLR 2022
IUPAC Name
SMILES
OH
OH
Oa
2-Bostyloxybenzaic acid
DCL-0/0011-00-00--11000
Figure 2: Visual representations of IUPAC names and SMILES. Tokens in IUPAC names
spond to well-known functional groups and moieties. In contrast, tokens in SMILES correspond to individual atoms and bonds.
Generating molecules directly with machine learning addresses this limitation, but de novo genera- tion can be of limited use in domains like drug discovery, where experts' intuitions about structure- activity relationships and external factors like patentability are important to consider in the design process. These constraints structure; for example a domain expert may be interested in a particular scaffold because it has favorable intellectual property attributes, or certain parts of drug may be needed for the desired biological activity, while other parts can be modified to increase bioavailability.
often be expressed by providing known portions of the molecular
To address this real-world setting, we consider the problem of learning make localized modi- fications to a molecule that change its physical properties in a desired way. We propose C5T5: Citraattrrssii-CConntiiiee Chemical Changer with T5 (Raffel et al., 2019), novel method for generative modeling of organic molecules that gives domain experts fine- sene-grained con- trol over the molecular optimization process while also providing more understandable predictions than prior methods (Figure T). Our two key contributions are 1) recasting molecular modeling as language modeling on the semantically rich IUPAC name base representation, and 2) the develop- ment of a novel conditional language modeling strategy using transformers that supports targeted modifications to existing molecules.
Controllable
IUPAC Names. The IUPAC naming system is systematic way of naming organic molecules based on functional groups and moietics, or commonly occurring clusters of connected atoms that have known chemical behaviors. Organic chemists have discovered countless chemical reactions that operate on functional groups, and they use these reactions to develop synthesis routes for novel molecules. Despite this, existing generative methods for organic molecules have ignored IUPAC names as a representation, instead opting for atom-based representations like SMILES (Weininger) 1988) and molecular graphs (Duvenaud et al., 2015). See Figure 2 for comparison of these rep- resentations. We argue for several advantages of IUPAC names in Section 3.1 To the best of our knowledge we are first to use IUPAC names as a base representation for molecular modeling.
Sele-Supervised Objective for Zero-Shot Editing. To enable targeted modifications of molecules without predefined edit pairs, we train transformers with a conditional variant of a self-supervised infilling task, where the model must replace masked-out tokens in the IUPAC name. As described in Section 3.2 we condition the model by prepending IUPAC names with discretized molecular property values; the model then learns the conditional relationships between the property value and molecular structure. To the best of our knowledge, C5T5 is the first method to use conditional infilling for select-and-replace editing; controlled generation contexts, such as modeling affect, politeness, or topic of natural language (Ghosh et al. 2017 Ficler & Goldberg 2017 Niu & Bansal 2018 Keskar al. 2019).
anticipate this method could be broadly applied in other
As we show in Section 4 CST5 is able to make interpretable targeted modifications to molecules that lead desired changes across several physical properties important in drug design.
RELATED WORK
Modeling. A number of machine learning methods have been developed for the task of design- ing organic molecules, but most do not allow a user to make targeted modifications to a molecule. Some methods, like generative adversarial networks and unconditional sequence models, provide no
2

---

Under review as a conference paper at ICLR 2022
control over generated molecule's 's structure (Grisoni al. 2020 Guimaraes et al. 2017 Sanchez Lengeling et al,; [e Cao & Kipi, 2018 Kajino 2019). and are therefore more useful for generating candidate libraries than optimizing particular molecule. Other methods, like variational autoen- coders or sequence models that are conditioned on a base molecule, allow specifying that generated molecules should be similar
starting molecule in some learned space, but there is no way to specifically target a certain part of the molecule to modify (He et al.] 202 lb. Jin et al. 2209;; Shinn al. 2018 Lim al. 2018
al. [202]] Yang et al., 2020 Kotsias et al., 2020; Gómez-Bombarelll
Dollar et al.
20221 Liu et al.
2018: Jin et al., 20188] Maziarka et al. [0200] Olivecrona et al. 2017
Bagal et al.
2021 You et al.
2018 Shi al., 2020). Recognizing the importance of leveraging
domain experts' intuition about -tructure-activity relationships, several methods, published mostly
in chemistry venues, have explored constraining generated molecules to contain a scaffold, or sub- graph of the full molecular graph (Li et al. 2019 Lim et al., 2020; Maziarz et al. (201)) However, these methods append to scaffolds arbitrarily instead of allowing domain experts to specify which part of the molecule they would like to modify or append to, limiting their utility for human-in-the- loop molecular optimization.
A few methods have explored allowing targeted modifications, where domain expert can mask out a portion of a starting molecule and ask the model to replace the mask with a novel side chain (Arás-Pous et al., 2020; Langevin et al., 20200 He et al. 2021a). These methods are limited because they only support masking parts of the molecule that can be truncated by cutting a single bond, and because they require a dataset of paired molecules (scaffolds & decorators) that must be constructed using hand-crafted rules. In contrast, CST5 learns in an entirely unsupervised fashion and therefore requires no paired data; the only limit to what can be masked is what can be represented using IUPAC tokens.
Representation. Existing methods all use SMILES (or a derivative representation) or graphs to represent molecules. There are number of drawbacks to using the SMILES representation: a small change in a molecule can lead to a large change in the SMILES string (Jin et al. 2018);; flattening the graph into a list of atoms artificially creates variable- and long-range dependencies between bonded atoms; and it is difficult to reason about common substructures, because the same structure can be represented in many different ways depending on how the graph was flattened. And although graphs seem like natural representation for molecules, graphs do poor job encoding symmetry, long- range interactions between atoms that are many bonds apart but nearby in 3D space, and long-range interactions that arise from conjugated systems (Duvenaud et al. 2015). CST5 operates instead of IUPAC names, which we argue in Section 3.1 is a more suitable representation for molecular optimization because tokens have much more semantic meaning. See Appendix A for more details on how CST5 relates to prior work.
Transformers for Molecular Modeling Outside of molecular optimization, transformers have found number of applications in molecular modeling tasks, including property prediction (Wang et al. 2019; Rong et al. 2020), chemical reaction prediction (Schwaller et al. al.] 2019)), retrosynthesis (Karpov et al.]| 2019)) and generating proteins (EInaggar et al., [2220] Grechishnikova 2021). A few works have explored using transformers for generative modeling of organic molecules (He et al.) 2021b Shin et al. 202T: Dollar et al. al., 202T}. D. Some works have also proposed using transformers for scaffold-conditionee generative modeling (He et al. [201ia] [Bagal et al.) [20T]). This work extends these efforts by proposing a simple yet effective training and zero-shot adaptation method, and by using IUPAC names instead of SMILES strings.
IUPAC Names Although we are unaware of prior work using IUPAC names as a base representa- tion for molccular modeling, several works have explored using machine learning to convert between IUPAC names and other molecular representations (Rajan et al. 2021 Handsel et al. 2021 Krasnov
et al. 2021
3 METHOD
Molecular optimization is difficult problem because it requires modifying molecule that al- ready satisfies a number of requirements. Modifications need to improve a particular aspect of the molecule without degrading its performance on other metrics, and without making it too difficult
3

---

Under review as a conference paper at ICLR 2022
52.5
50.0
47.5
45.0
Mg
42.5
-5
40.0
-6
37.5
-10.0
-7.
-5.0
-2.5
2.5
60
Other
Change
40
Group
Count/Mult:
Ring loc.
20
Stereo
Elements
-0
27
28
30
440
440
-20 -10
30
50
-45
-50
64
555
-5.0
-2.5
0.0
2.5
5.0
7.5
10.0
Figure 3: T-SNE visualization of the word2vec embedding space. "Charge": tokens that indi- cate formal charge. "Group": functional groups and moieties. "Count/Mult": multipliers. "Ring loe.": fused-ring locants. "Stereo": stereochemistry markers. "Locants": simple locants. "Ele- ments": single-atom tokens. As shown, the 2D location of tokens carries high semantic meaning; for example, locants only collocated, but approximately in order.
to synthesize. We argue that by using IUPAC names (Section 3.1 and by allowing users to target particular parts a molecule to modify (Section 3.2), C5T5 has the potential support human- in-the-loop molecular editing that complements domain experts" intuitions about structure-activity relationships and synthetic accessibility.
3.1 IUPAC NAMING
The International Union of Pure and Applied Chemistry (IUPAC) publishes a set of rules that allow systematic conversion between chemical structure and human-readable name (Favre Powell, 2013). For example, 2-chloropentane refers unambiguously to five carbons ("pent") connected by single bonds ("ane") with a chlorine atom ("chloro")) bonded the second carbon from one end ("2-"). IUPAC names are used ubiquitously in scholarly articles, patents, and educational materials. In contrast other linear molecular representations like SMILES and its derivatives, where single tokens mostly refer to individual atoms and bonds, tokens in IUPAC names generally have rich semantic meaning. For example, the token "ic acid" denotes a carboxylic acid, which is a com- mon functional group that has well-known physical and chemical properties; there are many known chemical reactions that either start with or produce carboxylic acids. Other tokens denote additional functional groups (e.g. "imide," "imine," " "al," "one"), locants (c.g. "1," "2," "N"), which indicate connectivity, alkanes (e.g. "meth," "eth," "prop"), which denote the lengths of carbon chains, poly- cyclic rings (e.g. "naphthalene," "anthrcceee" stereochemistry markers ("R," "S"), and multipliers

---

Under review as a conference paper at ICLR 2022
(e.g. "di," "tri",, which concisely represent duplicated and symmetric structures. Figure 2 shows the relationships between IUPAC names, graph representations, and SMILES.
For molecular optimization, C5T5 supports qualitatively different molecular edits compared to graph- and SMILES-based methods by virtue of its use of IUPAC names: editing a locant token corresponds to moving functional group along carbon backbone changing the connectivity of a fused ring system; and editing a multiplier token corresponds to creating or eliminating du- plicated and symmetric structures. For example, changing "ethylbenzene" to "hexaethylbenzene" replicates the ethyl structure around the entire benzene ring with a single token edit. These sorts of modifications require much more extensive editing for SMILES and graph-based methods !
We argue that IUPAC names are especially attractive for molecular optimization, since the process requires interaction between the algorithm and a domain expert, so interpretability paramount. Compared to graph- or SMILES-bar based models, C5T5 makes predictions that can be traced back to moieties and functional groups that domain experts are more likely to understand, trust, and know how to synthesize than either arbitrary collections of atoms and bonds or motifs decided upon by machine learning practitioners.
In addition to improved interpretability, we argue that using IUPAC names has advantages purely from the standpoint of modeling data, since moving from SMILES to IUPAC names is akin to moving from character-basee to word-based sequence model. Modeling at this higher level of abstraction enables the network to direct more of its capacity to structure at the relevant semantic level, instead of relearning lower-level details like the specifie atomic composition of functional groups. In this vein, we demonstrate the potential of IUPAC names by learning word2vec represen- tations of IUPAC name tokens (Mikolov et al. 2013), drawn from list of over 100 million names in the PubChem repository (Kim et al. 2016 and tokenized using list of tokens in OPSIN-an open-source IUPAC Name parser library (MIT License) (Lowe et al. 201T). For example, as shown acid" gets tokenized to ["2", "_"" "acet", "yl", "oxxy,, "benzo". "i aeid"). As with natural language modeling, we find that the embedding space learned by word2vec encodes the semantic meaning of the tokens, as shown in Figure 3 Different classes of tokens tend to be clustered together, and similar tokens within clusters are located nearby. For example, aromatic compounds with two rings are clearly separated from those with three, locants are ordered roughly correctly from to 100, and multiplier tokens are also roughly in order (zoom not shown). Following Mikolov et al. (2013). embedding vector space correspond to semantic analogies between tokens. For example, the nearest neighbor of "phosphonous acid"- "nitrous acid" + "nitroso" is the embedding for "phosphoroso." O The nearest neighbor of "diphosphate" "disulfate" + "sulfate" is "phosphate." Likewise for "sele- nate"- "tellurate" + "tellurite" being closest to "selenite."
in Figure 2 the chemical "2-acetyloxybent
also find that simple arithmetic operations in the
3.2 CONDITIONAL LANGUAGE MODELING WITH TRANSFORMERS
We now present the C5T5 objective, which trains a model to alter the properties of a molecule through localized edits. Importantly, this behavior does not require training on human-defined edit pairs, a strategy limited by either a fixed set of hand-specified chemical alterations or an expensive experimentallly-driie derived training set. Instead, this editing behavior emerges as a zero-shot side-effect of our conditional language modeling objective, requiring only a simple forward pass using our pretrained model without additional gradient updates.
Given a property value P, fragment of molecule F, and the rest of the molecule C' (the context), we wish to learn the conditional distribution P(FIC, P). Then, for a new molecule, one could alter the molecule towards desired property value by redacting the original F, changing P to pd and sampling a new F" ~ P(F[C, P'). Intuitively, this asks our model what kinds of molecular fragments the model would expect given the context and the new property value.
'Moving the attachment point of functional group requires only small edit of graph (i.e. changing one hond), but most graph- molecular generation methods sequentially generate one node time, followed by any bonds that connect the new node to the molecular graph so far. Moving a side chain therefore requires removing the entire chain and regenerating it node by node. Fignoring the emhedding for "nitmss""
5

---

Under review as a conference paper at ICLR 2022
To learn this conditional distribution, we propose conditional generalization of the infilling objec- tive used to train T5 (Raffel et al. 2019) and ILM (Donahue et al. 2020). This process consists of several steps, also illustrated in Figure
1. Replacing random spans of the tokenized IUPAC name with sentinel tokens.
2. Prepending the resulting sequence with token indicating the original molecule's is computed property value. To obtain these property value tokens, we discretize the distribution of property values into three buckets, specified in T
3. Training the model in T5 to produce the sequence of redacted tokens, prepended by their corresponding sentinel tokens.
This conditional infilling objective incentivizes the model to learn the relationship between the com- puted property value and the missing tokens. To make localized edit to molecule, we then replace the desired fragments with sentinel tokens, change the property value token, and sample autoregres- sively from the predictive distribution. Thus, our approach hybridizes the flexible editing capabilitiess of ILM (Donahue et al., (020)) with the controllability CTRL (Keskar et al. 2019). See Appendix C for experimental details. Code is available at redacted,
4 RESULTS
To demonstrate the promise of combining IUPAC names with conditional modeling using T5, we explore several molecular optimization tasks relevant to drug discovery. Specifically, we train CST5 to make localized changes that affect the octanol-water partition and distribution coefficients (logP, logD), polar surface area (PSA), and refractivity four properties commonly used to estimate bioavailability of candidate drug (Ghose al. 1999: Veber et al.; 2002 Bhal et al.) 2007). logP and logD measure the ratio of a compound's 's solubility in octanol, a lipid-like structure, and water; drugs need be somewhat soluble in both to be orally absorbed. PSA and refractivity both relate to charge separation within the molecule. As shown in Appendix
B CST5 generates mostly valid and novel molecules, with values of logP that lie outside of the range of "best in dataset" baseline for targeted modifications.
4.1 C5T5 SUCCESSFULLY MODIFIES PROPERTIES
First, we demonstrate that the localized changes proposed by C5T5 do in fact control the property value as desired. C5T5 allows domain experts to choose where to make changes to a molecule based on their intuition and particular application: the user masks out tokens in the IUPAC name that can be modified, and the model fills in the masked spans in way that changes the property value as directed. There is no canonical way to choose particular tokens to mask for evaluation purposes, so simply choose number of starting molecules randomly from PubChem (Kim et al. 2016), and then we iteratively mask all length-one to length-five spans (to match the training distribution) and inference. We also experiment with masking multiple spans per molecule during inference, as done during training. This is useful in practice when there are multiple areas of the molecule that can be changed in tandem to achieve the desired property value, but in our evaluation we observe qualitatively similar results when masking only single span, so for computational efficiency we limit ourselves single spans. We expect multi-span masks be much more important when optimizing for more complex properties.
F shows that CST5 successfully generates molecules with higher property values when passed <high>, and with lower property values when passed <low>, The model is much more successful raising property values than lowering them, especially for refractivity and polar surface area. For both of these properties, increasing the property value is straightforward: just add polar groups to replace whatever tokens were masked. In contrast, lowering these properties is only possible when the mask coincides with a polar group, in which case the model must find non-polar substitute while still maintaining the molecule's 's validity. Even if unsuccessful at lowering these two property values average, C5T5 can still be used in this case to suggest number of candidate edits, and the one with the lowest property value can be selected using property prediction model. This improvement over high-throughput screening and untargeted machine-learning methods for

---

Under review as a conference paper at ICLR 2022
20
Target: <high>
20
Target: <high>
Target: <low>
Target: <low>
Generated logp
15
Generated logD
15
10
5
-5
-10
10
-5
10
Orig. logp
Orig. logD
400
250
Target: <high>
Target: <high>
Generated Refractivity
350
Target: <low>
Target: <low>
200
Generated TPSA
250
150
200
100
150
100
50
100
25
Orig, Refractivity
Orig. TPSA
Figure 4: Calculated property values of optimized molecules vs. original. Values computed for 30 randomly chosen starting molecules. Top left: octanol-water partition coefficient. Top right: octanol-water distribution coefficient at pH 7. Bottom left: molar refractivity. Bottom right: polar surface area. Blue violins show the distribution of generated molecule properties when the model was asked complete molecules to achieve <high> property value, and red for <low>. Cutoffs between <high> and <med> and between <med> and <low> are shown as dashed blue and red lines, respectively. The black dashed line is y=x. Black vertical lines show best in dataset baseline for each trial molecule.
molccular optimization, since it isn't restricted to a predefined library candidate molecules, and still allows the user choose particular parts of the molecule modify.
4.2 MODIFIED TOKENS ARE CHEMICALLY INTUITIVE
One main advantage of C5T5 is that suggested edits are in the intuitive language of TUPAC names, rather than arbitrary combinations of atoms. Table I shows the tokens that the model most pref- erentially adds a molecule when asked to produce low vs. high logP. Unsurprisingly, the most common tokens added when increasing logP are generally long carbon chains (pentadecyl, unde- cyl, heptadecyl) and other hydrocarbons (trityl, perylen). Conversely, when the model asked to produce low-logP modifications, hydrophilic groups are added. LogP is simple metric, and by proposing molecular edits for logP that are obvious and easily understandable domain experts, we expect users to gain confidence that C5T5 will suggest reasonable edits for more complex properties.
To further investigate the types of optimizations C5T5 suggests, Figure F visualizes two of the start- ing molecules from Figure with logP values of -2.4 and 5.8. For each molecule, we mask spans as usual and generate after prepending with <low> (molecules on the left) and <high> (molecules on the right). The IUPAC name of the top starting molecule is (3.3-bisaminommettyypeeellee diol," where "bis" signifies that the CNH2 group should be duplicated, and "diol" means dupli- cate OH groups at the ends. By virtue of the IUPAC name encoding symmetry, C5T5 is easily able to generate similarly symmetric molecules. For example, the molecule in the top left is "3,3- hisaaminnmetthyypeenne---- 1,5-iu-fiiii acid," where the "ol" has been replaced with "sulfinic acid."
7

---

Under review as a conference paper at ICLR 2022
Table Tokens most preferentially added when C5T5 is asked to make modifications resulting high vs. low logP values. Multipliers compare the actual rate of adding tokens compared to the expected number if the model drew randomly from the data distribution independent of property value. Blue tokens are hydrocarbons (i.e. lipophilic groups). Red tokens contain hydrogen bonding donors or acceptors (i.e. hydrophilic groups)
Target:
<high>
Target: <low>
trityl
77.0x
phospho
48.1x
pentadecyl
20.2x
phosphonato
44.7x
Z.
17.7x
sulfinam
41,2x
perylen
11.8x
hydrazon
34.3x
undecyl
8.1x
sulfinato
26.3x
heptadecyl
7.9x
Z
17.9x
ylium
7.6x
oxonium
10.3x
isoindolo
5.9x
amoyl
9.3x
bH
5.9x
carbamic acid
8.6x
iod
5.8x
sulfin
6.9x
IngP: 10.9
I5gP:-5
lngP: 5.2
logP:
-3
Original lopP:-24
lugP: 14.2
Decrease logP
Increase logP
logP: 8.1
Original log!:
lopp: 8.P
logF:
logP: 1.4
Figure 5: Visualizations of base and logt-mptimized molecules. Two molecules from the logP plot Figure with three C5T5-optimized molecules for each of <low> and <high> logP.
Although symmetry is often highly desired, C5T5 is not limited to generating symmetric molecules. For example, the middle molecule on the right is 9.3=bis(aminomethyll hhapp/d----hhheddeeeeeeeettaa 1,5- dios" added an additional carbon chain non-symmetrically at the end of the pentane
Sometimes C5TS generates valid but unstable compounds. For example, the neighboring NH₃ groups in the bottom left molecule in the top half of Figure 5 are unstable, and would turn into aldehydes in an aqueous solution. All machine learning methods are susceptible to this sort mis- take, underscoring the importance of the type of human-in-the-loop optimization that C5T5 enables.
"tthhough this is a preferred IUPAC name, is still unambiguous, and therefore valid and parseable.
8

---

Under review as a conference paper at ICLR 2022
CSTS-IUPAC C5T5-SMILES
CSTS-JURAC
800
-UPA- CSTS-SMILES
600
600
CSTS-SMILES
Molecules
HierG2G
Molecules
HierG2G
Molecules
600
HierG2G
400
400
400
200
200
200
0-
0-
0.0
0.5
1.0
10
Tanimoto Similarity
Synthelic Accessibillty
LogP Increase
Figure 6: Histograms of Tanimoto similarity to base molecules, synthetic accessibility of generated molecules (both as computed in (Jin et al. 2020)), and logP increase for 30 random base molecules from the Zinc test set.
4.3 C5T5 MAKES SMALL AND SYNTHETICALLY ACCESSIBLE EDITS
C5T5's proposed molecular edits are not only intuitive, but also small and synthetically accessible. Unlike methods that train on edited pairs of molecules or that optimize molecules to be similar and synthetically accessible in latent space, CST5 achieves this purely from self-supervision.
For 30 base molecules randomly chosen from the Zinc dataset (Sterling & Irwin 2015), we compute the Tanimoto similarity between generated and base molecules and the synthetic accessibility of generated molecules. Results are shown in Figure along with comparison to Hierarchical Graph- to-graph VAE (HierG2G), state-of-the-art molecular graph translation model that trains on pairs of Zinc molecules (Tin et al.) 2020). We also compare to an ablation of C5T5 that uses SMILES instead of IUPAC names. C5T5 generates molccules that are more synthetically accessible and similar to base molecules and that have wider range of logp increases than HierG2G. An additional advantage of C5T5 not shown in Figure 0 is that C5T5 allows the user to choose what part of the molecule to edit, whereas HierG2G makes edits wherever it sees fit.
Unlike C5T5, methods like HierG2G that train on pairs of similar molecules are fundamentally limited by a paucity of experimental molecule pairs that have high similarity and high property value improvement. See Appendix B.2 for a quantitative analysis.
DISCUSSION AND CONCLUSION
We propose C5T5, a simple and effective zero-shot method for targeted control of molecular proper- ties with transformers. Unlike prior approaches that make user-targeted modifications, method requires no database of paired edits; instead, it simply trains in a self-supervised fashion on large dataset of molecules and coarse estimates of their molecular property values. Core to our method is the use of IUPAC names as data representation, which captures molecular structure at an ap- propriate level of abstraction and enables an intuitive editing interface for domain experts. C5T5 successfully rediscovers themically-intuitiv strategies for altering four drug-related properties in molecules, a notable feat given the absence of any human demonstration of these editing strategies.
Our work also has several limitations. The select- reeeelaed- interface provided by the infilling objective may not always match the needs or preferred design process of domain experts. The interface also only suggests how to fill in missing parts of a molecule, relying on domain expertise or enumeration to decide which parts of the molecule should be changed to begin with. addition, we explore only coarsc grasine bucketing of property values, leaving more fine-grain fine-grained treatment for future work. IUPAC names might also be too limiting in cases where a user wants to edit a subgraph of a molecule that does not correspond neatly to small number of IUPAC tokens. Finally, training C5TS is computationally expensive and sample inefficient.
Future work will investigate using C5T5 with more molecular properties, such as the power conver- sion efficiency of solar cells. We also leave to future work extending C5T5 to jointly model multiple properties, and adding more flexible editing interface.

---

Under review as a conference paper at ICLR 2022
6 REPRODUCIBILITY
In Appendix C we describe how to download and prepare the training dataset, our tokenization strategy, masking strategy, optimizer, learning rate schedule, number of iterations, and generation strategy. Our code is also available online at REDACTED.
One barrier to reproducibility that we use ChemAxon's proprietary software to compute molecular property values. However. ChemAxon offers free academic license, and we suspect that all results would be similar if property values were computed with RDKit instead.
REFERENCES
Josep Arús-Pous, Atanas Patronov, Esben Jannik Bjerrum, Christian Tyrchan, Jean-Louis Reymond, Hongming Chen, and Ola Engkvist. Smiles-basee deep generative scaffold decorator for de-novo drug design. Journal of cheminformaties, 12:1-18, 2020.
Viraj Bagal, Rishal Aggarwal, PK Vinod, and U Deva Priyakumar. Liggpt: Molecular generation using a transoormer-decoder model. 2021.
Sanjivanjit Bhal, Karim Kassam, Ian G Peirson, and Greg M Pearl. The rule of five revisited: applying log in place of log p in drug-likeness filters. Molecular pharmaceutics, 4(4):556-560, 2007.
G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012.
Nicola De Cao and Thomas Kipf. MolGAN: An implicit generative model for small molecular graphs. ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018.
Orion Dollar, Nisarg Joshi, David Beck, and Jim Pfaendtner. Attention-based generative models for de novo molecular design. Chemical Science, 2021.
Chris Donahue, Mina Lee, and Percy Liang. Enabling language models to fill in the blanks. In ACL, 2020.
David Duvenaud, Dougal Maclaurin, Jorge Aguilera-lparaguirre, Rafael Gómez-Bombarelli, Tim- othy Hirzel, Alán Aspuru- Aspuru-Guzik, and Ryan Adams. Convolutional networks on graphs for learning molecular fingerprints. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume NIPS' 15, pp. 2224 2224-2232, Cambridge, MA. USA, 2015. MIT Press.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Debsindhu Bhowmik, and Burkhard Rost. Prot- trans: Towards cracking the language of life's code through self supervied deep learnin and high performance computing. bioRxiv, 2020. doi:
URL https: 07. 12. 199554
// 7/www. www..biorxe.. www.biiiiiiv..ooo org/ /ttp://n///alll//22/////////////
Henri A Favre and Warren H Powell. Nomenclature of organic chemistry: IUPAC recommendations and preferred names 20/3. Royal Society of Chemistry, 2013.
Jessica Ficler and Y. Goldberg. Controlling linguistic style aspects in neural language generation. ArXiv, abs/1707.00233, 2017.
Arup K Ghose, Vellarkad N Viswanadhan, and John Wendoloski. A knowledge-based approach in designing combinatorial or medicinal chemistry libraries for drug discovery. 1. a qualitative and quantitative characterization of known drug databases. Journal of combinatorial chemistry. 1(1): 55-68, 1999.
Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Louis-Philippe Morency, and Stefan Scherer. Affect-Im: A neural language model for customizable affective text generation. arXiv preprint arXiv: 06851, 2017.
10

---

Under review as a conference paper at ICLR 2022
Daria Grechishnikova. Transformer neural network for protein-specific de novo drug generation as a machine translation problem. Scientific reports, 11(1):1-13, 2021.
Francesca Grisoni, Michael Moret, Robin Lingwood, and Gisbert Schneider. Bidirectional molecule generation with recurrent neural networks. Journal of chemical information and modeling, 60(3): 1175-1183, 2020.
2020.
Gabriel Lima Guimaraes, Benjamin Sanchez-Lenggelng Carlos Outeiral, Pedro Luis Cunha Farias, and Alán Espuru-Guzios Objective-reinfores generative adversarial networks (organ) for se- quence generation models. arXiv preprint arXiv: 10843, 2017.
Rafael GómezzBombarelli, Jennifer N. Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sänchez-Lengelingg Dennis Sheberla, Jorge Aguilera-Iparraguirre. Timothy D. Hirzel, Ryan P. Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven con- tinuous representation of molecules. ACS Central Science, 4(2):268 276, 2018. doi: 10. 1021/acscentsei..b00000 7b005722. URL PMID: 29532027.
wtt.:////......./...o.. 1ww.///sceeesscococooc 7b00572]
Jennifer Handsel, Brian Matthews, Nicola Knight, and Simon Coles. Translating the molecules: adapting neural machine translation to predict iupac names from a chemical identifier. 2021.
Jiazhen He, Felix Mattsson, Mareus Forsberg, Esben Jannik Bjerrum, Ola Engkvist, Christian Tyr- chan, Werngard Czechtizky, et al. Transformer neural network for structure constrained molecular optimization. 2021a.
Jiazhen He, Huifang You. Emil Sandström, Eva Nittinger, Esben Jannik Bjerrum, Christian Tyrchan, Werngard Czechtizky, and Ola Engkvist. Molecular optimization by capturing chemist's intuition using deep neural networks. Journal of cheminformatics, 13(1):1-17, 2021b.
Stephen A Hitchcock and Lewis D Pennington. Structure- brain exposure relationships. Journal of medicinal chemistry, 49((6))7559-775555 2006.
James P Hughes, Stephen Rees, S Barrett Kalindjian, and Karen L Philpott. Principles of early drug discovery. British journal of pharmacology,
2011.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junetion tree variational autoencoder for molecular graph generation. In International Conference on Machine Learning, pp. 2323-2332. PMLR, 2018.
Wengong Jin, Kevin Yang. Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to- graph translation for molecule optimization. In International Conference on Learning Represen- tations, 2019. URL http:////.pppaaeeeeeeeeeee/eeaaa
Wengong Jin, Dr. Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th Inter- national Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Re- search, pp. 4839-4848, PMLR, 13-18 Jul 2020. URL www.tttp://proceeeingge www.mrrrrss........ v119//1n20a. html
Hiroshi Kajino. Molecular hypergraph grammar with its application to molecular optimization. In International Conference Machine Learning, pp. 3183-3191. PMLR, 2019.
Pavel Karpov, Guillaume Godin, and Igor V Tetko. A transformer model for retrosynthesis. In International Conference Artificial Neural Networks, pp. 817-830. Springer, 2019.
N. Keskar, Bryan McCann, L. Varshney, Caiming Xiong, and R. Socher. Ctrl: A conditional trans- former language model for controllable generation. ArXiv,
2019.
Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. Pubchem substance and compound databases. Nucleic acidy research, 44(D1):D1202-D121) 2016.

---

Under review as a conference paper at ICLR 2022
Panagiotis-Chrin Christos Kotsias, Josep Arús-Pous, Hongming Chen, Ola Engkvist, Christian Tyrchan, and Esben Jannik Bjerrum. Direct steering of de novo molecular generation with descriptor con- ditional recurrent neural networks. Nature Machine Intelligence, 2(5):254- 2020.
Lev Krasnov, Ivan Khokhlov, Maxim Fedorov, and Sergey Sosnin. Stroczlupaa-traanffomer--sss artificial neural network for the conversion between chemical notations. 2021.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP (Demonstration), 2018
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.097000 2019.
Maxime Langevin, Hervé Minoux, Maximilien Levesque, and Mare Bianciotto. Scaffold- constrained molecular generation. Journal of Chemical Information and Modeling, 2020.
Yibo Li, Jianxing Hu, Yanxing Wang, Jielong Zhou, Liangren Zhang, and Zhenming Liu. Deep- scaffold: A comprehensive tool for scaffold-based de novo drug discovery using deep learning. Journal of chemical information and modeling. 60(1):77-91, 2019.
Jaechang Lim, Seongok Ryu, Jin Woo Kim, and Woo Youn Kim. Molecular generative model based on conditional variational autoencoder for de novo molecular design. Journal of cheminformatiis, 10(1):1-9, 2018.
Jacchang Lim, Sang- Sang-Yeon Hwang. Scokhyun Moon, Seungsu Kim, and Woo Youn Kim. Scaffold- based molecular design with a graph generative model. Chemical Science, 11(4):1153-1166 2020.
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander L. Gaunt. Constrained graph varia- tional autoencoders for molecule design. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 7806-7815, Red Hook, NY, USA, 2018, Curran Associates Inc.
Daniel M Lowe, Peter T Corbett, Peter Murray-Ruse and Robert C Glen. Chemical name to struc- ture: Opsin, an open source solution, 2011.
Eukasz Maziarka, Agnieszka Pocha, Jan Kaczmarczyk, Krzysztof Rataj, Tomasz Danel, and Michal Warchot. Mol-cyclegan: generative model for molecular optimization. Journal of Cheminfor- matics, 12(1):1-18, 2020.
Krzysztof Maziarz, Henry Jackson-Flux, Pashmina Cameron, Finton Sirockin, Nadine Schneider, Nikolaus Stiefl, and Mare Brockschmidt. Learning to extend molecular scaffolds with structural motifs. arXiv preprint arXiv:2 103.022000.03 2021.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv: 1301.3781, 2013.
Tong Niu and Mohit Bansal. Polite dialogue generation without parallel data. Transactions of the Association for Computational Linguistics, 6:373-389 2018.
Mareus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of cheminformatics, 9(1):1-14, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. arXiv preprint arXiv:1912.0170; 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Seikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830. 2011.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 19/0. 10683, 2019.
12

---

Under review as a conference paper at ICLR 2022
Kohulan Rajan, Achim Zielesny, and Christoph Steinbeck. Stout: Smiles iupac names using neural machine translation. Journal of Cheminformatiis, 13(1):1-14, 2021.
Jean-Louis Reymond, Lars Ruddigkeit, Lorenz Blum, and Ruud van Deursen. The enumeration of chemical space. Wiley Interdisciplinary Reviews: Computational Molecular Science, 2(5): 717-733, 2012.
Yu Rong. Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances Neural Information Processing Systems, 33, 2020.
Benjamin Sanchez-Lengeling. Carlos Outeiral, Gabriel L. Guimaraes, and Alan Hapuru-Guzik Opti- mizing distributions over molecular space. an objective-reinforeed generative adversarial network for inverse-design chemistry (organic).
Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher Hunter, Costas Bekas, and Alpha A Lee. Molecular transformer: a model for uncertainty-calibaali chemical reaction prediction. ACS central science, 5(9):1572-1583, 2019.
Chence Shi*, Minkai Xu*, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: flow-based autoregressive model for molecular graph generation. In International Confer- ence on Learning Representations, 2020. URL http:://openreview net/ forum?id
S1eaMkHYPr
Bonggun Shin, Sungsoo Park, Jin Yeong Bak, and Joyce C Ho. Controlled molecule generator for optimizing multiple chemical properties. In Proceedings of the Conference on Health, Inference, and Learning, pp. 146-153, 2021.
Teague Sterling and John J Irwin. Zine 15 15-li discovery for everyone. Journal of chemical information and modeling, 55(11) (((1)):324--337, 2015.
Daniel F Veber, Stephen R Johnson, Hung- Yuan Cheng, Brian R Smith, Keith W Ward, and Ken- neth Kopple. Molecular properties that influence the oral bioavailability of drug candidates. Journal of medicinal chemistry, 45(12):2115-262, 2002.
Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. Smiles-bert: large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informaties, pp. 429-436, 2019.
David Weininger. Smiles, chemical language and information system. 1. introduction method- ology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: satte-of-the-att natural language processing. arXiv preprint 1rXiv:1910.03771, 2019.
Kevin Yang. Wengong Jin, Kyle Swanson, Regina Barzilay, and Tommi Jaakkola. Improving molec- ular design by stochastic iterative target augmentation. In International Conference Machine Learning, pp. 10716-10726. PMLR, 2020.
Jiaxuan You, Bowen Liu, Rex Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 6412-6422, Red Hook, NY, USA, 2018. Curran Associates Inc.
13

---

Under review as a conference paper at ICLR 2022
Method
Base Repr.
Model
T7
UD
CO
CSTS
IUPAC
T5
He al.
2021b
SMILES
Seq2Seq/Transfomn
Langevin et al
2020
SMILES
Any
X
Arus-Pous et al.
2020
SMILES
LSTM
X
Jin et al.
2019
Graph/Motifs
JT-VAE+GAN
X
X
Li ct al.
2019
Graph/Atoms
GNN+VAE
2
X
et
22020
Graph/Atoms
VAE+GNN
MaziarZ et al.
202
Graph/Motifs
VAE+GNN
A'
Bagal et
202
SMILES
GPT
Shin et al.
202T
SMILES
Transformer+LSMM
X
He ct al.
2021a
SMILES
Transformer
Koisıas et a.
2020
SMILES
cRNN
X
et al.
2018
SMILES
VAE+ConNNet+GRU
X
Lim ct
20188
SMILES
cVAE
X
Dollar et al.
2021
SMILES
Liu
(0018)
Graph/Atoms
VAE+GNN
You ct al.
2018
Graph/Atome
GCN+GAN+RL
N
Jin
20118
Graph/Motif's
JT-VAE
Maziarka ct al.
2020
Graph/Motifss
CycleGAN+JTe
N
Shi et
2020
Graph/Atoms
FlOw+RL
X
Olivecrona et al.
2017
SMILES
RNN+RL
26
Table 2: Comparison of C5T5 to prior methods
"Whether the model makes targeted modifications i.e. whether the specify which part of the molecule the model should modify.
"Whether not the model can train without using paired molecular data.
"Whether not the model can switch objectives without re-training re-optimizing.
"The authors propose filtering results hased on where user wants to add side chain, the method itself does not target specific attachment points.
A QUALITATIVE COMPARISON To PRIOR METHODS
Table 2 shows how CST5 and prior methods for molecular optimization differ along several axes.
B ADDITIONAL EXPERIMENTS
B. NOVELTY, VALIDITY, & BEST-IN-
CEST-IN-DTTASET DATASET
Section 4.1 shows that C5T5 successfully modifies property values. Here, for molecules generated to optimize logP, we show the novelty and validity of the generated molecules, along with compar- ison to baseline of the best eligible compound in PubChem. and how new molecules were generated, eligible compounds are those that could be generated by masking any consecutive span of at most 5 IUPAC name tokens and replacing the masked tokens with any number of replacement tokens Results are shown in Tables and 4 Percent validity is the fraction of generated molecules that T5 generated with valid sentinel tokens that were consid- ered chemically valid by the ChemAxon logP caleulator. Percent novelty the fraction of distinct generated molecules that do not appear in PubChem (excluding when CST5 re-generated the source molecule). As shown, despite the comparative difficulty of learning IUPAC name syntax compared to SMILES syntax, C5T5 consistently finds novel and valid molecules that significantly outperform the best-in-dataset baseline.
match how the model was trained
'For computational efficiency, we filter out molecules that differ in length by more than 15 tokens, or that have more than 15 non-overlapping tokens in their hag of tokens, before checking whether they could indeed he generated by masking some length-5 sequence of tokens.
14

---

Under review as a conference paper at ICLR 2022
Table For each source molecule, show the percent of generated molecules that novel (not in PubChem, including invalid names) and valid (can be parsed by ChemAxon), the number of gen- erated molecules, the min/max logP of any generated molecule, the number of eligible compounds in PubChem and the min/max logP of all molecules in PubChem that could be generated by masking up to consecutive tokens. IUPAC names of source molecules are listed Table 4
Src.
gen.
% novel
% valid
max gen.
min gen.
# elig.
max PC
min PC
95.1%
81.7%
14.22
-5.04
26
10.81
-2.7
2
133
93.2%
91.7%
9.41
-3.82
28
1.92
-1.66
3
217
98.6%
91.7%
9.15
-2.17
4
3,01
1.66
4
140
100.0%
94.3%
10.21
1.15
3
8.18
3.78
5
128
92.2%
88.3%
6.91
0.86
19
4.24
2.35
6
160
100.0%
75.6%
8.16
0.47
2.56
1.6
7
159
99.4%
86.2%
9.05
-2.22
8
3,91
-133
8
137
99.3%
84.7%
14.74
-2.97
1.08
1.08
9
122
95.1%
81.1%
8.57
-4,89
34
4.75
-0.07
10
112
92.9%
88.4%
8.44
-2.6
112
5.03
-2.3
11
127
97.6%
81.1%
9.52
0.11
9
5.13
2.28
12
114
90.4%
85.1%
7.71
-3.32
515
6.13
-2.55
13
115
94.8%
90.4%
8.12
-1.1
36
4.05
0.3
14
149
97.3%
85.2%
14.09
-4.3
7
2.1
0.44
15
135
100.0%
83.0%
7.52
-0.7
2
5.08
4.12
16
214
100.0%
97.7%
6,86
-1.83
3
1.91
-0.05
17
156
99.4%
92.3%
10.58
-288
2
0.76
0.76
18
231
98.7%
83.1%
9.79
-1.35
6
5.6
3.46
19
148
93.2%
82.4%
9.98
-0.19
63
7.01
0.76
20
143
96.5%
72.7%
14.12
0.33
14
4.61
1.31
21
232
99.1%
85.3%
9.9
-1.6
3.15
1.14
22
150
96.0%
92.7%
7.96
-0.74
22
4.95
2.71
23
127
94.5%
87.4%
7.54
-3.1
4.67
0.37
24
160
99.4%
82.5%
7.54
-1.53
2
1.28
1.28
25
274
100.0%
95.6%
10.82
1.12
2
6.12
5.82
B.2
EDITED MOLECULE PAIRS IN ZINC
Methods like HierG2G (Jin al. 2020) train model to translate between "unoptimized" and "op- timized" molecules, where the training set consists of pairs of similar molecules in an underlying dataset (e.g. Zinc) that have different property values.
Training model in this way faces the fundamental limitation that there are very few pairs of molecules that are both similar (e.g. in Tanimoto similarity space) and have very different prop- erty values. To quantify this effect, we compute similarity and logP increase between all pairs of molecules the training set of Zine used by Jin et (2020). Results are shown in Figure
C EXPERIMENTAL DETAILS
Dataset Preparation We download PubChem from ftp ncbi momiiinmm.mmm. nih. www.ggovopbbbeee Compound/ /CURRENT-FUl /XLL/ and extracted each molecule's Preferred IUPAC Name and computed octanol- water partition coefficient (logP). There are 109M total compounds in the version of PubChem we downloaded in January 2021. For experiments using logP, we used the XLogP3 val- ues from PubChem, which were computed using OpenEye's software. For logD (pH=7), refractivity, and polar surface area, we computed values using ChemAxon' 's calculator with default parameters. Separately for each target property. excluded chemicals that had no logP value in PubChem or that were not parseable by ChemAxon's calculator. Of the remaining molecules, we randomly split into training set with 90M compounds and validation set with ~10M-19M compounds.
15

---

Under review as a conference paper at ICLR 2022
Table 4: IUPAC Name lookup table for Table
ID
Source Molecule
3,3-
--hooc/h2222224--ooooooyyyyyyyen Jeth
5 http://www......aa..o....../aaaa
10 [[leecaappprdiiiiyyyyyyyyllaaaao acid
11
[6 hwtp://wwoyy444c/lloooooo//////p
14
-iimmii
15 ShtteppiccooyyySSddooovoeeeeeeee
acid
3-phenylpropannoic acid
-aw.oiidddi-----aroaoaaiiii 1-carboxamide
18
oalp:/omeccttlpiiimmimmmeaaaaea 44-diiiine
20 4ealoos//2aaasssslllaaoooa/adaar
21
22
23
2- 2i1-2HHicclllooeeeeyyyyeyyyyyyee acid
24
logP diff
109
logP diff
logP diff > 2
107
logP diff >
Pairs
logP diff > 4
10
103
10a
0.0
0.2
0.4
0.6
0.8
1.0
Tanimoto Similarity Cutoff
Figure 7: Number of molecular pairs in Zine that pass Tanimoto similarity threshold (x-axis) and logP-difie LogP-difference threshold (different curves). Even for moderate similarity and logP thresholds, there quickly become vanishingly few usable molecular pairs. There are 24 x 109 total unique molecular pairs.
Tokenization We use HuggingFace's T5Tokenizer, which is based on the SentencePiece algorithm Kudo Richardson (2018). Because our goal is to have tokens that domain experts are familiar with, do not train SentencePiece substrings of moiety names. Instead, we manually specify the tokens to be all the keywords in the Opsin IUPAC name parsing library Lowe al. (201T). To these keywords, we add locants 1-100,
the IUPAC names, since doing so learns both combinations of and
16

---

Under review as a conference paper at ICLR 2022
stereochemistry markers (R, S. E. Z...), and a few miscellancous tokens e.g. handle spiro centers. This leads to a total of 1274 tokens. After tokenization, we truncate all names to at most 128 tokens.
We choose which property value token to use based the cutoffs specified in Table 5
Table 5: Numerical ranges across properties for each property value token. CutofI's for logP, PSA, and refractivity were chosen as common thresholds for druglikeness screening following (Ghose et al., 1999 Veber al. 2002 Hitchcock Pennington 2006). We use the same cutoff for logD as logP.
Property
<low>
<med>
<high>
Octanol-water partition coeff. (logP)
(-00-0,,4 -0.4)
(-0.4,5.6)
(5.6, x))
Octanol-water distribution coeff. (logD)
-0.4)
(-0.4.5.6)
(5.6, 20)
Polar surface area (PSA)
(0, 90)
(90, 140)
(140, 00)
Refractivity
(0, 40)
(40,
(130, 00)
Training We train 15-large model (~700M params) available from HuggingFace that was pre- trained on English text. We keep the first 1274 embeddings from the pretrained embedding table, along with the pretrained embeddings for the 100 sentinel tokens. When training, we mask 15% of the tokens in each input in spans of mean length 3 tokens, with minimum span length of 1. We use a linear warmup of the learning rate for 10,000 steps followed by a 1/ll decay. All models were trained using the AdamW W optimizer. We train the logP model for 2.5M iterations using max learn- ing rate 10-3, We train the refractivity/log/PPlaaa SA models using a maximum learning rate of 2 x 10 1/1 4/2 10- starting from the logP model after 1M/2.5M/2.5M iterations. (We trained using the latest model available when we started a run.) All models were trained on 8 NVIDIA A100s with batch size of 16 per GPU.
Generation We generate novel molecules greedily from the output of T5's decoder. We discard any generations where the sentinel tokens do not line up, and we further discard any molecules that cannot be parsed by ChemAxon's 's calculators. We also discard instances where C5T5 regenerates the base molecule. Doing so inflates our novelty score, but because we choose random masks (including, comma between locants), in many cases there few other ways to fill in the masked section besides regenerating the original tokens.
e.g. masking
Comparison to HierG2G eters from (Jin al. 2020 We generated all pairs of Zine training molecules with Tanimoto similarity of at least 0.4 and an increase in logP of at least 2.5, yielding ~200k pairs. We trained HierG2G for 32k iterations; further training did not improve results. To choose the similarity and logp thresholds, we attempted to use the average similarity and logp increase achieved by C5TS (similarity of 0.55, logP increase of 2.4). However, there are only 7000 pairs of molecules Zinc meeting these constraints, so we relaxed the cutoffs to achieve a reasonably sized training set. See Appendix B.2 for further details.
We trained HierG2G on Zine using the authors' code and hyperparam-
We were not able to fully reproduce the results in (Jin et al. 2020), either by running the authors' code as-is or by modifying the hyperparameters match the description the paper. Specifically, for penalized logP with a similarity cutoff of 0.4, get average improvement of ~ 2.8, whereas the results in the paper show an improvement reproduce the paper's results, but we do not believe the conclusions of our comparison will change qualitatively even with updated results from HierG2G.
3.98. We will update Figure 6 once we are able to
Cloud Computing Cost We train T5-Large models Raffel et al. (2019) on PubChem for each investigated property value using local computing resources and AWS 24x.2lre instances in the us-east-1 region. The logP model was trained for 2.5M iterations over 8 days; the logD and refractivity models for 350k iterations over 1 day each (initialized from the logP model after 2.5M iterations); and the Polar SA model for 1.6M iterations over 5 days (initialized from the logP model after 1M iterations). The SMILES logP ablation model was trained for 1M iterations over 12 days.
17

---

Under review as a conference paper at ICLR 2022
Total equivalent on-demand AWS cost for the models presented here is $19,000, and total carbon dioxide-equivalent emissions is ~ 570 kg Lacoste et al. (2019).
Software We use HuggingFace 4.2.2 (Apache 2.0 license) (Wolf al. 2019), PyTorch 1.8.0 (BSD) (Paszke et al. 2019), ChemAxon 20.17.0 (Academic License), scikit-learn 0.24.2 (New BSD) (Pedregosa al. 201 and python 3.9 (PSFL).
18


Reviews:
[{"review_id": "upUbumEZD4E", "reviewer": "Reviewer_pKJo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Comments**\n\nOverall the paper presents an interesting treatment for the molecule property optimization (e.g., lead optimization, etc). The approach is simple but seems to be effective. The IUPAC naming alternative to SMILES can be of its own contribution to a broader context of chemical engineering. \n\nHowever I have several concerns regarding the current draft.\n\n1. The technique contribution is not very significant. The IUPAC naming is interesting but itself is not solid enough to be considered as technical innovation. The masked pretraining has been seen in other context like NLP, program language, etc. Nevertheless, I think it is ok for an application paper if the quality of experiments is high and the results are solid. So I would focus more on the experimental part. \n\n2. The only baseline studied is Hierarchical G2G (Jin et.al) in Figure 6. Given that there has been many works in the space of molecule optimization, it would be necessary to include more baseline results, especially those with quantitative comparisons. Although some of the baseline methods may not be able to complete all the tasks, it would still be necessary to compare in the situations where baseline methods apply. In this way we can appreciate how significant the results are, especially given that the method proposed in this paper is self-supervised. \n\n3. I would also like to learn more about the limitations of this method. For example, the IUPAC limits the possibility of generating new functional groups. There must be a trade-off between model capacity and the variance of results. \n\n4. The extrapolation is not for free. I’d like to learn more about the failure cases where the model is not able to generate the molecules with desired properties. \n\n5. (Optional) while the model presents an unsupervised way of learning, it would be interesting to see if it can be further fine-tuned with the paired molecule data for lead optimization. In this way, one can compare the results directly with G2G in their benchmarks.\n", "summary_of_the_paper": "**Summary**\n\nThis paper proposed a way to modify the molecule based on language pretraining techniques. The sequence representation of molecules is based on IUPAC names, which can be more semantically meaningful and much easier to model than the SMILES or graph based molecule representation. The pretraining is done via a conditional text generation model where the model predicts the fragment names based on the remainder of the molecule and corresponding property values. The application on downstream molecule property optimization tasks show that the proposed approach is effective at obtaining high quality molecules.\n", "summary_of_the_review": "**Review summary**\n\nThis is an interesting application paper in the domain of molecule optimization with some minor technical contributions. The preliminary results are interesting, but it would be more solid with more quantitative comparison with existing methods on existing benchmarks. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V9hRg3ZS_Pu", "reviewer": "Reviewer_gkwv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrenghts\nThe idea of using the IUPAC nomenclature is indeed innovative and has not received attention in the area. The use of local modification of tokenized IUPAC names is exciting and seems to work for scaffold derivatization better than other methods\n\nThe T5 model is a powerful addition to language-based addition to molecule generation\n\nWeaknesses\nAt a higher level, this paper proposes a new generative approach but does not evaluate it upon the mutually agreed benchmarks in the field. Guacamol or Moses have lots of meaningful tasks and splits, including some scaffold-based ones. It does not seem appropriate to switch architectures and tasks and then just compare with the one baseline. If the authors believe those benchmarks are relevant, then it is more appropriate to make a case why explicitly (and probably use them anyway)\n\nFurthermore, making local modifications to starting scaffolds has a long history in cheminformatics. Approaches like genetic algorithms, including the graph-based genetic algorithm, can do a great job at local optimization challenges like this https://pubs.rsc.org/en/content/articlelanding/2019/sc/c8sc05372c And since all the properties come from a surrogate model oracle, it does not matter how many train pairs or in-time calls need to be made. LogP (or logD, i think for all the molecules shown they are __extremely__ correlated) is an additive property that can be improved by just appending more carbons to an alkyl chain, so it is a poor performance metric. One could very easily make a genetic-algorithm version of the token substitution approach as a baseline.\n\nI have a number of smaller comments below\n- The advantages of using IUPAC names are not clear at all. \n\n\"a small change in a molecule can lead to a large change in the SMILES string\" This is the case too for IUPAC names - taking one carbon away and closing a ring in the suggested structure in Figure 2 completely changes the IUPAC name\n\n\"flattening the graph into a list of atoms artificially creates variable- and long-range dependencies between bonded atoms\" It does in the IUPAC naming too, since there's a canonical atom ordering that is needed to refer to for writing the molecular name\n\n\"It is difficult to reason about common substructures, because the same structure can be\nrepresented in many different ways depending on how the graph was flattened\"\n\nSame in IUPAC nomenclature - the ester functionality is described through a completely different naming in the example below\n\n\nOC(C1=C(OC(C)=O)C=CC=C1)=O\n2-acetoxybenzoic acid\n\nO=C(O1)C2=C(OC1=O)C=CC=C2\n4H-benzo[d][1,3]dioxine-2,4-dione\n\n\n\"graphs do a poor job encoding symmetry, long-range interactions between atoms that are many bonds apart but nearby in 3D space, and long-range interactions that arise from conjugated system\" Again, the IUPAC naming convention has many of these flaws too. There's no 3D whatsoever in IUPAC either.\n\n\"locants (e.g. “1,” “2,” “N”), which indicate connectivity,\" These locants are also based on arbitrary ordering, just like SMILES\n\nAgain just like SMILES, while there's only one canonical choice of IUPAC nomenclature based on prioritization rules, it is entirely possible to write the IUPAC name for a molecule correctly but based on another prioritization rule. As a matter of fact this paper's approach to tokenization likely does that, depending on the nature of the transformations it perform on the starting name.\n\nSection 3.1 addresses few of the issues raised about SMILES / graphs. (for instance, it does address the multiplier, which is indeed very powerful in theory, but does not seem to be used by the generator in practice. \"pentatrytilbenzene\" would break the bank in logP)\n\n\"ignoring the embedding for “nitroso”\" I would make this part of the sentence, rather than a footnote\n\n\"We discard any generations where the sentinel tokens do not line up, and we further discard any molecules that cannot be parsed by ChemAxon’s calculators. We also discard instances where C5T5 regenerates the base molecule\" I think I understand the logic about needing to seed with something, but discarding the same from the statistics seems antithetic to the nature of a novelty metric. What is the fraction of molecules being discarded at every filter?\n\n\"Although this is not a preferred IUPAC name, it is still unambiguous, and therefore valid and parseable\" This is exactly the point i was making earlier. It seems the approach struggles with synonyms just like SMILES.\n\n\"C5T5 generates molecules that are more synthetically accessible and similar to base molecules and that have a wider range of logP increases than HierG2G.\" I am not sure I'm looking at the plots right. How was synthetic accessibility computed? Is 0 high or low accessibility? The scale in RDKIT has 1 for easy molecule and 10 for difficult, So C5T5 is producing more difficult molecules. Same for the logP increase, i see HierG2G producing a mean of 4 or so, and C5T5 is a little over 1? \n\n\n\n\"Methods like HierG2G that train on pairs of similar molecules are fundamentally\nlimited by a paucity of experimental molecu pairs that have high similarity and high property\nvalue improvement\" This seems misleading since all these methods are using surrogate models to make their labels, there's essentially N squared pairs for the training library. \n\n\n\nQuestions\n\n\"The nearest neighbor of “diphosphate” - “disulfate” + “sulfate” is “phosphate.\" I don't follow. Are diphosphate and disulphate tokens? Does the tokenization procedure not split out the di? Or the \"-ite\" termination? Those are literally rule-based tokens in IUPAC nomenclature.", "summary_of_the_paper": "This paper utilizes a transformer architecture based on T5 to generate tokenized IUPAC names of molecules. This allows sampling molecules that are local modifications of starting ones, or are local in IUPAC name but might be chemically large while respecting an initial scaffold.\n", "summary_of_the_review": "I think the idea of using IUPAC names is intriguing, but neither the theoretical arguments not the empirical results (due to lack of benchmarks) are convincing. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DE3sqv1oTJJ", "reviewer": "Reviewer_MJUv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The summary of the paper is concluded in the above contents. Here are the strengths and weaknesses of the paper from my view. \n\nStrengths:\n1. The first attractive point is the IUPAC names representation method. As mentioned in the paper, this method is a standardized molecular representation that encodes the rich structural information but that has been largely ignored. The common approach of the representation for the molecule is the SMILES or graph. As introduced, this representation can handle more semantic information, and also the interpretability is better. \n2. The method is straightforward and simple, which uses a pre-training method T5 to model a conditional language model, where the condition is the property value. As a result, C5T5 does not require paired dataset for the molecule property improvement. \n3. The results on properties show the desired optimization is satisfied, and the different studies give a more clear understanding.\n\nweaknesses:\nThough I personally like the simple idea, there are still concerns and questions from my point,\n1. The first concern is about the specific property value distribution. As shown in Table 5, each property only splits into three buckets. Then each model can only have three different types of property, which makes the model to be limited. For example, it can not clearly generate a better molecule if the initial molecule with logP is larger than 5.6. \n2. The second concern is about the IUPAC names. Though I feel happy to see a new representation method that seems to be better than other methods, the comparison is not clear and there seems no strong evidence or numbers can be shown in the paper. In this way, this can not clearly convince the superiority of IUPAC. However, I am pretty interested to see the goodness of IUPAC.\n3. One specific question is about the pre-trained T5 model. The authors mentioned that they do not use sentencepiece, and only keep the IUPAC names as the vocabulary, which is different from the pre-trained embedding of T5 model. It is said that the first 1274 embeddings of English pre-trained embedding are kept. I am not sure about this process, what is the difference between this method and directly replacing the vocabulary with the 1274 new randomly initialized embeddings? Since the vocabulary is different at all. \n4. It is required to make a result comparison to see the advantage of this method. Otherwise, it is hard to evaluate. \n5. Other concerns have been discussed by the authors, for example, the training cost is high; it is not always possible to generate valid and satisfying molecules since there are no constraints in the modeling. \n\n", "summary_of_the_paper": "This paper proposes a method C5T5, a self-supervised pre-training method based on the T5 pre-trained model, which is able to make zero-shot select-and-replace edits to satisfy specific property values. The specific difference of this paper is the IUPAC names (a standardized molecular representation), and the method is totally self-supervised. The experiments are evaluated on octanol-water partition, distribution coefficients, polar surface area, and refractivity, above four properties. Experiments show that the designed methods are able to achieve the optimization objective. ", "summary_of_the_review": "I personally feel good about this work, but unclear parts are main concerns and there are no comparison with other works. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes to use a transformer model and language model inspired self-supervised training techniques to generate local modifications of organic molecules. The use of IUPAC names coupled with language inspired pre-training is indeed an interesting idea worthy of exploration. The paper has a lot of promises in this regard but needs more work to deliver it through the finish line. In the rebuttal, the authors have provided strong arguments toward the advantages of using IUPAC representation. While these arguments make sense, they are more or less conceptual and better and more clear empirical evidences are required to back them up.
Decision: Reject