C5T5: Controllable Generation of Organic Molecules with Transformers

1 INTRODUCTION . Organic molecules are used in countless applications across human society : as medicines , industrial chemicals , fuels , pesticides , plastics , television screens , solar cells , and many others . Traditionally , new molecules are designed for particular tasks by hand , but the space of all possible molecules is so vast ( e.g . the total number of drug-like molecules may be as high as 1060 ) that most useful materials are probably still undiscovered ( Reymond et al. , 2012 ) . To automate materials discovery , domain experts have turned to high-throughput screening , in which a large library of potentially useful molecules is generated heuristically , and the most promising molecules are chosen for further study using computational models that estimate how effective each substance will be for the target application ( Hughes et al. , 2011 ) . Unfortunately , even high-throughput methods can still only screen a tiny fraction of all possible molecules . Generating molecules directly with machine learning addresses this limitation , but de novo generation can be of limited use in domains like drug discovery , where experts ’ intuitions about structureactivity relationships and external factors like patentability are important to consider in the design process . These constraints can often be expressed by providing known portions of the molecular structure ; for example a domain expert may be interested in a particular scaffold because it has favorable intellectual property attributes , or certain parts of a drug may be needed for the desired biological activity , while other parts can be modified to increase bioavailability . To address this real-world setting , we consider the problem of learning to make localized modifications to a molecule that change its physical properties in a desired way . We propose C5T5 : Controllable Characteristic-Conditioned Chemical Changer with T5 ( Raffel et al. , 2019 ) , a novel method for generative modeling of organic molecules that gives domain experts fine-grained control over the molecular optimization process while also providing more understandable predictions than prior methods ( Figure 1 ) . Our two key contributions are 1 ) recasting molecular modeling as language modeling on the semantically rich IUPAC name base representation , and 2 ) the development of a novel conditional language modeling strategy using transformers that supports targeted modifications to existing molecules . IUPAC Names . The IUPAC naming system is a systematic way of naming organic molecules based on functional groups and moieties , or commonly occurring clusters of connected atoms that have known chemical behaviors . Organic chemists have discovered countless chemical reactions that operate on functional groups , and they use these reactions to develop synthesis routes for novel molecules . Despite this , existing generative methods for organic molecules have ignored IUPAC names as a representation , instead opting for atom-based representations like SMILES ( Weininger , 1988 ) and molecular graphs ( Duvenaud et al. , 2015 ) . See Figure 2 for a comparison of these representations . We argue for several advantages of IUPAC names in Section 3.1 . To the best of our knowledge we are the first to use IUPAC names as a base representation for molecular modeling . Self-Supervised Objective for Zero-Shot Editing . To enable targeted modifications of molecules without predefined edit pairs , we train transformers with a conditional variant of a self-supervised infilling task , where the model must replace masked-out tokens in the IUPAC name . As described in Section 3.2 , we condition the model by prepending IUPAC names with discretized molecular property values ; the model then learns the conditional relationships between the property value and molecular structure . To the best of our knowledge , C5T5 is the first method to use conditional infilling for select-and-replace editing ; we anticipate this method could be broadly applied in other controlled generation contexts , such as modeling affect , politeness , or topic of natural language ( Ghosh et al. , 2017 ; Ficler & Goldberg , 2017 ; Niu & Bansal , 2018 ; Keskar et al. , 2019 ) . As we show in Section 4 , C5T5 is able to make interpretable targeted modifications to molecules that lead to desired changes across several physical properties important in drug design . 2 RELATED WORK . Modeling . A number of machine learning methods have been developed for the task of designing organic molecules , but most do not allow a user to make targeted modifications to a molecule . Some methods , like generative adversarial networks and unconditional sequence models , provide no control over a generated molecule ’ s structure ( Grisoni et al. , 2020 ; Guimaraes et al. , 2017 ; SanchezLengeling et al . ; De Cao & Kipf , 2018 ; Kajino , 2019 ) , and are therefore more useful for generating candidate libraries than optimizing a particular molecule . Other methods , like variational autoencoders or sequence models that are conditioned on a base molecule , allow specifying that generated molecules should be similar to a starting molecule in some learned space , but there is no way to specifically target a certain part of the molecule to modify ( He et al. , 2021b ; Jin et al. , 2019 ; Shin et al. , 2021 ; Yang et al. , 2020 ; Kotsias et al. , 2020 ; Gómez-Bombarelli et al. , 2018 ; Lim et al. , 2018 ; Dollar et al. , 2021 ; Liu et al. , 2018 ; Jin et al. , 2018 ; Maziarka et al. , 2020 ; Olivecrona et al. , 2017 ; Bagal et al. , 2021 ; You et al. , 2018 ; Shi * et al. , 2020 ) . Recognizing the importance of leveraging domain experts ’ intuition about structure-activity relationships , several methods , published mostly in chemistry venues , have explored constraining generated molecules to contain a scaffold , or a subgraph of the full molecular graph ( Li et al. , 2019 ; Lim et al. , 2020 ; Maziarz et al. , 2021 ) . However , these methods append to scaffolds arbitrarily instead of allowing domain experts to specify which part of the molecule they would like to modify or append to , limiting their utility for human-in-theloop molecular optimization . A few methods have explored allowing targeted modifications , where a domain expert can mask out a portion of a starting molecule and ask the model to replace the mask with a novel side chain ( Arús-Pous et al. , 2020 ; Langevin et al. , 2020 ; He et al. , 2021a ) . These methods are limited because they only support masking parts of the molecule that can be truncated by cutting a single bond , and because they require a dataset of paired molecules ( scaffolds & decorators ) that must be constructed using hand-crafted rules . In contrast , C5T5 learns in an entirely unsupervised fashion and therefore requires no paired data ; the only limit to what can be masked is what can be represented using IUPAC tokens . Representation . Existing methods all use SMILES ( or a derivative representation ) or graphs to represent molecules . There are a number of drawbacks to using the SMILES representation : a small change in a molecule can lead to a large change in the SMILES string ( Jin et al. , 2018 ) ; flattening the graph into a list of atoms artificially creates variable- and long-range dependencies between bonded atoms ; and it is difficult to reason about common substructures , because the same structure can be represented in many different ways depending on how the graph was flattened . And although graphs seem like a natural representation for molecules , graphs do a poor job encoding symmetry , longrange interactions between atoms that are many bonds apart but nearby in 3D space , and long-range interactions that arise from conjugated systems ( Duvenaud et al. , 2015 ) . C5T5 operates instead of IUPAC names , which we argue in Section 3.1 is a more suitable representation for molecular optimization because tokens have much more semantic meaning . See Appendix A for more details on how C5T5 relates to prior work . Transformers for Molecular Modeling Outside of molecular optimization , transformers have found a number of applications in molecular modeling tasks , including property prediction ( Wang et al. , 2019 ; Rong et al. , 2020 ) , chemical reaction prediction ( Schwaller et al. , 2019 ) , retrosynthesis ( Karpov et al. , 2019 ) and generating proteins ( Elnaggar et al. , 2020 ; Grechishnikova , 2021 ) . A few works have explored using transformers for generative modeling of organic molecules ( He et al. , 2021b ; Shin et al. , 2021 ; Dollar et al. , 2021 ) . Some works have also proposed using transformers for scaffold-conditioned generative modeling ( He et al. , 2021a ; Bagal et al. , 2021 ) . This work extends these efforts by proposing a simple yet effective training and zero-shot adaptation method , and by using IUPAC names instead of SMILES strings . IUPAC Names Although we are unaware of prior work using IUPAC names as a base representation for molecular modeling , several works have explored using machine learning to convert between IUPAC names and other molecular representations ( Rajan et al. , 2021 ; Handsel et al. , 2021 ; Krasnov et al. , 2021 ) . 3 METHOD . Molecular optimization is a difficult problem because it requires modifying a molecule that already satisfies a number of requirements . Modifications need to improve a particular aspect of the molecule without degrading its performance on other metrics , and without making it too difficult to synthesize . We argue that by using IUPAC names ( Section 3.1 ) and by allowing users to target particular parts of a molecule to modify ( Section 3.2 ) , C5T5 has the potential to support humanin-the-loop molecular editing that complements domain experts ’ intuitions about structure-activity relationships and synthetic accessibility . 3.1 IUPAC NAMING . The International Union of Pure and Applied Chemistry ( IUPAC ) publishes a set of rules that allow systematic conversion between a chemical structure and a human-readable name ( Favre & Powell , 2013 ) . For example , 2-chloropentane refers unambiguously to five carbons ( “ pent ” ) connected by single bonds ( “ ane ” ) with a chlorine atom ( “ chloro ” ) bonded to the second carbon from one end ( “ 2- ” ) . IUPAC names are used ubiquitously in scholarly articles , patents , and educational materials . In contrast to other linear molecular representations like SMILES and its derivatives , where single tokens mostly refer to individual atoms and bonds , tokens in IUPAC names generally have a rich semantic meaning . For example , the token “ ic acid ” denotes a carboxylic acid , which is a common functional group that has well-known physical and chemical properties ; there are many known chemical reactions that either start with or produce carboxylic acids . Other tokens denote additional functional groups ( e.g . “ imide , ” “ imine , ” “ al , ” “ one ” ) , locants ( e.g . “ 1 , ” “ 2 , ” “ N ” ) , which indicate connectivity , alkanes ( e.g . “ meth , ” “ eth , ” “ prop ” ) , which denote the lengths of carbon chains , polycyclic rings ( e.g . “ naphthalene , ” “ anthracene ” ) , stereochemistry markers ( “ R , ” “ S ” ) , and multipliers ( e.g . “ di , ” “ tri ” ) , which concisely represent duplicated and symmetric structures . Figure 2 shows the relationships between IUPAC names , graph representations , and SMILES . For molecular optimization , C5T5 supports qualitatively different molecular edits compared to graph- and SMILES-based methods by virtue of its use of IUPAC names : editing a locant token corresponds to moving a functional group along a carbon backbone or changing the connectivity of a fused ring system ; and editing a multiplier token corresponds to creating or eliminating duplicated and symmetric structures . For example , changing “ ethylbenzene “ to “ hexaethylbenzene ” replicates the ethyl structure around the entire benzene ring with a single token edit . These sorts of modifications require much more extensive editing for SMILES- and graph-based methods.1 We argue that IUPAC names are especially attractive for molecular optimization , since the process requires interaction between the algorithm and a domain expert , so interpretability is paramount . Compared to graph- or SMILES-based models , C5T5 makes predictions that can be traced back to moieties and functional groups that domain experts are more likely to understand , trust , and know how to synthesize than either arbitrary collections of atoms and bonds or motifs decided upon by machine learning practitioners . In addition to improved interpretability , we argue that using IUPAC names has advantages purely from the standpoint of modeling data , since moving from SMILES to IUPAC names is akin to moving from a character-based to a word-based sequence model . Modeling at this higher level of abstraction enables the network to direct more of its capacity to structure at the relevant semantic level , instead of relearning lower-level details like the specific atomic composition of functional groups . In this vein , we demonstrate the potential of IUPAC names by learning word2vec representations of IUPAC name tokens ( Mikolov et al. , 2013 ) , drawn from a list of over 100 million names in the PubChem repository ( Kim et al. , 2016 ) and tokenized using a list of tokens in OPSIN—an open-source IUPAC Name parser library ( MIT License ) ( Lowe et al. , 2011 ) . For example , as shown in Figure 2 , the chemical “ 2-acetyloxybenzoic acid ” gets tokenized to [ “ 2 ” , “ - ” , “ acet ” , “ yl ” , “ oxy ” , “ benzo ” , “ ic acid ” ] . As with natural language modeling , we find that the embedding space learned by word2vec encodes the semantic meaning of the tokens , as shown in Figure 3 . Different classes of tokens tend to be clustered together , and similar tokens within clusters are located nearby . For example , aromatic compounds with two rings are clearly separated from those with three , locants are ordered roughly correctly from 1 to 100 , and multiplier tokens are also roughly in order ( zoom not shown ) . Following Mikolov et al . ( 2013 ) , we also find that simple arithmetic operations in the embedding vector space correspond to semantic analogies between tokens . For example , the nearest neighbor of “ phosphonous acid ” - “ nitrous acid ” + “ nitroso ” is the embedding for “ phosphoroso. ” 2 The nearest neighbor of “ diphosphate ” - “ disulfate ” + “ sulfate ” is “ phosphate. ” Likewise for “ selenate ” - “ tellurate ” + “ tellurite ” being closest to “ selenite . ”

Summary: **Summary**  This paper proposed a way to modify the molecule based on language pretraining techniques. The sequence representation of molecules is based on IUPAC names, which can be more semantically meaningful and much easier to model than the SMILES or graph based molecule representation. The pretraining is done via a conditional text generation model where the model predicts the fragment names based on the remainder of the molecule and corresponding property values. The application on downstream molecule property optimization tasks show that the proposed approach is effective at obtaining high quality molecules.