C5T5: Controllable Generation of Organic Molecules with Transformers

1 INTRODUCTION . Organic molecules are used in countless applications across human society : as medicines , industrial chemicals , fuels , pesticides , plastics , television screens , solar cells , and many others . Traditionally , new molecules are designed for particular tasks by hand , but the space of all possible molecules is so vast ( e.g . the total number of drug-like molecules may be as high as 1060 ) that most useful materials are probably still undiscovered ( Reymond et al. , 2012 ) . To automate materials discovery , domain experts have turned to high-throughput screening , in which a large library of potentially useful molecules is generated heuristically , and the most promising molecules are chosen for further study using computational models that estimate how effective each substance will be for the target application ( Hughes et al. , 2011 ) . Unfortunately , even high-throughput methods can still only screen a tiny fraction of all possible molecules . Generating molecules directly with machine learning addresses this limitation , but de novo generation can be of limited use in domains like drug discovery , where experts ’ intuitions about structureactivity relationships and external factors like patentability are important to consider in the design process . These constraints can often be expressed by providing known portions of the molecular structure ; for example a domain expert may be interested in a particular scaffold because it has favorable intellectual property attributes , or certain parts of a drug may be needed for the desired biological activity , while other parts can be modified to increase bioavailability . To address this real-world setting , we consider the problem of learning to make localized modifications to a molecule that change its physical properties in a desired way . We propose C5T5 : Controllable Characteristic-Conditioned Chemical Changer with T5 ( Raffel et al. , 2019 ) , a novel method for generative modeling of organic molecules that gives domain experts fine-grained control over the molecular optimization process while also providing more understandable predictions than prior methods ( Figure 1 ) . Our two key contributions are 1 ) recasting molecular modeling as language modeling on the semantically rich IUPAC name base representation , and 2 ) the development of a novel conditional language modeling strategy using transformers that supports targeted modifications to existing molecules . IUPAC Names . The IUPAC naming system is a systematic way of naming organic molecules based on functional groups and moieties , or commonly occurring clusters of connected atoms that have known chemical behaviors . Organic chemists have discovered countless chemical reactions that operate on functional groups , and they use these reactions to develop synthesis routes for novel molecules . Despite this , existing generative methods for organic molecules have ignored IUPAC names as a representation , instead opting for atom-based representations like SMILES ( Weininger , 1988 ) and molecular graphs ( Duvenaud et al. , 2015 ) . See Figure 2 for a comparison of these representations . We argue for several advantages of IUPAC names in Section 3.1 . To the best of our knowledge we are the first to use IUPAC names as a base representation for molecular modeling . Self-Supervised Objective for Zero-Shot Editing . To enable targeted modifications of molecules without predefined edit pairs , we train transformers with a conditional variant of a self-supervised infilling task , where the model must replace masked-out tokens in the IUPAC name . As described in Section 3.2 , we condition the model by prepending IUPAC names with discretized molecular property values ; the model then learns the conditional relationships between the property value and molecular structure . To the best of our knowledge , C5T5 is the first method to use conditional infilling for select-and-replace editing ; we anticipate this method could be broadly applied in other controlled generation contexts , such as modeling affect , politeness , or topic of natural language ( Ghosh et al. , 2017 ; Ficler & Goldberg , 2017 ; Niu & Bansal , 2018 ; Keskar et al. , 2019 ) . As we show in Section 4 , C5T5 is able to make interpretable targeted modifications to molecules that lead to desired changes across several physical properties important in drug design . 2 RELATED WORK . Modeling . A number of machine learning methods have been developed for the task of designing organic molecules , but most do not allow a user to make targeted modifications to a molecule . Some methods , like generative adversarial networks and unconditional sequence models , provide no control over a generated molecule ’ s structure ( Grisoni et al. , 2020 ; Guimaraes et al. , 2017 ; SanchezLengeling et al . ; De Cao & Kipf , 2018 ; Kajino , 2019 ) , and are therefore more useful for generating candidate libraries than optimizing a particular molecule . Other methods , like variational autoencoders or sequence models that are conditioned on a base molecule , allow specifying that generated molecules should be similar to a starting molecule in some learned space , but there is no way to specifically target a certain part of the molecule to modify ( He et al. , 2021b ; Jin et al. , 2019 ; Shin et al. , 2021 ; Yang et al. , 2020 ; Kotsias et al. , 2020 ; Gómez-Bombarelli et al. , 2018 ; Lim et al. , 2018 ; Dollar et al. , 2021 ; Liu et al. , 2018 ; Jin et al. , 2018 ; Maziarka et al. , 2020 ; Olivecrona et al. , 2017 ; Bagal et al. , 2021 ; You et al. , 2018 ; Shi * et al. , 2020 ) . Recognizing the importance of leveraging domain experts ’ intuition about structure-activity relationships , several methods , published mostly in chemistry venues , have explored constraining generated molecules to contain a scaffold , or a subgraph of the full molecular graph ( Li et al. , 2019 ; Lim et al. , 2020 ; Maziarz et al. , 2021 ) . However , these methods append to scaffolds arbitrarily instead of allowing domain experts to specify which part of the molecule they would like to modify or append to , limiting their utility for human-in-theloop molecular optimization . A few methods have explored allowing targeted modifications , where a domain expert can mask out a portion of a starting molecule and ask the model to replace the mask with a novel side chain ( Arús-Pous et al. , 2020 ; Langevin et al. , 2020 ; He et al. , 2021a ) . These methods are limited because they only support masking parts of the molecule that can be truncated by cutting a single bond , and because they require a dataset of paired molecules ( scaffolds & decorators ) that must be constructed using hand-crafted rules . In contrast , C5T5 learns in an entirely unsupervised fashion and therefore requires no paired data ; the only limit to what can be masked is what can be represented using IUPAC tokens . Representation . Existing methods all use SMILES ( or a derivative representation ) or graphs to represent molecules . There are a number of drawbacks to using the SMILES representation : a small change in a molecule can lead to a large change in the SMILES string ( Jin et al. , 2018 ) ; flattening the graph into a list of atoms artificially creates variable- and long-range dependencies between bonded atoms ; and it is difficult to reason about common substructures , because the same structure can be represented in many different ways depending on how the graph was flattened . And although graphs seem like a natural representation for molecules , graphs do a poor job encoding symmetry , longrange interactions between atoms that are many bonds apart but nearby in 3D space , and long-range interactions that arise from conjugated systems ( Duvenaud et al. , 2015 ) . C5T5 operates instead of IUPAC names , which we argue in Section 3.1 is a more suitable representation for molecular optimization because tokens have much more semantic meaning . See Appendix A for more details on how C5T5 relates to prior work . Transformers for Molecular Modeling Outside of molecular optimization , transformers have found a number of applications in molecular modeling tasks , including property prediction ( Wang et al. , 2019 ; Rong et al. , 2020 ) , chemical reaction prediction ( Schwaller et al. , 2019 ) , retrosynthesis ( Karpov et al. , 2019 ) and generating proteins ( Elnaggar et al. , 2020 ; Grechishnikova , 2021 ) . A few works have explored using transformers for generative modeling of organic molecules ( He et al. , 2021b ; Shin et al. , 2021 ; Dollar et al. , 2021 ) . Some works have also proposed using transformers for scaffold-conditioned generative modeling ( He et al. , 2021a ; Bagal et al. , 2021 ) . This work extends these efforts by proposing a simple yet effective training and zero-shot adaptation method , and by using IUPAC names instead of SMILES strings . IUPAC Names Although we are unaware of prior work using IUPAC names as a base representation for molecular modeling , several works have explored using machine learning to convert between IUPAC names and other molecular representations ( Rajan et al. , 2021 ; Handsel et al. , 2021 ; Krasnov et al. , 2021 ) . 3 METHOD . Molecular optimization is a difficult problem because it requires modifying a molecule that already satisfies a number of requirements . Modifications need to improve a particular aspect of the molecule without degrading its performance on other metrics , and without making it too difficult to synthesize . We argue that by using IUPAC names ( Section 3.1 ) and by allowing users to target particular parts of a molecule to modify ( Section 3.2 ) , C5T5 has the potential to support humanin-the-loop molecular editing that complements domain experts ’ intuitions about structure-activity relationships and synthetic accessibility . 3.1 IUPAC NAMING . The International Union of Pure and Applied Chemistry ( IUPAC ) publishes a set of rules that allow systematic conversion between a chemical structure and a human-readable name ( Favre & Powell , 2013 ) . For example , 2-chloropentane refers unambiguously to five carbons ( “ pent ” ) connected by single bonds ( “ ane ” ) with a chlorine atom ( “ chloro ” ) bonded to the second carbon from one end ( “ 2- ” ) . IUPAC names are used ubiquitously in scholarly articles , patents , and educational materials . In contrast to other linear molecular representations like SMILES and its derivatives , where single tokens mostly refer to individual atoms and bonds , tokens in IUPAC names generally have a rich semantic meaning . For example , the token “ ic acid ” denotes a carboxylic acid , which is a common functional group that has well-known physical and chemical properties ; there are many known chemical reactions that either start with or produce carboxylic acids . Other tokens denote additional functional groups ( e.g . “ imide , ” “ imine , ” “ al , ” “ one ” ) , locants ( e.g . “ 1 , ” “ 2 , ” “ N ” ) , which indicate connectivity , alkanes ( e.g . “ meth , ” “ eth , ” “ prop ” ) , which denote the lengths of carbon chains , polycyclic rings ( e.g . “ naphthalene , ” “ anthracene ” ) , stereochemistry markers ( “ R , ” “ S ” ) , and multipliers ( e.g . “ di , ” “ tri ” ) , which concisely represent duplicated and symmetric structures . Figure 2 shows the relationships between IUPAC names , graph representations , and SMILES . For molecular optimization , C5T5 supports qualitatively different molecular edits compared to graph- and SMILES-based methods by virtue of its use of IUPAC names : editing a locant token corresponds to moving a functional group along a carbon backbone or changing the connectivity of a fused ring system ; and editing a multiplier token corresponds to creating or eliminating duplicated and symmetric structures . For example , changing “ ethylbenzene “ to “ hexaethylbenzene ” replicates the ethyl structure around the entire benzene ring with a single token edit . These sorts of modifications require much more extensive editing for SMILES- and graph-based methods.1 We argue that IUPAC names are especially attractive for molecular optimization , since the process requires interaction between the algorithm and a domain expert , so interpretability is paramount . Compared to graph- or SMILES-based models , C5T5 makes predictions that can be traced back to moieties and functional groups that domain experts are more likely to understand , trust , and know how to synthesize than either arbitrary collections of atoms and bonds or motifs decided upon by machine learning practitioners . In addition to improved interpretability , we argue that using IUPAC names has advantages purely from the standpoint of modeling data , since moving from SMILES to IUPAC names is akin to moving from a character-based to a word-based sequence model . Modeling at this higher level of abstraction enables the network to direct more of its capacity to structure at the relevant semantic level , instead of relearning lower-level details like the specific atomic composition of functional groups . In this vein , we demonstrate the potential of IUPAC names by learning word2vec representations of IUPAC name tokens ( Mikolov et al. , 2013 ) , drawn from a list of over 100 million names in the PubChem repository ( Kim et al. , 2016 ) and tokenized using a list of tokens in OPSIN—an open-source IUPAC Name parser library ( MIT License ) ( Lowe et al. , 2011 ) . For example , as shown in Figure 2 , the chemical “ 2-acetyloxybenzoic acid ” gets tokenized to [ “ 2 ” , “ - ” , “ acet ” , “ yl ” , “ oxy ” , “ benzo ” , “ ic acid ” ] . As with natural language modeling , we find that the embedding space learned by word2vec encodes the semantic meaning of the tokens , as shown in Figure 3 . Different classes of tokens tend to be clustered together , and similar tokens within clusters are located nearby . For example , aromatic compounds with two rings are clearly separated from those with three , locants are ordered roughly correctly from 1 to 100 , and multiplier tokens are also roughly in order ( zoom not shown ) . Following Mikolov et al . ( 2013 ) , we also find that simple arithmetic operations in the embedding vector space correspond to semantic analogies between tokens . For example , the nearest neighbor of “ phosphonous acid ” - “ nitrous acid ” + “ nitroso ” is the embedding for “ phosphoroso. ” 2 The nearest neighbor of “ diphosphate ” - “ disulfate ” + “ sulfate ” is “ phosphate. ” Likewise for “ selenate ” - “ tellurate ” + “ tellurite ” being closest to “ selenite . ”

Summary: This paper proposes a method C5T5, a self-supervised pre-training method based on the T5 pre-trained model, which is able to make zero-shot select-and-replace edits to satisfy specific property values. The specific difference of this paper is the IUPAC names (a standardized molecular representation), and the method is totally self-supervised. The experiments are evaluated on octanol-water partition, distribution coefficients, polar surface area, and refractivity, above four properties. Experiments show that the designed methods are able to achieve the optimization objective.