Conditional Generative Modeling for De Novo Hierarchical Multi-Label Functional Protein Design

1 INTRODUCTION . Designing proteins with a target biological function is an important task in biotechnology with highimpact implications in pharmaceutical research , such as in drug design or synthetic biology ( Huang et al. , 2016 ) . However , the task is challenging since the sequence-structure-function relationship of proteins is extremely complex and not yet understood ( Dill & MacCallum , 2012 ) . Functional protein design is currently done by traditional methods such as directed evolution ( Arnold , 1998 ) , which rely on a few random mutations of known proteins and selective pressure to explore a space of related proteins . However , this process can be time-consuming and cost-intensive , and most often only explores a small part of the sequence space . In parallel , data characterizing proteins and their functions is readily available and constitutes a promising opportunity for machine learning applications in protein sequence design . Moreover , the hierarchical organisation of protein functions in a complex ontology of labels could help machine learning models capture sequence-information relationships adequately . Recently , generative models have attempted to design proteins for different tasks , such as developing new therapies ( Muller et al. , 2018 ; Davidsen et al. , 2019 ) or enzymes ( Repecka et al. , 2019 ) . Nonetheless , most of the de novo protein sequence design methods , which generate sequences from scratch , focus on a specific function or on families of short proteins . Instead , we would like to focus on modeling several different biological functions at the same time to eventually be able to freely combine them . To this end , one first requires a model that is able to deal with and to understand the inherent label structure . We concern ourselves with the development of such a generative model . In this work , we introduce the general-purpose generative model ProteoGAN , a conditional generative adversarial network ( cGAN ) that is able to generate protein sequences given a large set of functions in the Gene Ontology ( GO ) Molecular Function directed acyclic graph ( DAG ) ( Gene On- tology Consortium , 2019 ) . To the extent of our knowledge , we are the first to propose a hierarchical multi-label de novo protein design framework , which does not require prior knowledge about the protein , such as seed sequence fragments or structure . Our contributions can be summarized as follows : ( i ) we propose a data-driven approach to de novo functional protein generation that leverages a large set of annotated sequences , ( ii ) we present a new extensive evaluation scheme to assess validity , conditional consistency , diversity , and biological relevance of the generated sequences , and ( iii ) we conduct an in-depth model optimization to derive actionable insights on architectural choices and efficient conditioning mechanisms while outperforming existing state-of-the-art protein generators . We focus on generative adversarial networks , due to their promising performance on specific sequence design tasks ( Repecka et al. , 2019 ) . We choose a conditional setting not to rely on oracles nor on multiple rounds of training-generation-measurement , since to this date a well performing general-purpose predictor of protein function remains elusive ( Zhou et al. , 2019 ) . As opposed to most existing methods ( see Section 2 ) , we aim to generate a comprehensive variety of proteins exhibiting a wide range of functions , rather than focusing on optimising a single function within a unique protein family . As this is a different task from the ones found in the literature , we need to define an adequate evaluation pipeline . Therefore , we establish a multiclass protein generation evaluation scheme centered around validity and conditional consistency . The model should generate protein sequences whose distribution resembles that of natural proteins and hence have similar chemo-physical properties , and it should do so conditionally , namely generating proteins of a given functional class without off-target functions . We are hence confronted with the problem of assessing i ) the performance of the generative model in a general sense , which is defined by how well the generated distribution fits the training data distribution , and ii ) the conditional performance of the model which we define as a special case of the general performance , where we compare sequence feature distributions between labels . We therefore require distribution-based evaluations . A natural choice to evaluate the performance of a generative model is a two-sample test , which allows to answer whether a generated and a real set of samples ( i.e . the dataset ) could originate from the same distribution . The difficulty here is to define a measure that can handle the structured data , in our case protein sequences . To this end , we design Maximum Mean Discrepancy ( MMD ) -based evaluation criteria ( Gretton et al. , 2012 ) , which ensure good model performance and a functioning conditioning mechanism by measuring differences in empirical distribution between sets of generated and real protein sequences . To ensure diversity , we monitor the duality gap ( Grnarova et al. , 2019 ) , a domain-agnostic indicator for GAN training . Lastly , we use a series of biologically-driven criteria in the evaluation phase that confirms the biological validity of the generated protein by relying on the standard protein feature software ProFET ( Ofer & Linial , 2015 ) . With this arsenal of measures , and given the low computational complexity of our MMD-based criteria , we compare different architectural choices and hyperparameters in an extensive and efficient Bayesian Optimization and HyperBand ( BOHB ) ( Falkner et al. , 2018 ) search . In particular , we develop improved variants of two existing conditional mechanisms on GANs ( Odena et al. , 2017 ; Miyato & Koyama , 2018 ) and show for the first time that the previously unexplored combination of both is beneficial to conditional generation . Moreover , the selected model outperforms ( i ) de novo conditional model CVAE ( Greener et al. , 2018 ) , repurposed and trained towards functional protein generation , other introduced baselines ( HMM , n-gram model ) , and ( ii ) models specifically built to challenge the necessity of a conditional mechanism . The remainder of the document is organized as follows . First , the background and related work section gives a concise overview of the biological mechanisms underlying the function of proteins , summarises the state-of-the-art generative models applied to protein design , details some conditional mechanisms in GANs and identifies existing evaluation criteria for GANs and cGANs . Subsequently , the method section describes ProteoGAN and its components and explains our protein generation evaluation framework . Finally , the results obtained by conditioning the generation of new sequences on 50 GO classes are presented and discussed before concluding with some final remarks . 2 BACKGROUND AND RELATED WORK . Biological mechanisms underlying protein functions . Proteins are biological structures that serve a wide variety of purposes in organisms . They are composed of chains of amino acids and can therefore be represented as simple sequences . However , the relationship between physico-chemical properties of amino-acids , three dimensional structure and resulting biological activity of the macromolecule is highly complex ( see supplementary Section A.1 ) . Nevertheless , since the advent of modern sequencing techniques , millions of proteins have been registered in databases , along with curated descriptions of their function . For example , the GO is a species-agnostic ontology that aims at classifying genes ( and the resulting proteins ) according to their functions , locations , and governing biological processes using a hierarchical structure of functional labels . As such , it represents an ideal interface between scientists who wish to design proteins with descriptive and modular labels , and a generative model that captures the complex relationships of sequence , structure and function . Guided and conditional generative models . Machine learning models and more recently deep generative models ( Eddy , 2004 ; Goodfellow et al. , 2014 ; Kingma & Welling , 2014 ; Rezende et al. , 2014 ; Vaswani et al. , 2017 ; Li et al. , 2017a ) have been used to design in silico biological sequences , such as RNA , DNA or protein sequences ( R. Durbin & Mitchinson , 1998 ; Davidsen et al. , 2019 ; Brookes et al. , 2019 ; Hawkins-Hooker et al. , 2020 ; Costello & Martin , 2019 ; Anand & Huang , 2018 ) . Among them , several approaches have been developed in order to control sequence generation . They can be sorted in three categories , guided , conditional or combinations thereof . Guided approaches use a predictor oracle in order to guide the design towards target properties , through iterative training , generation and prediction steps ( Brookes et al. , 2019 ; Gane et al. , 2019 ; Angermueller et al. , 2019 ; Gupta & Zou , 2019 ; Killoran et al. , 2017 ; Repecka et al. , 2019 ) . While these guided methods have the theoretical advantage to produce proteins with specific characteristics , for example brightness ( Brookes et al. , 2019 ) , they require an independent oracle . This oracle can be itself hard to train and remains imperfect , even for highly specialized prediction tasks . Moreover , the lack of well-functioning predictors for large numbers of labels impairs the usage of guided-generation techniques to multiclass applications such as functional protein generation ( Zhou et al. , 2019 ) . On the contrary , conditional approaches integrate the desired properties in the generation mechanism , eliminating the need for an oracle . Karimi et al . ( 2019 ) provided a guided conditional WassersteinGAN to generate proteins with novel folds . Interestingly , Madani et al . ( 2020 ) developed ProGen , a conditional transformer that enables a controlled generation of a large range of functional proteins . However , the method ’ s need for sequence context can be experimentally constraining and is not compatible with de novo design . Ingraham et al . ( 2019 ) present a graph-based conditional generative model that unfortunately needs only sparsely available structural information . Das et al . ( 2018 ) and Greener et al . ( 2018 ) train conditional VAEs in order to generate specific proteins , such as metalloproteins . Conditional mechanisms in GANs . Several conditional mechanisms have been proposed to conditionally generate samples with GANs . Among the most successful ones in conditional image generation tasks , Odena et al . ( 2017 ) introduced the auxiliary classifier GAN ( AC-GAN ) , which uses a third integrated network , in addition to the generator and the discriminator , to predict labels of both real and generated inputs to the discriminator . Miyato & Koyama ( 2018 ) proposed an alternative conditioning mechanism , where the label information is introduced to the network as the inner product of the embedded label vector and an intermediate layer of the network , a mechanism they refer to as projection . Projections can be seen as an alternative to simple concatenations of label information to the network input ( Mirza & Osindero , 2014 ) , in a way that respects the underlying probabilistic model . Generative models evaluation . To this date , there is no definitive consensus on the best evaluation measures for the evaluation of quality , diversity and conditional consistency of the output of a ( conditional ) generative model ( Papineni et al. , 2002 ; Salimans et al. , 2016 ; Heusel et al. , 2017 ; Shmelkov et al. , 2018 ; Kynkäänniemi et al. , 2019 ; DeVries et al. , 2019 ) . Most measures that stand out in computer vision such as the Inception Score ( IS ) ( Salimans et al. , 2016 ) , the Frechet Inception Distance ( FID ) ( Heusel et al. , 2017 ) , GAN-train and GAN-test ( Shmelkov et al. , 2018 ) depend on an external , domain-specific predictor . On the contrary , the domain-agnostic duality gap can be computed during training and at test time , and has been shown to correlate well with FID ( Grnarova et al. , 2019 ) . In functional protein prediction , results obtained by state-of-the-art classification mod- els are encouraging but still not good nor fast enough to entirely rely on them when evaluating and training GANs ( Fmax = 0.631 ( Radivojac et al. , 2013 ; Zhou et al. , 2019 ; You et al. , 2019 ) .

Summary: In this manuscript, the authors present a conditional GAN for generating protein sequences given specified GO terms. They argue that this approach to conditional protein generation is more appropriate than sequence-based generation, because it gets directly at functional specification. At a high level, this is an interesting idea though it has already started to be explored by other works. The authors are correct that these works focus primarily on optimize a single function of interest. However, there doesn’t seem to be any specific reason that guided design approaches could not generalize to multiple criteria. Regardless, controlled generation of proteins with pre specified functions is certainly interesting.