Conditional Generative Modeling for De Novo Hierarchical Multi-Label Functional Protein Design
Authors: Tim Kucera
Karsten Michael Borgwardt
Matteo Togninalli
Laetitia Papaxanthos
Venue:  (2021)
Keywords: protein design
conditional generative adversarial networks
gene ontology
hierarchical multi-label
GO
GAN

Abstract: The availability of vast protein sequence information and rich functional annotations thereof has a large potential for protein design applications in biomedicine and synthetic biology. To this date, there exists no method for the general-purpose design of proteins without any prior knowledge about the protein of interest, such as costly and rare structure information or seed sequence fragments. However, the Gene Ontology (GO) database provides information about the hierarchical organisation of protein functions, and thus could inform generative models about the underlying complex sequence-function relationships, replacing the need for structural data. We therefore propose to use conditional generative adversarial networks (cGANs) on the task of fast de novo hierarchical multi-label protein design. We generate protein sequences exhibiting properties of a large set of molecular functions extracted from the GO database, using a single model and without any prior information. We shed light on efficient conditioning mechanisms and adapted network architectures thanks to a thorough hyperparameter selection process and analysis. We further provide statistically- and biologically-driven evaluation measures for generative models in the context of protein design to assess the quality of the generated sequences and facilitate progress in the field. We show that our proposed model, ProteoGAN, outperforms several baselines when designing proteins given a functional label and generates well-formed sequences.

Under review as a conference paper at ICLR 2021
CONDITIONAL GENERATIVE MODELING FOR DE Novo HIERARCHICAL MULTI-LABEL FUNCTIONAL PROTEIN DESIGN
Anonymous authors Paper under double-bling review
ABSTRACT
The availability of vast protein sequence information and rich functional annota- tions thereof has a large potential for protein design applications biomedicine and synthetic biology. To this date, there exists no method for the general-purpose design of proteins without any prior knowledge about the protein of interest, such costly and rare structure information the Gene Ontology (GO) database provides information about the hierarchical or- ganisation of protein functions, and thus could inform generative models about the underlying complex sequence- funecton relationships, replacing the need for structural data. We therefore propose use conditional generative adversarial networks (cGANs) on the task of fast de novo hierarchical multi-label protein de- sign. We generate protein sequences exhibiting properties of large set of molec- ular functions extracted from the GO database, using single model and without any prior information. We shed light on efficient conditioning mechanisms and adapted network architectures thanks to thorough hyperparameter selection pro- cess and analysis. We further provide statistically- and biologically-driven eval- uation measures for generative models in the context of protein design to assess the quality of the generated sequences and facilitate progress in the field. We show that proposed model, ProteoGAN, outperforms several baselines when designing proteins given a functional label and generates well-formed sequences.
seed sequence fragments. However,
INTRODUCTION
Designing proteins with target biological function is an important task in biotechnology with high- impact implications in pharmaceutical research, such as in drug design or synthetic biology (Huang
et al. 2016). However, the task is challenging since the
relationship of proteins is extremely complex and not yet understood (Dill & MacCallum, 2012). Functional
protein design is currently done by traditional methods such directed evolution (Amnold 1998), which rely on a few random mutations of known proteins and selective pressure to explore a space of related proteins. However, this process can be time-consuming and cost-intensive, and most of- ten only explores a small part of the sequence space. In parallel, data characterizing proteins and their functions is readily available and constitutes promising opportunity for machine learning ap- plications in protein sequence design. Moreover, the hierarchical organisation of protein functions complex ontology of labels could help machine learning models capture sequence-inie innfommatioooooooonnnnnooo relationships adequately. Recently, generative models have attempted to design proteins for dif- ferent tasks. such as developing new therapies (Muller al.] 2018 Davidsen al.) 2019) or en- zymes (Repecka et al. 2019). Nonetheless, most of the de novo protein sequence design methods, which generate sequences from scratch, focus on specific function or on families of short proteins. Instead, we would like to focus modeling several different biological functions at the same time to eventually be able to freely combine them. To this end, one first requires a model that is able to deal with and understand the inherent label structure. We concern ourselves with the development of such a generative model.
In this work, introduce the ceneral-purpose generative model ProteoGAN, conditional gen- erative adversarial network (cGAN) that is able to generate protein sequences given a large set of functions in the Gene Ontology (GO) Molecular Function directed acyclic graph (DAG) (Gene On-

---

Under review as a conference paper at ICLR 2021
tology Consortium 2019). To the extent of our knowledge, we the first to propose hierarchical multi-label de novo protein design framework, which does not require prior knowledge about the protein, such as seed sequence fragments structure.
Our contributions can be summarized as follows: functional protein generation that leverages large set of annotated sequences, (ii) present new extensive evaluation scheme to assess validity, conditional consistency, diversity, and biolog- ical relevance of the generated sequences, and (iii) we conduct an in-depth model optimization to derive actionable insights on architectural choices and efficient conditioning mechanisms while out- performing existing state-of of-the-rrt protein generators.
(i) we propose a data-driven approach to de novo
We focus on generative adversarial networks, due to their promising performance on specific se- quence design tasks (Repecka al. 2019). We choose a conditional setting not to rely on oracles nor on multiple rounds of training-gent cnmeerationmmassrrmenn
since to this date well performing
general- purposs predictor of protein function remains elusive (Zhou et al. (019). As opposed to most existing methods (see Section D
we aim to generate a comprehensive variety of proteins ex- hibiting a wide range of functions, rather than focusing on optimising a single function within a unique protein family. As this is a different task from the ones found in the literature, we need to define an adequate evaluation pipeline.
Therefore, establish multiclass protein generation evaluation scheme centered around validity and conditional consistency. The model should generate protein sequences whose distribution re- sembles that of natural proteins and hence have similar chemo- pheesill properties, and it should do so conditionally, namely generating proteins of given functional class without off- target functions.
We hence confronted with the problem of assessing i) the performance of the generative model in a general sense, which is defined by how well the generated distribution fits the training data distribution, and ii) the conditional performance of the model which define a special case of the general performance, where we compare sequence feature distributions between labels. We therefore require distribution- generative model is a two-sample test, which allows to answer whether a generated and a real set of samples (i.e. the dataset) could originate from the same distribution. The difficulty here is to define a measure that can handle the structured data, in our case protein sequences. To this end, we design Maximum Mean Discrepancy (MMD)-based evaluation criteria (Gretton et al., 2012). which ensure good model performance and functioning conditioning mechanism by measuring differences in empirical distribution between sets of generated and real protein sequences. To ensure diversity, we monitor the duality gap (Grnarova et al., 2201)), domain-agnoste indicator for GAN training. Lastly, we use a series of biologically-drive criteria in the evaluation phase that confirms the biological validity of the generated protein by relying on the standard protein feature software ProFET Ofeer& & Linial 2015).
evaluations. A natural choice evaluate the performance of
With this arsenal of measures, and given the low computational complexity of our MMD-based criteria, we compare different architectural choices and hyperparameters in an extensive and efficient Bayesian Optimization and HyperBand (BOHB) (Falkner et al., 2018) search. In particular, we develop improved variants of two existing conditional mechanisms on GANs (Odena et al.) 2017;
Miyato & Koyama| 2018) and show for the first time that the previously unexplored combination of both is beneficial to conditional generation. Moreover, the selected model outperforms (i) de novo conditional model CVAE (Greener al.) (2088), repurposed and trained towards functional protein generation, other introduced baselines (HMM, n-gram model), and (ii) models specifically built to challenge the necessity of a conditional mechanism.
The remainder of the document is organized as follows. First, the background and related work section gives a concise overview of the biological mechanisms underlying the function of proteins, summarises the state-of-the-art generative models applied to protein design, details some condi- tional mechanisms in GANs and identifies existing evaluation criteria for GANs and cGANs. Sub- sequently, the method section describes ProteoGAN and its components and explains our protein generation evaluation framework. Finally, the results obtained by conditioning the generation of new sequences on 50 GO classes are presented and discussed before concluding with some final remarks.
2

---

Under review as a conference paper at ICLR 2021
2 BACKGROUND AND RELATED WORK
Biological mechanisms underlying protein functions. Proteins are biological structures that serve a wide variety of purposes in organisms. They are composed of chains of amino acids and can therefore be represented as simple sequences. However, the relationship between physico-chemiee properties of amino-acids, three dimensional structure and resulting biological activity of the macro- molecule is highly complex (see supplementary Section I. Nevertheless, since the advent of modern sequencing techniques, millions of proteins have been registered in databases, along with curated descriptions of their function. For example, the GO is a species-agnostie ontology that aims at classifying genes (and the resulting proteins) according to their functions, locations, and govern- ing biological processes using hierarchical structure of functional labels. As such, it represents an ideal interface between scientists who wish to design proteins with descriptive and modular labels, and generative model that captures the complex relationships of sequence, structure and function.
Guided and conditional generative models. Machine learning models and more recently deep
generative models (Eddy 2004 Goodfellow et al. 2014 Kingma & Welling 2014 Rezende al.,
2014 Vaswani et al. 2017 Li et al. (017a)) have been used to design in silico biological sequences,
such RNA, DNA or protein sequences (R. Durbin & Mitchinson, 1998, Davidsen et al. 2019
Brookes et al., 2019; Blawkis-HHokee et al.] 2020 Costello & Martin 2019:] Anand & Huang
2018). Among them, several approaches have been developed in order to control sequence genera- tion. They can be sorted in three categories, guided, conditional or combinations thereof. Guided
approaches use predictor oracle in order to guide the design towards target properties, through iter- ative training. generation and prediction steps (Brookes et al. 20199; Gane et al. 2019 | Angermmllll jet al. 2019; Gupta & Zou, 2019; Killoran et al. 2017: Repecka et al. 2019). While these guided methods have the theoretical advantage produce proteins with specific characteristics, for exam- ple brightness (Brookes et al. 2019), they require an independent oracle. This oracle can be itself hard to train and remains imperfect, even for highly specialized prediction tasks. Moreover, the lack of funlifunctionnng predictors for large numbers of labels impairs the usage of guided-gent generation techniques multiclass applications such as functional protein generation (Zhou et al.] (019). On the contrary, conditional approaches integrate the desired properties in the generation mechanism, eliminating the need for an oracle. Karimi et al. (2019) provided guided conditional Wasserstein- GAN to generate proteins with novel folds. Interestingly, Madani et al. (2020) developed ProGen, a conditional transformer that enables a controlled generation of large range of functional pro- teins. However, the method's ' need for sequence context can be experimentally constraining and not compatible with de novo design. [Ingraham et al. 2019) present a graph-based conditional generative model that unfortunately needs only sparsely available structural information. Das et al.) (018)) and Greener et al. (2018) train conditional VAES in order to generate specific proteins, such as metalloproteins.
Conditional mechanisms in GANs. Several conditional mechanisms have been proposed to con- ditionally generate samples with GANs. Among the most successful ones in conditional image generation tasks, Odena et al.] (2017) introduced the auxiliary classifier GAN (AC-GAN), which uses third integrated network, in addition the generator and the discriminator, to predict labels of both real and generated inputs to the discriminator. MMyato & Koyamal (2018) proposed an al- ternative conditioning mechanism, where the label information is introduced to the network as the inner product of the embedded label vector and an intermediate layer of the network, a mechanism they refer as projection. Projections can be seen an alternative to simple concatenations of label information to the network input (Mirza & Osindero, 2014), in way that respects the underlying probabilistic model.
Generative models evaluation. To this date, there is no definitive consensus on the best evalua- tion measures for the evaluation of quality, diversity and conditional consistency of the output of a (conditional) generative model (Papineni et al.] 2002) Salimans et al. 2016 Heusel et al.) 2017 Shmelkov et al. 2018 Kynkäänniemi et al. 2019 DeVries et al. 2019). ).. Most measures that stand out in computer vision such as the Inception Score (IS) (Salimans et al. 2016). the Frechet Incep- tion Distance (FID) (Heusel et al., 2017). GAN-train and GAN-test (Shmelkov et al. 2018) depend on an external, domain-specific predictor. On the contrary. the Nomain-agnoste computed during training and at test time, and has been shown to correlate well with FID (Grnaroval et al. 2019). In functional protein prediction, results obtained by state-of-the-an classification mod-
duality gap can be
3

---

Under review as a conference paper at ICLR 2021
Denae
Source
Conv
Generator
Discriminator
Figure I: A schematic overview of the general model structure that we screened during hyperpa- rameter search. Architectural features such as layer number and skip connections can vary. We note that the architecture of our final model differs from the one depicted here.
els are encouraging but still not good nor fast enough entirely rely on them when evaluating and training GANs (Fmax 0.631 (Radivojac et al. 2013 Zhou et al. 2019 You et al. 2019).
3 METHODS
3.1 MODEL ARCHITECTURE
We chose the framework of Generative Adversarial Networks for our model, specifically the Wasserstein with Gradient Penalty (Arjovsky et al. [017] Gulrajani et al. 2017). Our con- volutional architecture resembles the funnel-like structure of DCGAN (Radford et al. 2013). We propose variants of existing models in order to adapt the framework to sequence generation and to guide future model development the field. An extensive parameter search is performed on a vali- dation set to select the best variants and hyperparameters (see Section 4 while schematic view of the model can be found in Figure
Conditioning mechanism. We allow for the insertion of three types of conditioning mechanisms: projections, auxiliary classifiers, or a combination of both. While projection or auxiliary classifiers not exclusive, did not encounter any work that used both in one model. The conditioning mechanisms are further explained in the supplementary Section A.3. We also allow for more than one projection from different layers of the discriminator, with the rationale that in this way could utilize protein sequence information of the different abstraction levels of the convolutional layers. Finally, the embedded label is always concatenated the latent noise vector input of the generator.
Hierarchical label encoding, Given the hierarchical structure of the functional labels, we allow for three types of label embeddings: a) one-hot encoding,
common encoding for labels, b) Poincaré encoding (O neill 2014), as the hyperbolic space is well-suited for hierarchical data and c) node2vec encoding (Grover & Leskovec (0166). which preserves neighbourhood relationships by encoding the nodes of the GO DAG based on random walks. One protein can present several GO labels, and these embeddings aim capture the relations between the labels in the DAG and to incorporate this information into the generative process. We compare against a baseline model without this hierarchical information (named Non-Hiirarchical), where sequences are fed to the model for each label independently.
We further allow to concatenate chemophysical properties the respective amino-acids to the one- hot encoding of the sequences. Other architectural and optimizer hyperparameters subject to optimization, whose descriptions and value ranges are detailed in the supplementary Section A.2.3
3.2
MODEL EVALUATION
3.2.
MMD TO MEASURE SEQUENCE VALIDITY CONDITIONAL CONSISTENCY
Computation of MMD. We propose to use the kernel two-sample test Maximum Mean Discrep- ancy (MMD) Gretton et al. 2012 Sutherland et al. 2016) to build evaluation criteria for conditional sequence design. The test has been shown to be suited for distinguishing distributions structured data such protein sequences (Borgwardt et al. 2006). MMD has also been explored in the context of GANs, where it was shown to be able to function as discriminator (Li et al.] 2015) Dziugaite
et al. 2015). Here use it assess model quality based samples. Let X
[3:]1_ and
4

---

Under review as a conference paper at ICLR 2021
Y = {y, 1-= be samples from the distributions of generated and real proteins sequences, respec- tively Pg and P... When using MMD, we compute the empirical squared MMD statistic between these two distributions using equation (2) of Gretton et al. (2012)
2
n (x:) n [ m
1
m Σ
MMD
ll
where o(.) E F is the variant of the mapping funetion of the spectrum kernel proposed by Leslie jet al.] (200T), which accounts for the presence or absence of k-mers in the sequences of interest. The size of the k-mers was set 3, as suggested for protein sequences by [eslie et al. (200T). This expression is fast to compute as it scales linearly with the number of sequences and can consequently be used an early stopping criterion during the training process and for model selection. We also report the result of a more powerful Gaussian kernel on top of the 3-mer embeddings. We used the statistic itself rather than the resulting p-values because these latter are too sensitive as soon as more than 3% of random noise is added to the sequences (Table A6 in the Supplementary Material).
Generation of in-silico validated sequences with MMD. In order to assess to which extent our model captures the unconditional distribution of our sequences, we use MMD as described above. We therefore first ensure that the proteins generated by the model resemble existing ones. We could show that a 3-mer embedding is sufficient for our context, as the functional classes we are concerned with (see Section for description of functional classes) can be linearly separated in feature space. The functional annotations can be classified with 94% accuracy based on hyperplanes in our embedding space in a one-vs-all scheme.
Generation of functional sequences with MRR conditional MMD. We compute the condi- tional performance by measuring, for each set of generated proteins for given target label, how many sets of real proteins with off-target label are closer than the set of real proteins of the targeted label. The distances between sets are estimated using MMD as defined above. Let R be a set of real sequences, which is composed of the sets {R, h= of sequences with annotated la- bels (Le) where d is the number of labels. Let G = {G; f be an equally structured set of generated sequences. We want to maximise the following mean reciprocal rank (MRR):
MRR R. G) - rank (OnDMM (R G&))
where (MMD( G,)) is the rank of MMD(R,, G.) among sorted elements of the list [MMD(R; (R; G1), MMD( G2)...,, MMD Ga)]. MRR(G) maximal and of value when the generated sets of proteins for a given label are the closest to the set of real proteins with the same label. Variants of MRR that give more insight on conditional performance for closely-related functions in the label hierarchy are described in the supplementary Sections A.3.2 and A.3.3
3.2.2 MEASURES TO ASSESS QUALITY AND DIVERSITY OF GENERATED SEQUENCES
We monitor the duality gap (Grnarova al. 2019) of our GAN model. A small duality gap indi- cates good convergence and common failure modes, such as mode collapse, can be detected. We follow the authors' suggestion to approximate the latter with past snapshots of the training. Ad- ditionally, to provide a protein-centric evaluation we also report Kolmogorov-Smirnot (KS) two sample tests (Masscy] 195I) between generated and real samples from the 500 (not k-mer re- lated) sequence-basee features from the feature extractor library ProFET (Ofer & Linial 2015).
4 EXPERIMENTAL SETUP
Data. Sequence data was acquired from the UniProt Knowledgebase (UniProtKB, Consortium 2 Out of the 180 million entries, we selected sequences with experimental evidence and at least one annotated GO label. Then we restricted to the standard amino acids and sequence- lenghh of 2,048, which covers ca. 98.5% of the remaining data points. The resulting dataset contains 4,90 sequences. The ontology is structured as a DAG and labels have hierarchical relationship.
5

---

Under review as a conference paper at ICLR 2021
Le. proteins with given functional label inherit automatically the labels of their parents in the DAG. We restricted the number of labels to the 50 most common molecular functions, imposing a threshold of at least 5,375 sequences per functional label. This is sufficient for proof-of -frinnciie and would even enable the design of experimental assays for validation. Figure 4 illustrates the selected subset of labels and their hierarchical relationships. We randomly split the dataset in training, validation and test sets keeping ca. 15,000 (10%) sequences each in the validation and test sets, During model optimization, we use smaller splits with ca. 3.000 sequences each. We ensure that all labels have minimum amount of samples in the test and validation sets, and use the same number of sequences per class for our MRR measure (1.300 and 300 sequences, respectively). Further details about the dataset and splits are available in the supplementary Section 1 and Figure
Comparison partners. We compare ProteoGAN to several baselines to put its performance into perspective. We first use CVAE (Greener et al. 2018). We performed a bayesian optimization hyperparameter search over 1,000 models. The model was adjusted to incorporate the 50 labels of our problem setting. We could not compare fairly to PepCVAE (Das et al. 2018) as the model does not scale to sequence lengths of 2048 amino-acids. We refer the reader to the respective papers and to Section A.2.2 Tables A2 A3 and Figure A2 A2 for further information and results on both baselines.
Additionally, we constructed several baselines with the goal to assess the usefulness of the condi- tional generation mechanism. The first baseline, referred to as Unconditional, consists of as many unconditional GANs as there are labels. To do so, we remove the conditioning mechanism from our model and train multiple instances label. We generate sequences for a target label by sampling them from the GAN trained on the se- quences annotated with the same label. Our second alternative conditioning replaces conditional model with the combination of an unconditional model trained on the full data and an established predictor of protein function, NetGO (You et al., 2019), used to predict the labels of generated se- quences which replaces the need for conditioning. We refer to this baseline as Predictor-Guide Further, we assess whether the model utilizes the hierarchical label structure by training a Non- Hierarchical baseline which only sees the labels independently for each sequence. This is done by replicating and annotating a sequence for each associated label, while keeping the number of gradient updates the same across all models.
the fifty subsets of data that annotated with particular
Lastly, we compare two traditional generative models for sequences, a (profile) HMM and an n- gram model (n=3). Since they without conditioning mechanism, we train one model for each label combination in our testset (called 'on label set' in Table | in this case a protein's 's GO labels can contain several GO terms and their parent terms), as well as one model for each of the 50 labels we are dealing with (called "on single label' in Table [ in this case we discard the hierarchy of labels). Further description of most of the baselines available in the Supplementary Material
Section A.2.2
Hyperparameter optimization. We conducted two Bayesian Optimization and HyperBand (BOHB) searches (Falkner et al. 2018) on six Nvidia GeForce GTX 1080, first a broad search among 23 hyperparamaters and second, smaller and more selective, among 9 selected hyperpa- rameters. The optimization objective was set to maximize the ratio of the evaluation measures MRR/MMD, which are detailed in Section T.2.1 to balance the validity and the conditional con- sistency of the generated sequences. Both searches were complemented by a functional analysis of variance ([ANOVA) (Hutter et al.) (2014). The results of the second optimization, for which we evaluated 1, 000 models for a maximum duration of 27 epochs in our experiments, are shown Fig- ure 2 The 27 best selected models of the second hyperparameter search were then trained for a prolonged duration of 100 epochs, where the conditioning mechanism and an associated weighing factor became most important. Further details about hyperparameter optimization are available in the supplementary Section A.2.3 and Table A4
5 RESULTS AND DISCUSSION
5.1 MODELS SELECTED BY THE BOHB OPTIMIZATION
Insights cGANs architecture. The results of the FANOVA and of a prediction of hyperparameter marginals on the second automatic hyperparameter optimization led to the following observations. We could show that adding chemophysical features did generally decrease model performance, and

---

Under review as a conference paper at ICLR 2021
Hyperparameter impor- tance. The bars show individual im- portance of each hyperparameter in terms of the variance they explain. We conducted the analysis for all tri- als of the optimization (left) and for the selected models that were trained for prolonged time (right). The total variance explained by the main effects was 36% and 88%, respectively.
Figure 2:
All
auxilliary weighing factor
conditicning mechanism
discriminator learning rate-
generator learning rate
kernel size
embedding
projections
strides
use chemaphysical features
0.0
0.2
0.2 Importance
Importance
4MD
HND
4B0.
M00
9.000
HTT
6.0
Reedciad VW4
0.005
0.00
0.14
9:40
012
0.4
3.40
000
1856
10U0
poincrr
projection
bor
(a) Use biochemical features
(b) Label embedding
(c) Conditioning mechanism
Figure 3: Marginal predictions of biochemical features and the label embedding based on optimiza- tion data and of the conditional mechanism based on data of the 27 best selected models. Predictions were obtained training on MMD and MRR. Lower is better for blue and higher is better for red.
that the most suitable label embedding is simple one-hot encoding (Figure B-b)). More interest- ingly. the conditioning mechanism showed different impacts when analysing with respect to either MMD and MRR (Figure 3c). Performance increased when changing from projection, auxiliary classifier, to both when evaluating with respect to MRR, but the opposite occurred for MMD. This indicates that there is trade-off between conditional generation and general sequence quality. A combination of both conditioning mechanisms is the best option when aiming at conditional perfor- mance, although others might prefer an emphasis towards better sequence quality over conditional generation. We further conclude that small and simple model architectures show best results from an analysis of our first optimization, and that many of the various proposed extensions the GAN framework did not show significant impact performance in our context (see supplementary Fig-
ures
A4 A3 A6
Selection of the final model. We selected the best model checkpoint over training phase of 300 epochs based on the validation set. The final hyperparameter configuration of our model (Pro- teoGAN), well as loss plots and real-time evaluations during training can be found in supplemen- tary Section AA..44 Table A3 and Figure A7 Most notably, the final model had both conditioning mechanisms, multiple projections, and one-hot encoding of label information. Additionally, the duality gap evaluations during training (Figure A7 showed no signs of mode collapse.
5.2 PERFORMANCE EVALUATION OF PROTEOGAN
We report measures on the test set of the best model in Table A8 performance per individual GO label in Figure 4 ProteoGAN can reliably generate sequences con- ditioned various GO labels, where many of the labels can be very well generated without major off-target effects (23 (resp. 32) of 50 labels were average ranked first or second (resp. or third) compared all other labels). The overall conditional performance (MRR=0.545) is reasonably
We additionally report the model
Figure 4: Reciprocal rank for each individual la- bel. The structure represents the relations the GO DAG. Nodes are colored by how well the model can target them. Blue to light orange in- dicates that the model ranks the target function the first three positions average. Values are averaged over 5 different data splits.
7

---

Under review as a conference paper at ICLR 2021
close to reference set of natural protein sequences (MRR=0.889). With respect general quence quality, ProteoGAN reaches MMD values of 0.040, which corresponds to roughly 20% of random mutations in set of natural sequences (compare supplementary Table While this num- ber is high compared to traditional protein engineering approaches, we note that systematic studies showed that proteins can tolerate a high mutation rate (up to 59% (Markiewicz al.) 1994 Ng
& Henikoff 200Th) as long as critical residues are not affected, and notably Reeecka et al. (2019) could experimentally validate GAN-generated sequences with up 34% difference to its closest homolog. Additionally, we show that the sequences generated by the model are not closer to the training set than the test set is, as measured by squared euclidean distance in kernel feature space (Supplementary Figure A8 This implies that our model is not overfitting on the training set and is able to generate novel sequences.
Table D shows the performance under MMD, Gaussian MMD and MRR for ProteoGAN and various baselines. In general, MMD and Gaussian MMD give similar rankings for the different models. We first note that ProteoGAN outperforms all multi-label conditional baselines by significant margin in both overall model performance and conditional generation. We could also show that the (condi- tional) ProteoGAN is comparable to the Unconditional model, which implies that the conditioning mechanism can substitute the training of many models per individual label. It shows that the con- ditioning mechanism is working well. ProteoGAN scores also better than the Non-tfierarchical model, which shows that it could incorporate the hierarchical multi-label structure into the genera- tion and that it is beneficial. It remains to be shown that this sufficient to enable out of distribution functional label generation.
The weak conditional performance of the Predictor-Guided model (MRR = 0.109) suggests that the state-of-the-art predictor used (NeIGO) is not able to predict the right label for the generated sequences, and therefore fails at guiding conditional generation, possibly because the generated sequences do not have close homologs in the protein databases.
As an outlook, we provide some results of ProteoGAN trained on more specific labels (lower panel in Table We kept the same architecture as the one optimised for 50 labels and retrained the model in three different situations. We observe that the performance is still reasonable when the number of labels is doubled (100 labels). With 200 labels the performance starts to drop. It may be that the model capacity is too low in this case, which could be alleviated by tuning the hyperparameters we have identified to be critical. When training the model on the more specific labels of the depth-first sampling of labels the performance stays good, however we note that with increasing specificity the classes get very small.
ProteoGAN further outperformed CVAE and the HMM and n-gram baselines. This becomes evident especially in conditional generation (MRR). Figure 5 confirms these results. It can generally be seen that ProteoGAN shows good agreement with the real protein sequences of the testset in several important biological sequence features. We show exemplary feature distributions for some of the ca. 500 features analyzed by ProFET, For a summary statistic of all features, we report the distribution of KS-statistics between generated and real data across all features. Also here, the ProteoGAN variants outperformed the other models.
6 CONCLUSION
In this work, we develop conditional generative adversarial model, ProteoGAN, which generates sequences validated in-silico according to statistically- and biologically-drvenn measures. We iden- tify useful architectures and extensions of cGANs for proteins. The measures we propose are fast to compute and therefore can be used during optimization and training, as well as for model assess- ment. We show that the conditioning on hierarchical Gene Ontology labels was successful, as we could show that number of labels can be well targeted. Generally, it remains to be shown that the class of multi-label generative models can not only generate the correct feature distributions, but also experimentally valid proteins with the specified functions. This requires further development of evaluation measures, which we hope to have set basis for. Future improvements to the model might also come from a larger number of labels, more specific targeting for small classes and proof that such conditional models are able to combine the modular GO labels into new and unseen functions, which would be tremendously useful for biotechnological applications.
8

---

Under review as a conference paper at ICLR 2021
Prefenddn
Usconsh
Uicoodd Donal
Rechiaradd
19HN
(Singhe
1950 (lrgh0
D-gra Cirele)
1000 1906 15909 10100 21003 1900
230
2:00 2.5
(a) Molecular Weight
(b) Isoelectric Point
(c) Total Entropy
Pransnsinn
Messedilaad
Predicteruudeed
Pradictorgaieee
100
Reeblimen
0.6
Protosalll
Uncersttionl
CYAE
sonti (tet)
Hoat (irple))
inee (Seel
tigram (irglee)
engiaas (Ringhes
Higiase (Mlgg00)
6.44 055
300
3.90
3.02
100
(d) GRAVY
(e) Flexibility
(f) KS Statisties
Figure 5: (a-e) Distributions of some selected features of ProFET, dark blue is the reference. f) KS statistics over 500 ProFET features, lower is better.
Table 1: Evaluation of ProteoGAN and various baselines with our proposed measures (MMD, MRR and the more powerful gaussian kernel variant of MMD) on the testset. An arrow indicates that lower (J) higher (f) is better. Best results in bold, second best underlined. Given are mean values and standard deviation over five different data splits. Due to the computational effort, the Unconditional model was only trained one split. The positive control is sample of real sequences and simulates a perfect model, the negative control is sample that simulates the worst possible model for each metric (constant sequence for MMD, randomized labels for MRR). The second panel shows models without multi-label conditioning, which were conditioned on simplified label sets. Also shown are variants of ProteoGAN that were trained on different datasets with more specific labels.
Model
MMD4
Gaussian MMD)
MRRI
Positive Control
0.011 0.000
0.009 0.000
0.889 + 0.019
Negative Control
1.024 + 0.000
0.807 + 0.000
0.092 + 0.003
ProteoGAN
0.040 + 0.001
0.026 + 0.001
0.545 = 0.017
Predictor Guided
0.025 0.001
0.017 + 0.001
0.109 + 0.005
Nonhierarchical
0.228 + 0.056
0.155 + 0.039
0.271 + 0.086
CVAE (Greener et al.)
0.166 + 0.033
0.108 + 0.022
0.356 + 0.027
HMM (on label set)
0.166 + 0.003
0.1 0.002
0.107 + 0.006
n-gram (on label set)
0.064 + 0.004
0.042 + 0.002
0.261 + 0.029
Unconditional
0.032
0.021
0.543
HMM (on single label)
0.192 + 0.003
0.130 0.002
0.105 + 0.003
n-gram (on single label)
0.094 + 0.004
0.067 + 0.002
0.090 + 0.000
ProteoGAN (100 Labels)
0.034
0.023
0.470
ProteoGAN (200 Labels)
0.161
0.111
0.116
ProteoGAN (depth-first)
0.091
0.070
0.270
Code Availability. We make source code for ProteoGAN and the evaluations available at https: 7/githube httpa//......oeeeeeeeeaer//erogn

---

Under review as a conference paper at ICLR 2021
REFERENCES
Namrata Anand and Possu Huang. Generative modeling for protein structures, In Advances in Neural Information Processing Systems, pp. 7494-7505, 2018.
Christof Angermueller, David Dohan, David Belanger. Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In International Conference Learning Representations, 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning-Volume Voluuee 70, pp. 214- 223, 2017.
Frances Arnold. Design by directed evolution. Accounts of chemical research, 31(3):125-131, 1998.
Karsten M Borgwardt, Arthur Gretton, Malte Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22 (514 2006.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a contimuous space. In Proceedings of The 20th SIGNLL Confer- on Computational Natural Language Learning, 2016.
David Brookes. Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. volume 97 of Proceedings of Machine Learning Research, pp. 773-782, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
UniProt Consortium, Uniprot: a worldwide hub of protein knowledge. Nucleic acids research, 47
(DI):D506-DSI) 2019.
Zak Costello and Hector Garcia Martin. How to hallucinate functional proteins. arXiv preprint arXiv:1903.00458, 2019.
Payel Das, Kahini Wadhawan, Oscar Chang, Tom Sercu, Cicero Dos Santos, Matthew Riemer, Vijil Chenthamarakshan, Inkit Padhi, and Aleksandra Mojsilovic. Pepovae: Semi-supervised targeted design of antimicrobial peptide sequences, arXiv preprint arXiv:1810.077433 2018.
Kristian Davidsen, Branden Olson, William S DeWitt III, Jean Feng, Elias Harkins, Philip Bradley, and Frederick A Matsen IV. Deep generative models for t cell receptor protein sequences. Elife, 8:e46935, 2019.
Terrance Vries, Adriana Romero, Luis Pineda, Graham W Taylor, and Michal Drozdzal. On the evaluation of conditional gans. arXiv preprint urXiv:1907.08/77 2019.
Ken A Dill and Justin L MacCallum. The protein-folding problem, 50 years Science, 338(6110): 1042-1046, 2012.
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint
2015.
Sean Eddy. What is hidden markov model? Nature biotechnology. 22(10):1315-1336, 2004.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter opti-
mization at scale. ICML, 2018.
Andreea Gane, Babak Alipanahi, Christof Angermueller, David Belanger, David Dohan, Lucy Col- well, Olivier Chapelle, Ramya Deshpande, and Suhani Vora. comparison of generative models for sequence design. 2019.
Gene Ontology Consortium.
The gene ontology resource: 20 years and still GOing strong. Nucleic acids research, 47300): 2019.
10

---

Under review as a conference paper at ICLR 2021
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farkes Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-266622 26800, 2014.
Joe G Greener, Lewis Moffat, and David T Jones. Design of metalloproteins and novel protein folds using variational autoencoders. Scientific reports, 8(1):1-12, 2018.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola.
A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773: 2012.
Paulina Grnarova, Kfrr Y Levy, Aurelien Lucchi, Nathanaël Perraudin, Ian Goodfellow, Thomas Hofmann, and Andreas Krause. A domain agnostic measure for monitoring and evaluating gans. In Advances in Neural Information Processing Systems, pp. 12092-12102, 2019.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference Knowledge discovery and data mining, pp. 855-864, 2016.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp- 5767-5777, 2017.
Anvita Gupta and James Zou. Feedback gan for dna optimizes protein functions. Nature Machine Intelligence, 1(2):105-111, 2019.
Alex Hawkins-Hooker Florence Depardieu, Sebastien Baur, Guillaume Couairon, Arthur Chen, and David Bikard. Generating functional protein variants with variational autoencoders. BioRxiv, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626 0666-6637,22 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. International Conference on Learning Representations.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Toward controlled generation of text. In International Conference on Machine Learning, pp. 1587-1596, 2017.
Po-Ssu Huang, Scott E Boyken, and David Baker. The coming of age of de novo protein design. Nature, 537(7620):30-322)) 2016.
Frank Hutter, Holger Hoos, and Kevin Leyton- Brown. An efficient approach for assessing hyper- parameter importance. In International conference on machine learning, pp. 754-762. PMLR, 2014.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. In Advances in Neural Information Processing Systems, pp. 15820-15831, 2019.
Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Artificial Intelligence and Statistics, pp. 240-248, 2016.
Mostafa Karimi, Shaowen Zhu, Yue Cao, and Yang Shen. De novo protein design for novel folds using guided conditional wasserstein generative adversarial networks (gewgan). bioRxiv, pp. 769919, 2019.
Nathan Killoran, Leo J Lee, Andrew Delong, David Duvenaud, and Brendan Frey. Generating and designing dna with deep generative models. arXiv preprint arXiv:1712.06144 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.

---

Under review as a conference paper at ICLR 2021
Diederik P. Kingma and Max Welling. Auto Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, 2014.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, pp. 3927-3936, 2019.
Christina Leslie, Eleazar Eskin, and William Stafford Noble. The spectrum kemnl:: A string kernel for svm protein classification. In Biocomputing 2002, pp. 564-575. World Scientific, 2001.
Chun-Liang Li, Wei-Cheng Chang. Yu Cheng, Yiming Yang, and Barnabás P6ezos. Mmd gan: Towards deeper understanding of moment matching network. In Proceedings of the 31st Interna- tional Conference on Neural Information Processing Systems, pp. 2200-2210, 2017a.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765-6811 16, 2017b.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International Conference Machine Learning, pp-
2015.
Ali Madani. Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004. 03497, 2020.
Peter Markiewicz, Lynn G Kleina, Christina Cruz, Susannah Ehret, and Jeffrey H Miller. Genetic studies of the Lac repressor. XIV. Analysis of 4000 altered Escherichia coli Lac repressors reveals essential and non non-essential residues, as well as" spacers" which do not require a specific sequence. Journal of molecular biology, 240(5):421-433 1994.
Frank J Jr Massey. The kolmogorov-smirov test for goodness of fit. Journal of the American statistical Association, 46(253):68-777 1951.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 1411.111.. 2014.
Takeru Miyato and Masanori Koyama. egans with projection discriminator. International Confer- ence on Learning Representations, 2018.
Alex T Muller, Jan Hiss, and Gisbert Schneider. Recurrent neural network model for constructive peptide design. Journal of chemical information and modeling, 58(2):472-477 2018.
Pauline C Ng and Steven Henikoff. Predicting deleterious amino acid substitutions. Genome search, 11(5):863-874 2001.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier GANs. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2642-2651. JMLR. org, 2017.
Dan Ofer and Michal Linial. Profet: Feature engineering captures high-level protein functions. Bioinformaticx, 31(21):349-33336 2015.
Barrett O' neill. Elementary differential geometry. Academic press, 2014.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
A. Krogh R. Durbin, S. Eddy and G. Mitchinson. Biological sequence analysis. probabilistic models of proteins and nucleic acids. 1998.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, abs/151 1. 106434, 2015.
12

---

Under review as a conference paper at ICLR 2021
Predrag Radivojac, Wyatt T Clark, Tal Ronnen Oron, Alexandra M Schnoes, Tobias Wittkop, Artem Sokolov, Kiley Graim, Christopher Funk, Karin Verspoor, Asa Ben-Hur, et al. A large-scale evaluation of computational protein function prediction. Nature methods, 10(3):221-2111 2013.
Donatas Repecka, Vykintas Jauniskis, Laurynas Karpus, Elzbieta Rembeza, Jan Zrimec, Simona Poviloniene, Irmantas Rokaïtis, Audrius Laurynenas, Wissam Abuajwa, Otto Savolainen, et al. Expanding functional protein sequence space using generative adversarial networks. bioRxiv, pp. 789719, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. International Conference in Muchine Learning. 2014.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Ilan Samish. Computational protein design. Springer, 2017.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? In Proceedings of the European Conference Computer Vision (ECCV), pp. 213-229, 2018.
Dougal J Sutherland, Hsiao-Yi Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. arXiv preprint arXiv: /6/ 1,04488, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Ronghui You, Shuwei Yao, Yi Xiong, Xiaodi Huang, Fengzhu Sun, Hiroshi Mamitsuka, and Shan- feng Zhu. Netgo: improving large-scale protein function prediction with massive network infor- mation. Nucleic acids research,
13::339-WWW) 2019.
Naihui Zhou, Yuxiang Jiang, Timothy R Bergquist, Alexandra J Lee, Balint Z Kacsoh, Alex W Crocker, Kimberley Lewis, George Georghiou, Huy N Nguyen, Md Nafiz Hamid, al. The cafa challenge reports improved protein function prediction and new functional annotations for hundreds of genes through experimental screens. Genome biology, 20(1):1-23, 2019.
13

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A. BACKGROUND: BIOLOGICAL. MECHANISMS UNDERLYING PROTEIN FUNCTIONS
Proteins are complex biological structures but can be simply represented as chains of amino-acids, 20-character alphabet. While this is useful for modeling approaches, it hides the functional complex- ity of proteins. fact, amino acids, with their diverse ahysico-chemica properties, fold and assem- ble into complex three dimensional constructs at local and global level, giving rise to the overall protein structure. In turn, the structure of the protein is responsible for its function, where shape and clectrochemical properties dictate the behavior and biological activity of the macromolecule. The link between sequence and function is therefore highly complex and still not understood by the research community. The design of proteins has consequently been far mainly based rela- tively uninformed trial-and-error processes with slight random alterations of the protein sequence and subsequent assay of functionality (Arnold (1988) entail computationally heavy simulations of molecular dynamics (Samishl 2017). However, advances in machine learning have enabled the development of novel in silico design methods.
A.2 EXPERIMENTAL SETUP
In this section, describe in detail the dataset and preprocessing steps, the basclines and the hy- perparameter searches performed.
A.2.1 DATA
Sequence data is acquired from the UniProt Knowledgebase (UniProtKB, Consortium (219)) ). The database contains more than 180 million protein sequences with rich annotations such as structure and function. Nonetheless, most of these entries are only automatically annotated. To ensure high quality data for our model, we filter for sequences that are manually curated and have experimental evidence of some form. There are also some specialized proteins that have very long sequences, we only keep the sequences whose length is not exceeding 2048 amino acids, which covers ca. 98.5% of the data points. The resulting dataset contains 149, 390 sequences. The cut-off at 2048 amino-acids is multiple times longer than other approaches in this field, which is between 30- to 500-long (Das) et al. 2018 Davidsen et al., 2019: Greener et al. 2018 Repecka et al. 2019), and allows for a more complete model of the known sequence space.
Functional labels are collected from the same database. The gene ontology (GO) resource is com- posed of three branches, molecular function, cellular component and biological process. We focus on the molecular function ontology, which contains thousands of terms ranging from description like binding (GO:0005488) to very specific terms such as microtubule-sevvring ATPase activity (GO:0008568). Each protein is annotated with a set of GO labels describing the molecular func- tion of protein in modular way. The ontology is structured as a directed acyclic graph with a single root. Further, labels have hierarchical relationship. i.e. protein with given functional label in- herits automatically the labels of its parents in the DAG (is-a relationship). The molecular function ontology resource currently contains more than ten thousand labels, many of which have highly specific meaning and only few representatives. We therefore restrict the number of labels to the 50 largest classes, the smallest class containing 8659 proteins. We argue fifty labels is sufficient for a proof-of- -friinccileee and would even enable the design of experimental assays for validation. Figure AT illustrates the selected subset of labels and their relationships.
Train, validation and test splits were created to preferably represent all labels uniformly in the test and evaluation sets. We use 80-10-10 split for evaluation in the main body, and a 94-2-2 split for hyperparameter optimization and the results detailed in the supplement. For the validation and test set, we randomly sample sequences until there is at least 1.300 (300 in the optimization) sequences per class. The selections of hyperparameters by the BOHB hyperparameter optimizations for ProteoGAN and by the hyperparameter searches for the baselines are done on the validation set, while the results presented in the main text were acquired on the test set. For sequence sample generation, the model was conditioned on the label combinations of the evaluation/test set and the resulting sequences then compared with the respective set by MMD and MRR.
14

---

Under review as a conference paper at ICLR 2021
Figure A1: GO DAG of the labels selected for this project.
A.2.2 BASELINES
We implement four baselines to put the performance of our model into perspective. In this section, would like to give additional details concerning the two baselines that gathered from the literature.
PepCVAE (Das al. 2018) a VAE framework with single layer gated recurrent unit (GRU) RNN for both the encoder and the decoder, in a semi-supervised setting. Conditioning is performed by concatenating the encoded target labels to the latent code. In the paper, the sequences of interest are antimicrobial peptides, with a maximum length of thirty amino-acids. The conditioned on label was binary, i.e. antimicrobial activity not. The RNN component of the model highly resource- consuming, therefore we had to trim protein sequences to the first 32 amino-acids to run the model on our data. For fair assessment, we also run our model ProteoGAN on sequences of 32 amino- acids. We modify PepCVAE by introducing multiplying factor in front the the KL term of the ELBO loss as suggested by Higgins et al., to increase the stability the model. We optimise the model with Bayesian Optimization hyperparameter search, for which tried 1, 000 combinations of hyperparameters. We do not use BOHB because the early stopping would interfere with the different training phases of the model. The hyperparameters and their value ranges, well as the final model configuration can be found in Table [1] We refer the reader to Das et al. (2088) Hu al. (201)) and Bowman et al. (2016) for more information on the model.
CVAE (Greener et al.) 2018) uses conditional VAE (CVAE) in order to generate either metallo- proteins with desired metallo-binding sites or fold properties. In the of fold properties, the authors introduce iterative sampling and guidance steps in the latent space. The decoder and en- coder are both MLPs and the number of layers is chosen with hyperparameter search. Here also, we introduced a KL-balancing term to stabilize training. As for PepCVAE, the model presents loss scheduling scheme and therefore we could not use the BOHB optimization. However, we per- formed Bayesian Optimization hyperparameter search, for which we tried 1, 000 combinations of hyperparameters. Notably, we allowed for an optimization of network architecture by optimizing over the layer numbers for both encoder and decoder, and by optimizing the number of units in the first layer of the encoder and the last layer of the decoder. The unit number then halved towards the latent space with each layer. The hyperparameters and their value ranges, as well the final model configuration can be found in Table A2 We refer the reader to Greener et al. (0188) for more information on the model.
15

---

Under review as a conference paper at ICLR 2021
Table Al: PepCVAE hyperpramttssr subject to BO optimization.
Name
Values
Final Value
Learning rate
[1e-5,1e-2]
4e-3
Pretrain iterations
[1,5000]
3181
Latent dimension
[10,1000]*
101
Word dropout keep rate
[0.1]
0.43
Classifier loss balancing
[le-3,100]
9.9e-3
Latent loss balancing
Az
[1e-3,100]
1.9e-1
KL. balancing
B
[1e-3,100]
1.5e-2
Values were sampled logarithmic scale.
Table A2: CVAE of Greener et al. hyperparameters subject to BO optimization.
Name
Values
Final Value
Learning rate
[1e-5.1e-2]
7.8e-4
Pretrain start
[1,5000]
2598
Pretrain end
[1,5000]
1251
Latent dimension
[10,1000]*
761
KL balancing
B
[1e-3,100]
1.1e-3
Encoder layer number
[1,5]
3
Decoder layer number
[1,5]
1
(onno(Ee first layer units)
[4,10]
Log2 (Decoder last layer units)
[4,10]
9
Values were sampled on a logarithmic scale.
The HMM baselines were implemented based on HMMER. For HMM (on label set), all sequences in the training dataset containing specific label combination were aggregated, for each label set of the test set. For HMMM (on single label), all sequences in the training dataset containing spe- cific label were aggregated, for each the 50 labels. The resulting sequence sets were aligned with MAFFT (with parameters --retree 1 --maxiterate 0 --ep C. 023).. Because of the time-intense multiple sequence alignment the sequences sets were randomly sampled to have maximum size of 5000 sequences. From the alignment, profilHHMM was built with HMMER which was then sampled to generate sequence.
In the n-gram baseline also, sequences were selected according to label and single labels. Here the full data was used. n was set to 3. The sequence lengths were sampled from the training data length distribution.
The Predictor Guided baseline was a variant of ProteoGAN without conditioning mechanism trained the whole data. A sample generated like in the other models, however the labels were annotated by sampling the per-label probabilities outputted by the NetGO protein function predictor.
Table A3: Evaluation of ProteoGAN and PepCVAE truncated sequences with length 32. Shown are mean and standard deviation of five different data splits. The arrows indicate that lower (() or higher (1) is better.
Model
MMD
Gaussian MMDJ
MRRI
PepCVAE (L=32)
0.122 + 0.019
0.077 +: 0.012
0.139 +: 0.022
ProteoGAN (L=32)
0.033 + 0.002
0.022 0.001
0.321 + 0.029
16

---

Under review as a conference paper at ICLR 2021
0.6
KS Statistic
0.4
0.2
0.0
ProteoGAN (L=32)
PepCVAE (L=32)
Figure A2: Sequence feature analysis of the models trained on truncated data (Sequence length 32). KS statisties over 500 ProFET features, lower is better.
A.2.3 HYPERPARAMETER SEARCH
Description of the BOHB:
For ProteoGAN, we conducted hyperparameter searches with the Bayesian Optimization and HyperBand (BOHB) algorithm. The Hyperband (Li al. 2017b) al- gorithm uses successive halving (Jamieson & Talwalkar 2016) to evaluate number of models given budget of resources. The better half of the models are then evaluated on twice the budget, et cetera. Hyperband is independent optimization algorithm that has been combined with Bayesian optimization to form Bayesian optimization and Hyperband (BOHB) (Falkner et al. 2018), the op- timization strategy used in this project.
We conducted two BOHB optimizations. For both we evaluated 1, 000 models. All networks were trained with the Adam optimizer (Kingma & Ba 2015) with 31 and B2 = 0.9 (following (Gulrajani et al. 2017)). ). The optimization consisted first of broad search among 23 hyperparametters and second, of smaller and more specific search, among 9 selected typerparameters. For the first BOHB optimization, an optimization iteration was defined as two epochs which we found through pilot experiments was the minimum time to observe a viable trend in the metrics. The parameters R and 7 (in the notation (Li et al., (017bb) were set to 9 and respectively, which allowed for maximum training time of 18 epochs (22.5K gradient updates). The optimization objective was to maximize, in the validation set, the ratio of metrics MMD/MRR which are introduced the main document Section 3.2.T During the optimization, BOHB selected the models based on evaluations at the end of training period. For the second optimization, reduced the number of hyperparameters to only 9. We selected values for the other hyperparameters based the analysis of the hyperparameter importance of the first optimization (see paragraph below). The hyperparameters that showed either no importance or that were detrimental to training were removed. For this second optimization, the smaller network size allowed for 3 epochs per iteration, resulting in maximum training time of 27 epochs (1.2K gradient updates). The list of hyperparameters of the two BOHB optimizations and their ranges presented in Table A4 The parameters of the best models selected by the two BOHB optimizations are presented Table A5
Hy Hyperparameter optimization with BOHB:
Quantification of hyperparameter importance: After the optimization, we analyze hyperparam- eter importance with the approach presented in (Hutter et al. 2014). A surrogate model (random forest) is trained on the parameter configurations and the respective evaluation scores. This enables functional analysis of variance (fANOVA) which allows for a quantification of hyperparameter importance in terms the variance they explain. It also provides marginal predictions for each hyperparameter which gives insights about their optimal value setting. For the random forest, used 1, 0000 trees with maximum depth of 64, and repeat the estimation 100 times. We do so for all evaluated models of the first and second BOHB optimizations. The hyperparameter importances obtained from the first optimization (and resp. second optimization) are presented Figure [A (resp. Figure 2 The first fANOVA showed that parameters related the discriminator (learning
17

---

Under review as a conference paper at ICLR 2021
Table A4: Hyperparameters subject to BOHB optimization.
Name
Symbol
Values
Use chemophysical properties
Yes, No, only
Label embedding
one-hot, node2vee, Poincaré
Conditioning mechanism
projection, AC, both
AC weighting factor
2
[1, 1000]
Label smoothing factor
[0, 0.5]
Latent noise dimension
dz
[1, 1000]
Input noise standard deviation
a
[0, 1]
Generator learning rate
GG
[le-5. 1c-2]
Generator learning rate 2
7G2
[le-5, 1e-2]
Discriminator learning rate
?D
[le-5, 1c-2]
Discriminator learning rate 2 Training ratio
QD22
[le-5, 1e-2]
Peritic
[1, 10]
Learning rate schedule
constant, cosine, exponential
Schedule interval (in epochs)
[1, 18]
Generator layer number
ng
[1, 10]
Discriminator layer number
nD
[1, 10] 10]
Strides
8
1, 2, 4, 8
Filter size
[3, 12]
Generator skip
hg
[0, 10]
Diseriminator skip
hD
[0, 10]
Number of projections
np
[1.5]
Output source layers
os
[1.3]
Output label layers
oL
[1,3]
Values were sampled on a logarithmic scale. AC auxiliary classifier.
Table A5: Hyperparameters found in the first and second BOHB optimization. Values with an
asterisk indicate the preset configurations in the second optimization.
Name
First
Second
Use chemophysical properties
Yes
No
Label embedding
one-hot
one-hot
Conditioning mechanism
both
both
AC weighting factor
178
135
Label smoothing factor
0.28
_*
Latent noise dimension
91
100*
Input noise standard deviation
0.29
_
Generator learning rate
2.0e-3
4.1e-4
Generator learning rate 2
Discriminator learning rate
8.5e-4
4.0e-4
Discriminator learning rate 2 Training ratio
1
Learning rate schedule
constant
constant*
Schedule interval (in epochs)
Generator layer number Discriminator layer number
2
2
3
2
Strides
4
Filter size
8
12
Generator skip
Discriminator skip
.
Number of projections
1
2
Output source layers
Output label layers
2
1
AC auxiliary classifier.
18

---

Under review as a conference paper at ICLR 2021
rate, number of layers) are most important for model performance and helped to select poten- tially important hyperparameters for the second analysis. Noticeably, the best model of the first optimization already well-performing model but chose to second optimization to better understand the role of key hyperparameters, to gain insight in potential good practice when designing conditional generative adversarial networks and to further improve the performance of our model. The second YANOVA clarified the importance of the remaining hyperparameters, such as use of chemophysical features and label embeddings among others (Figure 2).
We also show marginal predictions for hyperparameters of the first optimization in Figure A6 and for the second optimization in Figure A3 and Figure A4
Obtainment of the final model: The 27 best selected models of the second hyperparameter search were then trained for prolonged duration of 100 epochs, where the conditioning mechanism and an associated weighing factor became most important, according to the last fANONA study (Fig- (r4 We evaluated twice per epoch and selected the weights the final model the checkpoint that showed the best (smallest) ratio MMD/MRR in the validation set. The final model, ProteoGAN, convolutional conditional generative adversarial network, with two conditioning mechanisms: an auxiliary classifier and projections. The dimensions of the convolutional layers are following the pyramidal architecture of DCGAN (Radford et al., 2015), i.c. with increasing output length and decreasing filter depth for the generator, and vice versa for the discriminator. The other hyperpa- rameters are presented Table A5
'Some other important factors were learning rate manedule-related parameters such as Generator learning rate 2 or schedule. We realized that these were detrimental to model performance as the short duration of training in the optimization did not allow to estimate long term effects seen in the selected models that were trained for 100 epochs.
19

---

Under review as a conference paper at ICLR 2021
M00
0.20
Deedeee ven
1000 OPE cathou
019-
5.7-
0.3
0.36-
2.300
944-
2.40
1310
034
0.2
bath
0,20
1.00
0.066
0008
2006
0025
(a) Auxiliary Classifier weighing
factor
(b) Conditioning mechanism
(c) Discriminator learning rate
030
0.40
0.10
MAR
Fwo0k0dd MDD
0.00
0.1
0.5
815
9:56
0.14
0.22
0.3
535
9.80
6002
0.004
6608
0.000
0013
10
unshot
rediOees
pohoerrs
(d) Generator learning rate
Kernel size
Label embedding
0.44
Dredgted NM0
0-000-
c1500
Prokktdd MD
0.50
0.42
3:40
63525
0.00
20
0.66
-m-
0.000
01/50
2.24
01455
039-
0.200
030
(g) Projections
(h) Strides
Use chemophysical features
Figure A3: Marginal predictions of hyperparameters based optimization data in the second op- timization. Predictions were obtained training on MMD and MRR. Note that for MMD, lower is better.
20

---

Under review as a conference paper at ICLR 2021
1.900
4M0
316
Predcred WiCc
1.055
100 OPE Setic
0.000
1904
HPU
M5X
051
0.9
0.00-
0.00
3856
0.00
0.4
9050
046
0640
3040
proection
0003
0022
0054
0.008
od8s
0.30
(a) Auxiliary Classifier weighing
factor
(b) Conditioning mechanism
(c) Discriminator learning rate
0.00
9,000
44A
051
0.1
6.06
1000 MRD
8.006
0.50
0.50
6.54
3.064
1ore
0000
3.500
0.44
0800
0352
0094
0838
0.08
0.02
20
pohicete
(d) Generator learning rate
Kernel size
Label embedding
MVD
1.50
0.56-
0/56-
(N606de MND
5.04
0.864
0.866
0.000
0.00
0.77
3.000
0.47
0048
DA6
9140
0.40
gg) Projections
(h) Strides
Use chemophysical features
Figure A4: Marginal predictions of hyperparameters based the data of the 27 best selected models the second optimization. Predictions were obtained training on MMD and MRR. Note that for
MMD, lower is better.
21

---

Under review as a conference paper at ICLR 2021
All Trials
Selection
auxilliary class fier weigh ng factor
conditioning mechanism discriminator layer number
discriminator learning
discriminetor learning rete
discriminator skip
generator layer numbar
gererator learning rate-
generator leamning
generator skip
input noise
kornel size
label embedding
label smoothing
critic
output label layers
output source layers
projections
schedule
schedule interval
strides
use chemoolysical features
dim-
0.000
0.025 Importance
0.0
0.1 Importance
Figure A5: Hyperparameter importance for the first BOHB optimization. Shown are all hyperpa- rameters subject to optimization for all models (left), and manual selection of models that was trained for 100 epochs (right).
A.3 METHOOS
In this section, we first describe in details the state-of-the-ar conditioning mechanisms and con- structed variants that used in this project. Second, we introduce three variants of MRR suited for hierarchically-struettii labels. Finally, we assess proposed evaluation measures cGANs by estimating empirical worst and best bounds for our experimental setting.
A.3.1 CONDITIONING MECHANISM
In this section, we first detail how we adapted the Wasserstein loss to the conditional setting, then describe state-of-the-an th--rar conditional GANs' objective functions and variants used in this project.
Loss function of conditional GANs: Our models are trained with the Wasserstein objective with gradient penalty from (Gulajjani al. 2017). As a reminder, the WGAN-GP losses can be written as follows:
Cp Ey(z) [xx)] [D())
+ XBm(()) (2) [(] V2 D2 - 1)
(3)
La -
where x p(x) is the data distribution and x ~ q(x) is the generator model distribution, è is an interpolated sample between real sequence and generated one, is the distribution of interpo- lated samples, D is the discriminator (or critic), Lo the loss of the discriminator and LG the loss of the generator. The [(I()](I V[ D(+) 1) ensures that the discriminator is Lipschitz continuous.
To be able use the Wasserstein objective with gradient penalty in the conditional setting of projec- tion cGAN (Miyato & Koyama 2018) (see below), we had to adapt the objective formula to include the label information. Let (xt ~ p be sample from the dataset, where the sequence and the label. Let D be the discriminator and G' the generator. Let g be the generator model distribution. such that - 4(y) is defined by the user and z - q (y) is learned. In practice, g(y) follows the label distribution of the data p Let & be interpolated sequence between real sequence and generated one. We call & + m m the distribution of interpolated sequences given label y. Let be weighing factor introduced in (Gulrajani et al. 2017). Taking conditional information into
22

---

Under review as a conference paper at ICLR 2021
016
310-
450
017
10 DM TENNA
0.60
935
1.15
Preskood
01A
0.2
105
100
0.10
340
1:00
200e
(a) Auxiliary Classifier weighing
factor
(h) Discriminator layer number
(c) Discriminator learning rate
3345
1.50
MND
1140
250-
$100
056
10100
052
0.44
640
050
0.3
052
2.0
1.20
612
250
(d) Generator layer number
(c) Input noise
(l) Training ratio
MM0
1340
PM0
M00
0.10
0.7
beettee WDD
hootcod NDD
2330
9.6
0.5
1.00
56-
055
015
2.4
1.10
2.20
0.10
951
Thue
107
(g) Strides
(h) Use chemophysical features
(i) Latent dimensionality
0.20
M40
0.00
6m
Auxiliary Classifier weighing
factor (Selected)
Figure A6: Marginal predictions of hyperparameters based data in the first optimization. We show some selected predictions that allowed for interpretation, all others were inconclusive. not otherwise noted, data comes from all trials in the optimization. Predictions were obtained training on MMD and MRR. Note that for MMD, lower is better.
23

---

Under review as a conference paper at ICLR 2021
account, the discriminator and generator losses can be expressed follows: LD Eg( ( (D y)ll Eptn) (pp()) (D(r, [D(x
+ p Em (11 D(ē y)112 - 1)
(4)
Lg = - B g (xx y)]l-
This formulation ensures that the Lipschitz constraints imposed on the discriminator in the uncon- ditional WGAN-GP objective holds for each class.
Case of the projection cGAN model (Miyato & Koyama, 2018): In the conditional GAN with projection discriminator model, the discriminator decomposed into a sum two terms, one being the inner product between a label embedding and an intermediate transformation of the input, and the second term being solely depending on the input æ. The new expression of the projection dis- criminator can be derived by assuming that the label is categorical and that both the log-likelihoods of the data and target distribution can be written as log linear models. Let y - v(y) be a linear pro- jection of the label into label embedding. Let Φo be vector output function applied the input and th, a scalar function applied to the vector output function Φu (x), Let A be an activation function of choice. The projection discriminator in (Miyato Koyama 2018) can therefore be written as:
D p) = A(f(xx))))
= A(v(()) (x)() + tby (o)(())
The label information is therefore introduced via an inner-prode inner-product. In practice, the discriminator equipped with projection layer that takes the inner product between the embedded label and an intermediate output of the discriminator. This formulation leads to a more stable algorithm compared simple concatenation of the label with the input, potentially thanks the introduction of form of regularization on the discriminator.
In this project, we also tested the possibility to include several projections in the discriminator. In addition to the previous notations of this paragraph, let assume that we have k projections. Let { be l neural networks, which can be decomposed n, layers y; = 15, 1h o ..... -1 li.
Let {px ) be the layer number at which the inner product with the output of the projection (v, } occurs in each neural network. The expression of the discriminator given by:
D(x, y) = A(f(z,))
&
(o. O =
o- (z( + (x))))
In practice we share lower layer parameters and allow for up to four projections. Our BOHB hyper- parameter searches did not show evidence of the superiority of projection mechanisms for condition- ing purposes when they are the unique type of conditional mechanism in the network. However, the projection models were able to generate sequences similar to naturally occurring ones (low MMD).
Case of the eGAN model with auxiliary classifier (Odena et al., 2017): As opposed to projection cGANs, cGANs with auxiliary classifier add term to the generator and discriminator losses to incorporate the log- likelihood of the correct labels (compare Equation 4 In addition to notations introduced for Equation 4 let Cp be the auxiliary classifier, ce the cross entropy and a weighting factor. The loss function of cGANs with auxiliary classifiers can be written as:
CD = Ey(=)) [Dx)] - Bp(r, (xx)
AlE [() V (=))l 1) + (et (CC(() La - Euyas) [xx)] + >)E [ (ce(CD(z), v)]l-
On typically shares weights with and is trained when minimising Ln but is fixed when minimis- ing Le.
In our work, we compare both types of conditional GANs (GAN equiped with auxiliary classifier or with multiple projections at several layers (see Equation ()) to a third proposed model that com- bines both mechanisms. It is important to note that in this case the label information introduced in the projection may not be shared with the auxiliary classifier (compare Figure I). The fANOVA analysis performed on the second BOHB optimization results shows that the combination of both mechanisms helps to obtain better performing conditioning mechanism, measured by MRR.
24

---

Under review as a conference paper at ICLR 2021
A.3.2 EVALUATION MEASURES FOR CONDITIONAL GENERATIVE MODELS IN THE CASE OF HIERARCHICAL LABELS
In addition to the evaluation measures MMD and MRR described in the main document Sec- tion [.2.1] we built three variants of MRR in order to better characterise the effectiveness of the conditional generation and its ability to handle hierarchical multi-label settings. We look sub- measures of MRR where either all parent nodes (M)Rp), all child nodes (MRRc), or both (MRRB) are ignored in the ranking of a conditional MMD term. Removing parent or child terms attempts at understanding to which degree the conditioning mechanism leads to severe off-target generation of sequences of unrelated labels. Indeed, these three alternative measures do not penalize if the gen- erated distribution of sequences for given label is closer to either parents' or children's 's sequences than it is its target's sequences, which is less severe than off-target generation of sequences of an unrelated label. Therefore, getting good MRRx values would indicate that our model is able to conditionally generate sequences up to closely related (parent's or child's) s) functions. Additionally, comparing MRRx values between themselves and with MRR would give some insight in closely- related conditional generation performance. For example, a MRR (resp. MRRc) value much larger than MRR could indicate that sequences that are generated with a target function are often closer to the natural sequences exhibiting the parent (resp. child) functional label, i.e. the model is too general, or too specific, respectively.
A.3.3 ASSESSMENT OF THE EVALUATION MEASURES FOR CGANS
We assess the quality of the proposed evaluation measures by constructing "best case" and "worst case" scenarios, to understand what would represent a perfect success or failure mode of our model. We consider as "best case" the case where the generative model generates sequences that are ob- served. The "worst case" scenario differs depending on the evaluation measure.
Scenarios for MMD: MMD has theoretical bounds of [0, v2) if the sequences in both sets are self- similar and totally dissimilar from each other. As we aim to compare real sequences to generated sequences that resemble real sequences, we therefore fix one set to be a collection of n natural protein sequences and the second set be other natural protein sequences modified with different percentages of random noise. In practice, the set of natural sequences is the test set used to report the results in the main document, and the random noise is injected in the form of single-point mutations to sequences of the second set. The results are reported in Table A6 and indicate that MMD is a proxy for the quality of the generated sequences. We observe that MMD increases with the amount of noise injected in the sequences of the second set. The generation of close to constant sequences a plausible failure mode of the GAN and would lead to a very high MMD value (last row). The lengths of the sequences of the mutated set were conserved, however we also report MMD with respect to fully random sequences of maximum length. The MMD value between two sets of real sequences is around 0.0237, adding 1 1% of noise to the sequences in one of the set leads to an MMD value of 0.0240, 10% of noise to 0.0324 and 20% of noise to 0.0484. In comparison, in biology, proteins have been shown to be viable up to 30- 60% of mutations in the amino-acids of their sequences (Repecka et al. 2019; Markiewicz et al. 1994| Ng & Henikoff] 2001). We also report empirical p-values following |Borwwardt et al. (2006). under the null hypothesis that the two sets are from the same distribution. These were obtained by ranking the original MMD statistic in 1000 iterations of statistics where the aggregated sequences were randomly assigned to each of the two sets.
Scenarios for MRR: Since MRR is conditional measure, we constructed the "worst case" sample as a set of natural protein sequences with randomized label assignments. This aims to simulate a generative model that produces well-formed sequences, but ignores the conditioning objective. One could also construct a scenario that simulates an antagonistic model that actively assigns wrong labels, instead of random ones. This will likely not occur in practice, though. Table shows the MRR values for a real data sample and the same sample with randomized labels. The reference for MRR was again the test set and the evaluated sample an equally structured set ("Positivee Control", Table A7) where the label annotations were randomly shuffled among the sequences ("Negative Control". T Table A7). The MRR evaluates a set of sequences with respect to the 50 selected labels. We also look at the sub-measures of MRR where either all parent terms (MRRp), all child terms (MRRc). or both (MRR,) are ignored in the ranking of a term. This gives additional insights on how well the model works with respect to the up- and downstream labels in the GO DAG.
25

---

Under review as a conference paper at ICLR 2021
Sample
MMD
p-value
Dataset Sample
0.0237
0.1499
Dataset Sample 1 noise
0.0240
0.0370
Dataset Sample + 2% noise
0.0243
0.0050
Dataset Sample + 3% noise
0.0248
Dataset Sample + 5% noise
0,0262
0
Dataset Sample + 10% noise
0.0324
Dataset Sample + 20% noise
0.0484
0
Dataset Sample + 30% noise
0.0660
Dataset Sample 50% noise
0.1009
Dataset Sample + 100% noise
0.1788
100% noise (maximum length)
0.3044
Constant (all leucine)
1.0258
Table A6: MMD values with different percentage of mutations, p-values
Table A7: Best and worst case MRR, as well as model evaluations of the main text with the extended set of MRR measures. Positive Control simulates perfect model, Negative Control model that ignores conditional information.
Model
MRR
MRR
MRRc
MRR,
Positive Control
0.7887
0.8260
0.8520
0.8925
Negative Control
0.0909
0.1177
0.0923
0.1196
ProteoGAN (ours)
0.5956 0.0237
0.7018 = 0.0180
0.6494 + 0.0237
0.7588
0.0166
Unconditional
0.5219 + 0.0195
0.6089 + 0.0242
0.5729 + 0.0205
0.6643
0.0225
redictor-guidde
0.1071
0.1367
1262
0.1562
Greener et al.
0.3132 + 0.0161
0.3658 + 0.0157
0.3775 + 0.0154
0.4306
0.0150
PepCVAE (I=32)
0.1910 + 0.0137
0.2038 + 0.0156
0.2103 + 0.0195
0.2267
0.0194
ProteoGAN (L=32)
0.3159 0.0205
0.3464 + 0.0207
0.3532 + 0.0240
0.3892
0.0221
Results for MRR variants: Table shows the results for the MRR variants for some models. The results confirm that our model is better at conditional generation than the basclines, including the baseline that consists of training 50 unconditional models. The comparison between MRR variants suggests that proteins often resemble proteins in the target class. When this is not the case, proteins are often similar their parent class, which makes sense as the class is then more general. The small difference between the MRR value of our model and of the positive control indicates that the model rarely generates sequences that resemble proteins an unrelated class. Additionally, compared to the controls, the MRRc are relatively low compared to MRR, which suggests that the model does not tend to create more specific child labels, which would be detrimental in a biological application.
A.3.4 LOSSES AND REAL-TIME EVALUATION OF THE FINAL MODEL
The loss function of the final model presented in the main document, combining projection and auxiliary classifier, is shown Figure A7 We monitored the duality gap (red), for which we split the training data into adversary finding set and a test set of 1% of the train set each. The duality gap is well-behaved, with a fast convergence to 0, indicating that there is no mode collapse and suggesting that the samples of reasonable quality. Also, the evaluations of MMD and MRR can be seen during training (evaluated twice per epoch) which provides valuable information for model selection and early stopping.
A.4 RESULTS WITH SMALLER TESTSET SIZE
26

---

Under review as a conference paper at ICLR 2021
Table A8: Evaluation of ProteoGAN and various baselines with our proposed measures (MMD and MRR) and NetGO (Fmix) smaller testset testset (n=300, ca. 2% of the data). An arrow indicates that lower (L) or higher (f) is better. Given are mean values of n 10 (MMD/MRR) and n = max different random seeds for the latent variable of the model. Note that models marked with (L=32) have been trained and evaluated on a set of truncated sequences and are hence not directly comparable to the other values. Also, since without multi-label conditioning, the Unconditional
model was conditioned on different label sets as the other models and controls.
Model
MMD
MRR
Fmax 1
Positive Control
0.0237
0.7887 0.0909
0.7705
Negative Control
1.0258
0.3485
ProteoGAN (ours)
0.0463 + 0.0003
0.5956 + 0.0237
0.4178 0.0004
Unconditional
0.0380 + 0.0010
0.5219 + 0.0195
0.3050 + 0.0024
Predictor-guides
0.0428
0.1071
0.4776
Greener al.
0.1611 + 0.0012
0.3132 0.0161
0.4658 0.0020
PepCVAE (I=32)
0.1504 + 0.0054
0.1910
0.0138
0.4140
0.0003
ProteoGAN (L=32)
0.0372 + 0.0005
0.3160 + 0.0205
0.4147 + 0.0005
Generator Loss
Discriminator Loss
Duality Gap
Generator-AC Loss
Discriminator-AC Loss
MMD
Generator-W Loss
Discriminator-W Loss
MRR
0.6
100
M wmmmmmmn Mmn myy MM Mag Amedyihnn An
75
0.5
50
0.4
25
0.3
0
-25
0.2
50
0.1
75
0
50000
100000
150000
200000
250000
Parameter Update
Figure A7: Losses and evaluations at training time. W Wasserstein, AC Auxiliary Classifier
27

---

Under review as a conference paper at ICLR 2021
train test
train generated
1200
1000
800
counts
400
200
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
min. sq. euclidean distance between each sequence in the test / generated set and all sequences the training set
Figure A8: Distributions of pairwise distances kernel feature space between testset and the training set (red) and a generated set of ProteoGAN and the training set (green). It can be seen that the generated sequences are not closer the training set than the testset (which would indicate overfitting). Further the generated sequences are about as far, but not further, away from the training set than the testset.
28


Reviews:
[{"review_id": "Y9N-atH_YLx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Brief Summary: The authors describe a method of generating sequences from neural networks conditioned on sequence annotations using novel statistics about the distribution of sequences. However, the evaluation criteria are lacking, requiring additional work.\n\nPros: \n- I think using MMD to determine the similarity between the true and generated distributions was very interesting. I am also a big fan of the kmer kernel for the MMD test statistic.\n\n-I also think understanding mode collapse with the duality gap was a good idea.\n\n- I appreciated the hyperparameter optimization studies to understand which parameters were important to tune to get working.\n\nCons:\n- I am concerned by the repeated usage of \"valid protein sequences\" throughout the paper. I'm not sure if the authors intended to use the word \"functional\". The real test of \"valid\" would be experimental validation in the lab to determine if the sequences still performed the functions of the GO categories, but I understand that this is not a particularly viable option. However, to make such a claim, the authors could calculate various energy statistics from Rosetta, or other mutation effect predictors to determine if the sequences are likely to be functional.\n- I am particularly troubled by the statement in the paper: “With respect to general sequence quality, ProteoGAN reaches MMD values of 0.0463, which corresponds to roughly 20% of random mutations in a set of natural sequences (compare supplementary Table A5). In comparison, previous in vitro biological experiments showed that proteins with 34% (resp. 59%) of mutated positions compared to the wild-type were viable…”\nThis, in no way, proves that sequences generated by this method are valid or functional. A single mutation can abrogate protein function, let alone twenty.\n-I am also concerned about the train-test-validation splits in this paper: It seems like this was done randomly. This is not how to prove the method can generalize to new sequences. I worry that generated sequences in the training set can have homologues in the test set. Train/test splits using CATH or SCOP would be more appropriate.\n-Since there is no control for homology in this work, it would be interesting to just use HMMER generated sequences of a family in a given GO category to compare against (simplest baseline model).\n-In Figure 5, I don't understand where the \"Random\" values came from. Are you sampling from sequences of the same length or kmer composition? Truly random amino acids is not a useful comparison.\n- MRR is an odd statistic. Why not just BLAST, or an alignment-based approach? How does MRR compare to e-value, or closest in sequence ID?\n\nNeutral:\n- I am surprised the authors are only using manually annotated sequences for this task, or not leveraging some sort of pretraining. Homologous sequences could improve model performance.\n- As someone working in biotechnology, I think there is great interest in the refinement of molecular properties to highly-specific functions, and little interest in generating sequences generally from GO categories, which are much too broad.", "extra_scores": {}, "extra_text": {}}, {"review_id": "N25ji9eoUEi", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this manuscript, the authors present a conditional GAN for generating protein sequences given specified GO terms. They argue that this approach to conditional protein generation is more appropriate than sequence-based generation, because it gets directly at functional specification. At a high level, this is an interesting idea though it has already started to be explored by other works. The authors are correct that these works focus primarily on optimize a single function of interest. However, there doesn’t seem to be any specific reason that guided design approaches could not generalize to multiple criteria. Regardless, controlled generation of proteins with pre specified functions is certainly interesting.\n\nThat said, the work presented here is too preliminary with too many missing baselines, missing or poorly/confusingly described experiments, and the dataset is not fully described. No comparisons are made against HMMs or other typical autoregressive generative models. Furthermore, there appear to be critical problems with the experiments. Specific comments follow below (in no particular order).\n\n1.\tIf I understand correctly, the MMD method used by the authors compares k-mer distributions between generated and real sequences. They set the k-mer size to 3 (“The size of the k-mers was set to 3”). Therefore, MMD is only measuring whether the sampled 3-mer frequency matches the observed 3-mer frequency. This metric seems too simple and can’t capture complex dependencies that exist in protein sequences. In fact, a simple 3-gram sequence model can perfectly optimize this metric. A simple baseline model in which sequences are generated from a 3-gram model conditioned on the function labels would be informative here. No evidence is otherwise provided that MMD is a good measure of the quality of a generative model.\n2.\tThe GAN seems like overkill for this problem. Have the authors considered any autoregressive generative models that can be trained by maximum likelihood?\n3.\tGANs also have the problem that likelihoods cannot be computed which makes it difficult, maybe even impossible, to use them to rank candidates or otherwise as priors over sequence space. If one wanted to use this model in practice, how would I choose a set of sequences to synthesize and experimentally validate? Strictly by random sampling? Is there a way to choose the top-k most likely sequences? In practice, experimental throughput is not large enough to explore many random samples, which is why focused design approaches have received so much attention. Is there a way to resolve this with GAN-based models?\n4.\tTrain/val/test splits: the val and test splits are very small. Much smaller than typically used for evaluating ML models (only 2% for val and test). Also, there is no analysis of sequence complexity within these groups. Typically, models are only of interest if they can generalize to relatively distant sequence. How similar are the train/val/test distributions? \n5.\tTable 1 reports standard deviations (I assume. It isn’t stated what the error values are in the caption.) for random samples from the latent variables, but this doesn’t take into account variability due to model training or the small test set. I encourage the authors to report standard errors over multiple data splits instead.\n6.\tI propose a simple generative model: given some set of GO terms, retrieve proteins with those terms from the training set. Then, to generate new proteins I simply propose those sequences. As a second approach, I fit an HMM to those sequences and sample new sequences from the HMM. How does this compare to the proposed approach? These baselines would also reveal how easy the val/test sets are due to similarity to the training set. \n7.\tIn Table 1, “Positive Control” is described confusingly: “The reference for MRR was again the test set and the evaluated sample an equally structured set (”Positive Control”)” on p.23 is the only definition for positive control I can find. What is the positive control? \n8.\tThe primary use case for a model like this seems to be the ability to generate proteins with combinations of functions not found naturally. Otherwise, choosing a protein to start from is trivial: retrieve it from the database. Have the authors considered that use case?\n9.\tSequences lengths for the baseline models were limited to 32 amino acids whereas the proposed model has a maximum sequence length of 2048. Not only that, but only the first 32 amino acids were used? This does not seem like a fair comparison at all. Typical proteins are much longer than 32 amino acids, so it isn’t surprising that the baselines perform poorly. Furthermore, this limitation makes no sense. RNNs can easily be applied to protein sequences of length 2048.\n10.\tProteins can have multiple GO terms associated with them. How was that dealt with? Do these proteins occur multiple times in the dataset for each term? Are proteins generated conditioned only a single term or multiple terms simultaneously? \n11.\tAs far as controlled sequence generation is concerned, “Generative Models for Graph-Based Protein Design” - Ingraham 2019 seems highly relevant but is not mentioned.\n\nThings that would improve my rating:\n1.\tFully describe the train/val/test splits with analysis of sequence similarity between splits within functional categories. Increasing the size of the val and test splits is also a good idea.\n2.\tAdd reasonable baselines to the model evaluation and provide proof that MMD with k-mer size of 3 is a good measure of generative model quality. Baselines must be trained to generate complete sequences, rather than only the N-terminal 32 amino acids.\n3.\tProvide a more rigorous description of the model and experiments. A reader should, ideally, be able to understand the conditioning structure from the main text.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "E1j-baUn6_V", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript describes the development of a conditional GAN (ProteoGAN) for performing protein design.  The key idea is to design sequences while taking into account the hierarchical structure of the Gene Ontology.  Other contributions include the development of novel measures of assessment in this domain.\n\nThis is an ambitious piece of work.  Just describing everything that was done took a lot of space, and the authors have done a good job of relegating some of the important but less critical details to the supplement.\n\nOverall, the experimental design seems to be quite rigorous.  For example, it is nice to see significant time spent on hyperparameter optimization for the competing methods.\n\nOne of the main contributions of the paper is to propose a couple of performance measures that allow a set of designed proteins to be compared to a set of reference proteins. This is done using the MMD measure, based on a 3-mer spectrum kernel.  While the measures themselves seem reasonable, in order to propose a novel measure it seems that the paper should include some meta-evaluation of the measures themselves.  This could be achieved, e.g., by using 3D structure information to assess how well the proposed measures capture \"real\" similarity between proteins.  As it stands, the paper simply proposes the measures and then evaluates various protein design methods using those measures.\n\nThe general idea of taking into account the hierarchical structure of the GO is sensible, but not much effort seems to go into evaluating how well this worked.  Three embedding modes are mentioned briefly on p. 4, but the embeddings themselves are not discussed in the Results section, and there is no comparison to a variant of the proposed method that does not use this hierarchical information. \n\nThe results section also mentions, with respect to CVAE, that \"close inspection of the sequences and of the predicted chemophysical properties (Figure 5) of the sequences indicates that the sequences are not well-formed.\"  I can see from the figure that the sequences exhibit unusual characteristics, but more work should have been done to delve into this.\n\nMinor: In the second sentence of the abstract, I did not initially understand the meaning of the prhase \"or sequence fragments.\"  Based on the recurrence of this phrase on p. 2, I gather that the authors are talking about scenarios in which a protein is given a sequence fragment as the starting point for optimization, but this should be clarified.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CR5r2U5xBNC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper applied conditional generative adversarial networks (cGANs) on the task of protein sequence design with respect to GO molecular functions.\n\nOverall the contribution was based on empirical numerical experiments. It was good practice yet the contribution or the novelty seemed fairly limited.\n\nUnderstood that one challenge of evaluating protein sequence design method at a large scale is the high cost of wet lab experimentation. Yet it would be nice to provide case studies, either through small scale lab experiments or deep diving into example outputs of the proposed method.\n\nIn addition, the experiment setup specified that the number of labels was restricted to 50 most common CF; as a result, the selected CFs would all be very high level and hence lack of specificity. It’d be good to see how the propose method would generalize to more specific CF terms.\n\nLastly, GO is a directed graph; yet it’s unclear whether the authors used the 50 CF terms as independent labels or in a DAG fashion. It’d be nice to incorporate the correlation among labels into protein design.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to use conditional GANs to generate protein sequence with respect to GO molecular functions. The idea is nice. But the reviewers find that there are many things that are not clear. For example, some sentences, phrase, the model and experiments pointed by the reviewers that should be  rigorous described. The technical contribution is also limited. The author are encouraged to revise the paper according to the comments.
Decision: Reject