Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: In this manuscript, the authors present a conditional GAN for generating protein sequences given specified GO terms. They argue that this approach to conditional protein generation is more appropriate than sequence-based generation, because it gets directly at functional specification. At a high level, this is an interesting idea though it has already started to be explored by other works. The authors are correct that these works focus primarily on optimize a single function of interest. However, there doesn’t seem to be any specific reason that guided design approaches could not generalize to multiple criteria. Regardless, controlled generation of proteins with pre specified functions is certainly interesting.

That said, the work presented here is too preliminary with too many missing baselines, missing or poorly/confusingly described experiments, and the dataset is not fully described. No comparisons are made against HMMs or other typical autoregressive generative models. Furthermore, there appear to be critical problems with the experiments. Specific comments follow below (in no particular order).

1.	If I understand correctly, the MMD method used by the authors compares k-mer distributions between generated and real sequences. They set the k-mer size to 3 (“The size of the k-mers was set to 3”). Therefore, MMD is only measuring whether the sampled 3-mer frequency matches the observed 3-mer frequency. This metric seems too simple and can’t capture complex dependencies that exist in protein sequences. In fact, a simple 3-gram sequence model can perfectly optimize this metric. A simple baseline model in which sequences are generated from a 3-gram model conditioned on the function labels would be informative here. No evidence is otherwise provided that MMD is a good measure of the quality of a generative model.
2.	The GAN seems like overkill for this problem. Have the authors considered any autoregressive generative models that can be trained by maximum likelihood?
3.	GANs also have the problem that likelihoods cannot be computed which makes it difficult, maybe even impossible, to use them to rank candidates or otherwise as priors over sequence space. If one wanted to use this model in practice, how would I choose a set of sequences to synthesize and experimentally validate? Strictly by random sampling? Is there a way to choose the top-k most likely sequences? In practice, experimental throughput is not large enough to explore many random samples, which is why focused design approaches have received so much attention. Is there a way to resolve this with GAN-based models?
4.	Train/val/test splits: the val and test splits are very small. Much smaller than typically used for evaluating ML models (only 2% for val and test). Also, there is no analysis of sequence complexity within these groups. Typically, models are only of interest if they can generalize to relatively distant sequence. How similar are the train/val/test distributions? 
5.	Table 1 reports standard deviations (I assume. It isn’t stated what the error values are in the caption.) for random samples from the latent variables, but this doesn’t take into account variability due to model training or the small test set. I encourage the authors to report standard errors over multiple data splits instead.
6.	I propose a simple generative model: given some set of GO terms, retrieve proteins with those terms from the training set. Then, to generate new proteins I simply propose those sequences. As a second approach, I fit an HMM to those sequences and sample new sequences from the HMM. How does this compare to the proposed approach? These baselines would also reveal how easy the val/test sets are due to similarity to the training set. 
7.	In Table 1, “Positive Control” is described confusingly: “The reference for MRR was again the test set and the evaluated sample an equally structured set (”Positive Control”)” on p.23 is the only definition for positive control I can find. What is the positive control? 
8.	The primary use case for a model like this seems to be the ability to generate proteins with combinations of functions not found naturally. Otherwise, choosing a protein to start from is trivial: retrieve it from the database. Have the authors considered that use case?
9.	Sequences lengths for the baseline models were limited to 32 amino acids whereas the proposed model has a maximum sequence length of 2048. Not only that, but only the first 32 amino acids were used? This does not seem like a fair comparison at all. Typical proteins are much longer than 32 amino acids, so it isn’t surprising that the baselines perform poorly. Furthermore, this limitation makes no sense. RNNs can easily be applied to protein sequences of length 2048.
10.	Proteins can have multiple GO terms associated with them. How was that dealt with? Do these proteins occur multiple times in the dataset for each term? Are proteins generated conditioned only a single term or multiple terms simultaneously? 
11.	As far as controlled sequence generation is concerned, “Generative Models for Graph-Based Protein Design” - Ingraham 2019 seems highly relevant but is not mentioned.

Things that would improve my rating:
1.	Fully describe the train/val/test splits with analysis of sequence similarity between splits within functional categories. Increasing the size of the val and test splits is also a good idea.
2.	Add reasonable baselines to the model evaluation and provide proof that MMD with k-mer size of 3 is a good measure of generative model quality. Baselines must be trained to generate complete sequences, rather than only the N-terminal 32 amino acids.
3.	Provide a more rigorous description of the model and experiments. A reader should, ideally, be able to understand the conditioning structure from the main text.

review_title: Reasonable idea, but experiments are unconvincing

final_score: rating: 7: Good paper, accept
review_content: This manuscript describes the development of a conditional GAN (ProteoGAN) for performing protein design.  The key idea is to design sequences while taking into account the hierarchical structure of the Gene Ontology.  Other contributions include the development of novel measures of assessment in this domain.

This is an ambitious piece of work.  Just describing everything that was done took a lot of space, and the authors have done a good job of relegating some of the important but less critical details to the supplement.

Overall, the experimental design seems to be quite rigorous.  For example, it is nice to see significant time spent on hyperparameter optimization for the competing methods.

One of the main contributions of the paper is to propose a couple of performance measures that allow a set of designed proteins to be compared to a set of reference proteins. This is done using the MMD measure, based on a 3-mer spectrum kernel.  While the measures themselves seem reasonable, in order to propose a novel measure it seems that the paper should include some meta-evaluation of the measures themselves.  This could be achieved, e.g., by using 3D structure information to assess how well the proposed measures capture "real" similarity between proteins.  As it stands, the paper simply proposes the measures and then evaluates various protein design methods using those measures.

The general idea of taking into account the hierarchical structure of the GO is sensible, but not much effort seems to go into evaluating how well this worked.  Three embedding modes are mentioned briefly on p. 4, but the embeddings themselves are not discussed in the Results section, and there is no comparison to a variant of the proposed method that does not use this hierarchical information. 

The results section also mentions, with respect to CVAE, that "close inspection of the sequences and of the predicted chemophysical properties (Figure 5) of the sequences indicates that the sequences are not well-formed."  I can see from the figure that the sequences exhibit unusual characteristics, but more work should have been done to delve into this.

Minor: In the second sentence of the abstract, I did not initially understand the meaning of the prhase "or sequence fragments."  Based on the recurrence of this phrase on p. 2, I gather that the authors are talking about scenarios in which a protein is given a sequence fragment as the starting point for optimization, but this should be clarified.


review_title: This is a well executed protein design project with several good ideas but also several important missing pieces

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper applied conditional generative adversarial networks (cGANs) on the task of protein sequence design with respect to GO molecular functions.

Overall the contribution was based on empirical numerical experiments. It was good practice yet the contribution or the novelty seemed fairly limited.

Understood that one challenge of evaluating protein sequence design method at a large scale is the high cost of wet lab experimentation. Yet it would be nice to provide case studies, either through small scale lab experiments or deep diving into example outputs of the proposed method.

In addition, the experiment setup specified that the number of labels was restricted to 50 most common CF; as a result, the selected CFs would all be very high level and hence lack of specificity. It’d be good to see how the propose method would generalize to more specific CF terms.

Lastly, GO is a directed graph; yet it’s unclear whether the authors used the 50 CF terms as independent labels or in a DAG fashion. It’d be nice to incorporate the correlation among labels into protein design.
review_title: Review for "Conditional Generative Modeling for De Novo Hierarchical Multi-Label Functional Protein Design"

final_score: rating: 3: Clear rejection
review_content: Brief Summary: The authors describe a method of generating sequences from neural networks conditioned on sequence annotations using novel statistics about the distribution of sequences. However, the evaluation criteria are lacking, requiring additional work.

Pros: 
- I think using MMD to determine the similarity between the true and generated distributions was very interesting. I am also a big fan of the kmer kernel for the MMD test statistic.

-I also think understanding mode collapse with the duality gap was a good idea.

- I appreciated the hyperparameter optimization studies to understand which parameters were important to tune to get working.

Cons:
- I am concerned by the repeated usage of "valid protein sequences" throughout the paper. I'm not sure if the authors intended to use the word "functional". The real test of "valid" would be experimental validation in the lab to determine if the sequences still performed the functions of the GO categories, but I understand that this is not a particularly viable option. However, to make such a claim, the authors could calculate various energy statistics from Rosetta, or other mutation effect predictors to determine if the sequences are likely to be functional.
- I am particularly troubled by the statement in the paper: “With respect to general sequence quality, ProteoGAN reaches MMD values of 0.0463, which corresponds to roughly 20% of random mutations in a set of natural sequences (compare supplementary Table A5). In comparison, previous in vitro biological experiments showed that proteins with 34% (resp. 59%) of mutated positions compared to the wild-type were viable…”
This, in no way, proves that sequences generated by this method are valid or functional. A single mutation can abrogate protein function, let alone twenty.
-I am also concerned about the train-test-validation splits in this paper: It seems like this was done randomly. This is not how to prove the method can generalize to new sequences. I worry that generated sequences in the training set can have homologues in the test set. Train/test splits using CATH or SCOP would be more appropriate.
-Since there is no control for homology in this work, it would be interesting to just use HMMER generated sequences of a family in a given GO category to compare against (simplest baseline model).
-In Figure 5, I don't understand where the "Random" values came from. Are you sampling from sequences of the same length or kmer composition? Truly random amino acids is not a useful comparison.
- MRR is an odd statistic. Why not just BLAST, or an alignment-based approach? How does MRR compare to e-value, or closest in sequence ID?

Neutral:
- I am surprised the authors are only using manually annotated sequences for this task, or not leveraging some sort of pretraining. Homologous sequences could improve model performance.
- As someone working in biotechnology, I think there is great interest in the refinement of molecular properties to highly-specific functions, and little interest in generating sequences generally from GO categories, which are much too broad.
review_title: Interesting conditional generative model, but lacking proper validation criteria

Meta-review: The paper proposes to use conditional GANs to generate protein sequence with respect to GO molecular functions. The idea is nice. But the reviewers find that there are many things that are not clear. For example, some sentences, phrase, the model and experiments pointed by the reviewers that should be  rigorous described. The technical contribution is also limited. The author are encouraged to revise the paper according to the comments.
Decision: Reject