Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a multi-scale VQ-VAE model to generate antimicrobial peptides (AMPs). The model both train on the amino-acid (aa) sequences and on secondary structure. The author evaluate the model by comparing attributes of the generated sequences with real AMPs. The authors also perform an ablation study of their model, an extensive benchmark of existing generative models that have been used for peptides. 

The main contributions of the authors are:
- Developing a generative model that accounts for both the aa sequences as well as the secondary structure of the AMPs. 
- A large set of experiments to validate their approach, comprising the study of attributes of the generated sequences, benchmark of existing methods, evaluation of the generated sequences with existing predictors and an ablation study. 


main_review: I believe that pretraining the model on a larger dataset, and the usage of both the aa sequence and secondary structure, are items that contribute to the strength of the paper. More generally, the ablation study Table 3 is very interesting in order to understand better the performance of the proposed model. Additionally, the model seems to be well-thought to fit the sequences under study.

However, the work of the authors show some weaknesses, which are described below. 

1. Some points are unclear and could be better explained. For example, the sizes of the datasets of LSSAMP and LSSAMP without condition Table 1 are very similar, did you resample between the two or did you remove peptides that do not verify the condition? Also, I am not sure how the AA Acc and SS Acc were computed, Table 3. Could you give a bit more details?

2. We could question the comparison partners. For example, the authors do not compare to PepCVAE (Das et al 2018), while mentioning it at the beginning of the manuscript. Does PepCVAE correspond to the VAE proposed Table 2? Also, while the model of the author uses pretraining, I am not sure to understand whether the proposed comparison partners do. I believe that it could be fair to use pretraining for at least a few of the comparison partners in order to have a fairer comparison, especially because pretraining in this application is not a new idea (see Das et al 2018).

3. The results of the proposed approach are not entirely convincing. First, the proposed model seems to underperform compared to MLPeptide (Table 2). Second, it seems that the author have to filter, after generation, peptides with more than 30% coil structure or helix structure lengths shorter than 4. Given the simplicity of these rules, it seems surprising that the model is not able to largely capture them, if they are largely present in the training set. Does this filtering also improve the results of the comparison partners? If yes, by how much? And if no, does it mean the comparison partners better capture these simple rules? Third, in the APD dataset, very few peptides (~6%) seem to verify the boundary conditions defined for the three C, H, and uH attributes. Therefore, I am wondering if it is reasonable to evaluate the generated sequences based on the defined ranges for these attributes ("combination" Table 1), or based on these attributes at all. Fourth, in the paragraph Codebook Number, the authors claim that the reconstruction accuracy achieves the best performance when the codebook number is 3. To me, it seems that the AA Acc performance could be indistinguishable between 2, 3 or 4 codebooks as the numbers are very close. Could you provide standard deviations to be able to better understand the results?

4. While the authors analyse the similarity between the generated sequences and the real ones, the authors do not mention possible overfitting (how similar the generated sequences are to the real ones) or mode collapse (how similar the generated sequences are between themselves). In particular, it seems that the generated secondary structure is not diverse as we can see Table 5 in the appendix. I believe that these two points (similarity and diversity) could be interesting to study to have a better understanding of the performance of the proposed approach.



Typos:

several times 'mutliply' is used instead of 'multiple'

Section 3.1: to reconstruct x -> x should be bold

Section 3.1: j \in K -> j \in {1, ..., K}

Section 3.4: the second D_r should be D_s

Section 3.4: the sum (large sigma signs) should indicate the letter on which the sum is done. 

Section 4.1: will be padding -> will be padded

Section 4.1: codebook -> codebooks

Section 4.3: 6.27% should be 6.15%? (to match Table 1)

There are a few other typos, I think it could be worth rereading carefully the manuscript. 


Unclear sentences/plots:

Section 3.4: "the number that the embedding ... via Eqn.1 " -> maybe it should be " the number of times that the embedding vector e_k is ..."

Table 3/4: the caption is very confusing as there is no space between Table 3 and 4. 

The 3D plot Figure 3 is very hard to read as all the points are overlapping.



summary_of_the_review: To summarise, I believe that the paper is below the acceptance threshold as:
- the proposed method do not seem to beat the state of the art (MLPeptide)
- some analyses of the generated sequences are missing, such as similarity study between the generated sequences and the real ones, and diversity of the generated sequences.  
- the value of some validation experiments (for example Table 1) or of the filtering step is unclear.

I could increase the score if the authors address these points.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Generating antimicrobial peptide (AMP) is a challenging task. This paper assumes a latent representation is both associated with the amino acid sequence and its secondary structure of peptide. This paper adopts a VQ-VAE network to learn the latent representation of a given peptide sequence, as well as its secondary structure. Then a transformer-based language model is used to model the prior distribution of the latent representations. Random peptide sequences can be generated at this point. The paper evaluates the physical properties of the random peptide sequences, and compared the proportions of these random peptide sequences that are classified as AMP. The comparison shows that secondary structure is important for the generation and the quality of the generated peptide sequences is high.

main_review: I think the idea of utilising secondary structure to aid AMP generation is useful, which could be seen from the improved performance from LSSAMP w/o to LSSAMP in Table 2. This result suggests that secondary structure is an important factor to consider in AMP design. The physical properties of generated peptides (table 1) are also impressive.

I feel the paper is on the boarder line due to the following concerns. 
1. In table 2, the performance is generally worse than MLPeptide, which suggests that the proposed method have space to improve.
2. Are secondary structure really important? Could you use pep-fold (https://bioserv.rpbs.univ-paris-diderot.fr/services/PEP-FOLD3/) to get the secondary structure of peptides generated by other methods such as Random, VAE, AMP-GAN, MLPeptide, then filter out peptide sequences as LSSAMP. This will help to answer the improvement is universal, rather than unknown modelling glitches in LSSAMP.
3. Page 6, right before sec 4.2, The batch size for Dr (n=57k) and Ds (n=46k) are 30,000 and 10,000, which are very large. What is the consideration here?
4.  Table 1, the combination score of LSSAMP is much higher than that of APD, which is suspicious. In theory APD performance should be the upper bound of LSSAMP. Is this suggesting LSSAMP collapse to a local mode, i.e. generated pep sequences mimicking a subset of APD? Page 13, supplementary A2, all structures are long stretches of H. I suggest the authors convert each pep sequence to a vector with each dimension representing the probability of a certain amino acid of the given pep sequence. Then make a tSNE plot of generated pep sequences together with APD dataset, this will help to clarify the diversity of generated pep sequences.
5. Page 5, prior model: I think the dependency is already modelled by the encoder, what is the motivation of  imposing a further layer of dependency on the hidden layer?
6. It will be nice if a section could be added regarding the reproducibility of experiments in the paper. 

Minors
1. Page 2, “Experimental results show that LSSAMP can …. with multiply”: multiply to multiple
2. Page 5, sec 3.4, paragraph 2, “a peptide dataset Dr” to “ Ds”

summary_of_the_review: I think this paper is on the boarder line. I tend to accept it if the secondary structure information could be proven to be useful also for other methods.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This manuscript addresses the problem of generating novel antimicrobial peptides (AMPs).  The approach uses a deep learning framework, and the primary novelty is to include in the model information about peptide secondary structure.  The hypothesis that such information could improve the ability to capture valuable information about AMP function is plausible a priori.

main_review: Unfortunately, I think this work suffers from a major conceptual flaw, namely, that there is not satisfactory gold standard to evaluate the utility of the proposed method.  The task being performed involves training a model from a set of known AMPs, simulating novel AMPs, and trying to ascertain whether these simulated AMPs are "good" in some sense.  The manuscript does this by asking whether the sequences capture some of the known properties of AMPs and whether they get assigned high scores by existing AMP classifiers.  This is not a useful mode of evaluation, for two reasons.  First, if these are truly the right performance measures to use, then it would make sense to simply optimize the model w.r.t. those measures directly.  Second, it is certainly plausible that a synthetic AMP sequence could do well according to these measures yet not have AMP function.  In practice, the only way to know whether an AMP simulator works is to synthesize that peptides and test them.  Short of that, I do not think it's possible to evaluate the accuracy of this type of model.

Another problematic aspect of this work is the use of deep models for modeling very small datasets.  The paper mentions that the AMP dataset is small and that some additional data is used for training.  I would like to see specifics about how large these datasets are and how they compare in size to the number of parameters in the model.

An aspect of evaluation that is missing here is the degree of sequence similarity to any member of the training set.  In principle, it should be easier to generate functional AMPs if we don't require that they differ much from known AMPs.  For example, if I randomly substitute a single amino acid in a known AMP, then the resulting "simulated" AMP is very likely to be functional.  Thus, the "random replacement" baseline method would be expected to do very well when p is small.  In practice, a good objective should probably include a term that captures the degree of novelty, by e.g. weighting sequences according to their nearest neighbor in the training set.

Minor:

Abstract: "few studies consider structure" Clarify in this sentence, rather than several sentences later, that this refers to secondary structure of the peptide.

On p. 2, I do not understand what "the conditioning vector" refers to.

Write out "VQ-VAE" when it's first used, and similarly for other abbreviations such as "FFN."

I think the footnote on p. 4 should be incorporated into the main text, and the meanings of the eight letters should be mentioned.

What does it mean to "kill bacteria in a physical way"? It's hard to
imagine any other way to kill them.

It is not clear to me exactly how the parameter p in the random baseline is interpreted: is this probability computed independently for each amino acid in the peptide?

"multiply ideal" -> "multiple ideal"

"proprieties" -> "properties"


summary_of_the_review: This paper addresses a problem for which no good gold standard is available, short of experimental synthesis and testing of the proposed sequences.
review_title: null

Meta-review: Reviewers agreed that taking into account the secondary structure in addition to the amino acid sequence, although not new in bioinformatics, may be a good idea in the context of deep generative models of peptides. On the other hand, all reviewers also agreed that the experimental results do not allow concluding about the potential benefit of the method, i.e., whether it is likely to produce potent AMPs (and whether it does it better than existing methods). Indeed, the proposed computational criteria can not replace a proper experimental validation, and it is not clear whether a "better method" on the computational criteria will be "better" in the real world. Second, the results on the computational criteria are not convincing: regarding the physical properties, it remains debatable to claim that a method is good if it outputs many AMPs that fulfill the criterion, while less than 7% of the true AMPs do; and regarding the computational prediction of being an AMP, the proposed method is outperformed by existing ones. In conclusion, we consider that the paper is not ready for publication at ICLR, since there is no significant methodological novelty nor significant experimental results if this is an application paper, and we encourage the authors to consider a publication with wet lab experiments to demonstrate the relevance of the method.
Decision: Reject