Generating Antimicrobial Peptides from Latent Secondary Structure Space
Authors: Danqing Wang
Zeyu Wen
Lei Li
Hao Zhou
Venue: ICLR 2022 Submitted (2022)
Keywords: Antimicrobial Peptides
Drug Discovery
Secondary Structure
VQ-VAE

Abstract: Antimicrobial peptides (AMPs) have shown promising results in broad-spectrum antibiotics and resistant infection treatments, which makes it attract plenty of attention in drug discovery. Recently, many researchers bring deep generative models to AMP design. However, few studies consider structure information during the generation, though it has shown crucial influence on antimicrobial activity in all AMP mechanism theories. In this paper, we propose LSSAMP that uses the multi-scale VQ-VAE to learn the positional latent spaces modeling the secondary structure. By sampling in the latent secondary structure space, we can generate peptides with ideal amino acids and secondary structures at the same time. Experimental results show that our LSSAMP can generate peptides with multiply ideal physical attributes and a high probability of being predicted as AMPs by public AMP prediction models.

Under review as a conference paper at ICLR 2022
GENERATING ANTIMICROBIAL PEPTIDES FROM LA- TENT SECONDARY STRUCTURE SPACE
Anonymous authors
Paper under double-blind review
ABSTRACT
Antimicrobial peptides (AMPs) have shown promising results in broad-spectrum antibiotics and resistant infection treatments, making it magnet for the field of drug discovery. Recently, an increasing number of researchers have been intro- ducing deep generative models to AMP design. However, few studies consider secondary structure information during the generation, though it has shown cru- cial influence on antimicrobial activity in all AMP mechanism theories. This paper proposes LSSAMP which uses the multi-scale VQ-VAE to learn the positional la- tent spaces modeling the secondary structure. By Sampling in the latent secondary structure space, we can generate peptides with ideal amino acids and secondary structures simultancouslly. Experimental results show that our LS LSSAMP can gener- ate peptides with multiple ideal physical attributes and a high probability of being predicted as AMPs by public AMP prediction models.
INTRODUCTION
Developing neural networks for drug discovery has attracted increasing attention recently. It can fa- cilitate the discovery of potential therapies and reduce the time and cost of drug development (Stokes) al. 2020). Plenty of works have been done to employ deep generative models in searching for drug- like molecules with desired properties and achieved great success (Jin et al.] 20188; Shi et al.) 2019 SShwalbe--Kode & Gömez- Bombarell 2020; Xie al.) [220)). However, these works mainly focus on small molecules, and more complicated biochemicals, such as proteins, are still rarely explored.
Antimicrobial peptides (AMPs), defined as short proteins of less than 50 amino acids with potent antimicrobial activity, are an emerging category of therapeutic agents. AMPs exist widely in the nat- ural immune system for all species and kill bacteria in a physical way (Aronica et al. 2021; Cardoso jt al. 2020). They attach to the bacterial membrane and insert into the membrane to form pores, which leads to the death of bacteria by allowing cytoplasmic leakage. This mechanism makes them more promising for handling extensively drug-resistant bacteria than traditional antibiotics (Mahla- puu al. 2016). However, the theoretical chemical space of peptides is enormous and the sequence number grows exponentially as the length increases. Thus. it is challenging to search for valid peptides with antimicrobial properties from such huge sequence space.
Several factors can affect the antimicrobial activity of peptides (Boman, 2003). Amino acids with positive charges are more likely bind with bacterial membrane as most bacterial surfaces are anionic, while those with high hydrophobicities tend to move from the solution environment to the bacterial membrane. However, the mechanisms of antibacterial peptides need not only a reasonable sequence but also an appropriate structure. For example, by forming the helix structure, peptide can gather the hydrophobic amino acids on one side and hydrophilic ones on the other. This ability, named amphipathy, helps it insert into the membrane and maintain stable hole with other peptide molecules the membrane, as shown in Figure I The hole will drain the cytoplasm and finally kill the bacteria. This mechanism of killing the bacteria is called 'barrel stave'. Amphipathy plays an important role deciding the antibacterial activity of peptides and is closely related to the secondary structure of the peptide (Aronica al. (021).
According to the antimicrobial mechanism, a new AMP should meet the following criteria. Cl: possesses several ideal physical attributes (e.g. positive charge and high hydrophobicity). C2: It

---

Under review as a conference paper at ICLR 2022
Fold
Attach
Attach
An example of the 'barrel stave' antimicrobial mechanism. The blue indicates the hy- drophobic amino acids, and the red ones are hydrophilic. On the left, although the peptides with reasonable amino acids has attached to the bacterial membrane, they still can not insert into it. How- ever, by folding into the helix structure, as shown on the right, the peptides maintain a stable hole which breaks the membrane of the bacterium.
Figure 1:
has appropriate secondary structures (e.g. alpha-helix). C3: It differs from existing AMPs some extent.
The existing works mainly focus sequential features of amino acids and ignore the secondary structure. The traditional methods replace subsequences with patterns from the pattern database in given template (Porto al.) 2018). Inspired by success in deep neural networks. many researchers apply neural generative models to AMP discovery. They often use the physical attributes as the extra input to control the generation phase (Das al., 2018: Van Oort al., 202T), or train classifiers on each attribute to filter the peptides after the generation (Capecchi et al. 2021 Das et al.) [2021). The former ones usually generate peptides that have low correlation with the input attributes and the filter phase of the latter ones make the sampling inefficient.
As described above, the antimicrobial activity is determined by both the amino acid composition and secondary structure of the peptide. Thus, propose LSSAMP generate antimicrobial pep- tides from the latent semantic and structure space. Taking the peptide sequence as the time series, assign latent variable each position. Since it is computationally intractable to autoencoder (VQ-VAE) (van den Oord al., (017)) learn the discrete distribution for each position and fur- ther design multi-scale codebooks strategy to capture different local patterns to fit various length ranges for amino acid and structure sequences. During the generation process, LSSAAMP will sample a backbone from the secondary structure latent space and generate the amino acid sequence simul- tancously. We evaluate LSSAMP and several baselines through physical properties that are closely related to the antibacterial mechanism. Besides, we use some public AMP prediction models to predict generated sequences being AMPs previous works did (Das et al. 2020 Van Oort et al.) 2021).
tinuous latent variables over all positions, we employ the vector quuatized-vvaiatioone
To conclude, our contributions are as follows:
We propose ISSAMM, generative model which samples peptides from the latent secondary struc- ture space to control the peptide properties.
We develop multi-scale VQ-VAE to learn positional latent spaces from different aspects and model semantic sequences and structural sequences in the same space.
Experimental results show that LSSAMP can generate peptides with multiple ideal features such as positive charge, better hydrophobicity, and better amphipathicity. The results public AMP classifiers also verify that our model can generate peptides with high AMP probability.
RELATED WORK
Antimicrobial Peptides Generation Traditional methods for AMP design can be divided into three
approaches (Tores & de Fuente-Nunez 2019): sequential pattern database from existing AMPs, and then pick template peptide and replace local sequence with patterns (Loose et al. 2006;) Porto et al.) (018).. (ii) The genetic algorithms analyze
(i) The pattern recognition algorithms build a
the AMP database and design some antibiotic activity functions (Maccari al. 2013). (iiiYThe
2

---

Under review as a conference paper at ICLR 2022
Pattern Selector1
codebook
Generator
Encoder
Pattern Selector N
aodshool.
Classifier
Sample seed
Condition
The framework of LSSAMP. The input sequence is first fed into the encoder to get the hidden representation h. Then, we use N pattern selectors to select local patterns with different scales and use the corresponding codebooks to obtain discrete latent variables for each position. The codebooks are made to learn the latent space of the secondary structure by employing the secondary structure prediction task on them. For inference, we generate peptides from the latent structure space by sampling the index sequences on the codebooks. We also try use some conditions restrict the structure motifs to further improve the generation performance.
Figure 2:
molecular modeling and molecular dynamics methods build 3D models of peptides and analyze activity (Matyus al.] 2007 Bolintineanu Kaznessis 2011). Deep generative models take rapid growth in recent years. Dean & Walper (2020) encodes the peptide into the latent space and interpolates across predictive vector between known AMP and its scrambled version to generate novel peptides. The PepCVAE (Das et al. 2018) and CLaSS (Das et al. (021) employ the variational auto-encoder model to generate sequences. The AMPGAN (Van Oort et al. 2021) uses the generative adversarial network to generate new peptide sequences with conditions. To our knowledge, this is the first study to take secondary structure information into consideration during the generative phase, which is conducive to effectively generate well-structured sequences with desired properties.
Sequence Generation via VQ-VAE The variational auto-encoders (VAEs) were first proposed by Kingma & Welling (2014) for image generation, and then widely applied to sequence genera- tion tasks such as language modeling (Bowman et al. (2066), paraphrase generation (Gupta et al., 2018), machine translation (Bao et al. 2019) and so on. Instead of mapping the input to continuous latent space in VAE, the vector quantized variaationa autoencoder (VQ-VAL VAE) (van den Oord al., 2017 learns the codebook to obtain discrete latent representation. It can avoid issues of posterior collapse while has comparable performance with VAEs. Based on it, Razavi et al. (2019) uses multi-scale hierarchical organization to capture global and local features for image generation. |Bao] jtt al. (202T) learns implicit categorical information of target words with VQ- VQ-VAE and models the categorical sequence with conditional random fields in non-autoregressive machine translation. In this paper, we employ the multi-scale vector quantized technique to obtain the discrete representa- tion for each position of the peptide.
3 METHOD
Given peptide sequence acids and L the sequence length, the corresponding secondary structure can be denoted as (11, 1/2, ... ...,,,,) .9L). Following the definition in Kabsch & Sander (1983). there are 8 secondary structure types, including one unknown label, so ys E (H, G.G.G.G,TTT.
{ui, a..). where a belongs to the 20 common amino
I We first employ VQ-VAE for the sequence reconstruction task to learn the sequential latent space (Section 3.1 Then, we enforce the latent space to model the structure information by the secondary structure task (Section 3.2 Besides, design the multi-scale codebooks capture different local patterns (Sec-
H, G, 7 denote the alpha, 3-10, and pi helix. E, are the strand and The others coil structures.
3

---

latent sequence Under review as a conference paper at ICLR 2022
tion 3.3 Finally, describe the training and inference phase in Section 3.4 The overview of our model Is shown in Figure
3.1 MODELING PEPTIDE SEQUENCES
For sequential information, we embed the input peptide {ai, 02, ... ,.. to the latent space via the encoder and use the generator to reconstruct æ. We assume that each a; is determined by latent variable 24, and the input sequences a a1:L will be assigned
21:1. Since it is computationally intractable to sum continuous latent variables over the sequence, we use VQ-VAE (van den Oord et al.) 2017) to lookup the discrete embedding vector Ey = (zg(a)), (aL)) for each position by vector quantization.
Specifically, the encoder output (a;) E Rd will be replaced by the codebook embedding (q)))) _d via nearest neighbors lookup from the codebook B E I2 xd;
=q(a;) = Ek: and k = argmin .K3 |l (a,) - e,112-
(I)
Here, K is the size of the codebook and the dimension of the codebook entry e. Then, the generator will take 2yg (a:) as its input and reconstruct æ. The training objective L, is defined as:
Ly = log (a. 2 (a.)) + ||sg [a)(a - 2g (a.))) [|ze(a.) sg]z,(a.)112.
(2)
Here, sg(.) is the stop gradient operator, which becomes 0 at the backward pass. is the commit coefficient to control the codebook loss.
3.2 MODELING SECONDARY STRUCTURES
In order to model the categorical information of the secondary structure, we define an 8-category sequence labeling task on the latent space, which takes æ as the input and the structure label sequence as the target. Similar with sequence reconstruction, we use the same encoder to get ze (a.) and employ VQ-VAE to obtain discrete representation. Then, = (as) is fed to separate classifier for the secondary structure prediction:
L log (a.)) + |s [a:)] - (a)2)
(3)
Peptide sequences and structures have distinctive local features, which are often utilized in tradi- tional design algorithms. The patterns of amino acids are often used for template-bes feature-based recognition. For structure motifs such as a-helix with at least 3.6 consecutive amino acids, they will determine the position of amino acids in the 3D space and affect the function of peptides. However, the structure motifs are often much longer than sequence patterns. Therefore, we establish codebooks of multiple scales to learn latent spaces for different local patterns.
design and
3.3 MULTI-SCALE VQ- VAE
For the encoder output h; 20 (a:), first N pattern selectors F(x) extract local patterns from different scales, which will get 2 (a:) = F(1) (h;). Then, we establish the codebook for each pattern, and Eqn. to look up the nearest codebook embedding 2. (a.).
We share the codebooks between the sequence reconstruction and secondary structure prediction to capture common features and relationships between the amino acid and its structure. Specifically, we make N, codebook for the sequence information and N₂ for the structure. The 2g. (a;) is the concatenation of N, codebook embeddings and is fed the sequence generator:
2g. (a:) IlNEN, 2("
(4)
Based on Eqn. 2 the reconstruction training objective for multi-scale VQ-VAE can be denoted as:
L, log (a. (2/l/2g- (a))) + Σ
sg 2()) (a) 24 (0s)
2 +
((()) (a:) sglz, (a.)) (a>))(2)
NE

---

Under review as a conference paper at ICLR 2022
Algorithm Training phase of LSSAMP
Require: A pre-training dataset D.. peptide dataset with secondary structure Da. and the AMP dataset Damp- The model Ms.
1: Pre-train sequence reconstruction on D, and update M2 via Eqn. 5
2: Finctunc sequence reconstruction as well as secondary structure prediction on D. and update the Mo via Eqn. 6
3: Further Finetune Mr Damp-
for 1.2.... .N do
5: Create an empty datasel Cu.
6: for 24 Damp do
7: Save the 11-th codebook index of via Eqn. to C..
end for
9: Train an auto-regressive language model Mprior, on Cn-
10:
end for
The loss of secondary structure prediction task Lg can be formulated in the same way from Eqn. And the total training loss is:
L == L, + Tbe
(6)
where the 7 the weight of the secondary structure prediction.
3.4 TRAINING AND INFERENCE
We discuss the training and sampling process in this section. Since the AMP dataset Damp is very small, we use two datasets D, and D, to pre-train the reconstruction and prediction task. The whole training process is described in Algorithm[]
Training To enable the generator to generate valid peptide sequences, we first pre-train our model on large protein dataset D, for the sequence construction task. Then, we jointly train the se- quence reconstruction and the structure prediction on peptide dataset D, with secondary structure information. This phase maps the positional latent space to entangled sequential and structural distribution. Finally, we finetune our model on the small AMP dataset Damp to capture the charac- teristies of AMPs.
Following Kaiser et al. (2018), we use Exponential Moving Average (EMA) to update the embedding vectors in the codebooks. Specifically, keep count C₁ measuring the number of times that the embedding vector e₁ is chosen as the nearest neighbor of 2. (a:) via Eqn. [ Thus, the counts are updated with sort of momentum: Ck Xoe
(1 X) (a.) = ck). with the embedding ex
being updated as: + dex + E) (a, s. Here, À is the decay parameter.
ea
Prior Model The prior distribution over the codebook is a categorical distribution and can be made auto-regressive by the extra prior model. In order to model the dependency between 21:6, we train Transformer-basee language models on the embedding entries. We extract the index sequences generated by Eqn. for each codebook n and then train Mprior, on them.
Inference We sample several index sequences from the prior models for each codebook n, and then lookup the codebook get the embedding vector =(")_ Finally, =(") is fed to the generator and classifier to generate the sequence with its secondary structure. We also try to control the secondary
structure by existing AMP structure patterns further improve the generation quality.
EXPERIMENT
4.1 EXPERIMENT SETUP
Dataset The Universal Protein Resource (UniProt a comprehensive protein dataset. We down- load reviewed protein sequences (550k) with the limitation of 100 in length D, (57k examples). Then we use ProSP Billings et al. 2019) to predict the secondary structure for D, and filter some
hhtps://www.uijnootorrr//
5

---

Under review as a conference paper at ICLR 2022
low-quality examples. Therefore, we get Dg with 46k examples, which have both sequence and sec- ondary structure information. For antimicrobial peptide dataset, we download from Antimicrobial Peptide Database (APD3) (Wang et al. 2016) and filter repeated ones to get 3222 AMPs Damp- We randomly extract 3,000 examples as validation and 3,000 as test on D, and D,. For Damp- the size of validation and test is both 100. Following Veltri et al. (2018), negative examples that do not have antimicrobial activities to examine the evaluation metric. It first removes peptide sequences with anti-microbe activity from Uniprot, and then removes sequences with length < 10 and sequences with > 40 sequence identity with AMP sequences, resulting in 2021 non-AM AMP sequences.
establish decoy set with
Baseline We implement several baselines for our experiment. Traditional methods usually randomly substitute several amino acids on the existing AMPs and conducting biological experiments on them. Thus, we use the Random replacement with probability p as the baseline. Following Dean & Walper (020), we use VA to embed the peptides into the latent space and sample latent variable = from the standard Gaussian distribution p N (0, 1). For a fair comparison, we use the same Transformer architecture as our model LS LSSAMP and train on the Uniprot D, and APD dataset D Damp AMP- GAN is proposed by Van Oort et al. (202I), which uses BiCGAN architecture with convolution layers. It consists of three components: the generator, discriminator, and encoder. The generator and 49k false negative sequences from UniProt
Dean
discriminator share the same encoder. It is trained and 7k positive AMP sequences. PepCVAE (Das et al., 2018 is a VAF-based generative model with a structured variable to control the attribute of the sequence. Since the authors did not release their code, we use the model architecture from Hu et al. 2017) and modify the reproduced code for AMPs. The original paper use 93k sequences from UniProt and 7960/6948 positive/negative AMPs for training. For comparison, we use our UniProt dataset D, and ADP dataset D Damp to train it. MLPeptide (Capecchi et al., (021) RNN-based generator. It is first trained on 3580 AMPs and then transfer against specific bacteria.
Implementation Details There are three main modules for LSSAMP. The encoder and decoder are based on 2-layer Transformer (Vaswani et al., 2017) with dmooee = 128 and head = 8. The size of FFN projection is dy = 512 and all drouput rate are 0.1. For the classifier, we use the same CNN block as Billings et al. 2019)) with 32 input channels and dilation scale of [1, 2, 4, 8, 10). For multi-scale codebooks, we first apply CNN F(() to extract features. We set n 4 and kernel 8). The features will be padded to the same length as the input sequence. width ranging in [1. 2, 4,
Then, we use 4 codebooks with K = 8 and d 128. The reconstruction and prediction share the same codebooks, which means N, = N₂ = 4. The commit coefficient is set to 3 0.05.
We use Py for implementation and train the single Tesla- V100-32GB. We optimize the pa- rameter with Adam Optimizer (Kingma & Ba 2013). During pre- pre-training for sequence construction on D,,, we limit the max length 100 and set the maximum tokens in a batch bz 30,000, learning rate Ir as 0.01 with 8,000 warmup steps, and decoy weight for EMA as X = 0.8. For secondary structure prediction on D,, the max length limited 32, b2 10, 000, br 0.003, À 0.95, and the prediction loss coefficient = 1. Finally, we transfer to D Damp with the same hyperparameters except the Ir 0.001.
4.2 EVALUATION METRIC
To evaluate the antimicrobial activity of the generated peptide S = {a1, 42 aL), we use several physical features according to the mechanism of AMPs. Besides, we also apply some open-source AMP predictive models to estimate the probability of the generated peptides being AMPs.
4.2.1 PHYSICAL ATTRIBUTES
Charge The bacterial membrane usually takes the negative charge, so peptides with the positive charge prefer to bind with the bacteria. The whole net charge of the peptide sequence S is defined as the sum of the charge of all its amino acids C(a;) at pH 7.4, which is C(S) = En,es es C(a,) Hydrophobicity The hydrophobicity reflects the tendency bind lipids on the bacterial membrane. is more likely for the peptide with a high hydrophobicity to move from the solution environment
"https//aps.ummc..dd http://ggttc..ammmmieeedcclomooe

---

Under review as a conference paper at ICLR 2022
to the bacterial membrane. We use the hydrophobicity scale H (a,) in Eisenberg et al. (9844) to calculate the hydrophobicity of a sequence, which is H()) -o, H (a.)
Amphipathicity Hydrophobic Momentum The amphipathicity measures the ability of the pep- tide to bind water and lipid at the same time, which is a definitive feature of antimicrobial pep- tidesoHancock & Rozek 2002). It be quantified by the hydrophobic momentum (S, defined by Eisenberg et al. (1984). The hydrophobic momentum is determined by the hydrophobic- ity H (a:) of each amino acid a,, along with the angle 0 between amino acids. The angle can be estimated by the secondary structure. For the a-helix structure, 0 is 100 and for 8-sheet, 0 is 180".
wH(
Σ H(a;) * cos(i 0))²
H(a,)sin(( * 0))²
For each peptide, we calculate the above properties to measure its antimicrobial activity. For com- parison, draw the distribution APD and decoy dataset and choose range for each property f We use the percentage of peptides in the range for each attribute to leverage the generation perfor- mance and Combination measure the percentage of those that satisfy three conditions the same time. The detailed information is attached in the appendix.
Uniq
C
H
uH
Combination
APD Decoy
3222
68.75%
27. 27.96%
4.72%
6.15%
2020
21.83%
8.81%
1.98%
0.10%
Random p 0.1
4978
66.16% = 0.21%
26.92% + 0.24%
23.12% + 0.58%
4.40%
0.16%
Random p 0.2
5000
61.70%
0.39%
24.87% + 0.29%
20.79% + 0.76%
2.47%
0.17%
VAE
4988
38.54%
0.36%
21.37% + 0.58%
12.60% + 0.67%
0.34%
0.11%
AMP-GAN
4976
88.07% + 00.42%
17.39% + 0.75%
23.55% + 0.72%
1.93%
0.05%
PepCVAE
1346
58.89% + 1.05%
14.54% + 0.55%
11.65% + 0.23%
2.75%
= 0.25%
MLPeptide
4486
86.59%
0.55%
9.01% = 0.33%
36.56% 1 0.62%
3.22%
0.19%
LSSAMP
4886
75.06% + 0.37%
39.51% + 0.64%
27.18% + 0.41%
9.96%
0.07%
LSSAMP w/o condition
4893
60.88% 0.44%
38.75% T 0.26%
24.05% 1 0.71%
6.42% 0.29%
Table 1: Physical attributes of generated sequences. We use the percentage of peptides meeting the
range to measure the performance. Uniq is the number of unique generated sequences. C. H. uH correspond to charge, hydrophobicity, hydrophobic moment described in Section 4.2.] Combina- tion is the percentage satisfying three ranges at the same time. The best results are bold.
4.2.2 AMP CLASSIFIERS
Following previous works (Das et al. 2020) Van Oort al.) 2021), we also use several open-source AMP prediction tools to predict the AMP probability of generated sequence. Since these prediction tools trained and report results in different AMP datasets, first use them predict sequences in APD and decoy dataset as a reference of their performance.
Thomas et al. (2010) trained on the AMP database of 3782 sequences with random forest (RF). discriminant analysis (DA), support vector machines (SVM) and artificial neural network (ANN)) B respectively. AMP Scanner v Veltri Veltri et al. 2018). short Scanner, is CNN-&LSTM based
deep neural network trained on 1778 AMPs picked from APD. AMPMIC 10 WWitten & Witten 2019)
trained a CNN-based regression model on 6760 unique sequences and 5T345 MIC measurement to
predict MIC values. IAMPI (Kavousi et al. (2200) is model based on Xtreme Gradient Boosting. achieves the highest correct prediction rate on a set of ten more recent AMPs (Aronica et al., 2021). ampPEF LLwrrnnce et al. 2021 is random forest based model which is trained 3268 AMP sequences. It has the best performance across multiple data sets (Aronica et al. 2021).
'The requirements C E [2, 10j, H [0.25, oo), and u/l E [0.5, 0.75) [1.75, oo).
7 "htt://www.camm3..bl hammm.3..innirrh.e... wtw.:hhhee.../ppeiittioooooo
'We drop the ANN model becasue its accuracy APD is low (82.83%).
'htt:://cbbl... wt..ac wta.c.r/AAMCCllasfff//neee
Phaps://gitub..co http:://ggitih.ccc////wrreeeea/m

---

Under review as a conference paper at ICLR 2022
4.3 RESULTS AND ANALYSIS
We generate 5000 sequences for each baseline and filter the repeated ones. During the generation, add some structural restrictions based on the antimicrobial mechanism the secondary structure to improve the performance. Specifically, we reject peptides with more than 30% coil structure ('-),, which can hardly fold in the solution environment and insert into the bacterial membrane. Besides, we limit the minimum length of a helix structure ('H') to 4 based on physical rules. We name our model with structural control as LSSAMP and the model without extra conditions as LSSAMP w/o condition. We discuss the structure conditions and give some generated peptides in the appendix.
SVM
RF
DA
Scanner
AMPMIC
LAMPE
amPEE
Average
APD Decoy
87.78%
91.24%
86.24%
94.66%
98.42%
97.83%
91.50%
92.52%
17.43%
13.71%
16.04%
0.25%
18.07%
23.53%
52.92%
20.28%
Random p = 0.1
86.06%
86. 12%
84.01%
93.23%
19.14%
95.60%
91.74%
87.99%
Random = 0.2
76.66%
76.64%
74.83%
86.95%
68.57%
91.14%
87.89%
80.38%
VAE
24.90%
15.30%
13.83%
15.12%
15.25%
40.31%
24.30%
21.29%
AMP-GAN
78.62%
87.29%
83.82%
82.17%
89.58%
93.88%
80.52%
85.
PepCVAE MLPeptide
82.84%
85.96%
93.33%
85.44%
98.44%
98.14%
80.77%
89.27%
90.43%
92.55%
93.08%
93.72%
96.34%
97.05%
91.37%
93.51%
LSSAMP
91.78%
89. 89.86%
91.09%
90.51 %
93.40%
90.42%
93.23%
91.47%
LSSAMP w/o condition
77.98%
78.31 %
78.00%
85.72%
76.71%
90.05%
84.32%
81.58%
Table 2: The percentage of generated sequences being predicted AMP. The classifiers are de- scribed in Section 4.2.2 The first part is the prediction results on AMP and non-AMM dataset as the reference. The bold ones are the best model results.
Physical Attributes As listed in Table [ LSSAMP outperforms on the combination percentage by large margin (5.56%), which indicates that our model generate sequences satisfying multiple properties at the same time. If we do not control the structure, the combination percentage is similar APD, which implies our model learns the latent distribution of APD. For the specific attribute, LSSAMP tends to generate peptides with higher hydrophobicity, while AMP-GAN and MLPeptide have more positive sequences. Compared with other models, PepCVAE tends to generate redundant sequences, which results in significant decline in the number of unique sequences. Also, we can find that by control the secondary structure, all metrics can be improved. This verifies that secondary structure information has a great influence peptide properties and it is beneficial to take it into consideration during the peptide generation.
AMP Prediction The results of prediction tools shown in Table 2 By the comparison of LSSAMP and LSSAMP w/o condition, we can draw the same conclusion as above that control- ling the secondary structure can further improve the generation performance. For AMPMIC and IAMPE, PepCVAE has significant advantage over other methods, but has poor performance in other classifiers. LSSAMP performs better than MLPeptide in SVM and amPEP but does not perform as well as it in other classifiers. On the classifier with the largest performance gap IAMPE, can find that the results of LSSAMP and LSSAMP w/o condition are similar, which implies that different secondary structures do not affect the results of IAMPE. However, in real scenarios, the structure very important for the antimicrobial mechanism, which makes the classifier less reliable.
PPL 1
Loss 1
AA Acc.t
SS Acc.
Codebook PPL
Loss
AA Acc.t
SS Acc.t
LSSAMP
3.12
1.14
99.93
86.76
[]]
19.04
2.94
65.49
83.41
w/o D.
11.56
2.45
66.06
82.78
1,2]
3.84
1.35
99.40
85.39
w/o D.
3.83
1.34
99.58
85.87
1,2, 4]
3.32
1.20
100.00
85.95
w/o subbook
3.49
1.25
99.86
86.61
[1,2,4.8]
3.24
1.17
99.79
87.20
Table 3: Ablation Study on validation set of D Damp- "w/o" means that we remove the mod- ule from LSSAMP. 1 means lower is better, while t means higher is better. The detailed descriptions are in Section 4.3
Table 4: The influence of the number of code- books. "[1,2,4,8]' indicates that we use code- books capture local features with window sizes of 1,2,4.8. The meanings of symbols are the same Table 3
8

---

Under review as a conference paper at ICLR 2022
ARD
APB 15SAMP _eo,
0.12
LSSAMP LSSAMYP_WL_COm wac.ond LOSAMP
15SAUP
0.10
Dacey
Daccy
Sequence Langth
0.125
0.303
15-
0.375
3.01
0.250
3.02
0.525
A ACDEFGH MMPPO Q VWW
0.000
10
15
APO
LSSAMP
Global Charge
LSSAMP
1.0
bal Hytrophobbe Moment
3.5
4.0
APD
LSSAMP Jo 201d
LSSAMP
Gotal Hytrophobiccyy
Decoy
2.5-
3.0
3.5
1.0
-19 2
-1.0
charje
-25
10
-10
LSSAMP
LSS
Dacoy
LSSAMP,
LSSAMP
Dacoy
Figure 3: The distribution of amino acids, charge, sequence length, hydrophobicity, hydrophobic momentum, and a 3D visualization for three physical attributes.
Ablation Study We conduct the ablation study for our LSSAMP and show the results in Table 3 PPL is the perplexity of generated sequences that can measure fluency. Loss is the model loss on the validation set. AA Acc. is the reconstruction accuracy of amino acids sequences and SS Acc. is the prediction accuracy of the secondary structure. We can find that without the first pre- training phase D, for sequence reconstruction, the model can hardly generate valid sequences. The second phase to train the model on the large-scale secondary structure dataset D. will affect the prediction performance on the target AMP dataset. If remove multiple sub-codebooks and use single codebook with the same size, the performance will decline a little.
Codebook Number We explore the effect of different numbers codebooks on generation perfor- mance. From Table we find that single small codebook can hardly learn enough information to reconstruct the sequences. The PPL, loss, and secondary structure accuracy become better with the increase of codebook items. However, the reconstruction accuracy achieves the best performance when the codebook number is 3. This may be due the relatively short local patterns for the amino acid sequence, which makes the window of size 8 too long for it.
Visualization We plot the distribution of amino acids, charge, sequence length, hydrophobicity. and hydrophobic momentum for APD, Decoy, and our models in Figure 3 Without condition, the distribution of LSSAMP is similar to APD, which indicates that LSSAMP successfully learns the latent space of APD. However, if we control the secondary structure, it is more likely to generate sequences with a longer length and more positive charges. For hydrophobicity and hydrophobic momentum, the distribution of generated sequences is more concentrated.
CONCLUSION
In this paper, we propose a multi-scale VQ-VAE to generate peptides. Motivated by the important role of structure in the antimicrobial mechanism, SSMA leamss the latent spaces for each position with both sequential and structural features. Our model shows excellent performance on physical attributes related to antimicrobial activities and has high probability to be predicted AMPs by public classifiers. As the structural feature determines the function of macromolecules, LSSAMP has great potential for macromolecules design, including proteins and RNAs.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Pietro GA Aronica, Lauren M Reid, Nirali Desai, Jianguo Li, Stephen J Fox, Shilpa Yadahalli, Jonathan W Essex, and Chandra Verma. Computational methods and tools in antimicrobial peptide research. Journal of Chemical Information and Modeling, 6177:3117-311 2021.
Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xinyu Dai, and Jiajun Chen. Generating sentences from disentangled syntactic and semantic spaces. In Proc. of ACL, July 2019.
Yu Bao, Shujian Huang, Tong Xiao, Dongqi Wang, Xinyu Dai, and Jiajun Chen. Nor-autoreeressive translation by learning target categorical codes. In Proc. of NAACL-HLT, pp. 5749-5759, 2021.
Wendy M Billings, Bryce Hedelius, Todd Millecam, David Wingate, and Dennis Della Corte. Prospr: democratized implementation of alphafold protein distance prediction network. bioRxiv, pp. 830273, 2019.
Dan S Bolintineanu and Yiannis N Kaznessis. Computational studies of protegrin antimicrobial peptides: a review. Peptides, 32(1):188-201. 2011.
HG Boman. Antibacterial peptides: basic facts and emerging concepts. Journal of internal medicine, 254(3):197-215, 2003.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. In 20th SIGNLL Conference on Compu- lational Natural Language Learning, CONIL. 2016, pp. 10-21. Association for Computational Linguistics (ACL), 2016.
Alice Capecchi, Xingguang Cai, Hippolyte Personne, Thilo Kohler, Christian van Delden, and Jean- Louis Reymond. Machine learning designs non-hemolytic antimicrobial peptides. Chemical Science. 2021.
Marlon H Cardoso, Raquel Q Orozco, Samilla B Rezende, Gisele Rodrigues, Karen GN Oshiro, Elizabete Cândido, and Octávio I. Franco. Computer-aided design of antimicrobial peptides: are we generating effective drug candidates? Frontiers in microbiology. 10:3097, 2020.
Payel Das, Kahini Wadhawan, Oscar Chang, Tom Sercu, Cicero Dos Santos, Matthew Riemer, Vijil Chenthamarakshan, Inkit Padhi, and Aleksandra Mojsilovic. Pepovae: Semi-supervised targeted design of antimicrobial peptide sequences. arXiv preprint arXiv:1810.07743. 2018.
Payel Das, Tom Sereu, Kahini Wadhawan, Inkit Padhi, Sebastian Gehrmann, Flaviu Cipcigan, Vijil Chenthamarakshan, Hendrik Strobelt, Cicero dos Santos, Pin-Ye Chen, et al. Accelerating an- timicrobial discovery with controllable deep generative models and molecular dynamics. arXiv preprint arXiv:2005.1 /248, 2020.
Payel Das, Tom Sereu, Kahini Wadhawan, Inkit Padhi, Sebastian Gehrmann, Flaviu Cipcigan, Vijil Chenthamarakshan, Hendrik Strobelt, Cicero Dos Santos, Pin-Yu Chen, et al. Accelerated an- timicrobial discovery via deep generative models and molecular dynamics simulations. Nature Biomedical Engineering, 56)):6133623, 2021.
Scott N Dean and Scott A Walper. Variational autoencoder for generation of antimicrobial peptides. ACS omega, 5(33):00746-222 20754, 2020.
David Fisenberg, Robert M Weiss, and Thomas C Terwilliger. The hydrophobic moment detects periodicity in protein hydrophobicity. Proceedings of the National Academy of Sciences, 81(1): 140-144, 1984.
Ankush Gupta, Arvind Agarwal, Prawaan Singh, and Piyush Rai. A deep generative framework for paraphrase generation. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proc. of AAAI, pp. 5149-5156, 2018
Robert EW Hancock and Annett Rozek. Role membranes in the activities of antimicrobial cationic peptides. FEMS microbiology letters, 206(2):143-149, 2002.
10

---

Under review as a conference paper at ICLR 2022
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov. and Eric P Xing. Toward controlled generation of text. In International Conference on Machine Learning. pp. 1587-1596. PMLR, 2017.
Wengong Jin, Kevin Yang, Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to- graph translation for molecule optimization. In International Conference on Learning Represen- tations, 2018.
Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: pattern recog- nition of hydrogen-bonded and geometrical features. Biopolymers: Original Research on Biomolecules, 22(12):557-2263 1983.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In International Con- ference Machine Learning, pp. 1390-2399, PMLR, 2018.
Kaveh Kavousi, Mojtaba Bagheri, Saman Behrouzi, Safar Vafadar, Fereshteh Fallah Atanaki, Ba- hareh Teimouri Lotfabadi, Shohreh Ariaeenejad, Abbas Shockravi, and Ali Akbar Moosavi- Movahedi. lampe: Nmr-assisted computational prediction of antimicrobial peptides. Journal of Chemical Information and Modeling, 60(10):4691-470)) 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), Proc. of ICLR, 2015.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. 2014.
Travis Lawrence, Dana I. Carper, Margaret K Spangler, Alyssa A Carrell, Tomás A Rush, Stephen J Minter, David Weston, and Jessy I. Labbé. ampeppy 1.0: a portable and accurate antimicrobial peptide prediction tool. Bioinformatics, 37(14)22058-266) 2021.
Christopher Loose, Kyle Jensen, Isidore Rigoutsos, and Gregory Stephanopoulos. A linguistic model for the rational design of antimicrobial peptides. Nature, 443 13:66-86) 2006.
Giuseppe Maccari, Mariagrazia Di Luca, Riccardo Nifosí, Francesco Cardarelli, Giovanni Signore, Claudia Boccardi, and Angelo Bifone. Antimicrobial peptides design by evolutionary multiob- jective optimization. PLoS computational biology. 9(9)=10032)) 2013.
Margit Mahlapuu, Joakim Hákansson, Lovisa Ringstad, and Camilla Björn. Antimicrobial peptides: an emerging category of therapeutic agents. Frontiers in cellular and infection microbiology, 6: 194, 2016.
Edit Matyus, Christian Kandt, and D Peter Tieleman. Computer simulation of antimicrobial pep- tides. Current medicinal chemistry, 14(26))2789-2798, 2007.
William F Porto, Isabel CM Fensterseifer, Suzana M Ribeiro, and Octavio L Franco. Joker: An algorithm to insert patterns into sequences for designing antimicrobial peptides. Biochimica et Biophysica Acta (BBA)-Geneeee Subjects, 1862(9))0043-2052 2018.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in neural information processing systems, pp. 14866-14876, 2019.
Schrödinger, LLC. The AxPyMOL molecular graphics plugin for Microsoft PowerPoint, ver- sion 1.8. November 2015a.
Schrödinger, LLC. The JyMOL molecular graphics development ceveloomeettcompppeen, component, version 1.8. November 2015b.
Schrödinger, LLC. The PyMOL molecular graphics system, version 1.8. November 2015c.
Daniel Schwalbe-Kode and Rafacl Gömez-Bombarelll Generative models for automatic chemical design. In Machine Learning Meets Quantum Physics, pp. 445-467. Springer, 2020.
Yimin Shen, Julien Maupetit, Philippe Derreumaux, and Pierre Tuffery. Improved pep-fold approach for peptide and miniprotein structure prediction. Journal of chemical theory and computation, 10 (10):4745-4755 2014.
11

---

Under review as a conference paper at ICLR 2022
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: flow-based autoregressive model for molecular graph generation. In International Conference on Learning Representations, 2019.
Jonathan M Stokes, Kevin Yang, Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina M Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar Bloom-Ackermann, et al. A deep learning approach to antibiotic discovery. Cell, 180(4))688-705) 2020.
Shaini Thomas, Shreyas Karnik, Ram Shankar Barai, Vaidyanathan Jayaraman, and Susan Idicula-Thomas. Camp: a useful resource for research on antimicrobial peptides. Nucleic acids research, (11uppp1) 2010.
Marcelo Der Torossian Torres and Cesar de Fuente-Nunez. Toward computer-made artificial
antibiotics. Current opinion in microbiology, 51:30-38, 2019.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, pp. 6309-631 6318, 2017.
Colin Van Oort, Jonathon B Ferrell, Jacob M Remington, Safwan Wshah, and Jianing Li, Ampgan v2: Machine learning-guus guided design of antimicrobial peptides. Journal of Chemical Information and Modeling, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Proc. of NeurlPS, pp. 5998-6008, 2017.
Daniel Veltri, Uday Kamath, and Amarda Shehu. Deep learning improves antimicrobial peptide recognition. Bioinformutics, 34(16):2740-7447 2018 8.
Guangshun Wang, Xia Li, and Zhe Wang. Apd3: the antimicrobial peptide database tool for research and education. Nucleic acids research, 4401))11087-D1093, 2016.
Jacob Witten and Zack Witten. Deep learning regression model for antimicrobial peptide design. BioRxiv, pp. 692681, 2019.
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. In International Conference on Learning Representations, 2020.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 ATTRIBUTE DISTRIBUTION
To determine the threshold of net charge, hydrophobicity, and hydrophobic moment, we analyze the distribution of sequences in APD and decoy in Figure For charge, we follow the rule summarized by specialists and choose sequences whose net charge is +2 to +10. For two characters left, we plot the histogram and compare the proportion in every bin. If the proportion of APD is larger than that in decoy, we add the bin into the acceptation range of the evaluation metric. The final ranges are C' E [2, 10). H [0.25, od], and uH [0.5, 0.75] U [1.75, oo].
9.200
APD
APD
0.200
APD
9.175
0.175
0.100
0.10
0.150
Fraction
2.125
0.08
2.00
2.075
3.050
0.050
3.025
0.025
3.000
10
15
0.00
-1.0
0.000
0.0
0.5
15
25
Global Charge
Global Hydrophobic
Global Hydrophobic Moment
Figure 4: The histogram of charge, hydrophobicity and hydrophobic moment on APD and decoy dataset.
A.2 GENERATED AMPs
We show 10 peptides generated by LSSAMP in Table 5 and build 3D models for four of them by PEPFold (Shen et al. 2014) and draw the picture by PyMOL (Schrödinger, LLC) 2015c a in Figure We can find that all these peptides have long helix structures, which make them more likely to have the antimicrobial ability. At the same time, although the model predicts a long continuous helix structure for peptide ID = 4 and ID 9, in fact, they have small coil structure between two helix structures. indicates that our model tends to predict a long continuous secondary structure instead of several discontinuous small fragments.
ID
Sequence
Secondary Structure
H
uH
FLPLVRVWAKLI FLSTYPYVAFKWVPT. ECPPAKC
НHHHHHHНHH
2.0
0.471
0.723
2.0
0.446
1.812
PEGYLARGIKSVYKIVVAGLLGG
0.420
0.549
GYLPAATKOYLLGGIKKIIVKK
3.0
0.419
0.523
OPTLLGGAIIHIILGGGNVRRRSSSST
2.0
0.416
0.514
FYPGLIKKANVIGYTIICIIIXCCTT
ННННННННHH
3.0
0.394
1.815
ALNCOMLTGIGKLAGKA
0.344
0.506
LLTRIIVGAISAVTSSIKKSS
3.0
0,334
0,531
ALSYIKGEWWAASLLLKRRCATTTAA 10 H.NPIIKTATTOLIVTAICCCKKKKK
3.0
0,334
0.660
4.0
0.334
1.940
Table 5: Ten generated peptides and their physical attributes and predicted structures. 'H' the a-helix, 'T' is the Turn and '-" is the coil.
A.3 STRUCTURE CONDITION
As described above, controlling the secondary structure can affect the properties of generated pep- tides. Thus limit the percentage of the coil structure with different ratio and calculate the physical attributes of generated peptides. The results are shown Figure ( We can find that with the de- crease of number of coil structures, the percentage of positive peptides keep growing. However, for
13

---

Under review as a conference paper at ICLR 2022
(a) ID=1
(h) ID=4
ID=9
(d) ID=10
Figure 5: 3D structures for sequence FLPLYRYWAKLL CILPAAKKYLLPIIKIIVG TLSYKGGWAASSLKKOFCAAAAAA FLNPUIKATTOLLTAAKKIFLKKK ID corresponds to
Table 5
Uniq
C
H
uH
Combination
APD Decoy
3222
68.75%
27.96%
4.72%
6.15%
2020
21.83%
8.81% %
1.98%
0.10%
Random
0.11
2055
73.54% + 0.69%
37.93% + 0.44%
27.82% + 0.31%
8.74% 1 0.41 %
Random
0.2
1831
69.00% = 0.31%
34.39%
+ 0.31%
22.70% + 1.06%
4.66% i 0.66%
VAE
475
56.99% = 2.92%
24.05%
3.28%
9.82% 1.64%
0.63% 1 0.74%
AMP-GAN
1966
90.86% = 0.50%
19.55% + 0.34%
21.26% 0.53%
2.10% + 0.35%
PepCVAE
208
62.76% + 1.58%
12.61% + 1.61%
12.66%
+ 2.80%
6.68% + 1.82%
MLPeptide
2106
84.11% = 0.39%
11.02% + 0.57%
45.80% + 1.22%
4.34% + 0.38%
LSSAMP
4886
75.06% = 0.37%
39.51 + 0.64%
27. 18% = 0.41% %
9.96% + 0.07%
Table 6: Physical attributes of sequences filtered by secondary structures. same with Table
The notations the
hydrophobicity and hydrophobic moment, the percentage drop after 0.3. Therefore, we limit the length of the coil structure 30% our main experiments.
80%
70%
50%
40%
30%
20%
0%
1.0
0.7
0.4
0.3
0.2
UH
Combine
Figure 6: The physical attributes of peptides with different percentage of the coil structure. The x-axis is the maximum percentage and y-axis is the percentage of peptides that meet the attribute range.
A.4 FILTER STRATEGY
Different from the existing work, we control the secondary structures during the generation phase instead of filtering sequences after the generation. But does this filtering mechanism make sense for other methods as the post-prucess, To answer this question, we use the same rules to filter the generated peptides of comparison partners. The results shown in Table 6 Comparing Table
14

---

Under review as a conference paper at ICLR 2022
APD
LSSAMP WO cond
LSSAMP
Decay
tSNE Amino Acid
Figure 7: The tSNE plot for the distribution of amino acid in each sequence on four datasets.
6nd we can find that all results are improved by limited sequences the alpha-helical structures. implies that by controlling the structure, the physical attributes can be improved, which further verifies the importance of secondary structures. However, the sequence number have significantly declined, indicating that this senerate-then-fiie pipeline is inefficient.
A.S DISTRIBUTION ON AMINO ACIDS
To illustrate the distribution of amino acids in generated peptides, we use the tSNE plot shown in Figure 7 We convert use the vector with each dimension representing the probability of a certain amino acid to represent the peptide. Then we use tSNE convert the high-dimensional vector to 2D and visualize them. From Figure 7 we can obverse that there is large overlap between LSSAMP w/o condition and APD, which indicates that our model has captured the global distribution of APD instead of collapsing to a local mode. Furthermore, LSSAMP covers APD and has some outliers. implies that with the secondary structure condition, model not only learn the existing AMP space but also explore more possible space.
A.6 NOVELTY
To measure the novelty generated peptides, we define three evaluation metrics: Uniqueness, Diversity, and Similarity. Uniqueness is the percentage of the unique peptides. Diversity measures the similarity among generated peptides. We calculate the Levenshtein distance between every two sequences and normalize by the sequence length. Then average the normalized distance get the mean as its diversity. The higher the diversity, the more dissimilar the generated peptides are. Similarity is the similarity between the generated peptides and the training AMP set (APD). For each generated sequence, we find peptide from the training set which has the smallest Levenshtein distance with it and normalize the distance by length. And we caleulate the average length as the similarity.
From Table we obverse that VAE has the highest diversity and lowest similarity. However, in Table ] and VAE performs the worst, which implies we can not measure the generation quality based on the novelty alone. Besides, the limitation on secondary structures will cause a significant decline in diversity. However, it does not result in the increase of the number of redundant peptides since the uniqueness does not increase. It implies that the restrictions make the model capture similar local patterns but not generate the exact same sequence.
15

---

Under review as a conference paper at ICLR 2022
Uniqueness f
Diversity t
Similarity +
Random p 0.1
99.54% 0.02%
0,871 1 0.021
0.078 1 0.001
Random p ==
99.99%
0.02%
0.971 + 0.022
0.160 0.001
VAE
98.60% 0.05%
1.011 = 0.038
0.584 + 0.002
AMP-GAN
99.54% = 0.07%
0,907 + 0.023
0.565 + 0.007
PepCVAR MLPeptide
26.50% + 0.58%
0.367 0.007
0.423 + 0.005
90.02% + 0.30%
0.850 0.016
0.416 0.010
LSSAMP
98.10% + 0.09%
0.664 + 0.018
0.528 + 0.005
LSSAMP w/o condition
97.64% + 0.24%
0.828 + 0.013
0.580 0.008
Table 7: The novelty of generated sequences.
Codebook
PPL
Loss +
AA Acc.1
SS Ace.t
[1]
19.04 - 2.84
2.94 + 0.14
65,49 = 3.49
83.41 + 2.34
1,
3.84 = 0.09
1.35
99.40 = 0.45
85.39 0.26
1, 1,2,
3.32 + 0.03
1.20 0.01
100.00
0.00
85.95 0.42
1,2, 4,
3.24 + 0.16
1.17 + 0.05
99.79 + 0.20
87.20 0.62
Table 8: The mean and standard deviation of results on codebooks. the same as Table
The meanings of symbols are
A.7 REPRODUCTION
We deseribe the architecture of our model and the hyperparameters used in the experiments in detail in Section We run the model for several times and calculate the mean and variance. In Table add the and the standard deviation of each setting in Table We will also release code after the anonymous period.
16


Reviews:
[{"review_id": "d_h_9eNXRr1", "reviewer": "Reviewer_s64x", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe that pretraining the model on a larger dataset, and the usage of both the aa sequence and secondary structure, are items that contribute to the strength of the paper. More generally, the ablation study Table 3 is very interesting in order to understand better the performance of the proposed model. Additionally, the model seems to be well-thought to fit the sequences under study.\n\nHowever, the work of the authors show some weaknesses, which are described below. \n\n1. Some points are unclear and could be better explained. For example, the sizes of the datasets of LSSAMP and LSSAMP without condition Table 1 are very similar, did you resample between the two or did you remove peptides that do not verify the condition? Also, I am not sure how the AA Acc and SS Acc were computed, Table 3. Could you give a bit more details?\n\n2. We could question the comparison partners. For example, the authors do not compare to PepCVAE (Das et al 2018), while mentioning it at the beginning of the manuscript. Does PepCVAE correspond to the VAE proposed Table 2? Also, while the model of the author uses pretraining, I am not sure to understand whether the proposed comparison partners do. I believe that it could be fair to use pretraining for at least a few of the comparison partners in order to have a fairer comparison, especially because pretraining in this application is not a new idea (see Das et al 2018).\n\n3. The results of the proposed approach are not entirely convincing. First, the proposed model seems to underperform compared to MLPeptide (Table 2). Second, it seems that the author have to filter, after generation, peptides with more than 30% coil structure or helix structure lengths shorter than 4. Given the simplicity of these rules, it seems surprising that the model is not able to largely capture them, if they are largely present in the training set. Does this filtering also improve the results of the comparison partners? If yes, by how much? And if no, does it mean the comparison partners better capture these simple rules? Third, in the APD dataset, very few peptides (~6%) seem to verify the boundary conditions defined for the three C, H, and uH attributes. Therefore, I am wondering if it is reasonable to evaluate the generated sequences based on the defined ranges for these attributes (\"combination\" Table 1), or based on these attributes at all. Fourth, in the paragraph Codebook Number, the authors claim that the reconstruction accuracy achieves the best performance when the codebook number is 3. To me, it seems that the AA Acc performance could be indistinguishable between 2, 3 or 4 codebooks as the numbers are very close. Could you provide standard deviations to be able to better understand the results?\n\n4. While the authors analyse the similarity between the generated sequences and the real ones, the authors do not mention possible overfitting (how similar the generated sequences are to the real ones) or mode collapse (how similar the generated sequences are between themselves). In particular, it seems that the generated secondary structure is not diverse as we can see Table 5 in the appendix. I believe that these two points (similarity and diversity) could be interesting to study to have a better understanding of the performance of the proposed approach.\n\n\n\nTypos:\n\nseveral times 'mutliply' is used instead of 'multiple'\n\nSection 3.1: to reconstruct x -> x should be bold\n\nSection 3.1: j \\in K -> j \\in {1, ..., K}\n\nSection 3.4: the second D_r should be D_s\n\nSection 3.4: the sum (large sigma signs) should indicate the letter on which the sum is done. \n\nSection 4.1: will be padding -> will be padded\n\nSection 4.1: codebook -> codebooks\n\nSection 4.3: 6.27% should be 6.15%? (to match Table 1)\n\nThere are a few other typos, I think it could be worth rereading carefully the manuscript. \n\n\nUnclear sentences/plots:\n\nSection 3.4: \"the number that the embedding ... via Eqn.1 \" -> maybe it should be \" the number of times that the embedding vector e_k is ...\"\n\nTable 3/4: the caption is very confusing as there is no space between Table 3 and 4. \n\nThe 3D plot Figure 3 is very hard to read as all the points are overlapping.\n\n\n", "summary_of_the_paper": "The paper proposes a multi-scale VQ-VAE model to generate antimicrobial peptides (AMPs). The model both train on the amino-acid (aa) sequences and on secondary structure. The author evaluate the model by comparing attributes of the generated sequences with real AMPs. The authors also perform an ablation study of their model, an extensive benchmark of existing generative models that have been used for peptides. \n\nThe main contributions of the authors are:\n- Developing a generative model that accounts for both the aa sequences as well as the secondary structure of the AMPs. \n- A large set of experiments to validate their approach, comprising the study of attributes of the generated sequences, benchmark of existing methods, evaluation of the generated sequences with existing predictors and an ablation study. \n\n", "summary_of_the_review": "To summarise, I believe that the paper is below the acceptance threshold as:\n- the proposed method do not seem to beat the state of the art (MLPeptide)\n- some analyses of the generated sequences are missing, such as similarity study between the generated sequences and the real ones, and diversity of the generated sequences.  \n- the value of some validation experiments (for example Table 1) or of the filtering step is unclear.\n\nI could increase the score if the authors address these points.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_zcKHMdStti", "reviewer": "Reviewer_bThw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the idea of utilising secondary structure to aid AMP generation is useful, which could be seen from the improved performance from LSSAMP w/o to LSSAMP in Table 2. This result suggests that secondary structure is an important factor to consider in AMP design. The physical properties of generated peptides (table 1) are also impressive.\n\nI feel the paper is on the boarder line due to the following concerns. \n1. In table 2, the performance is generally worse than MLPeptide, which suggests that the proposed method have space to improve.\n2. Are secondary structure really important? Could you use pep-fold (https://bioserv.rpbs.univ-paris-diderot.fr/services/PEP-FOLD3/) to get the secondary structure of peptides generated by other methods such as Random, VAE, AMP-GAN, MLPeptide, then filter out peptide sequences as LSSAMP. This will help to answer the improvement is universal, rather than unknown modelling glitches in LSSAMP.\n3. Page 6, right before sec 4.2, The batch size for Dr (n=57k) and Ds (n=46k) are 30,000 and 10,000, which are very large. What is the consideration here?\n4.  Table 1, the combination score of LSSAMP is much higher than that of APD, which is suspicious. In theory APD performance should be the upper bound of LSSAMP. Is this suggesting LSSAMP collapse to a local mode, i.e. generated pep sequences mimicking a subset of APD? Page 13, supplementary A2, all structures are long stretches of H. I suggest the authors convert each pep sequence to a vector with each dimension representing the probability of a certain amino acid of the given pep sequence. Then make a tSNE plot of generated pep sequences together with APD dataset, this will help to clarify the diversity of generated pep sequences.\n5. Page 5, prior model: I think the dependency is already modelled by the encoder, what is the motivation of  imposing a further layer of dependency on the hidden layer?\n6. It will be nice if a section could be added regarding the reproducibility of experiments in the paper. \n\nMinors\n1. Page 2, “Experimental results show that LSSAMP can …. with multiply”: multiply to multiple\n2. Page 5, sec 3.4, paragraph 2, “a peptide dataset Dr” to “ Ds”\n", "summary_of_the_paper": "Generating antimicrobial peptide (AMP) is a challenging task. This paper assumes a latent representation is both associated with the amino acid sequence and its secondary structure of peptide. This paper adopts a VQ-VAE network to learn the latent representation of a given peptide sequence, as well as its secondary structure. Then a transformer-based language model is used to model the prior distribution of the latent representations. Random peptide sequences can be generated at this point. The paper evaluates the physical properties of the random peptide sequences, and compared the proportions of these random peptide sequences that are classified as AMP. The comparison shows that secondary structure is important for the generation and the quality of the generated peptide sequences is high.\n", "summary_of_the_review": "I think this paper is on the boarder line. I tend to accept it if the secondary structure information could be proven to be useful also for other methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KZvsv0wJ1lP", "reviewer": "Reviewer_n6Pg", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Unfortunately, I think this work suffers from a major conceptual flaw, namely, that there is not satisfactory gold standard to evaluate the utility of the proposed method.  The task being performed involves training a model from a set of known AMPs, simulating novel AMPs, and trying to ascertain whether these simulated AMPs are \"good\" in some sense.  The manuscript does this by asking whether the sequences capture some of the known properties of AMPs and whether they get assigned high scores by existing AMP classifiers.  This is not a useful mode of evaluation, for two reasons.  First, if these are truly the right performance measures to use, then it would make sense to simply optimize the model w.r.t. those measures directly.  Second, it is certainly plausible that a synthetic AMP sequence could do well according to these measures yet not have AMP function.  In practice, the only way to know whether an AMP simulator works is to synthesize that peptides and test them.  Short of that, I do not think it's possible to evaluate the accuracy of this type of model.\n\nAnother problematic aspect of this work is the use of deep models for modeling very small datasets.  The paper mentions that the AMP dataset is small and that some additional data is used for training.  I would like to see specifics about how large these datasets are and how they compare in size to the number of parameters in the model.\n\nAn aspect of evaluation that is missing here is the degree of sequence similarity to any member of the training set.  In principle, it should be easier to generate functional AMPs if we don't require that they differ much from known AMPs.  For example, if I randomly substitute a single amino acid in a known AMP, then the resulting \"simulated\" AMP is very likely to be functional.  Thus, the \"random replacement\" baseline method would be expected to do very well when p is small.  In practice, a good objective should probably include a term that captures the degree of novelty, by e.g. weighting sequences according to their nearest neighbor in the training set.\n\nMinor:\n\nAbstract: \"few studies consider structure\" Clarify in this sentence, rather than several sentences later, that this refers to secondary structure of the peptide.\n\nOn p. 2, I do not understand what \"the conditioning vector\" refers to.\n\nWrite out \"VQ-VAE\" when it's first used, and similarly for other abbreviations such as \"FFN.\"\n\nI think the footnote on p. 4 should be incorporated into the main text, and the meanings of the eight letters should be mentioned.\n\nWhat does it mean to \"kill bacteria in a physical way\"? It's hard to\nimagine any other way to kill them.\n\nIt is not clear to me exactly how the parameter p in the random baseline is interpreted: is this probability computed independently for each amino acid in the peptide?\n\n\"multiply ideal\" -> \"multiple ideal\"\n\n\"proprieties\" -> \"properties\"\n\n", "summary_of_the_paper": "This manuscript addresses the problem of generating novel antimicrobial peptides (AMPs).  The approach uses a deep learning framework, and the primary novelty is to include in the model information about peptide secondary structure.  The hypothesis that such information could improve the ability to capture valuable information about AMP function is plausible a priori.\n", "summary_of_the_review": "This paper addresses a problem for which no good gold standard is available, short of experimental synthesis and testing of the proposed sequences. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agreed that taking into account the secondary structure in addition to the amino acid sequence, although not new in bioinformatics, may be a good idea in the context of deep generative models of peptides. On the other hand, all reviewers also agreed that the experimental results do not allow concluding about the potential benefit of the method, i.e., whether it is likely to produce potent AMPs (and whether it does it better than existing methods). Indeed, the proposed computational criteria can not replace a proper experimental validation, and it is not clear whether a "better method" on the computational criteria will be "better" in the real world. Second, the results on the computational criteria are not convincing: regarding the physical properties, it remains debatable to claim that a method is good if it outputs many AMPs that fulfill the criterion, while less than 7% of the true AMPs do; and regarding the computational prediction of being an AMP, the proposed method is outperformed by existing ones. In conclusion, we consider that the paper is not ready for publication at ICLR, since there is no significant methodological novelty nor significant experimental results if this is an application paper, and we encourage the authors to consider a publication with wet lab experiments to demonstrate the relevance of the method.
Decision: Reject