Full-Atom Peptide Design with Geometric Latent Diffusion

Authors: xiangzhe  kong
yinjun  jia
wenbing  huang
yang  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:22:58

Abstract: Peptide design plays a pivotal role in therapeutics, allowing brand new possibility to leverage target binding sites that are previously undruggable. Most existing methods are either inefficient or only concerned with the target-agnostic design of 1D sequences. In this paper, we propose a generative model for full-atom Peptide design with Geometric LAtent Diffusion (PepGLAD) given the binding site. We first establish a benchmark consisting of both 1D sequences and 3D structures from Protein Data Bank (PDB) and literature for systematic evaluation. We then identify two major challenges of leveraging current diffusion-based models for peptide design: the full-atom geometry and the variable binding geometry. To tackle the first challenge, PepGLAD derives a variational autoencoder that first encodes full-atom residues of variable size into fixed-dimensional latent representations, and then decodes back to the residue space after conducting the diffusion process in the latent space. For the second issue, PepGLAD explores a receptor-specific affine transformation to convert the 3D coordinates into a shared standard space, enabling better generalization ability across different binding shapes. Experimental Results show that our method not only improves diversity and binding affinity significantly in the task of sequence-structure co-design, but also excels at recovering reference structures for binding conformation generation.

Main paper: 
 introduction: 
 Peptides are short chains of amino acids and acts as vital mediators of many protein-protein interactions in human cells. Designing functional peptides has attracted increasing attention in biological research and therapeutics, since the highly-flexible conformation space of peptides allows brand new possibility to target binding sites previously undruggable with antibodies or small molecules [17, 35] . The key of peptide design is to generate peptides that interact compactly with target proteins (see Figure 1 ), since they mostly exhibit flexible conformations [20] unless bound to these receptors [60] .Conventional simulation or searching algorithms rely on frequent calculations of physical energy functions [6, 9] , which are inefficient and prone to poor local optimum. Recent advances illuminate the remarkable success of exploiting geometric deep generative models, particularly the equivariant diffusion models [45, 72] , for molecule design [21, 39] , antibody design [29, 44, 34] and protein design [64, 28] , as well as latent diffusion models further enhancing the performance [71, 18] . Inspired by these successes, a natural idea is leveraging diffusion models for peptide design as well, which, yet, is challenging in two aspects. From the dataset aspect, existing databases (PepBDB [65] , Propedia [46] ) merely collect data from Protein Data Bank (PDB) [5] , neither performing filters according to practical relevance [49] and redundancy, nor providing an adequate split for evaluation.Figure 1 : Left: Peptide design requires generating peptides that form compact interactions with the binding site on the receptor. The intricacy of protein-peptide interactions demands efficient exploration in the vast space for sequence-structure co-design. Right: Different binding sites (a, b, c, d) adopt disparate center offsets and geometric shapes, approximating variable 3D Gaussian distributions that deviate from N (0, I). We propose to convert the geometry into a standard space approximating standard Gaussian, via an affine transformation derived from the binding site ( §3.3).Therefore, this paper first curates a benchmark from PDB [5] and the literature [59] , and then systematically evaluates the generative models in terms of diversity, consistency, and binding affinity.From the methodology aspect, it is nontrivial to adopt latent diffusion models to characterize the geometry of protein-peptide interactions. The first nontriviality stems from the full-atom geometry, which determines the comprehensive protein-peptide interactions in the atomic level, yet difficult to preserve. Throughout the generation process, the type of each amino acid always changes and thus requires us to generate different number of atoms, which is unfriendly to diffusion models that prefer fixed-size generation. Current latent diffusion models on molecules [71] or protein backbones [18] still tackle tasks with fixed number of atoms, thus leaving this challenge untouched. The second nontriviality lies in the variable binding geometry. Diffusion models are typically implemented directly in the data space, which might be suitable for regular data (e.g. images with fixed value range), yet ill-suited for our case on 3D coordinates where the value range is not fixed and even cursed with high variances due to the rich diversity in protein-peptide interactions. These variances define divergent target distributions of Gaussian with disparate expectation and covariance, which hinders the transferability of the diffusion process across different binding sites and thereby yields unsatisfactory generalization capability. Unfortunately, this point is seldom investigated previously.To address the above problems, we propose a powerful model for full-atom Peptide design with Geometric LAtent Diffusion (PepGLAD) with the following contributions:• We construct a new benchmark from PDB and literature based on practical relevance and non-redundancy, then systematically evaluate available sequence-structure co-design models on the task of target-specific peptide design. • To capture the full-atom geometry, we first learn a Variational AutoEncoder (VAE) to obtain a fixed-size latent representation (including a 3D coordinate and a hidden feature) for each residue of the input peptide, and then conduct the diffusion process in this latent space, both of which are conditioned on the binding site to better model protein-peptide interactions. Notably, the proposed design enables our model to accommodate full-atom input and output. • Regarding the variable binding geometry, we derive a shared standard space from the binding sites by proposing a novel skill-receptor-specific affine transformation. Such affine transformation is computed by the center offset and covariance Cholesky decomposition of the binding site coordinates, serving as a mapping from the binding site distribution to standard Gaussian. With the affine transformation applied to both the binding sites and the peptides, we are able to project the shape of all complexes into approximately standard Gaussian distribution, which facilitates generalization to diverse binding sites.Favorably, all the aforementioned models and processes meet the desired symmetry, i.e., E(3)equivariance, as proved by us. Experiments on sequence-structure co-design and complex conformation generation demonstrate the superiority of PepGLAD over the existing generative models. 
related work: 
 Peptide Design Conventional methods directly sample residues [6] or building blocks from libraries containing small fragments of proteins [26, 57, 9, 8] , with guidance from delicate physical energy functions [2] . These methods are time-consuming and easy to be trapped by local optimum. Recent advances with deep generative models mainly focuses on target-agnostic 1D language models [48] , antimicrobial peptides [13, 63] , or a subtype of peptides with α-helix [68, 69] . While geometric deep generative models are exhibiting notable potential in other domains of target-specific binder design (e.g. antibodies), their capability of target-specific peptide design remains unclear, which is the first problem we answer in this paper. Other contemporary work includes peptide design algorithms with flow matching frameworks [38, 40] .Geometric Protein/Antibody Design Protein design primarily aims to generate stable secondary or tertiary structures [27] , where diffusion models demonstrate inspiring performance [66, 58, 3, 72] . In particular, RFDiffusion [64] first generates backbones via diffusion, and then designs the sequences through cycles of inverse folding and structure refining with empirical force fields. Chroma [28] adopts a similar strategy, but further explores controllable generative process with custom energy functions. Antibody design, encompassing a special family of proteins in the immune system to capture antigens, mainly focuses on inpainting complementarity-determining regions (CDRs) at the interface between the antigen and the framework [33, 34, 61] , where the geometric diffusion models exhibit promising potential [44, 45] in co-designing sequence and structure. Unlike antibodies which are constrained by framework regions, peptides exhibit a more irregular binding pattern and greater flexibility, adapting to binding sites upon interaction [35] . Thus the target distributions are remarkably divergent on different binding sites, posing an urgent need for more robust generative modeling.Geometric Latent Diffusion Models Diffusion models learn a denoising trajectory to generate desired data distribution from a prior distribution, commonly standard Gaussian [54, 55, 25] . Recent literature extends diffusion to 3D small molecules satisfying the E(3)-equivariance [70] , which triggers subsequent advances in geometric design of macro molecules (e.g. antibody, protein) as aforementioned. Further efforts are made to latent diffusion models [53, 71, 18] , which implement the generative process in the compressed latent space of pretrained auto-encoders, to improve the performance. Compared to the literature that either encodes atom-wise representation in the latent space for small molecule generation [71] , or compress a fixed number of atoms into one latent node for protein backbone generation [18] , we explore compression of the full-atom geometry by directly generating different residues with variable number of atoms in the latent space. Moreover, we propose a novel technique, namely the data-specific affine transformations, to enhance the generalization ability of diffusion models, which is barely explored before.No single amino acid should constitute more than 25% of the fragment; Hydrophobic amino acids should comprise less than 45% of the fragment, with charged amino acids accounting for 25% to 45%.3. Isolated Stability: Instability [22] should be below 40; Considering the surrounding amino acids as interaction partners, the fragment should have a buried surface area (BSA) above 400Å2 [10] , with a relative BSA above 20%.We use FreeSASA [47] to calculate the surface area of fragments. Let SA bound represent the surface area of the isolated fragment when considering surrounding amino acids, and SA unbound represent the surface area when not considering them. The buried surface area is then calculated as BSA = SA unbound -SA bound , and the relative BSA is calculated as BSA rel = BSA/SA unbound . In total, we obtain 70,645 fragments meeting these criteria. 
our method: pepglad: 
 We first define the notations in the paper and formalize peptide design in §3.1. The overall workflow of our PepGLAD is presented in Figure 2 , which consists of three modules: (1) An autoencoder that defines the joint latent space for sequences and structures conditioned on the full-atom context of the binding site ( §3.2); (2) An affine transformation derived from the binding site to project the 3D geometry into a standard space approximating standard Gaussian distribution ( §3.3); (3) A latent diffusion model trained on the standard latent space ( §3.4). Finally, we summarize the training and the sampling procedures in §3.5.We represent binding sites and peptides as geometric graphs G = {(x i , ⃗ X i )}, where each node i is a residue with its amino acid type x i and the coordinates of all its c i atoms ⃗ X i ∈ R ci×3 . In later sections, we use the simplified notations i ∈ G to denote that a node i is in the geometric graph G, and |G| to denote the total number of nodes in G. We use G p and G b to represent the geometric graph of the peptide and the binding site, respectively. In this work, the binding site incorporates residues on the target protein within 10Å distances to the peptide residues based on C β atoms which alleviates leakage of the side-chain interactions. Note that the threshold (10Å) is chosen to be large to better reduce leakage of the peptide geometry. Task Definition Given the binding site G b , we aim to obtain a generative model p θ conforming to the distribution of binding peptides q(G p |G b ).The autoencoder [62] consists of an encoder E ϕ that encodes the peptide G p in the presence of the binding site G b into a latent state G z , and a decoder D ξ that reconstructs the peptide from the latent state to obtainG ′ p = {(x ′ i , ⃗ X ′ i )}.To encourage E ϕ to learn contextual representations of residues, we corrupt 25% of the residues in G p with a [MASK] type to obtain Gp as the input:EQUATIONwhereG z = {(z i , ⃗ z i )|i ∈ G p } contains the latent states z i ∈ R h (h = 8 in this paper) and ⃗ z i ∈ R 3 sampled from the encoded distribution N (z i ; µ i , σ i ) and N (⃗ z i ; ⃗ µ i , ⃗ σ i) using the reparameterization trick [32] . We borrow the adaptive multi-channel equivariant encoder in dyMEAN [34] for both E ϕ and D ξ to capture the full-atom geometry. In the decoder D ξ , we factorize the joint distribution of sequences and structures as follows:EQUATIONwhere the sequence is first decoded and then the all-atom geometry, initialized with replications of ⃗ z i , is reconstructed. The training objective of the autoencoder consists of the reconstruction loss L recon and the KL divergence L KL to constrain the latent space. The reconstruction loss includes cross entropy on the residue types, mean square error (MSE) on the full-atom structures, and an auxilary loss L aux on bond lengths and angles [31] :EQUATIONwhere H denotes cross entropy. We include details of L aux in Appendix A. The KL divergence constrains z i and ⃗ z i with the prior N (0, I) and N (⃗ r i , I), respectively, where ⃗ r i denotes the coordinate of the alpha carbon (C α ) in node i:EQUATIONwhere D KL denotes the KL divergence, λ 1 and λ 2 reweight the contraints on the sequence and the structure, respectively. L KL prevents the scale of z i from exploding and constrains ⃗ z i around C α to retain necessary geometric information. Such regularization also helps ensure consistent scales between the peptide latent coordinates and the pocket, mitigating potential issues arising from their different levels of abstraction. Then we have the overall training objective of the variational autoencoder as follows:EQUATIONWe have explored E(3)-invariant latent space, which appears to have difficulties in reconstructing the full-atom structures since it lacks information of geometric interactions with the pocket atoms (Appendix F).With the latent space given by the autoencoder, we further exploit a standard space obtained from receptor-specific affine transformations, which enhances the transferability of diffusions on disparate binding sites (see Figure 1 ). Most peptides fold into complementary shape upon binding on the receptor [60, 41] . Thus, the target distribution is inherently characterized by the shape of the binding site. Given the wide disparity in binding geometries, directly implementing diffusion in the data space yields minimal transferability among different binding sites. To address this deficiency, we propose to implement the diffusion process on a shared standard space converted via an affine transformation derived from the binding site. Formally, denoting the C α coordinates of the residues in a given binding siteG b as ⃗ R ∈ R 3×|G b | , we can derive their center ⃗ µ = E[ ⃗ R] ∈ R 3 and covariance ⃗ Σ = Cov( ⃗ R, ⃗ R) ∈ R 3×3, so that these coordinates can be regarded as sampled from the distribution N (⃗ µ, ⃗ Σ). We then calculate the Cholesky decomposition [19] of ⃗ Σ:EQUATIONwhere ⃗ L is a lower triangular matrix. ⃗ L is unique [19] and invertible since the covariance matrix is a real-valued symmetric positive-definite matrix2 . Then we can define the affine transformation F : R3 → R 3 , which enables the projection of the geometry into the standard space approximating standard Gaussian F ( ⃗ R) ∼ N (0, I). Further, we can easily obtain the inverse of F as:EQUATIONWith the above definitions, for each given binding site G b , we transform the geometry via the derived F to obtain the standard space, where the diffusion model is implemented, and recover the original geometry with F -1 (see Figure 2 ) after generation. Notably, we have the following proposition to ensure that the equivariance is maintained under the proposed affine transformation with scalarization-based equivariant GNNs [23, 16] : Proposition 3.1. Denote the invariant and equivariant outputs from a scalarization-based E(3)equivariant GNN as f ({h i , ⃗ x i }) and ⃗ f ({h i , ⃗ x i }), respectively. With the definition of F in Eq. 7, ∀g ∈ E(3), we have f({h i , F (⃗ x i )}) = f ({h i , F g (g • ⃗ x i )}) and g • F -1 ( ⃗ f ({h i , F (⃗ x i )})) = F -1 g ( ⃗ f ({h i , F g (g • ⃗ x i )})), where F g is derived on the coordinates transformed by g. Namely, the E(3)-equivariance is preserved if we implement the GNN on the standard space and recover the original geometry from the outputs.The proof is in Appendix B. This is vital since it indicates the Markov kernel is E(3)-equivariant, and thus ensures the E(3)-invariance of the probability density in the diffusion process [70] . Note that our variational autoencoder ( § 3.2) and latent diffusion model ( § 3.4) are already designed to be equivariant even without the proposed affine transformation here. The purpose of defining such component is to encourage better generalization of the diffusion processes. Indeed, it is nontrivial to analyze whether such an implementation will break the equivariance of our workflow. Luckily, Proposition 3.1 manages to prove that scalarization-based equivariant networks [23] , which is used in our autoencoder and diffusion model, are seamlessly compatible with such affine transformation, naturally preserving equivariance without any requirements of adaption.With the aforementioned preparations, the discrete residue types are encoded as continuous latent representations {z i }, and the full-atom geometry is also compressed and standardized into 3D vectors {⃗ z i } ∼ N (0, I). Therefore, we are ready to implement a diffusion model on the standard latent space to generate z i and ⃗ z i . The forward diffusion process gradually adds noise to the data from t = 0 to t = T , resulting in the prior distribution N (0, I). The reverse diffusion process generates data distribution by iteratively denosing the distribution from t = T to t = 0. We denote⃗ u t i = [z t i , ⃗ z t i ] and G t z = {(z t i , ⃗ z t i )}as the intermediate state for node i and the entire peptide at time step t, respectively. For simplicity, we assume both G t z and the binding site G b are already standardized via the transformation F b in Eq. 7. Then we have the forward process as:EQUATIONEQUATIONwhere β t is the noise scale increasing with the timestep from 0 to 1 conforming to the cosine schedule [51] , and ᾱt = s=t s=1 (1 -β s ). Then the state at timestep t can be sampled as:EQUATIONwhere ϵ i ∼ N (0, I). Following Ho et al. [25] , the reverse process can be defined with the reparameterization trick as:EQUATIONEQUATIONwhere α t = 1 -β t , and ϵ θ is the denoising network also implemented with the equivariant adaptive multi-channel equivariant encoder in dyMEAN [34] to retain full-atom context of the binding site during generation and preserve the equivariance under affine transformations (Proposition 3.1). Finally, we have the objective at time step t as MSE between the predicted noise and the added noise in Eq. 10, as well as the overall training objective L LDM as the expectation with respect to t:EQUATIONTraining The training of our PepGLAD can be divided into two phases where a variational autoencoder is first trained and then a diffusion model is trained on the standard latent space. We provide the overall training procedure in Algorithm 1 (see Appendix D). Note that a smooth and informative latent space is necessary for the consecutive training of the diffusion model, thus we resort to unsupervised data from protein fragments apart from the limited protein-peptide complexes for training the autoencoder, which we describe in Appendix E.The sampling procedure includes generative diffusion process on the standard latent states, recovering the original geometry with the inverse of F in Eq. 7, and decoding the sequence as well as the full-atom structure of the peptide (see Algorithm 2 in Appendix D). A problem here is that the unordered nature of graphs is not compatible with the sequential nature of peptides, thus the generated residues may have arbitrary permutation on the sequence order. Inspired by the concept of classifier-guided sampling [15] , we first assign an arbitrary permutation P on the sequence order to the nodes. Then we steer the sampling procedure towards the desired subspace conforming to P with the following empirical classifier p(1|{⃗ z t i }), which estimates the probability of the current coordinates belonging to the desired subspace:EQUATIONEQUATIONwhere µ d and σ d are the mean and variance of the distances of adjacent residues in the latent space measured from the training set. Intuitively, this classifier gives higher confidence if the adjacent (defined by P) residues are within reasonable distances aligning with the statistics from the training set. Nevertheless, the effect of the guidance is relatively minor, which is only a technical trick to enhance the robustness. We provide more details in Appendix G.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]Answer: [Yes] Justification: We provide codes and data for our model and the experiments.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
setup: 
 Task We evaluate our PepGLAD and baselines on the following tasks: (1) sequence-structure co-design ( §4.2) aims to generate both the sequence and the structure of the peptide given the specific binding site on the receptor (i.e. protein). ( 2) Binding Conformation Generation ( §4.3) requires to generate the binding state of the peptide given its sequence and the binding site of interest.Dataset We first extract all dimers from the Protein Data Bank (PDB) [5] and select the complexes with a receptor longer than 30 residues and a ligand between 4 to 25 residues [59] . Then we remove the duplicated complexes with the criterion that both the receptor and the peptide has a sequence identity over 90% [56] , after which 6105 non-redundant complexes are obtained. To achieve the cross-target generalization test, we utilize the large non-redundant dataset (LNR) from Tsaban et al. [59] as the test set, which contains 93 protein-peptide complexes with canonical amino acids curated by domain experts. We then cluster the data by receptor with a sequence identity threshold of over 40%, and remove the complexes sharing the same clusters with those from the test set. Finally, the remaining data are randomly split based on clustering results into training and validation sets, yielding a new bechmark calling PepBench. Further, we exploit 70k unsupervised data from protein fragments (ProtFrag) to facilitate training of the variational autoencoder. We also implement a split on PepBDB [65] based on clustering results for evaluation. We show details and statistics of these datasets in Appendix E.Baselines We first borrow three baselines from the antibody design domain. HSRN [30] autoregressively decodes the sequence while keeps refining the structure hierarchically, from the C α to other atoms. dyMEAN [34] is equipped with an full-atom geometric encoder and exploits iterative non-autoregressive generation. DiffAb [44] jointly diffuses on the categorical residue type, the coordinate of C α as well as the orientation of each residue. Next, we explore two baselines from the general protein design. RFDiffusion [64] exploits a pipeline that first generates the backbone via diffusion and then alternates between inverse folding [14] and structure refining based on a physical energy function [2] . AlphaFold 2 [31] is the well-known model for protein folding, which also shows certain abilities on peptide conformation prediction [59] . We also include two traditional methods.AnchorExtension [26] designs peptides by first docking an existing scaffold to the binding site, and then optimizing the peptide with cycles of mutations guided by energy functions. FlexPepDock [42] is designed for flexible peptide docking via optimization in the landscape of a physical energy function [2] . Implementation details are provided in Appendix I.Metrics A favorable generative model should produce diverse candidates while maintaining fidelity to the desired distribution. To comprehensively evaluate the models, we generate 40 candidates for each receptor and employ the following metrics: (1) Diversity. Inspired by [72] , we measure the diversity via unique clusters of sequences and structures. Specifically, we hierarchically cluster the structures based on pair-wise root mean square deviation (RMSD) of C α . The diversity of structures Div struct is defined as the number of clusters versus the number of candidates. A similar procedure can be applied to the sequences to obtain Div seq , utilizing the similarity [36] derived from alignment [24] . Then the co-design diversity is Div seq Div struct . (2) Consistency. We measure how well the models learn the 1D&3D joint distribution by the sequence-structure consistency, quantified via Cramér's V [12] association between the clustering labels (as in Diversity) of the sequences and the structures. High consistency indicates that candidates with similar sequences also have similar structures, implying that the generative model effectively captures the dependency between 1D and 3D.(3) ∆G. Aligned with the literature [34, 44] , we employ the binding energy (kcal/mol) provided by Rosetta [2] , a widely-used suite for biomolecular modeling with physical energy functions, to evaluate the binding affinity of the generated candidates. Lower ∆G indicates stronger binding between the peptide and the target. (4) Success. We report the proportion of successful designs (i.e. ∆G < 0, indicating no severe atomic clashes or twisted conformations) among all the candidates. For all metrics except ∆G, we first compute values for each receptor individually and then average the results across different receptors. For ∆G, we identify the best candidate on each receptor as the outputs and report the median value across different receptors. Details about the metrics are provided in Appendix H, including discussion on AAR (Appendix H.1) and consistency (Appendix H.2).Results Table 1 illustrates that our PepGLAD generates significantly more diversified and consistent peptides with better binding energy and success rates compared to the baselines. When benchmarking HSRN, dyMEAN, and DiffAb, which perform well on antibody CDR design, we observe a notable performance gap between non-diffusion baselines (i.e. HSRN, dyMEAN) and the diffusion-based baseline (i.e. DiffAb), suggesting the higher complexity in peptide design and the need for stronger modeling capabilities. Compared to DiffAb, which operates on categorical residue types, C α coordinates and orientations, our PepGLAD (1) better captures the dependency between sequence and structure, as indicated by higher diversity and consistency, since diffusion is implemented on the latent space where the representation of sequence and structure are nicely correlated by the autoencoder; (2) more effectively captures the intricate protein-peptide interactions, demonstrated by better ∆G and success rates, since we leverage the full-atom context of the binding site and enhances generalization capability by converting the geometry into a standard space. We showcase two candidates designed by our PepGLAD with favorable binding energy given by Rosetta in Figure 3 . Furthermore, the diversity within successful designs is 0.632, which is higher than that of all designs (0.506), indicating the high structural flexibility of peptides upon successful binding. Bottom: A generated candidate with complementary shape to the binding site (PDB=3pkn, ∆G=-33.32). Both candidates form compact interactions at the interface.PepGLAD PepGLAD Receptor Peptide Hydrophilic HydrophobicWe also evaluate our PepGLAD against two sophisticated pipeline systems in Table 2 . The traditional method (i.e. AnchorExtension) is limited by low efficiency, thus we can only afford outputting 10 candidates for each receptor. For a relatively fair comparison with RFDiffusion, we refine the structure of the generated candidates using the empirical force field in RFDiffusion. However, the comparison may still disadvantage our PepGLAD, given that RFDiffusion is finetuned from a model pretrained on a large-scale dataset [64] . Nevertheless, as demonstrated in Table 2 , our model still exhibits marvelous superiority on diversity, consistency, and success rate, while achieving competitive binding energy ∆G, with obviously higher efficiency. Results As shown in Table 3 , our PepGLAD surpasses all the baselines in terms of both RMSD Cα and DockQ by a large margin, highlighting the superiority of incorporating the full-atom context and the binding-site shape into the latent diffusion process. Additionally, we present the distribution of the best RMSD Cα on different test receptors using box plots and showcase a generated conformation highly resembling the reference in Figure 4 . The distribution reveals that our model achieves favorable performance on RMSD Cα with lower variance on the test set compared to other baselines, exhibiting robust generalization ability across disparate binding sites.  
analysis: 
 We conduct the following ablations: the full-atom geometry (Full-Atom); the affine transformation (Affine); the unsupervised data from protein fragments (ProtFrag) and the mask policy (Mask) when training the autoencoder. Note that generative performance is assessed from various aspects, and improvement in one aspect at the disproportionate expense of others might be meaningless. Thus, we additionally compute the average of all the metrics to evaluate the comprehensive effect of each module, where ∆G is normalized by the statistics on the test set. Table 4 demonstrates the following observations:  
limitations: 
 Despite the promising results, we acknowledge several limitations which might be addressed by future work. First, the binding affinity assessment relies on the Rosetta scoring function as a proxy for wetlab experiments. There may be discrepancies between the predicted and actual binding energies.The ultimate test of a peptide utility is its performance in vivo, which is too costly for large-scale evaluation. Nevertheless, this is a problem confronting the entire community, and we hope future research might propose more reliable in silico proxies to bridge the gap. Second, while this paper addresses peptide design from the aspect of proteins, it might also be reasonable to think from the aspect of small molecules if the peptides are short enough. Under such circumstances, it is also beneficial to further explore counterparts of methods for small molecule design [43, 52, 39] , which we leave for future work.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See § 4.1, Appendix E and Appendix I.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this paper, we first assemble a dataset from Protein Data Bank (PDB) and literature to benchmark generative models on target-specific peptide design in terms of diversity, consistency, and binding energy. Subsequently, we propose PepGLAD, a powerful diffusion-based model for full-atom peptide design. In particular, we explore diffusion on the latent space where the sequence and the full-atom structure are jointly encoded by a variational autoencoder. We further propose a receptor-specific affine transformation technique to project variable geometries in the data space into a standard space, which enhances the transferability of diffusion processes on disparate binding sites. Our PepGLAD outperforms the existing models on sequence-structure co-design and binding conformation generation, exhibiting high generalization across diverse binding sites. Our work represents a pioneering effort in the exploration of deep generative models for simultaneous design of 1D sequences and 3D structures of peptides, which could inspire future research in this field.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The size of the dataset is large, and we report results on two different datasets for more solid evaluation.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not pose such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: They are properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The curated PepBench and ProtFrag are available at https://zenodo.org/records/13373108. The codes for our PepGLAD are open-sourced at https://github.com/THUNLP-MT/PepGLAD.This paper aims to advance the field of peptide design through the construction of a benchmark and the development of a novel latent diffusion model, PepGLAD, which addresses key limitations in current methods. Our work represents a step forward in computational peptide design, with the potential to impact both scientific research and practical applications in various domains. For instance, more precise peptide design could lead to enhanced drugs in the pharmaceutical industry, and could facilitate the creation of new biomaterials, sensors, and other innovative technologies in biology and materials science. We wish our paper could inspire future research in this field.To better recover the all-atom geometry, we employ an auxilary structural loss similar to the violation loss in Jumper et al. [31] including the supervision on C α coordinates, bond lengths, and side-chain dihedral angles. First, since the C α is critical in deciding the global geometry of the peptide, we exert additional loss on its coordinates to distinguish it from other atoms:EQUATIONwhere ⃗ r ′ i and ⃗ r i are the reconstructed and the ground truth coordinates of C α in node i. Next, we implement L1 loss on the bond lengths:EQUATIONwhere B(i) includes all chemical bonds in node i, and b ′ denotes the reconstructed bond length. For simplicity, he bonds between residues are included into the bonds of the former residue. Finally, we supervise on the χ 1 to χ 4 side-chain dihedral angles [73] :EQUATIONwhere A(i) includes all side-chain dihedral angles in node i, χ ′ and χ denotes the reconstructed and the ground truth angles, respectively. The overall auxilary loss for node i is then given by:EQUATIONwhere we set λ CA = 1.0, λ bond = 1.0, λ angle = 0.5 in our experiments. We find that it is necessary to set λ angle with a relatively small value to make the training process stable.Preserving atom instances enhances the modeling of side chain interactions but introduces challenges due to potential twisting in local geometry. While in sequence-structure co-design, samples undergo fast relax by physical force field to ensure a valid local geometry, in binding conformation generation, we use an alignment technique to place the idealized side chains on the generated atom instances. Specifically, the idealized side chain can be represented as at most 4 dihedral angles (χ-angles), treating fragments like phenyl group as rigid bodies. Suppose there are n i χ-angles and c i atoms in node i, we can define a function to map from the χ-angles with the backbone coordinates ⃗ B i ∈ R 4×3 to atom instances as M i : R ni × R 4×3 → R ci×3 , which is luckily differentiable [28] . Thus we can optimize χ-angles via gradient descent to minimize the MSE between the coordinates constructed from χ-angles and those generated by the model:EQUATIONwhere ⃗ X i ∈ R ci×3 denotes the coordinates of c i atom instances in node i generated by the model. Now it should be easy to construct an idealized side chain maintaining fidelity to the generated atom instances by M (χ * i , ⃗ B i ). In the experiments of conformation generation on PepBench, such idealization gives slightly better DockQ and RMSD atom than Rosetta side-chain packing algorithm but with higher efficiency.B Proof of Proposition 3.1 Proposition 3.1. Denote the invariant and equivariant outputs from a scalarization-based E(3)equivariant GNN as f ({h i , ⃗x i }) and ⃗ f ({h i , ⃗ x i }), respectively. With the definition of F in Eq. 7, ∀g ∈ E(3), we have f({h i , F (⃗ x i )}) = f ({h i , F g (g • ⃗ x i )}) and g • F -1 ( ⃗ f ({h i , F (⃗ x i )})) = F -1 g ( ⃗ f ({h i , F g (g • ⃗ x i )})), where F g is derived on the coordinates transformed by g. Namely, the E(3)-equivariance is preserved if we implement the GNN on the standard space and recover the original geometry from the outputs.For simplicity, given F derived from a set of coordinates {⃗ x i } following Eq. 7, we use F g to indicate the affine transformation derived from {g • ⃗x i }, where g ∈ E(3). Additionally, we keep the terminology "standard space" for describing the space after the data-specific affine transformation F . We begin by proving a key lemma, that is, the E(3)-invariance of distances between two nodes in the standard space converted by F : Lemma C.1. Given two nodes i and j in the geometric graph G, denoting their coordinates as ⃗x i and ⃗ x j , their distance in the standard space is E(3)-invariant. Namely, ∀g ∈ E(3), ∥F (⃗x i ) -F (⃗ x j )∥ = ∥F g (g • ⃗ x i ) -F g (g • ⃗ x j )∥.Proof. ∀g ∈ E(3), g can be instantiated as an orthogonal matrix Q ∈ O(3) (including rotation and reflection), and a translation vector ⃗ t ∈ R 3 . Denoting all coordinates in the geometric graph as ⃗ X ∈ R 3×|G| , and the number of nodes in G as n, we can derive the E(3)-equivariance of the expectation of the coordinates:EQUATIONWith the E(3)-equivariance of the expectation, it is easy to derive the following equation on the covariance matrix:EQUATION)Based on the Cholesky decomposition used in the derivation of F , we denoteCov( ⃗ X, ⃗ X) = ⃗ L ⃗ L ⊤ and Cov(g • ⃗ X, g • ⃗ X) = ⃗ L g ⃗ L ⊤g , with which we can immediately derive:EQUATIONConsidering Q -1 = Q ⊤ , we further have the following equation:EQUATION)Given thatF (⃗ x) = ⃗ L -1 (⃗ x -E[ ⃗ X]), we are now ready to prove Lemma C.1 as follows:EQUATIONEQUATIONEQUATIONEQUATIONWith Lemma C.1, we are able to give the proof of Proposition 3.1 as follows.Proof. A first observation is that to prove Proposition 3.1, we only need to prove the equivariance in the 1-layer case, since the multi-layer case can be decomposed into the 1-layer case by inserting I = F • F -1 between layers, where I is the identical mapping. Generally, each layer in scalarizationbased E(3)-equivariant GNN has the following paradigm:EQUATIONEQUATIONEQUATIONwhere N (i) denotes the neighborhood of node i, and ϕ x outputs a scalar. Therefore, for the invariant part, we have:EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONFor the equivariant features, recallingF -1 (⃗ x) = ⃗ L⃗ x + E[ ⃗ X],we have:EQUATIONEQUATIONEQUATIONBy replacing g with identical element I in E(3), since F = F I , we can immediately derive:EQUATIONEQUATIONwhich concludes Proposition 3.1.We present the pseudo codes for training in Algorithm 1 amd sampling in Algorithm 2. Gp ← mask(G p ) {Mask 25% Residues} 6:{(µ i , σ i , ⃗ µ i , ⃗ σ i )} ← E ϕ ( Gp , G b ) {Encoding} 7: {(ϵ i , ⃗ ϵ i )} ∼ N (0, I) {Reparameterization} 8: G z ← {(µ i + ϵ i ⊙ σ i , ⃗ µ i + ⃗ ϵ i ⊙ ⃗ σ i )} 9: G ′ p ← D ξ (G z , G b ) {Decoding} 10: L AE = i∈Gp (L recon (i) + L KL (i))/|G p | 11:ϕ, ξ ← optimizer(L AE ; ϕ, ξ)12:end while Sample (G p , G b ) ∼ S 20: {(µ i , σ i , ⃗ µ i , ⃗ σ i )} ← E ϕ (G p , G b ) {Encoding} 21: G 0 z ← {(µ i , ⃗ µ i )} 22: F ← affine(G b ) {Affine Transformation} 23: G 0 z , G b ← F (G 0 z ), F (G b ) {Standard Geometry} 24: {(z 0 i , ⃗ z 0 i )} ← G 0 z 25: t ∼ U(1, T ), {(ϵ i , ⃗ ϵ i )} ∼ N (0, I) 26: G t z ← { √ ᾱt [z 0 i , ⃗ z 0 i ] + (1 -ᾱt )[ϵ i , ⃗ ϵ i ]} 27: L t LDM = i ∥[ϵ i , ⃗ ϵ i ] -ϵ θ (G t z , G 1 , t)[i]∥ 2 /ϵ θ ← TrainLatentDiffusion(E ϕ , D ξ , S) 36: return E ϕ , D ξ , ϵ θWe show details for constructing the datasets used in our paper here. Further statistics are presented in Table 5 and distribution of peptide lengths in Figure 5 .We exploit unsupervised data from monomer proteins to enrich the training of the autoencoder. Specifically, we first extract all single chains from the Protein Data Bank (PDB) before December 8th, 2023, and remove the duplicated chains on a sequence identity threshold of over 90%. Then, for each chain, we extract fragments satisfying the following criteria:Algorithm 2 Sampling Algorithm of PepGLAD input decoder D ξ , denoising network ϵ θ , binding site G b output peptide G p 1: F ← affine(G b ) {Affine Transformation} 2: G b ← F (G b ) {Standard Geometry} 3: {(z T i , ⃗ z T i )} ∼ N (0, I) 4: for t in T, T -1, • • • , 1 do 5: G t z ← {(z t i , ⃗ z t i )} {Latent Denoising Loop} 6: ⃗ u t i ← z t i , ⃗ z t i 7: ε i = [ϵ i , ⃗ ϵ i ] ∼ N (0, I) 8: ⃗ u t-1 i ← 1 √ α t (⃗ u t i -β t √ 1-ᾱt ϵ θ (G t z , G b , t)[i]) + β t ε i 9: [z t-1 i , ⃗ z t-1 i ] ← ⃗ u t-1 i 10: end for 11: G z ← {(z 0 i , ⃗ z 0 i )} 12: G z , G b ← F -1 (G z ), F -1 (G b ) {Data Geometry} 13: G p ← D ξ (G z , G b ) {Decoding} 14: return G p 1. Length:The fragment should consist of 4 to 25 residues.Here we illustrate the details for constructing the supervised dataset (PepBench) used in our paper. Similar to literature [65] , we also exploits available data from the Protein Data Bank (PDB) [5] . We first extract all dimers in PDB deposited before December 8th, 2023, and filter out the complexes with a receptor longer than 30 residues and a ligand between 4 to 25 residues, which aligns with Tsaban et al. [59] . Peptides with lengths in this range are more relevant to practical applications such as drug discovery, as they exhibit favorable biochemical properties [49] . Then we remove the duplicated complexes with the criterion that both the receptor and the peptide has a sequence identity over 90% [56] , after which 6105 non-redundant complexes are obtained. We use MMseqs2 for clustering based on sequence identity:# create database from the sequences mmseqs create seqs.fasta database # clustering with sequence identity above 90% mmseqs cluster database database_clusters results --min-seq-id 0.9 -c 0.95 --cov-mode 1To achieve the cross-target generalization test, we utilize the large non-redundant dataset (LNR) introduced by Tsaban et al. [59] as the test set, which is curated by domain experts. LNR originally includes 96 protein-peptide complexes. We obtain 93 complexes after excluding the ones with non-canonical amino acids. We then cluster the LNR along with the PDB data by receptor with a sequence identity threshold of over 40%. Subsequently, we remove the complexes sharing the same clusters with those from the test set and those including non-canonical amino acids in the peptides. Finally, the remaining data are randomly split based on clustering results into training and validation sets. The characteristics of different splits are presented in Table 5 . In addition, the binding site contains residues on the receptor within 10Å distances to the peptide, where the distance between two residues is measured by the distance between their C β coordinates.Similar to the split of PepBench, we use MMseqs2 for clustering and randomly split the data into training, validation, and test sets based on the clustering results. For the test set, we randomly select one protein-peptide complex in each cluster to avoid redundancy. We found that the E(3)-equivariant latent vectors were significant, which can convey sufficient geometric information. If we use E(3)-invariant latent space without these geometric latent vectors, the reconstruction ability (RMSD, DockQ) of the VAE deteriorates significantly (Table 6 ). It is reasonable since E(3)-invariant latent space lacks geometric interactions with the pocket atoms, leading to difficulties in reconstructing the full-atom structures on the binding site. Peptides consist of linearly connected amino acids, which exert constraints on the 3D geometry. Specifically, residues adjacent in the sequence should also be close in the structure, since they are connected by a peptide bond. However, 3D graphs are unordered and do not incorporate such induct bias, which means the generated nodes might have arbitrary permutation on sequence orders. To tackle this problem, we take inspiration from classifier-guided diffusion [15] , which adds the gradient of a classifier to the denoising outputs to guide the generative diffusion process towards the subspace where the classifier gives high confidence. We utilize an empirical classifier p(1|{⃗ z t i }) defined as follows:EQUATIONEQUATIONwhere µ d and σ d are the mean and variance of the distances of adjacent residues in the latent space measured from the training set. With p(1|{⃗ z t i }), we are able to assign an arbitrary permutation on sequence orders to the nodes, and steer the sampling procedure towards the desired subspace conforming to P, since this classifier gives higher confidence if the adjacent (defined by P) residues are within reasonable distances aligning with the statistics from the training set. In particular, the coordinate denoising outputs are refined as follows:EQUATIONwhere λ adjusts the weight of the guidance. Besides the constraints on the distance between adjacent residues, we can also include guidance on avoiding clashes between non-adjacent residues by defining the following energy term:EQUATIONSubsequently, we just need to revise the empirical classifier as:EQUATIONwhere ⃗ r j is the C α coordinate of node j in the binding site. We observe a slight improvement upon including the clash energy term. Nevertheless, the guidance is only a small technical trick with minor enhancement on the performance as shown in Table 7 . While the amino acid recovery (AAR) is widely used in antibody design [33, 34, 44] , we find it not informative enough for evaluating generative models on learning the distribution of binding peptides, due to the vast and highly diverse solution space. To elucidate this, we conducted an analysis of AAR using a sequence dataset from Xia et al. [67] including 328 receptors and 600k peptides with binary binding labels, from which we filter out 102 receptors with well-explored solution space (i.e. with at least 500 binders). For each receptor, we randomly select one binder as the reference and sample N candidates according to a specified positive ratio r, resembling the scenario of evaluating generative models for peptide design. For example, setting r = 0.1 and N = 100 involves sampling 10 binders and 90 non-binders to construct the candidates. Then we compute the best AAR of these candidates with respect to the reference as the result. This process is repeated for 10 times per receptor, and the results are averaged across different receptors, which can be interpreted as the evaluation score of AAR on a generative model that can generates candidates with a positive ratio of r. We select N = 10, 40, 100 and enumerate the choices of r to derive the relation plot in Figure 6 , where the results of a random sequence generator is also included for comparison. It can be derived that the gap of AAR between the worst model (r = 0.0) and the best model (r = 1.0) is insignificant. Furthermore, all models, including the best model (r = 1.0) which always produces positive samples, exhibit performance akin to the random sequence generator, with consistent trends regardless of the choice of N . We attribute this to the vast solution space of peptide design, where evaluating with dozens of candidates relative to a single reference is unreliable. In other words, achieving a high AAR is improbable since the model would need to fortuitously explore the subspace around the reference, which is arbitrary, on every test receptor; Conversely, a low AAR does not necessarily denote a poor generative model, as the model may be exploring distinct solutions from the single reference. Moreover, we calculate the Spearman correlation between the receptor-level best AAR and the positive ratio r of the candidates, yielding 0.23, 0.22 and 0.24 for N = 10, 40, and 100, respectively, indicating very weak correlation. Based on this analysis, we assert that AAR is unsuitable for evaluating target-specific peptide design. The comparison of AAR and success rates on PepBench (Table 8 ) also indicates that models with similar AAR can have distinct success rates. While it is common to evaluate consistency by comparing generated structures with AF2-predicted structures [7] , such a "supervision-based" method suffers from severe limitations in the situation where AF2 [31] fails to achieve an acceptable performance. As highlighted not only in Tsaban et al. [59] but also demonstrated by our experiments, even state-of-the-art models like AF2 struggles to consistently produce high-quality structures of protein-peptide complexes. Our findings reveal that only 36% of test samples could be accurately predicted within a 5Å RMSD (considered as near-native conformation) by AF2, indicating even the ground truth can only achieve 36% success rates if we use such supervison-based consistency for evaluation, making such evaluation not reliable in the domain of peptide design.In light of this limitation, we propose an "unspervised" evaluation framework to assess consistency, that is, the statistical association between the clustering results of sequences and structures. Theoretically, this serves the necessary condition for true consistency. Namely, if a model truly captures the consistency between sequence and structure, it will necessarily achieve a high score on the proposed metric. Conversely, if a model fails to attain a high score on the proposed metric, it is not possible to capture true consistency.In our experiments, we found that our proposed consistency metric effectively distinguishes nonconsistent modeling methods, such as HSRN, which tend to produce disparate sequences while sharing identical structures.Indeed, evaluating generative models comprehensively is crucial, requiring assessment from multiple perspectives. Generally, these evaluations can be categorized into two main aspects: diversity and fidelity to the desired distribution. Regarding diversity, we take inspiration from Yim et al. [72] and quantify it with the number of unique clusters relative to the number of candidates. This metric provides insight into the variety and richness of the generated samples. For fidelity, the primary focus should be the binding affinity, for which we adopt the physical energy from Rosetta [2] since it is widely used in various domains and exhibit robust generalization capability [1, 37, 64] . Further, considering the dependency between sequence and structure, we propose the consistency metric, which is critical for distinguishing whether the generative model is capturing the 1D&3D joint distribution and thereby truly facilitating "co-design". We have also discussed the reason for implementing the consistency metric with the "unsupervised" fashion in the above section. Below, we outline the implementation of these metrics.Diversity We hierarchically cluster the sequences and the structures with aligning score [36] and RMSD of C α , respectively. In particular, we implement the aligning score using Biopython [11] with BLOSUM62 matrix [24] and Needleman-Wunsch algorithm [50] . The thresholds for clustering is similarity above 0.6 and RMSD below 4.0 for sequence and structure, respectively. We provide the python codes below for clearer presentation: Denoting the diversity of the sequences and the structures as Div seq and Div struct , respectively, we calculate the co-design diversity as Div seq Div struct . Success Since a negative ∆G value typically indicates a potential for binding, we report the ratio of all generated candidates that satisfy this threshold. Moreover, candidates with ∆G < 0 usually are at least physically valid (i.e. without obvious atomic clash), thus it is meaningful to use this success rate to evaluation the generative ability of the models.We train PepGLAD on a GPU with 24G memory with AdamW optimizer. For the autoencoder, we train for 60 epochs with dynamic batches, ensuring that the total number of edges (proportional to the square of the number of nodes) remains below 60,000. The initial learning rate is 10 -4 and decays by 0.8 if the loss on the validation set has not decreased for 3 consecutive epochs. Regarding the diffusion model, we train for 500 epochs with the same batching strategy as for the autoencoder. The learning rate is 10 -4 and decay by 0.6 if the loss has not decreased for 3 consecutive validations, where the validation is conducted every 10 epochs. In the experiment of binding conformation generation, we only use the supervised dataset, thus we extend the training epochs for the autoencoder and the diffusion model to 500 and 1000, respectively. Consequently, the patience of learning rate decay is extended to 15 epochs for training the autoencoder. We keep other settings unchanged. The hyperparameters of PepGLAD used in our experiments are provided in Table 9 . For HSRN [30] , dyMEAN [34] , and DiffAb [44] , we directly integrate their official implementation into the same training framework as our PepGLAD, and adjust the batch size, learning rate, and training epochs to obtain the optimal performance, which we present in Table 10 .We outline the implementation of other baselines below: RFDiffusion [64] We follow the official instruction to randomly select 20% of residues on the binding site as "hotspots", and generate the backbone via diffusion followed by cycles of inverse folding with ProteinMPNN [14] and full-atom structural refining with the officially provided Rosetta protocol.AnchorExtension [26] It is implemented with the Rosetta suite, thus we resort to the official release of the pipeline protocols for docking and optimizing. We use the default parameters and generate 10 candidates for each receptor due to its limitation of efficiency. We randomly pick one peptide in the training set with the same length as the reference peptide as the initial motif for docking.FlexPepDock [42] We follow its official tutorial to implement this baseline in the C++ version of Rosetta.AlphaFold2 [31] We borrow the results from [59] , which explores two strategy to use AlphaFold2 on peptide conformation generation, including the receptor and the peptide as separate chains or link them together with long loops. The results contain a total of 10 candidates for each receptor, with 5 from the separate strategy and 5 from the linked strategy.NeurIPS Paper Checklist Justification: We have an independent section ( § 6) to discuss limitations. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.The binding site has at leastnonoverlapping nodes, namely rank( ⃗ R) = 3, thus we can ignore the corner case of semi-positive definite matrices.HSRN and dyMEAN generate homogeneous structures that are clustered together yet still sample very different sequences, leading to zero association between squence and structure. 


Decision: This paper focuses on the problem of designing peptides conditioned on a protein pocket. To that end, the authors first gather a dataset based on the Protein Data Bank, and then propose a diffusion-based model called PepGLAD, which combines several careful modelling and training choices appropriate for the domain. 

Reviewers generally considered PepGLAD to be a significant advancement on top of prior work, especially given its ability to do full-atom generation (instead of backbone only). The paper introduces several novel ideas, such as the affine transformation between the pocket position and a normal distribution, which most reviewers found interesting and appropriate for the downstream task. Apart from the model itself, this work also carefully curates a dataset with a reasonable train/test split; the reviewers noted the dataset is quite small, but they still considered it to be an important contribution to the field. While the initial version of the paper sparked many clarification questions, these were mostly addressed by the authors during the rebuttal period. In particular, one of the reviewers noted a subtle point around how the protein pocket is defined during data generation (which could potentially "leak" the shape of the ground-truth peptide); the authors did consider this nuance, and their current approach seems satisfactory. 

In summary, this paper curates a useful dataset for an important research problem, and builds an effective model on that dataset, combining a mix of novel and pre-existing ideas. I recommend the paper to be accepted, but I would invite the authors to take the clarification questions from the reviewers seriously, in order to make sure the manuscript is more accessible. In particular, for the final version of the paper I would encourage the authors to include a description of how the protein pocket is defined and what potential issues could arise from not doing this carefully.

Reviews: review: paper_summary: paper_summary: The authors proposed PepGLAD, a latent diffusion model for full-atom peptide design. This work mainly addressed two challenges for peptide design, (1) full-atom modeling, and (2) diverse binding geometry. Specifically, a variational auto-encoder was trained to learn latent representations of protein-peptide interactions. Then, a latent diffusion model was employed to capture the underlying distribution to sample binding site-conditioned peptide conformations. The authors curated a benchmark dataset to evaluate the model performance, and compared with existing models to showcase its effectiveness.
strength_weakness: strength_weakness: - Dataset is small (6105 non-redundant complexes). Data augmentation attempts (ProtFrag) do not seem to help (Table 4 ablation results). Although sequence-based clustering has been utilized to prevent data leakage, training/testing VAE and diffusion model on 6105/93 data points make the results less convincing, *especially considering the enormous design space for sequence-structure co-design tasks*.
- Guidance with an additional classifier (Sec 3.5) makes the model more complicated (which seems a bit ad hoc to me).
- From the results shown in Table1, it seems that the model is able to generate a variety of conformations (high Div.) which are reasonable ($\Delta G < 0$). Given the extremely limited size of dataset, I am not sure if the model can properly capture the *conformation space* at all. In other words, I am a bit skeptical about the proposed evaluation metrics.
questions: questions: - L#130. For each residue, the initial coordinates are initialized with the same $\vec{\boldsymbol{z}}_i$. How significant are these initial vectors? Can we use an SE(3)-invariant latent space and a regression model to generate atomic coordinates (similar to AlphaFold StructureModule, conditioned on the sampled residue type)?
- Classifier guidance seems ad hoc and brings more complication to modeling. Would an architecture similar to that used in [ESM3](https://www.biorxiv.org/content/10.1101/2024.07.01.600583v1) or [PVQD](https://www.biorxiv.org/content/10.1101/2023.11.18.567666v1) help?
- Table 1: out of 40 samples generated by PepGLAD, there are about 20 different conformations (Div.) and 20 valid peptides (Success). How should the readers interpret this? Peptides exhibit high structural flexibility upon binding?
- Did I miss sequence recovery metrics in the model benchmark section (excluding Consistency, which only models the joint distribution)?
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces a novel latent diffusion model, Peptide design with Geometric Latent Diffusion (PepGLAD), for the task of peptide design. The authors propose an affine transformation to project the raw Euclidean space into a standardized one, ensuring physical symmetry. Overall, I think it is a good paper, with solid experimental results and novelty in methods.
strength_weakness: strength_weakness: There are several doubts on method contributions, datasets, as shown in Questions.
questions: questions: 1. What are the key differentiating factors of PepGLAD compared to GeoLDM [1]? While I recognize the shift from latent molecular generation to full-atom peptide generation, are there any specific model design elements (apart from the affine transformation) that set it apart from GeoLDM? Furthermore, does the model incorporate any strategies to reduce the costs associated with generating full-atom peptide scenarios?

2. The most significant contribution made by the authors in this paper is the Receptor-Specific Affine Transformation. This transformation projects the geometry to approximately N(0,I), derived from the binding site. This approach appears to be the first of its kind applied to latent diffusion models for molecules and proteins. I am curious if there are any other works that have used this technique to ensure equivariance in their generation process, such as frame-averaging [6] [7].

3. Compared to previously established databases [2] [3], the datasets utilized in your study appear relatively modest in size. Could you comment on the unique advantages of your proposed datasets over these larger ones? Furthermore, have you considered applying your model to existing datasets to reinforce the validity of your results?

4. I suggest to add more citations on recently proposed related works on peptide design [3] [4] [5].

[1] M Xu, et al. Geometric Latent Diffusion Models for 3D Molecule Generation

[2] Z Wen, et al. Pepbdb: a comprehensive structural database of biological peptide-protein interactions.

[3] L Lin, et al. PPFlow: Target-Aware Peptide Design with Torsional Flow Matching

[4] Osama Abdin, et al. PepFlow: direct conformational sampling from peptide energy landscapes through hypernetwork-conditioned diffusion

[5] Colin A Grambow, et al. RINGER: Conformer Ensemble Generation of Macrocyclic Peptides with Sequence-Conditioned Internal Coordinate Diffusion

[6] W Jing, et al. DSMBind: SE(3) denoising score matching for unsupervised binding energy prediction and nanobody design

[7] Omri Puny, et al. Frame Averaging for Invariant and Equivariant Network Design
limitations: limitations: See Appendix. J Limitations
paper_summary: paper_summary: This paper explores the structure-based peptide design problem. A benchmark on this task and  a powerful diffusion-based model for full-atom peptide design, named PepGLAD, are proposed. PepGLAD explores the geometric latent diffusion, where the sequence and the full-atom structure are jointly encoded by a variational autoencoder. The proposed method outperforms the existing models on sequence-structure co-design and binding conformation generation.
strength_weakness: strength_weakness: 1. This paper failed to further explore the explicit interaction modeling in geometric latent space. I think it’s the key advantage compared to standard latent space (without the geometric latent).
2. The introduction to Receptor-Specific Affine Transformation can be more clear. I kindly suggest the authors can add one sentence or two summarizing the first paragraph in section 3.3 into the section 1 of Introduction. In such way readers may be more clear that this Affine Transformation is applied to the peptides and the pockets (i.e. binding site distribution in the introduction). Also the point 3 in line 62-65 has a mistake: according to the method section the affine transformation F is mapping binding site distribution (of peptides) into standard Gaussian (not the reverse)
3.  The conditioning method of pockets is not explored. The denoiser takes the geometric latents of peptides and the coordinates of pockets as input, but the latents are coordinates are different levels of abstraction and this may cause some underlying issues.
questions: questions: 1. What’s the motivation for using Latent Diffusion instead of standard diffusion? Just because it’s better in image generation domain?
2. How to incorporate geometric pocket condition (e.g. coordinates) into geometric latent diffusion process?
3. Is the geometric latent are just the Ca atoms of each residues? Will using all four backbone atoms as latent improve the perforemance？
4. Is the adaptive multi-channel equivariant encoder in dyMEAN a scalarization-based equivariant GNNs? It’s better to add some explainations in this paper.
5. Does this Receptor-Specific Affine Transformation beat simple gaussian normalization, common trick in Structure-based Drug Design field?
limitations: limitations: The authors didn’t discuss about the limitations of this method. This paper failed to further explore the explicit interaction modeling in geometric latent space. I think it’s the key advantage compared to standard latent space (without the geometric latent).
paper_summary: paper_summary: The paper presents a new diffusion model for generating peptide binders given protein pockets. It also presents a new benchmark dataset, created by selecting examples from PDB and ensuring sequence dissimilarity between training and test data.
strength_weakness: strength_weakness: In the introduction and abstract, please state clearly that PepGLAD needs to be given the binding site (not the whole protein structure).

Line 24 is hard to understand: ‘The key of peptide design …’.  It seems to say that strong binding requires the peptide to adopt a compact, inflexible shape, but I do not understand why.

The argument for the affine transformation is unclear.  I could not decipher line 48 ‘These variances define divergent target distributions of Gaussian…’.  Plenty of methods (AlphaFold, RFDiffusion, GeoDiff, …) are able to predict or design elongated protein and molecule shapes without any such transform, and here it should be easy because the binding site shape is given as a conditioning input. Surely a model can learn that the binder it generates should approximately match the shape of the binding site, even if the binding site is far from spherical?

I think that proposition 3.1 could be stronger. Please check, but I think that if $x$ is rotated, then $L$ undergoes the same rotation, and hence $F_g(g.x)=F(x)$. Thus, $f(h, F(x))$ is invariant and $F^{-1}\overrightarrow{f}(h, F(x))$ is equivariant even if $f$ and  $\overrightarrow{f}$ are not.

I am surprised that this transformation improves performance.  The score model in the transformed space now has to learn that bonds in different orientations should have different lengths, inferring from the inter-residue distances of the binding site how much each dimension is compressed or stretched.
questions: questions: Why do you need to generate all-atom designs? Do you think it could work just as well to generate C-alpha coordinates and discrete residue types? Mixed discrete/continuous diffusion models have shown some success for small-molecule drug design (DiffSBDD) and crystal design (MatterGen).

Line 223: does ‘filter out’ mean ‘keep’ or ‘discard’? 

The dataset seems small for deep learning and for the complexity of the task.   How could it be augmented?

Table 3: what counts as a good RMSD, for practical purposes? Are all the methods given the same inputs, or are some doing blind docking?

Equation (3): how is the MSE on full-atom structures defined when the residue type is wrong?
limitations: limitations: Like all peptide design in silico, this is hamstrung by lack of good in silico evaluation: dG values from Rosetta are not reliable estimates of real binding affinity. Ideally there would be some wet-lab evaluation of the generated designs.

The method requires binding site to be identified before peptide binders are designed.

Comments: title: Thanks for the constructive reviews
Thanks for the insightful reviews
Response to Rebuttal
Thanks for your rebuttal!
Thanks for the thoughtful reviews
Thank you for the clarifications
Thanks for your valuable reviews
Reply
comment: Thanks for the constructive comments and questions! We will add the responses, as well as the remaining limitations to the revision, to better enhance the quality of our submission. Thanks again for your efforts in reviewing our submission!
Thanks for the insightful reviews, which help refine our manuscript! We will add the responses to the revision to reflect the discussion process. Thanks again for your valuable efforts in reviewing our submission!
Dear Authors,

Thanks for your response. Most of my questions have been properly addressed. Although some limitations still remain, e.g., dataset size and benchmark metrics, I think researchers can refer to this work for some insights and know-how to solve their domain specific challenges. To that end, I have raised my rating and support the acceptance of this manuscript.

Best,
I have read all of the author rebuttal, which addressed most of my concerns. Thus I will raise my score. Thanks for the detailed response.
Thanks sincerely for the thoughtful reviews! For the binding site, we use all residues on the target protein within 10Å to any atoms on the peptide. The threshold 10Å is kind of large to avoid data leakage. We will make this point clear in the revision. Thanks again for your effort in reviewing our submission.
Thank you for the clarifications.

Regarding proposition 3.1: you're absolutely right. Sorry, and thank you for your patience.

One further detail I could not find in the paper is the exact definition of the binding site. Is this a potential source of data leakage? I am wondering if the binding site is defined in terms of the ground-truth pose of the same peptide.

I keep my score as-is.
We sincerely thank the reviewer for the efforts in providing the valuable comments and questions, which helps enhance our submission! We are glad to hear that your concerns have been successfully addressed. We will add the responses to the revision to improve the quality of our paper.
I am glad that my concerns are all address by the authors. And I will lift my rating.