Graph Denoising Diffusion for Inverse Protein Folding
Authors: Kai Yi
Bingxin Zhou
Yiqing Shen
Pietro Lio
Yu Guang Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: inverse folding
graph neural networks
roto-translation equivariance
diffusion model
TL;DR: We propose a swift graph denoising diffusion model to generating diverse protein sequences for inverse folding.

Abstract: Inverse protein folding is challenging due to its inherent one-to-many mapping characteristic, where numerous possible amino acid sequences can fold into a single, identical protein backbone. This task involves not only identifying viable sequences but also representing the sheer diversity of potential solutions. However, existing discriminative models, such as transformer-based auto-regressive models, struggle to encapsulate the diverse range of plausible solutions. In contrast, diffusion probabilistic models, as an emerging genre of generative approaches, offer the potential to generate a diverse set of sequence candidates for determined protein backbones. We propose a novel graph denoising diffusion model for inverse protein folding, where a given protein backbone guides the diffusion process on the corresponding amino acid residue types. The model infers the joint distribution of amino acids conditioned on the nodes' physiochemical properties and local environment. Moreover, we utilize amino acid replacement matrices for the diffusion forward process, encoding the biologically-meaningful prior knowledge of amino acids from their spatial and sequential neighbors as well as themselves, which reduces the sampling space of the generative process. Our model achieves state-of-the-art performance over a set of popular baseline methods in sequence recovery and exhibits great potential in generating diverse protein sequences for a determined protein backbone structure.

Graph Denoising Diffusion for Inverse Protein Folding
Kai Yi University New South Wales kai. yi @unsw. edu au
Bingxin Zhou Shanghai Jiao Tong University bingxin. zannnszhtusuu edu cn
Yiqing Shen Johns Hopkins University yshen920 edu
Pietro Liò University of Cambridge pl219@cam.ace
Yu Guang Wang Shanghai Jiao Tong University University New South Wales yuguang- wang@sjtu. edu en
Abstract
Inverse protein folding is challenging due its inherent one-to-many mapping characteristic, where numerous possible amino acid sequences can fold into a single, identical protein backbone. This task involves not only identifying viable sequences but also representing the sheer diversity of potential solutions. However, existing discriminative models, such transformer-basee -ased auto-regressive models, struggle to encapsulate the diverse range of plausible solutions. In contrast, diffusion probabilistic models, as emerging genre of generative approaches, offer the potential to generate diverse set of sequence candidates for determined protein backbones. We propose novel graph denoising diffusion model for inverse protein folding, where a given protein backbone guides the diffusion process on the corresponding amino acid residue types. The model infers the joint distribution of amino acids conditioned on the nodes' physiochemical properties and local environment. Moreover, we utilize amino acid replacement matrices for the diffusion forward process, encoding the biologically meaningful prior knowledge of amino acids from their spatial and sequential neighbors as well themselves, which reduces the sampling space of the generative process. Our model achieves state-of-the-art performance over set popular baseline methods in sequence recovery and exhibits great potential in generating diverse protein sequences for determined protein backbone structure. The code is available on https:/ // /gi thub.
Introduction
Inverse protein folding, inverse folding, aims to predict feasible amino acid (AA) sequences that can fold into a specified 3D protein structure [22]. The results from inverse folding can facilitate the design of novel proteins with desired structural and functional characteristics. These proteins can serve numerous applications, ranging from targeted drug delivery to enzyme design for both academic and industrial purposes [25, 31, 37, 52]. In this paper, develop a diffusion model tailored for graph node demoising to obtain new AA sequences given a protein backbone.
Despite its importance, inverse folding remains challenging due to the immense sequence space to explore, coupled with the complexity of protein folding. On top of energy- physical reasoning of protein's 's folded state [1], recent advancements in deep learning yield significant progress in learning the mapping from protein structures to AA sequences directly. For example, discriminative models
"qqul contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

prior
FFN
equiarant
X
Diffusion Process
condition
Denoising Process
H
Summation
1=1
Network Propagator
Adarogato"
Figure 1: Overview of GRADE-IF. the diffusion process, the original amino acid is stochastically transitioned to other amino acids, leveraging BLOSSM with varied temperatures as the transition kernel. During the denoising generation phase, initial node features are randomly sampled across the 20 amino acids with a uniform distribution. This is followed by a gradual denoising process, conditional on the graph structure and protein secondary structure at different time points. We employ roto-translation equivariant graph neural network as the denoising network.
formulate this problem as the prediction of the most likely sequence for a given protein structure [7, 10. 17, 50]. However, they have struggled to accurately capture the one-to-many mapping from the protein structure to non-unique AA sequences.
Due their powerful learning ability, diffusion probabilistic models have gained increasing attention. They are capable of generating diverse range of molecule outputs from fixed set of conditions given the inherent stochastic nature. For example, Torsion Diffusion [19] learns the distribution of torsion angles of heavy atoms to simulate conformations for small molecules. Concurrently, SMCDIFF [43] enhances protein folding tasks by learning the stable scaffold distribution supporting a target motif with diffusion. Similarly, DIFFDOCK [5] adopts generative approach protein-ligand docking, creating a range of possible ligand binding poses for a target pocket structure.
Despite the widespread use of diffusion models, their comprehensive potential within the context of protein inverse folding remains relatively unexplored. Current methods in sequence design are primarily anchored in language models, encompassing Masked Language Models (MLMs) [25, 23] and autoregressive generative models [17, 26, 28). By tokenizing AAs, MLMS formulate the sequence generation tasks as masked token enrichment. These models usually operate by drawing an initial sequence with certain number of tokens masked as a specific schedule and then learning to predict the masked tokens from the given context. Intriguingly, this procedure can be viewed as model when trained by a parameterized objective. Autoregressive
a discrete models, conversely, be perceived as deterministic diffusion processes [3]. It induces conditional distribution to each token, but the overall dependency along the entire AA sequence is recast via an mndependently-xeccute diffusion process.
On the contrary, diffusion probabilistic models employ an iterative prediction methodology that generates less noisy samples and demonstrates potential in capturing the diversity inherent in real data distributions. This unique characteristic further underscores the promising role diffusion models could play in advancing the field of protein sequence design. To bridge the gap, we make the first attempt at a diffusion model for inverse folding. We model the inverse problem as a denoising problem where the randomly assigned AA types in protein (backbone) graph is recovered to the wild type. The protein graph which contains the spatial and biochemical information of all AAs represented by equivariant graph neural networks, and diffusion process takes places graph nodes. In real inverse folding tasks, the proposed model achieves SOTA recovery rate, improve 4.2% and 5.4% recovery rate for single-chain proteins and short sequences, respectively, especially for conserved region which has a biologically significance. Moreover, the predicted structure of generated sequence is identical the structure of native sequence.
The preservation of the desired functionalities achieved by innovatively conditioning the model on both secondary and third structures in the form of residue graphs and corresponding node features. The major contributions of this paper are three-fold. Firstly, we propose GRADE-IIF, a diffusion model backed by roto-translate equivariant graph neural network for inverse folding. It stands out

---

from its counterparts for its ability to produce a wide array of diverse sequence candidates. Secondly, as a departure from conventional uniform noise in discrete diffusion models, we encode the prior knowledge of the response of AAs to evolutionary pressures by the utilization of Blocks Substitution Matrix as the translation kernel. Moreover, to accelerate the sampling process, we adopt Denoising Diffusion Implicit Model (DDIM) from its original continuous form to suit the discrete circumstances and back it with thorough theoretical analysis.
Problem Formulation
2.1 Residue Graph by Protein Backbone
residue graph, denoted as 9 = (X, E), aims to delineate the geometric configuration of a protein. Specifically, every node stands for an AA within the protein. Correspondingly, each node is assigned a collection of meticulously curated node attributes X to reflect its physiochemical and topological attributes. The local environment of a given node is defined by its spatial neighbors. as determined by the k-nearest neighbor (kNN) algorithm. Consequently, each AA node is linked to a maximum of k other nodes within the graph, specifically those with the least Euclidean distance amongst all nodes within a 30A contact region. The edge attributes, represented as E E RR illustrate the relationships between connected nodes. These relationships are determined through parameters such as inter-atomic distances, local N-C positions, and sequential position encoding scheme. We detail the attribute construction in Appendix C.
2.2 Inverse Folding as a Denoising Problem
The objective of inverse folding is to engineer sequences that can fold to a pre-specified desired structure. We utilize the coordinates of Ca atoms to represent the 3D positions of AAs in Euclidean space, thereby embodying the protein backbone. Based on the naturally existing protein structures, our model is constructed to generate protein's 's native sequence based on the coordinates of its backbone atoms. Formally we represent this problem as learning the conditional distribution p(Xaaa (xPos). Given a protein of length n and a sequence of spatial coordinates 20. pos a:pos) representing each of the backbone Ca atoms the structure,
X = {=1 pos
the target is to predict X {xf1 ..... ... a*)), ), the native sequence of AAs. This density is modeled in conjunction with the other AAS along the entire chain. Our model is trained by minimizing the negative log- likelihood of the generated AA sequence relative to the native wild-type sequence. Sequences can then be designed either by sampling or by identifying sequences that maximize the conditional probability given the desired secondary and tertiary structure.
2.3 Discrete Denoising Diffuuson Probabilistic Models
Diffusion models belong to the class of generative models, where the training stage encompasses diffusion and denoising processes. The diffusion process q (II...., ØT 2o) = II (zz 2τ-1) corrupts the original data æo ~ g (x) into series of latent variables ( 2T), with each carrying progressively higher levels of noise. Inversely, the denoising process pa (0o, 21,,.., 2r)
p (xT) II pa (xt-1 æx) gradually reduces the noise within these latent variables, steering them back towards the original data distribution. The iterative denoising procedure is driven by a differentiable operator, such as trainable neural network.
While in theory there is no strict form for q (ax; 2t-1) to take, several conditions are required to be fulfilled by pe for efficient sampling: (i) The diffusion kernel q( zn) requires a closed form to sample noisy data at different time steps for parallel training. (ii) The kernel should possess a tractable formulation for the posterior q (æ,-1 æt, #t, zp). Consequently, the posterior po(#;-1la:; (x:)) = { (xx-1 Zt, Tn)) dpo (ao) (..) and 2g can be used as the target of the trainable neural network. (iii) The marginal distribution ((T))) should be independent of æng This independence allows us to employ q(xT) as a prior distribution for inference.
The aforementioned criteria are crucial for the development of suitable noise-adding modules and training pipelines. To satisfy these prerequisites. we follow the setting in previous work [3]. For categorical data E {1...., K), the transition probabilities are calculated by the matrix [Qrly = q (x, = xt-1 i). Employing the transition matrix and on one-hot encoded categorical feature
3

---

we can define the transitional kernel in the diffusion process by:
q (x xz-1) 21-1Q: and q (z+ x) xQ:
where Q- Q2.....000 Q. The Bayes rule yields that the posterior distribution can be calculated in closed form as q (x;-1 Tt, 2) x Q, @ determined using the transition kernel, the model output at time t, and the state of the process zt. Through iterative sampling, we eventually produce the generated output Zu-
xQ1-1. The generative probability can thus be
The prior distribution p(ær) should be independent of the observation Zo- Consequently, the construction of the transition matrix necessitates the of a noise schedule. The most straightforward and commonly utilized method is the uniform transition, which can be parameterized as Qe = a,d + (L at)1.1 /d with I be the transpose of the identity matrix I, d refers the number of amino acid types (i.e., d = 20) and 1a denotes the one vector of dimension d. As t approaches infinity, undergoes progressive decay until it reaches O. Consequently, the distribution g(xT) asymptotically approaches a uniform distribution, which is essentially independent of æ.
3 Graph Denoising Diffusion for Inverse Protein Folding
In this section, we introduce a discrete graph denoising diffusion model for protein inverse folding, which utilizes a given graph 9 = {X, A. E) with node feature X and edge feature E as the condition. Specifically, the node feature depicts the AA position, AA type, and the spatial and biochemical properties X X X 8 X prop]. We define a diffusion process on the AA feature X and denoise conditioned on the graph structure which is encoded by equivariant neural nefworks [35]. Moreover, we incorporate protein-specific prior knowledge, including an AA substitution scoring matrix and protein secondary structure during modeling. We also introduce a new acceleration algorithm for the discrete diffusion generative process based transition matrix.
3.1 Diffusion Process and Generative Denoising Process
Diffusion Process To capture the distribution of AA types, we independently add noise each AA node of the protein. For any given node, the transition probabilities are defined by the matrix Qr. With the predefined transition matrix, we can define the forward diffusion kernel by
9 (X
= X Q and q (X:
X X
X x=a Q
where Q = Q Q: is the transition probability matrix up to step I.
Training Denoising Networks The second component of the diffuusion model is the denoising neural network fo, parameterized by 0. This network accepts noisy input G. (X,, E). where X the concatenation of the noisy AA types and other AA properties including 20 one-hot encoded AA type and 15 geometry properties, such as SASA, normalized surface-awane node features, dihedral angles of backbone atoms, and 3D positions. It aims to predict the clean type of AA X aa which allows to model the underlying sequence diversity in the protein structure while maintaining their inherent structural constraints. To train fo. we optimize the cross-entropy loss L between the predicted probabilities pi X 88) for each node's AA type.
Parameterized Generative Process A new AA sequence generated through the reverse diffusion iterations on each node x. The generative probability distribution Pa (x; l,x,) is estimated from the predicted probability ((x**));;) by the neural networks. We marginalize over the network predictions to compute for generative distribution at each iteration:
pe (x:-1 xx) x > Eq(E1-1/20) 12,, x x=))) po (x (z:),
(2)
where the posterior
x.Q ② x. 2
(#:-1 ";") = Cat E;-1
(3)
ar
Quar
can be calculated from the transition matrix, state of node feature at step / and AA type as The aa is the sample of the denoising network prediction *(***).

---

BLOSUU
BLOSUM
10.5
AA tpc
AMIYY EOW EUNERDN
AA bpp
00
semple time
0.0
Sesmigii Eitso
The middle two panels depict the transition probability of Leucine (L) from 0 T. Both the uniform and BLOSUM start as Dirichlet distributions and become uniform at time T. As shown in the two side figures, while the uniform matrix evenly disperses L's probability other AAs over time, BLOSUM favors AAs similar to L.
Figure 2:
3.2 Prior Distribution from Protein Observations
3.2.1 Markov Transition Matrices
The transition matrix serves as guide for discrete diffusion model, facilitating transitions between the states by providing the probability of moving from the current time step to the next. As reflects the possibility from one AA type another, this matrix plays a critical role in both the diffusion and generative processes. During the diffusion stage, the transition matrix is iteratively applied to the observed data, which evolves over time due to inherent noise. As diffusion time increases, the probability of the original AA type gradually decays, eventually converging towards a uniform distribution across all AA types. In the generative stage, the conditional probability p(x;-1 (xt) is determined by both the model's prediction and the characteristics of the transition matrix Q. as described in Equation
Given the biological specificity of AA substitutions, the transition probabilities between AAs are not uniformly distributed, making it illogical define random directions for the generative sampling process. As an alternative, the diffusion process could reflect evolutionary pressures by utilizing substitution scoring matrices that conserve protein functionality, structure, or stability in wild-type protein families. Formally, an AA substitution scoring matrix quantifies the rates at which various AAs in proteins are substituted by other AAs over time [44]. In this study, we employ the Blocks Substitution Matrix (BLOSUM) [12], which identifies conserved regions within proteins that are presumed to have greater functional relevance. Grounded in empirical observations of protein evolution, BLOSUM provides an estimate of the likelihood of substitutions between different AAs. We thus incorporate BL.OSUM into both the diffusion and generative processes. Initially, the matrix is normalized into probabilities using the softmax function. Then, we use the normalized matrix B with different probability temperatures to control the noise scale of the diffusion process. Consequently, the transition matrix at time t is given by Q, BT. By using this matrix refine the transition probabilities, the generative space to be sampled is reduced effectively, thereby the model's predictions converge toward a meaningful subspace. See Figure 2 for comparison of the transition matrix over time in random and BLOSUM cases.
3.2.2 Secondary Structure
Protein secondary structure refers to the local spatial arrangement of AA residues in a protein chain. The two most common types of protein secondary structure are alpha helices and beta sheets, which are stabilized by hydrogen bonds between backbone atoms. The secondary structure of protein serves as a critical intermediary, bridging the gap between the AA sequence and the overall 3D conformation of the protein. In our study, we incorporate eight distinct types of secondary structures into AA nodes as conditions during the sampling process. This strategic approach effectively narrows down the exploration space of potential AA sequences. Specifically, we employ DSSP (Define Secondary Structure of Proteins) predict the secondary structures of each AA and represent these structures using one-hot encoding. Our neural network takes the one-hot encoding as input and utilizes it to denoise the AA conditioned on it.
The imposition of motif conditions such as alpha helices and beta sheets on the search for AA sequences not only leads to a significant reduction in the sampling space of potential sequences, but also imparts biological implications for the generated protein sequence. By conditioning the sampling process of AA types their corresponding secondary structure types, guide the resulting protein
5

---

sequence towards acquiring not only the appropriate 3D structure with feasible thermal stability but also the capability to perform its intended function.
3.3 Equivariant Graph Denoising Network
Bio-molecules such as proteins and chemical compounds are structured in the 3-dimensional space, and it is vital for the model to predict the same binding complex no matter how the input proteins are positioned and oriented to encode robust and expressive hidden representation. This property can be guaranteed by the rotation equivariance of the neural networks. A typical choice of such network is an equivariant graph neural network [35]. We modify its SE(3)-equivariant neural layers to update representations for both nodes and edges, which reserves SO(3) rotation equivariance and E(3) translation invariance. At the Ith layer, an Equivariant Graph Convolution (EGC) inputs
a set of n hidden node embeddings H = 1 "h.,.... h! describing AA type and geometry
properties, edge embedding m (t) with respect connected nodes and j. X {
for node coordinates and for time step embedding of diffusion model. The target of a modified EGC layer is to update hidden representations H for nodes and M for edges. Concisely, MM t]. X
H(++), ((+1) = FGC H
To achieve this, an EGC layer defines
ma
11+1)
Φe
h h
!1)
xx -x,
x0011 .mij
(++1)
(1)
(1)
xx or mij
(1+1)
v;
x;
+
xi
(4)
71 jfi
h (++1) = on (h
(141))
where de, dh the edge and node propagation operations, respectively. The du is additional operation that projects the vector edge embedding maj to scalar. The modified EGC layer preserves equivariance rotations and translations the set of 3D node coordinates X and performs invariance to permutations on the nodes set identical to any other GNNs.
3.4 DDIM Sampling Process
A significant drawback of diffuuson models lies in the speed of generation process, which is typically characterized by numerous incremental steps and can be quite slow. Deterministic Denoising Implicit Models (DDIM) [39] frequently utilized counter this issue in continuous variable diffusion generative models. DDIM operates on a non-Markovvin forward diffusion process, consistently conditioning on the input rather than the previous step. By setting the noise variance each step to 0, the reverse generative process becomes entirely deterministic, given an initial prior sample.
Similarly, since we possess the closed form of generative probability 1 (ax)) in terms of predicted 88 and the posterior distribution p(x;-11 F a;**), we can also render the generative model deterministic by controlling the sampling temperature of p(= (x,). Consequently, we can define the multi-step generative process by
Pe (æi-k xx)
[ g(01-R/0, x "))x"""*
(5)
where the temperature T controls whether it is deterministic or stochastic, and the multi-step posterior distribution is
Q.Q.T ... QT 3 x: Q
q (Zi-k Ft, 22)
Cat
Zt-k
xa Q100
(6)
4 Experiments
We validate our GRADE-IF on recovering native protein sequences in CATH [30]. The performance mainly compared with structure-aware SOTA models. The implementations for the main

---

Table 1: Recovery rate performance of CATH on zero-shot models.
Perplexity +
Recovery Rate
CATH version
Model
Short
Single-chan
Short
Single-chain
4.2
STRUCTGNN [18]
8.29
8.74
6.40
29.44
28.26
35.91
GRAPHTRANS [18]
8.39
8.83
6.63
28.14
28.46
35.82
GCA [41]
7.49
6.05
32.62
31.10
37.64
GVP [20]
7.23
7.84
5.36
30.60
28.95
39.47
GVP-large [17]
7.68
6.12
6.17
32.6
39.4
39.2
ALPHADESIGN [9]
7.32
7.63
6.30
34.16
32.66
41.31
ESM-IFI [17]
8.18
6.33
6,44
31.3
38.5
38.3
PROTEINMPNN [6]
4.57
36.35
34.43
49.87
PIFOLD [10]
6.04
6.31
4.55
39.84
38.53
51.66
GRADE-IF
5.49
6.21
4.35
45.27
42.77
52.21
rale
0.6
0.5
0.5
0.5
0.4
0,3
all
short
situle chain
10
15
31 helbe holix helic H-bandes turn ext strand
fialicet hend coil
contact munber
Figure 3: Recovery rate and surface residues and different secondary structure
algorithms (see Appendix D) at https://githu: httpa//.o.oo/..iiii///i/eee 1tt are programmed with Byyorchh-eemeee (ver 2.2.0) and PyTorch (ver 1.12.1) and executed on an NVIDIA Tesla V100 GPU with 5, 120 CUDA cores and 32GB HBM2 installed on an HPC cluster.
4.1 Experimental Protocol
Training Setup We employ CATH v4.2.0- -...0--aedddee partitioning as conducted by GRAPHTRANS [18] and GVP [20]. Proteins categorized based CATH topology classification, leading to division of 18, 024 proteins for training, 608 for validation, and 1, 120 for testing. To evaluate the generative quality of different proteins, test model across three distinct categories: short, single-chain, and all proteins. The short category includes proteins with sequence lengths shorter than 100. The single-chain category encompasses proteins composed of single chain. In addition, the total time step of the diffusion model is configured as 500, adhering to a cosine schedule for noise [27]. For the denoising network, implement six stacked EGNN blocks, each possessing hidden dimension of 128. Our model undergoes training for default of 200 epochs, making use of the Adam optimizer. batch size of 64 and learning rate of 0.0005 are applied during training. Moreover, prevent overfitting, we incorporate a dropout rate of 0.1 into our model's architecture.
Evaluation Metric The quality of recovered protein sequences is quantified by perplexity and recovery rate. The former measures how well the model's predicted AA probabilities match the actual AA at each position in the sequence. lower perplexity indicates better of the model to the data. The recovery rate assesses the model's ability to recover the correct AA sequence given the protein's s 3D structure. It typically computed as the proportion AAs the predicted sequence that matches the original sequence. A higher recovery rate indicates better capability of the model to predict the original sequence from the structure.
4.2 Inverse Folding
Table compares GRADE-IFF performance on recovering proteins in CATH, with the last two columns indicating the training dataset of each baseline method, To generate high-confidence
7

---

Table Numerical comparison between generated sequence structure and the native structure.
Method
Success
TM score
PLDDT
avg RMSD
PIFOLD
85
0.80 + 0.22
0.84 0.15
1.67 + 0.99
PROTEINMPNN
94
0.86 + 0.16
0.89 + 0.10
1.36 + 0.81
GRADE-IF
94
0.86 + 0.17
0.86 + 0.08
1.47 + 0.82
Recovery 0.612
Rocovery 0.547
Recovery
RMSD: 0.915
RMSD: 1.211
RMSD: 0,999
3vg. PLDDT: 0.841
aVg. PLDDT: 0.835
avg. pLDDT: 0.830
Figure 4: Folding comparison GRADE--F--ennratte sequences and the native protein (in nude).
sequences, GRADE-IF integrates out uncertainties in the prior by approximating the probability p(x R Ei= (2/)))))/)(())))))) Notably, we observed improvement of 4.2% and 5.4% in the recovery rate for single-chain proteins and short sequences, respectively. We also conducted evaluations different datasets (Appendix E) and ablation conditions (Appendix F).
Upon subdividing the recovery performance based on buried and surface AAs, we find that the more conserved core residues exhibit higher native sequence recovery rate. In contrast, the active surface AAs demonstrate a lower sequence recovery rate. Figure examines AA conservation by Solvent Accessible Surface Area (SASA) (with SASA< 0.25 indicating internal AAs) and contact number (with the number of neighboring AAs within 8 Å in 3D space) [11]. The recovery rate of internal residues significantly exceeds that of external residues across three protein sequence classes, with the recovery rate increasing in conjunction with the contact number. We also present the recovery rate for different secondary structures, where we achieve high recovery for the majority secondary structures, with the exception of minor 5-turn helix structure that occurs infrequently.
4.3 Foldability
We extend our investigation to the foldability of sequences generated at various sequence recovery rates. We fold generated protein sequences (by GRADE-IF) with ALPUAAFLLD2 and align them with the crystal structure to compare their closeness Figure (PDB ID: 3FKF). All generated sequences are nearly identical to the native one with their RMSD Á over 139 AAs, which lower than the resolution of the crystal structure at 2.2 Å. We have also folded the native sequence by ALPHAFOLD2, which yields an average pLDDT of 0.91. In comparison, the average pLDDT scores of the generated sequences are 0.835, underscoring the reliability of their folded structures. conjunction with the evidence presented in Figure 3 which indicates our method's superior performance in generating more identical results within conserved regions, we confidently posit that GRADE-IFF can generate biologically plausible novel sequences for given protein structures (See Appendix G).
The numerical investigation is reported in Table where we pick the first 100 structures (ordered in alphabetical order by their PDB ID) from the test dataset and compare the performance of GRADE- IF with PROTEINMPNN and PIFOLD. We follow [48] and define the quality of a novel sequence by the TM score between its ALPHAFOLD22folle structure and the native structure, with an above-0.5 score indicating the design is successful. Overall, our GRADE-IF exhibits high pLDDT, low RMSD, and high foldability. There are several proteins that face challenges with both GRADE-IF and baseline methods for folding with a high TM score, iee., IBCT, IBHA, and ICYU, whose structural determination is based on NMR, an experimental technique that analyzes protein structure in buffer solution. Due to the presence of multiple structures for single protein in NMR studies, is reasonable for folding tools to assign low foldability scores.

---

Table Numerical comparison on diversity and recovery rate
low recovery rate
medium recovery rate
high recovery rate
Method
diversity
recovery
diversity
recovery
diversity
recovery
PIFOLD
0.37
0.47
0.25
0.50
0.21
0.50
PROTEINMPNN
0.51
0.42
0.27
0.46
0.26
0.52
GRADE-IF
0.61
0.33
0.54
0.47
0.25
0.53
step: ag PLDDT: 0.867
stop: ang. pL.DDT: 0,826
step: 5 avg. PLDDT: 0.811
step:
PLDDT: 0.778
TM score: 0.910
TM score: 0.879
T'M score: 0,861
T' score: 0.820
diversity: 0.259
diversity: 0.446
diversity: 0.547
diversity: 0.618
Figure Folding structures of generated protein sequences with different steps.
4.4 Diversity
Given the one-to-many mapping relationship between protein structure and its potential sequences, an inverse folding model must be capable of generating diverse protein sequences for a fixed backbone structure. To investigate the diversity of GRADE-IF in comparison to baseline methods, we define diversity as self-similarity, where higher diversity indicates a model's proficiency in generating distinct sequences. We employ PIFOLD, PROTEINMPNN, and GRADE-IF for the test dataset at low, medium, and high recovery levels. We vary the temperature for the former two baseline methods within the range (0.5, 0.1, 0.0001) } and adjust the sample step for GRADE- IF from {1, 10, 50). The average performance from sampling 10 sequences summarized in Table 3, revealing comparable results among the three models in both recovery rate and diversity. In general, increasing the probability temperature for PIFOLD and PROTEINMPNN (or decreasing the sample step in GRADE-IF) leads reduction in uncertainty and more reliable predictions, resulting in higher diversities and lower recovery rates for all three models. While all three methods demonstrate the ability to recover sequences at a similar level, particularly with high probability temperatures (or low sample steps), GRADE-IF produces significantly more diverse results when the step size is minimized. Our findings demonstrate that GRADE-IF can generate sequences with 60% difference in each sample, whereas PIFOLD and PROTEINMPNN achieve diversity rates below 50%.
generate new proteins
We next explore the foldability of these highly diverse protein sequences designed by GRADE-IF. Figure 5 compares the generated proteins (folded by ALPHAFOLD2) with the crystal structure (PDB ID: 3FKF). We vary the step size size to calculate the average DLDDT, TM score, and diversity. For simplicity, we visualize the first structure in the figure. Decreasing the step size results in the generation of more diverse sequences. Consequently, there is a slight reduction in both pLDDD and T'M scores. However, they consistently remain at a considerably high level, with both metrics approaching 0.8. This reduction can, in part, be attributed to the increased diversity of the sequences, as ALPHAFOLD2 heavily relies on the MSA sequences. MSA. Remarkably, when step= 1, the sequence diversity exceeds 0.6, indicating that the generated sequences share an approximately 0.3 sequence similarity compared the wild-type template protein sequence. This suggests the generation of protein sequences from substantially distinct protein family when both PLDDT and TM scores continue to exhibit high degree of confidence.
range of 1, 5. 10, 50, generating 10 sequences per step
expected that more dissimilar sequences would produce a more diverse
5
Related Work
Deep Learning models for protein sequence design Self-supervised models have emerged as pivotal tool in the field of computational biology, providing robust method for training extensive

---

protein sequences for representation learning. These models are typically divided into two categories: structure-based generative models and sequence-based generative models. The former approaches protein design by formulating the problem of fixed-backbone protein design as conditional sequence generation problem. They predict node labels, which represent AA types, with invariant or equivariant graph neural networks [2, 17, 18, 20, 40, 42]. Alternatively, the latter sequence-hased generative models draw parallels between protein sequences and natural language processing. They employ attention-base methods to infer residue-win wise relationships within the protein structure. These methods typically recover protein sequences autoregressively conditioned on the last inferred AA [25, 29, 36], employing BERT-style generative framework with masked language modeling objectives and enable the model to predict missing or masked parts of the protein sequence [23, 26, 33, 46].
Denoising Diffussin models The Diffusion Generative Model, initially introduced by Sohl- Dickstein et al. [38] and further developed by Ho et al. [13]. has emerged as a potent instrument for a myriad of generative tasks in continuous time spaces. Its applications span diverse domains, from image synthesis [34] to audio generation [49], and it has also found utility in the creation of high-quality animations [14], the generation of realistic 3D objects [24], and drug design [5, 43]. Discrete adaptations of the diffusion model, on the other hand, have demonstrated efficacy in a variety of contexts, including but not limited to, text generation [3], image segmentation [15], and graph generation [16, 47]. Two distinct strategies have been proposed to establish a discrete variable diffusion process. The first approach involves the transformation of categorical data into a continuous space and then applying Gaussian diffusion [4, 16]. The alternative strategy is to define the diffusion process directly on the categorical data, an approach notably utilized in developing the D3PM model for text generation [3]. D3PM has been further extended to graph generation, facilitating the joint generation of node features and graph structure [47].
Conclusion
Deep learning approaches have striven to address multitude of critical issues in bioengineering, such as protein folding, rigid-body docking, and property prediction. However, only a few methods have successfully generated diverse sequences for fixed backbones. In this study, we offered a viable solution by developing a denoising diffusion model to generate plausible protein sequences for a predetermined backbone structure. Our method, referred to GRADE-IF. leverages substitution matrices for both diffusion and sampling processes, thereby exploring a practical search space for defining proteins. The iterative denoising process is predicated the protein backhone revealing both the secondary and tertiary structure. The 3D geometry is analyzed by a modified equivariant graph neural network, which applies roto-translation equivariance protein graphs without the necessity for intensive data augmentation. Given a protein backbone, our method successfully generated a diverse set of protein sequences, demonstrating significant recovery rate. Importantly, these newly generated sequences are generally biologically meaningful, preserving more natural designs in the protein's 's conserved regions and demonstrating a high likelihood of folding back into a structure highly similar to the native protein. The design of novel proteins with desired structural and functional characteristics is of paramount importance in the biotechnology and pharmaceutical industries, where such proteins can serve diverse purposes, ranging from targeted drug delivery to enzyme design for industrial applications. Additionally, understanding how varied sequences can yield identical structures propels the exploration of protein folding principles, thereby helping to decipher the rules that govern protein folding and misfolding. Furthermore, resolving the inverse folding problem allows the identification of different sequences that fold into the same structure, shedding light on the evolutionary history of proteins by enhancing our understanding of how proteins have evolved and diversified over time while preserving their functions.
10

---

Acknowledgments and Disclosure of Funding
We thank Lirong Zheng for providing insightful discussions on molecular biology. Bingxin Zhou acknowledges support from the National Natural Science Foundation of China (62302291). Yu Guang Wang acknowledges support from the National Natural Science Foundation of China (62172370), Shanghai Key Projects (23440790200) and SH2DZXX000)) This project undertaken with the assistance computational resources and services from the National Computational Infrastructure (NCI), which is supported by the Australian Government.
References
[1] Rebecca Alford, Andrew Leaver-Fay, Jeliazko R Jeliazkov, Matthew O' O'Meara Frank DiMaio, Hahnbeom Park, Maxim V Shapovalov, P Douglas Renfrew, Vikram K Mulligan, Kalli Kappel, et al. The Rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation, 13(6):3031-3048 2017.
[2] Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205. 15019, 2022.
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 14:17981-17993. 2021.
[4] Ting Chen, Ruixiang ZHANG, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In The Eleventh International Conference on Learning Representations, 2023.
[5] Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi S. Jaakkola. Diffdock: Diffusion steps, twists, and turns for molecular docking. In The Eleventh International Conference on Learning Representations, 2023.
[6] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert Ragotte, Lukas Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning-basee protein sequence design using proteinmpnn. Science, 578(6615):49 49.49,956666 2022.
[7] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Wang Yu, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. ProfTrans: Towards cracking the language of lifes code through supervuee deep learning and high performance computing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[8] Octavian-Fugen Ganea, Xinyuan Huang, Charlotte Bunne, Yatao Bian, Regina Barzilay, Tommi S Jaakkola, and Andreas Krause. Independent SE(3)-equivariant models for end-to-end rigid protein docking. In International Conference on Learning Representations, 2021.
[9] Zhangyang Gao, Cheng Tan, and Stan Z Li. Alphadesign: A graph protein design method and benchmark on alphafolddb. arXiv preprint arXlv:2202.01077, 2022.
[10] Zhangyang Gao, Cheng Tan, and Stan Z. Li. Pifold: Toward effective and efficient protein inverse folding, In International Conference on Learning Representations, 2023.
[11] Hai'e Gong, Haicang Zhang, Jianwei Zhu, Chao Wang, Shiwei Sun, Wei-Mou Zheng, and Dongbo Bu. Improving prediction of burial state of residues by exploiting correlation among residues. BMC bioinformatics, 18(3):165-175 2017.
[12] Steven Henikoff and Jorja G Henikoff. Amino acid substitution matrices from protein blocks. Proceedings of the National Academy of Sciences, 89(22):00915--1911 1992.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 13:6840-6851, 2020.

---

[14] Jonathan Ho. Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[15] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454-12465, 2021.
[16] Emiel Hoogehoom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning. pages 8867-8887. PMLR, 2022.
[17] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8946-8970. PMLR, 17-23 Jul 2022.
[18] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. Advances in Neural Information Processing Systems, 32, 2019.
[19] Bowen Jing. Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffusion for molecular conformer generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[20] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2021.
[21] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583-58) 2021.
[22] George A Khoury, James Smadbeck, Chris A Kieslich, and Christodoulos A Floudas. Protein folding and de novo protein design for biotechnological applications. Trends in biotechnology. 92(2):99-109, 2014.
[23] Zeming Lin, Halil Akin, Roshan Rao, Brian Hic. Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379 2023.
[24] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[25] Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos Jr, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology. pages 1-8, 2023.
[26] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. In Advances in Neural Information Processing Systems, volume 34, pages 29287-29303 2021.
[27] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[28) Erik Nijkamp, Jeffrey Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. ProGen2: exploring the boundaries of protein language models. arXiv:2206. 13517, 2022.
[29] Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena Hurtado, Aidan N Gomez, Debora Marks, and Yarin Gal. Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval. In International Conference on Machine Learning, pages 17017. PMLR, 2022.
12

---

[30] CA Orengo, AD Michie, S Jones, DT Jones, MB Swindells, and JM Thornton. CATH hierarchic classification of protein domain structures. Structure, 5(8):1093-1109 1997.
[31] Robin Pearce and Yang Zhang. Deep learning techniques have significantly impacted protein structure prediction and protein design. Current opinion in structural biology, 68:194-207, 2021.
[32] Yifei Qi and John ZH Zhang. Denseepd: computational protein sequence design with densenet. Journal of chemical information and modeling, (005)122--222 2020.
improving the accuracy of neural-network-base
[33] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, (0((5))e2016299111 2021.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[35] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In International conference machine learning, pages 9323-9332. 2021.
[36] Jung-Eun Shin, Adam J Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C Kruse, and Debora S Marks. Protein design and variant prediction using autoregressive generative models. Nature Communicationss, 12( 12(1):2403, 2021.
[37] Pawel Šledž and Amedeo Caflisch. Protein structure-based drug design: from docking to molecular dynamics. Current opinion in strucfural biology, 48:93-102, 2018.
[38] Jascha Sohi-Dickstein, Eric Weiss, Niru Maheswaranathann and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages
PMLR, 2015.
[39] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations.
[40] Alexey Strokach, David Becerra, Carles Corbi- Corbi-Verge, Albert Perez-Riba, and Philip M Kim. Fast and flexible protein design using deep graph neural networks. Cell Systems, 11(4):002441, 2020.
[41] Cheng Tan, Zhangyang Gao, Jun Xia, and Stan Z Li. Generative de novo protein design with global context. arXiv preprint arXiv:2204. 10673, 2022.
[42] Yang Tan, Bingxin Zhou, Yuanhong Jiang, Yu Guang Wang, and Liang Hong. Multi-level protein representation learning for blind mutational effect prediction. arXiv:2306.0445: 2023.
[43] Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S. Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem. International Conference on Learning Representations, 2023.
[44] Rakesh Trivedi and Hampapathalu Adimurthy Nagarajaram. Substitution scoring matrices for proteins-an overview. Protein Science, (9(11):2150-216 2020.
[45] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[46] Jesse Vig, Ali Madani, Lav R Varshney, Caiming Xiong, Nazneen Rajani, et al. BERTology meets biology: Interpreting attention in protein language models. In International Conference on Learning Representations, 2021.
[47] Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. In The Eleventh International Conference on Learning Representations, 2023.
13

---

[48] Kevin E Wu, Kevin K Yang, Rianne van den Berg, James Y Zou, Alex X Lu, and Ava P Amini. Protein structure generation via folding diffusion. arXiv preprint 2xXiv:2209.16111 2022.
[49] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. JEEF/ACM Transactions Audio, Speech, and Language Processing, 2023.
[50] Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei YE, and Quanquan Gu. Structure- informed language models are protein designers. In International Conference Machine Learning, 2023.
[51] Bingxin Zhou, Outongyi Lv, Kai Yi, Xinye Xiong, Pan Tan, Liang Hong, and Yu Guang Wang. Accurate and definite mutational effect prediction with lightweight equivariant graph neural networks. arXiv:2304.082299 2023.
[52] Bingxin Zhou, Lirong Zheng. Banghao Wu, Kai Yi, Bozitao Zhong, Pietro Lio, and Liang Hong. Conditional protein denoising diffusion generates programmable endonucleases. bioRxiv, pages 2023-08, 2023.
14

---

A Broader Impact and Limitations
Broader Impact We have developed a generative model rooted the diffusion denoising paradigm. specifically tailored to the context of protein inverse folding. As with any other generative models, it capable of generating de novo content (protein sequences) under specified conditions (e-g., protein tertiary structure). While this method holds substantial potential for facilitating scientific research and biological discoveries, its misuse could pose potential risks human society. For instance, in theory, it possesses the capacity to generate novel viral protein sequences with enhanced functionalities. To mitigate this potential risk, one approach could be confine the training dataset for the model to proteins derived from prokaryotes and/or eukaryotes, thereby excluding viral proteins. Although this strategy may to some extent compromise the overall performance and generalizability of the trained model, it also curtails the risk of misuse of the model by limiting the understanding and analysis of viral protein construction.
The conditions imposed the sampling process gently guide the generated protein sequences. However, certain scenarios, stringent restrictions may be necessary to produce a functional protein. Secondary structure, as living example, actively contributes to the protein's 's functionality, For instance, transmembrane a-helices play essential roles in protein functions, such as passing ions or other molecules and transmitting a signal across the membrane. Moreover, the current zero-shot model is trained on general protein database. For specific downstream applications, such generating new sequences for particular protein protein family, it may necessitate the incorporation of auxiliary modules or the modification of training procedures to yield more fitting sequences.
Limitations
B Non-Markovian Forward Process
We give the derivation of posterior distribution g (#t-1 Øt, x²a) for generative process from step to step. The proof relies the Bayes rule, Markov property, and the pre-defined transition matrix for AAs.
Propsition
For y (xt-1 Øe; x) defined in Eq 3, we have
ziQT @ xx20((111
q (xt-1 Fz, x
Cat 0t-1
x Qioi
Proof. By Bayes rules, we can expand the original equation q (xx-1 xaaa (x: xz-1) (x xa)
Ite 2la to
q (x De, x² (I:e x.") q (x;-1
q (xx a)))
q (x) x.
As pre-defined diffusion process, we get y (#t, aaa) x ar. and (#t-1 aaa)) = x Q.-1.
For the term of (xx æt-1, by Bayes rule and Markov property, have (x: #1-1, 2n)) q (I: 11-1) q( (x:)) x.Q T
where the normalizing constant is Em,-, Q.Q @ (xx:) = (a, Em.,, QT) @ TT..)) ② m (xx)
Then q (xx æ1-1, x)) = and the posterior distribution is: zi
IiQU @ x0 Qi-1
(xt-1 a xx))
Cat xt-1
an Qii1
The following gives the derivation for the discrete DDIM which accelerates the generative process. Propsition 2. For (x+-k æt, "²) defined in Eq 6,
Q.Q. QT @ x2 Q
(xi-k Ii, (a)) = Cat Øi-k
15

---

Proof. By Bayes rules, we can expand the original equation (#t- x1, xna) to (x wt; x²a) q (Its kt x= (#z=
@ (2 x 1s) a (0t x)m)
g (3t x:a))
o (2t a:aa)
As pre-defined diffusion process, we get y (are maa)) aa Qc. and y (a+t- (0 x0 Q, Q
u.QTT..Q and the
Similarly with q (xx (0=11, in Proposition 1, q (x)e æ1 a posterior is
O. O. @ xr Qi
(Zt 11, x²n) = Cat Zi-k
was Q
C Graph Representation of Folded Proteins
The geometry of proteins suggests higher-level structures and topological relationships, which are vital protein functionality. For a given protein, create a k-nearest neighbor (kNN) graph G = [X. E) to describe its physiochemical and geometric properties with nodes representing AAs by X E R:39 node attributes with 20-dim AA type encoder, 16-dim AA properties, and 3-dim AA positions. The undirected edge connections are formulated via a &NN-graph with cutoff. In other words, each node is connected to up to k other nodes the graph that has the smallest Euclidean distance over other nodes and the distance is smaller than a certain cutoff (e.g., 30A). Edge attributes are defined for connected node pairs. For instance, if node i and j are connected to each other, their relationship will be described by Eij Eji 93
The AA types are one-hot encoded to 20 binary values by X On top of it, the properties of AAs and AAs' local environment are described by X prop, including the normalized crystallographic B-factor, solvent-accessible surface area (SASA), normalized surface-aware node features, dihedral angles of backbone atoms, and 3D positions. SASA measures the level of exposure of AA to solvent in protein by a scalar value, which provides an important indicator of active sites of proteins locate whether residue is on the surface of the protein. Both B-factor and SASA are standardized with AA-wise mean and standard deviation on the associate attribute. Surface-aware features [8] of an AA is non-linear projections to the weighted average distance of the central AA its one-hop neighbors i' E N,, i.e.
EVEN, WA pos.s X
P (x;; A) =
EVEN, WA X X ppN,
where the weights are defined by
exp (- (-1|Xpun,t X Xpux,y' x 112 (//2)
Wi,
Σ EPEN; exp
1-IXpuase |Xpusss X 11? /A)
with À E 21,2,5, 30}. The X E R denotes the 3D coordinates of the ith residue, which represented by the position of a-carbon. We also use the backbone atom positions to define the spatial conformation of each AA in the protein chain with trigonometric values of dihedral angles (sin, cos) o ( vi,, w:).
Edge attributes E E R on the other hand, include kernel-based distances, relative spatial positions, and relative sequential distances for pairwise distance characterization. For two connected residues and j. the kernel-based distance between them is projected by Gaussian radial basis functions (RBF) of exp ||,,-0011 with r = 1, 1.,
...,,, R. A total number of 15 distinct distance- based features
20
are created with o, (1.5 k 0, 14). corresponding residues' heavy atoms positions to define 12 relative positions. They represent local fine -ra-ind relations between AAs and the rigid property of how the two residues interact with each other. Finally, the residues' sequential relationship is encoded with 66 binary features by their relative position dj.j |s; - sjl, where 8; and sj are the absolute positions of the two nodes in the AA chain [51]. We further define a binary contact signal [18] to indicate whether two residues contact in the space, i.e., the Euclidean distance ||Ca; Ca.ll 8.
Next, local frames [8] are created from the
16

---

D Training and Inference
In this section, we elucidate the training and inference methodologies implemented in the diffusion generative model. As shown in Algorithm 1, training commences with random sampling of a time scale from uniform distribution between 1 and 7". Subsequently, we calculate the noise posterior and integrate noise dictated by its respective distribution. We then utilize an equivariant graph neural network for denoising predictions, using both the noisy amino acid and other properties as node features, and leveraging the graph structure for geometric information. This results in the model outputting the denoised amino acid type. Ultimately, the aross-entropy loss is computed between the predicted and original amino acid types, providing parameter for optimizing the neural network.
Algorithm 1 Training
1: Input: A graph 9 = (X, E}
2: Sample t (1,
3: Compute o( [X,,X X
= X a2 xt
4: Sample noisy X, ~ 9 X, X 88)
5: Forward pass: p (xx Se E,
6: Compute cross-entropy loss: L = LCE (p (p( _ X)
7: Compute the gradient and optimize denoise network fa
Upon completing the training, we are capable of sampling data using the neural network and the posterior distribution p( [2; ax*). As delineated the algorithm, we initially sample an amino acid uniformly from 20 classes, then employ our neural network to denoise X from time /. From here, we can calculate the forward probability utilizing the model output and the posterior distribution. Through iterative processing, the ultimate model sample closely approximates the original data distribution. We can also partially generate amino acids given some known AA within the structure. Analogous to inpainting task, each step, we can manually adjust the prediction at known positions to the known amino acid type, subsequently introducing noise, as illustrated in the Algorithm 4.
Algorithm 2 Sampling (DDPM)
Sample from uniformly prior XT p(Xr)
2: for in (T, T 1..... 1} do
3: Predict p X 8a X,) by neural network p0 xxxaa X2) fo(X,, x. E,t,ss)
4: Compute po ( X,) Σ xm q( X xa) ((x'" (i))
Ex.0
5: Sample X, Po (X,- X,)
6: end for
7: Sample X po X X1)
Algorithm 3 Sampling (DDIM)
1: Sample from uniformly prior XT p(XT)
2: for in {T, T k, ..., 1 do
3: Predict 6(X X,) by neural network p x Xi) fe(X,, E,
4: Compute pa X R X,)
= am
(X, x X,, X ) (x(Xaa) (x))
5: Sample X,- ~ Po (x-k X₂)
6: end for
7: Sample Xaa ~ Po( X aa X1)
17

---

Algorithm 4 Partial Sampling
1: Input Mask M indicate which position is fixed
2: Sample from uniformly prior XT p(Xr)
3: for in {T, T k, .... } do
4: Predict p(_X nas X²) ) by neural network p X X₂) fo(X;, E,),s))
5: Compute p( am X,) p( X
O M + p(X X₂) ① M)
6: Compute pa Xx -- X₂
X nm q( X X X p X,)
7: Sample X ~ po (X x
8: end for
9: Sample X (o X1)
E Inverse Folding Performance on TS50 and T500
In addition to the CATH dataset, also evaluated our model using the TS50 and T500 datasets. These datasets were introduced by DenseCPD [32]. encompassing 9888 structures for training, and two distinct test datasets comprising 50 (TS50) and 500 (T500) test datasets, respectively. The same preprocessing steps applied to the CATH dataset were utilized here. The denoising network comprises six sequentially arranged EGNN blocks, each boasting hidden dimension of 256. Our model's performance, outlined in Table 4, achieved an accuracy of 61.22.. 61.22% on T500, and 56.32% on TS50, respectively.
Table 4: Recovery rate performance of TSS0 and T500 zero-shot models.
TS50
T500
Model
Perplexity
Recovery
Perplexity
Recovery
STRUCTGNN [18]
5.40
43.89
4.98
45.69
GRAPHTRANS [18]
5.60
42.20
5.16
44.66
GVP [20]
4.71
44.14
4.20
49.14
GCA [41]
5.09
47.02
4.72
47.74
ALPHADESIGN [9]
5.25
48.36
4.93
49.23
PROTEINMPNN [6]
3,93
54.43
3.53
58.08
PIFOLD [10]
3.86
58.72
3.44
60.42
GRADE-IF(ours)
3.71
56.32
3.23
61.22
F
Ablation Study
We conducted ablation studies to assess the impact of various factors on our model's performance. These elements encompassed the selection of the transition matrix (uniform versus BLOSUM), the integration of secondary structure embeddings in the denoising procedure, and the function of the equivariant neural network. As demonstrated in Figure 6, incorporating equivariance into the denoising neural network substantially enhances the model's performance. Given that the placement of protein structures in space can arbitrary, considering symmetry helps to mitigate disturbances. Moreover, we found that including secondary structure as auxiliary information lessens uncertainty and improves recovery. Lastly, utilizing the BLOSUM matrix as the noise transition matrix boosted the recovery rate by 2%, highlighting the benefits of infusing biological information into the diffusion and generative processes. This approach reduces sample variance and substantially benefits overall model performance.
denoising neural network
In our sampling procedure, we accelerate the original DDPM sampling algorithm, which takes every step in the reverse sampling process, by implementing the discrete DDIM as per Equation 6. This discrete DDIM allows us to skip every steps, resulting
speed-up of the original DDPM by a factor of k. We conducted an ablation study on the impact of speed and recovery rate by trying different skip steps: 1, 2, 5, 10, 20, 25, 50, and 100. We compare the recovery rates achieved by these different steps. Our results revealed that the recovery rate performance decays the number of
18

---

Equivariance
Secondary Strueture
BLOSUM
0.5
0.5
0.5
yos
0.4
0.3
0.4
0.4
all
short
single choin
all
short
single chain
all
short
single chain
Figure 6: Recovery rate with the different selection of the transition matrix, whether considering equivariance and secondary structure.
skipped steps inereases. The best performance is achieved when skipping single step, resulting in recovery rate of 52.21%, but at a speed of 100 times slower than when skipping 100 steps, which yields recovery rate of 47.66%.
Prediction Confusion Matrix
BLOSUM
15
wild type BANAAT CNCRRIONNERNNN
10
5
0
DENDIRRR prediction
AGPDENQURKMI reference
Figure 7: Comparison of the distribution of mutational prior (BLOSUM replacement matrix) and sampling results.
We further compare the diversity of GRADE-IF with PIFOLD and PROTEINMPNN Figure For given backbone, we generate 100 sequences with self similaiill less than 50% and employ 1-SNE [45] for projection into a 2-dimensional space.
the same level of diversity, GRADE- IF encompasses the wild-type sequence, whereas the other two methods fail include the wild-type within their sample region. Furthermore, inspiring at a recovery rate threshold of 45% for this protein, GRADE-IF manages to generate substantial number of samples, whereas the other two methods revert to deterministic results. This further substantiates the superiority of our model in terms of achieving sequence diversity and high recovery rate concurrently.
I-SNE (temp=0.5)
L-SNE (temp=0.1)
I-SNE (temp=10
dimension 2
dimension
dimension
dimension
dimension
dimension
GraDe-IF
PiFold
ProteinMPNN
wild type
Figure 8: t-SNE of the generated sequences of GRADE-IF compared to PIFOLD and PROTEINMPNN,
19

---

G Additional Folding Results
We further analyzed the generated sequences by comparing different protein folding predictions. We consider the crystal structures of three native proteins with PDB IDs: 1ud9 (A chain), 2rem (B chain), 3drn chain), which we randomly choose from CATH dataset. For each structure, we generated three sequences from the diffusion model and used ALPHAFOLD [21] to predict the respective structures, As shown in Figure 9, these predictions (in purple) were then compared with the structures of the native protein sequences (in nude). We can observe that the RMSD for all cases is lower than the preparation accuracy of the wet experiment. The results demonstrate that our podel-genneated sequences retain the core structure, indicating their fidelity to the original structures.
PDB ID: 1ud9A00
PLDDI: 0.835
Recovery Rate: 0.593
Recovery Rate: 0.535
Recovery Rate: 0.494
RMSD: 0.819
RMSD: 0.955
RMSD: 0.864
avg. PLDDT: 0.841
avg. PLDDT: 0.824
avg. pLDDT: 0.785
PDB ID: 2remBo0
PLDDI: 0.879
Recovery Rate: 0.538
Recovery Rate: 0.505
Recovery Rate: 0.500
RMSD: 0.914
RMSD: 0.827
RMSD: 0.953
avg. PLDDT: 0.791
avg. pLDDT: 0.780
avg. pLDDT: 0.780
PDB ID: 3drnBo0
PLDDT: 0.8465
Recovery Rate: 0.624
Recovery Rate: 0,490
Recovery Rate: 0.470
RMSD: 0.730
RMSD: 0.810
RMSD: 0.829
avg. PLDDT: 0.750
aVg. PLDDT: 0.735
avg. PLDDT: 0.746
Figure 9: Folding comparisons between native sequence and generated sequence
20


Reviews:
[{"review_id": "Rw9NZBUNzl", "reviewer": "Reviewer_YmFn", "summary": "This work presents a denoising diffusion model for protein inverse folding: predicting the amino acid sequences that fold into the given 3D protein structure. The proposed method leverages a discrete denoising diffusion model with respect to the graph structure representing the protein backbone. This work proposes to use Blocks Substitution Matrix for the transition matrix considering the different transition probabilities between amino acids, and further utilize the distinct types of secondary structure as a condition during sampling that guides the sampling process to appropriate 3d structures.", "questions": "- Using DDIM for sampling, how long does it take to predict the sequence compared to the baselines, e.g. ProteinMPNN or ESM-IF1?", "limitations": "The limitations are discussed in the Supplementary file.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. \n\n- The approach to using (discrete) diffusion models on protein inverse folding is novel to the best of my knowledge. \n\n- Two main contributions are novel and well-motivated: Employing BLOSUM for considering transitions between AAs and using the secondary structure information as a condition during sampling injects biological prior knowledge into the diffusion process which further reduces the sampling space and results in plausible AA sequences.\n\n- The proposed method shows superior performance for the inverse folding tasks in terms of perplexity and recovery rate. The diversity analysis shows that the generated sequences are diverse without losing recovery rate.", "weaknesses": "- Using the distinct types of secondary structures as a condition for the sampling process is not clear. How is the information used? Is it an input to the model during each step of the sampling process?\n\n- The reason for not giving a higher rating is that although the approach to the task is novel, using BLOSUM and secondary structure conditions are novel, and the method shows superior performance, the components of the proposed methods are widely used in different domains: Diffusion models and E3 equivariant networks are widely used in the generation of protein structures given sequences (I understand that the task is not the same but in the similar domain) and DDIM is also used to reduce the sampling steps.  \n\n- Minor: line 143 \"orm.alized\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful feedback on our work, particularly your kind recognition of the quality of our presentation, the originality of our work, and its significance. We would like to address your questions and concerns as follows:\n\n**Weaknesses**:\n- *secondary structure representation*: In our approach, we employ an eight-dimensional one-hot encoded feature to represent nodes (amino acids) associated with one of the eight secondary structures. During the reverse diffusion process, this secondary structure information undergoes projection into the feature dimension and is subsequently integrated into the time embedding. This embedding is then added to the features of the output of each layer.\n- *novelty of the work*: We agree that many individual components, such as the forward/reverse diffusion model, equivariant graph neural networks, and DDIM, have been previously explored in research. However, what sets our work apart is the non-trivial combination of these components, achieved through significant additional effort. Notably, we derive a substitution matrix for multi-step transitions in discrete space, leading to a closed-form expression for this matrix, which in turn allows us to establish the Discrete Discrete-time Inhomogeneous Markov (DDIM) model for protein sequence generation. Furthermore, our approach's effectiveness in generating reliable protein sequences is attributed to the careful arrangement of auxiliary components, including the transition matrix for forward/reverse diffusion and the incorporation of protein secondary structures for conditional sampling. We contend that the overall framework we introduce to address the inverse folding problem, encompassing evaluation metrics and related aspects, constitutes an additional valuable contribution.\n- minor typo: Thank you for pointing out this typo. We have corrected it to “normalized” in the paper. \n\n**Questions**:\n- *sequence prediction time*: The sampling speed of DDIM is influenced by the choice of skip steps (as illustrated in Figure 5). To illustrate, we present the generation of 3FKF-A and provide a comparative analysis of inference times along with baseline methods, considering skip steps of 100, 20, and 1 for GraDe-IF, where the skip step of 1 reverts to the original DDPM sampling algorithm. The inference time of GraDe-IF decreases significantly with increasing step size, and it becomes faster than all baseline methods when step size=100.\n| Method                       | Inference Time (in seconds)  |\n|------------------------------|:--------------------------------------------:|\n| PiFold         \t        | 0.06                                              |\n| ProteinMPNN    \t        | 0.14                                              |\n| ESM-if1           \t        | 0.17                                              |\n| GraDe-IF (step=1)     | 6.14                                              |\n| GraDe-IF (step=20)   | 0.31                                              |\n| GraDe-IF (step=100) | **0.05**                                              |\n"}, {"review_id": "xzbmJ0ki6u", "reviewer": "Reviewer_j843", "summary": "The authors of the manuscript present GraDE-IF, a diffusion model based method for inverse protein folding given the backbone of the structure. The denoising network is a graph neural network that's equivariant to rotations and translations. Moreover, a biologically relevant inductive bias is incorporated into the discrete diffusion process by replacing the uniform amino acid transition probabilities with amino acid substitution scoring matrices. Finally, sampling is accelerated by deploying a variant of the Denoising Diffusion Implicit Model (DDIM).", "questions": "1. As mentioned in \"Weaknesses\", I missed some kind of quantitative metric to measure diversity / uniqueness of generated sequences accross proteins.\n2. Another valuable addition to the results, although it might not be possible due to time constraints, would be to show how this method could aid protein design, e.g. by providing a synthetic backbone and generating plausible corresponding sequences.\n3. Figure 1: the text in the \"prior\" and \"condition\" blocks is quite small. For me this caused some confusion because at a quick glance it seemed like the bottom part of \"condition\" was an amino acid sequence rather than secondary structure annotation. \n4. Table 1: Are the CATH versions correct? The first sentence of section 4.1 (line 233) mentions CATH v4.3.0.\n5. Figure 4: The 45% threshold seems quite conveniently chosen such that only one sequence remains for the other two models. It would be informative to include this plot for some different recovery rate cut-offs, if not in the main text then at leats in the supplementary. Additionally, for the figure on the right, no unit is given for \"speed\", and there is no comparison to other models (PiFold, ProteinMPNN).\n6. Missing references/proofs to support the connections made in line 45-51.\n7. Missing defenitions:\n    * FFN (Figure 1): mention feed forward network somewhere.\n    * $\\mathcal{E}$ (line 76) is never defined.\n    * $d$ (line 126) is never defined (number of categories).\n    * $\\mathbf{A}$ (line 131) is never defined.\n8. Confusing mathematical notation:\n    * The graph notation seems inconsistent accross the paper.\n    * I think there's a \"pos\" superscript missing in line 93 (i.e. $\\mathbf{X} \\rightarrow \\mathbf{X^{pos}}$)\n    * The expression in line 126 seems a bit confusing. Isn't the transpose of the identity matrix the same (i.e. $\\mathbf{I}^T=\\mathbf{I}$) and wouldn't $\\mathbf{I} \\ \\mathbf{I}^T=\\mathbf{I}$? It might be that this is the point you wanted to make, but it was not completely clear to me.\n9. Wrong figure references line 260, line 289.\n10. Not all appendices are referenced to in the main text.\n11. A grammar check would improve the flow of the paper.", "limitations": "The authors adequately dicuss the limitations of their work in the appendix, even though I would prefer it if some of this discussion was transferred to the main paper. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "To the best of my knowledge, this is the first work that explores the use of a discrete diffusion model for inverse protein folding. The fact that prior biological knowledge is incorporated directly into the discrete diffusion process increases the appeal of this method. The presented results look promising.", "weaknesses": "Even though I enjoyed reading the paper, there are still some concerns.\n\nFirst and foremost, in my opinion the reported metrics do not fully support the claim that the model is capable of generating diverse protein sequences. Figure 6 shows some qualitative examples of different sequences leading to plausible structures, but I'm missing quantitative results on diversity/uniqueness, and an average RMSD/pLDDT for a large batch of generated sequences folded by AlphaFold2 for different proteins. This would make the story much more convincing.\n\nAdditionally, the paper is quite sloppy in some places, with confusing mathematical notation, wrong figure references, and occasionally poor sentence constructions. I appreciate that this is likely due to time constraints, but some time should be spent on fixing this for the final paper in case it gets accepted.\n\nI will address the more minor concerns in the \"Questions\" part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your meticulous feedback, as well as your recognition of the novelty of our work. Below we respond to your concerns and questions point-by-point.\n\n**Weaknesses** :\n\n1. *Quantitative Results on Diversity*: We have incorporated quantitative evaluations of generated results using the `diversity` metric, defined as `1 - sequence identity`. We extended our assessments to average RMSD and pLDDT across a broader spectrum of 42 distinct proteins. Note that the reason we not investigating a larger volume of proteins is because that AlphaFold failed to fold more structures within the limited time for rebuttal. (1) Table 1 in PDF compares the recovery rate and diversity of the generation results by controlling temperature = {0.5, 0.1, 0.0001} for PiFold and ProteinMPNN and sample step ={20, 100, 250} for GraDe-IF. The three levels are divided roughly into low, medium, and high recovery rates in the table below. The results, quantified as average diversity and recovery rate, are depicted in Table 1 in PDF.  (2) We expanded the validation of RMSD and pLDDT across a wider selection of 42 proteins and reported the results in Table 2 in PDF. Notably, GraDe-IF exhibited the highest average pLDDT and lowest RMSD, highlighting its effectiveness over baseline methods.\n\n2. *Typo*: We have carefully gone through the main text to fix the typos and inappropriately defined notations, combining suggestions from all the reviewers. \n\n**Questions**:\n\n1. *quantitative metrics on the generated results*: Refer to the response provided above.\n2. *sequence generation for synthetic backbone*: Thank you for suggesting a new possibility for our design. There is no doubt that generating plausible sequences for synthetic backbones is of great significance for designing structural proteins. While the limited time for rebuttal prevents us from preparing a throughout investigation upon generating valid sequences for practically meaningful protein structures, it opens a promising direction for future investigation, combining in silico validation and wet experiments through structural characterization methods like NMR, X-ray crystallography, and cryo-EM.\n3. *conditions in Figure 1*: We have fixed the problem in the revision by magnifying the contents in both prior and condition blocks, mitigating any residual confusion.\n4. *CATH versions*: We utilized CATH4.2 for both training and evaluation. Thank you for highlighting this inconsistency. We have fixed the typo in the revised version. \n5. *45% threshold*: We selected the 45% threshold because it represents the best recovery rate that the other two models achieved when sampling at very low temperatures. To address your suggestion, we have added an additional plot at an intermediate recovery rate cut-off (see Figure 1 in PDF). We have added the unit for speed in Figure 5, which is ‘second’ in the revision. Moreover, Table 3 in PDF compares the sampling speed of our model at varying step sizes against other models. Note that the sampling speed of DDIM is influenced by the choice of skip steps (as illustrated in Figure 5). The inference time of GraDe-IF decreases significantly with increasing step size, and it becomes faster than all baseline methods when step size=100.\n6. *missing reference*: The claim was from D3PM [1]. We introduced the work in related work and added the reference in line 49  in the revised version.\n7. *missing definitions*: In the revised version, we: (1) added the full name of FFN in the caption of Figure 1; (2) added the definition of $d$ (line 126), which is the number of amino acid types, i.e., d=20; (3) For the other two notations, $\\mathcal{E}$ (the set of edges) and $\\mathbf{A}$ (the adjacency matrix), we updated the definition of a graph to $\\mathcal{G}=(\\mathbf{X}, \\mathbf{A}, \\mathbf{E})$, with $\\mathbf{X}, \\mathbf{A}$ and $\\mathbf{E}$ representing the node feature matrix, adjacency matrix, and edge feature matrix of the graph, respectively. \n8. *confusing mathematical notation*: In the revised version, we: (1) unified the notation of graphs by $\\mathcal{G}=(\\mathbf{X}, \\mathbf{A}, \\mathbf{E})$ (see the previous answer); (2) updated $\\mathbf{X}$ in line 93 to $\\mathbf{X}^{pos}$; (3) corrected the definition of $\\mathbf{Q}_t$ to $\\mathbf{Q}_t = \\alpha_t \\mathbf{I} + (1 - \\alpha_t)\\mathbf{1}_d\\mathbf{1}_d^{\\top}/d$, where $\\mathbf{1}_d$ denotes a d-dimensional one vector.\n9. *figure reference*: Thank you for identifying these typos. We have fixed both references in the revision. In lines 262&289 (of the revised version) the wrong number (“Figure 7”) has been corrected to “Figure 3”.\n10. *reference to Appendix*: We have added proper references in the revision. For instance, Appendix B (non-Markovian forward process) is now linked in Section 3.4 (DDIM sampling process); Appendix C (algorithm) is mentioned at the beginning of Section 4 (Experiment); and Appendix F (ablation study) is mentioned in Section 4.2. \n11. *grammar check*: We've conducted a thorough grammar check, correcting various grammatical errors and typos in the revised version.\n\n\n**Reference**:\n\n[1] Austin, Jacob, et al. \"Structured denoising diffusion models in discrete state-spaces.\" Advances in Neural Information Processing Systems 34 (2021): 17981-17993.\n\n"}, {"review_id": "jn3CSlVHPE", "reviewer": "Reviewer_ZARz", "summary": "The paper highlights that \"Existing discriminative models struggle to capture the diverse range of solutions, while generative diffusion probabilistic models offer the potential for generating diverse sequence candidates.\" They propose to use the denoise diffusion model together with the prior information from the secondary structure and BLOSUM matrix to train an inverse folding model. As shown in the experiments, their method achieves better results than the previous baselines, and they also show the folding results for the model predictions.", "questions": "1. The authors show the trade-off between speed and recovery ratio. Here, I wonder if the authors use a standard Euler solver or any other sampling schedule to do the inference. \n2. In Figure 6, the authors show the results of several PDB proteins,  do the authors make sure that these proteins have < 30% sequence similarity as the proteins in the training dataset? ", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[+] The proposed graph denoising diffusion model introduces a new perspective for addressing the challenging problem of inverse protein folding. It leverages the power of diffusion probabilistic models to generate a diverse set of sequence candidates for a given protein backbone. In protein design, diversity is a key point, since we want diverse candidates to build a library for wet-lab experiments. \n[+] The model achieves state-of-the-art performance compared to popular baseline methods in sequence recovery. \n[+] The method includes prior information during the model training, and the method is easy-to-follow.\n[+] The authors also check SASA and other properties, which is useful for protein design and engineering.", "weaknesses": "[-] Although the paper mentions diversity, however, they mainly measure the recovery ratio (wildtype accuracy) and the PPL, which cannot tell the readers whether their model captures the \"useful\" diversity. The diversity is only shown in Figure 4, and I think it is not enough to demonstrate the model's advances in diversity.\n[-] The baseline PiFold or ESMFold algorithm can also introduce diversity by controlling the temperature and do sampling based on the probability score, a detailed comparison with the baselines could improve the experiments. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful feedback on our paper and recognizing its contribution and performance. We have taken your remarks into careful consideration and offer the following responses to your concerns and questions:\n\n**weaknesses**: \n- *measurement of diversity, and additional comparison with PiFold and ProteinMPNN*: (1) The table below compares the recovery rate and diversity of the generation results by controlling temperature = {0.5, 0.1, 0.0001} for PiFold and ProteinMPNN and sample step ={20, 100, 250} for GraDe-IF. The three levels are divided roughly into low, medium, and high recovery rates in the table below. The results, quantified as average diversity and recovery rate, are depicted in the table provided. For clarity, the metric \"diversity\" is quantitatively defined as `diversity = 1-sequence identity`, where the average sequence identity is computed pairwise for generated sequences. For instance, in the first cell of the table, where the method is PiFold, the recovery level is low, and the metric is diversity, the diversity value is indicated as 0.3796.\n(2) In response to the task of aligning recovery rate levels, we have introduced an intermediary plot positioned between the two subfigures in Figure 4 (refer to Figure 1 in the attached PDF document). This supplementary plot effectively illustrates the intricate relationship between recovery rates and diversity across the three methods on a finer scale. Notably, the visualization highlights the rapid contraction of the sampling space for both PiFold and ProteinMPNN. In contrast, the samples generated by GraDe-IF exhibit a broader distribution that encompasses the wild type.\n\n|    | low recovery | | medium recovery | | high recovery  |  |\n|------------------|:-------:|:--------:|:-----:|:-------:|:------:|:------:|\n|   (Method)  | (diversity) | (recovery) |  (diversity) | (recovery) | (diversity) | (recovery) |\n| PiFold\t| 0.3796 | 0.4794 | 0.2535 | 0.5084 | 0.2181 | 0.5087 |\n| ProteinMPNN | 0.5159 | 0.4268 | 0.2735 | 0.4679 | 0.2657 | 0.4679 |\n| GraDe-IF | 0.5899 | 0.4142 | 0.5462 | 0.4586 | 0.5296 | 0.4755 |\n\n\n**Questions**:\n- *sampling schedule for inference*: While the standard Euler solver was not implemented in our study due to the distinct characteristics of our diffusion process, which involves a discrete transition matrix that does not follow a continuous stochastic differential equation, we acknowledge the potential for future exploration. In forthcoming research, there may be merit in formulating a score-based model and developing a sampling algorithm that operates within a continuous time framework. It's plausible that continuous solvers, such as the Euler solver, could play a role in this endeavor.\n- *sequence similarity of the generated protein sequences*: We have meticulously examined the generated sequences, confirming that none of them exhibit a sequence identity exceeding 30% with the training samples. It is essential to emphasize that the intention behind presenting Figure 6 is to showcase the capacity of our generated protein sequences, conditioned on a template protein backbone, to reliably fold back to the template structure (RMSD < resolution) with a high degree of confidence (pLDDT > 0.8). We want to underline that all the displayed proteins, including 3FKF in Section 4, along with 1ud9, 2rem, and 3drn in Appendix G, were selected from the test set, which was properly processed to ensure no overlapping with the training or validation set. Moreover, every visualized sequence underwent a thorough comparison against the wild-type template protein sequence, yielding sequence identity values that fall within the 40% to 60% range.\n"}, {"review_id": "Sodvu5Jmsm", "reviewer": "Reviewer_92pD", "summary": "The paper proposes discrete diffusion for the task of inverse protein folding (IPF). The many-to-one mapping of sequence to structure warrants a generative model of sampling multiple possible sequences that fold into a given structure. Nearly all prior works use autoregressive generative models so the exploration of non-autoregressive models is high interest for its improvements in accuracy and speed. The authors propose several IPF specific improvements to discrete diffusion that results in state-of-the-art performance on commonly used benchmarks. Additional analysis is performed on accuracy for different classes of residues such as surface exposed, diversity, and designability.\n\nThe contributions can be summarized as follows:\n\n- First exploration of discrete diffusion for IPF.\n- Discrete diffusion improvements such as a BLOSUM weighted transition kernel, secondary structure auxiliary prediction, DDIM for faster sampling.\n- New SOTA performance on IPF benchmarks.", "questions": "- As mentioned above, what is the relationship between sequence recovery and designability/foldability? [note to self of checking their foldability results.\n\n- Why does the method only use C-alpha positions? ProteinMPNN found using all backbone atoms to improve results. I would be curious if all backbone atoms helps and if noise augmentation improves.\n\n- Have the authors tried not providing hand-crafted geometric features? I would’ve suspected these can be inferred from the raw coordinates like in ProteinMPNN.\n\n- Why do the benchmark numbers not match those reported in ProteinMPNN?\n\n- There is are some typos and weird language I noticed.\n    - Line 78. “Meticulously curated” is a weird phrase. I would reword it.\n    - Line 88. Needs capitalization.\n    - Line 93. x_1^{pos} is bold face while the rest are not.\n    - Line 134. Equivariant is misspelled.\n    - Line 143. normalized is misspelled.", "limitations": "- There no limitations discussed. It would be interesting to see where the method fails and initial preliminary thoughts on how it can be improved.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "I enjoy the direction and development of ideas in this paper. Protein sequences are largely driven by their local environment based on the graph construction. It makes a lot sense to free the decoding order to be learned rather than left-to-right. Though the method is not new, the application is novel and well motivated. I appreciate the authors exploring this direction for IPF.\n\n- Well-motivated by taking advantage of the graph structure inherent in the problem to improve sampling.\n- Well written. I could follow the math and figures to understand the method. I appreciate the authors shared code.\n- Removes limitations of left-to-right decoding order of previous works which results in higher diversity.", "weaknesses": "- Comparison to alternative discrete diffusion methods. I hoped for more discussion on the merits and considerations of choosing D3PM over alternative methods of discrete diffusion such as analog bits and latent diffusion.\n\n- No in-depth analysis of foldability. There is a question of whether higher recovery rate is always better. For instance, foldingdiff [1] showed in Table S1 and S2 that foldability was much higher with ProteinMPNN than with ESM-1F -- the latter which has higher sequence recovery. Including a benchmark on foldability and not just a few examples would be important to understand foldability as a function of sequence recovery on a benchmark would be high importance to understand the relationship. I already believe the contributions are strong and this is merely a suggestion for higher impact.\n\n- No explanation of different benchmark numbers. I noticed the numbers for proteinmpnn don't match what is reported in this paper.\n\n\n[1] https://arxiv.org/abs/2209.15611\n[2] https://www.science.org/doi/10.1126/science.add2187 \n[3] https://www.biorxiv.org/content/10.1101/2022.04.10.487779v1 \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for recognizing the novelty and contribution of our work. Your insightful comments helped us enrich the analysis a lot. By the response below we hope we address your concerns properly. \n\n**weaknesses**:\n\n1. *comparison to alternative diffusion methods*: The question of choosing between discrete diffusion in original space (e.g., D3PM) and latent continuous space diffusion (e.g., analog bits) remains open. Each approach has its merits and limitations. Discrete diffusions provide transparency in the generative process, enabling observation of sequence transformation, and latent space diffusion benefits from established continuous diffusion methods used in image/video generation. The latter allows a similar treatment of discrete data diffusion within a continuous Gaussian diffusion framework.\n\n2. *foldability*: Thank you for highlighting this important aspect. We've incorporated your suggestion by adopting the TM score to assess foldability. We've provided Table 4 in the PDF, summarizing average TM scores (with standard deviations) between structures predicted by AlphaFold2 for generated sequences and their wild-type counterparts. GraDe-IF consistently achieves the highest TM scores with the least variance. Moreover, we've extended foldability evaluation to a wider set of 42 proteins (Table 5 in PDF), confirming GraDe-IF's superiority in terms of foldability.\n\n3. *different benchmark performance*: The benchmark discrepancies arise due to our distinct dataset split rules. Although both ProteinMPNN and our study utilize CATH4.2, ProteinMPNN used random splitting by 80:%10%:10% (according to their experimental details), and we follow Ingraham et al.'s [1] split, ensuring no overlap in CATH topology classification between training, validation, and test sets. This explains the divergent reported results. Same as [1,2,3], we implemented the latter splitting rule for our method. We have also tested the 80:%10%:10% splitting rule as established in ProteinMPNN’s paper, and it returns a higher recovery rate on the test set at 59.6%. As a reference, ProteinMPNN reported their recovery rate at 50.8%. Having said that, we would like to emphasize again that we do not optimize our diffusion model towards a close-to-1 recovery rate of the generated sequence. Instead, we expect a preferred model to perform higher diversity on the generated sequences while capturing the composition of amino acids at those conserved regions or positions (such as the buried regions in proteins.).\n\n**Questions**:\n\n1. *foldability*:  see weaknesses point 2.\n\n2. *coordinates*: (1) The objective of the designed model is to generate possible amino acid sequences for a given protein backbone. To this end, we make all the operations, including the diffusion process and the message passing propagators on residue levels. Since the atom types for different amino acids only differ on the side chain, atom types on the backbone are determined. Regarding the atom positions, we initially utilized C-alpha positions to represent the position of the residue, which is a widely applied choice for generating residue-level protein graphs. (2) While we did not employ the position of other atoms explicitly, we did include their spatial relationship by defining the relative position and local frames of local atoms. The detailed definitions were introduced in Appendix C.\n\n3. *hand-crafted features*: We agree that it is possible to infer the processed node features from the raw coordinates. However, there is no doubt that a much higher volume of input proteins would be required to train a more sophisticated (and potentially more powerful) model. Moreover, defining hand-crafted features has been used in several previous protein-related works, which indicates that it can be considered a promising choice for modeling protein topology. While we remain open to exploring this idea in future research, we conducted a preliminary ablation study that investigates the influence of the dihedral angle on the model's performance. The study revealed that omitting this hand-crafted feature led to a decrease in the recovery rate of the entire test dataset, from 52.21% to 51.47%, and an increase in perplexity from 4.35 to 4.58.\n\n4. *mismatched performance for ProteinMPNN*: see weaknesses point 3.\n\n5. *typos*: We've diligently corrected the typos you pointed out in the revised version.\n\nWe have followed your insightful comments and suggestions to address the concerns you raised and polish our work. We believe these changes have greatly enhanced the quality and contribution of this paper. We believe that the additional analysis and discussion now better support the merit of our work. We would be more than willing to take your further suggestions, and we kindly request you reconsider the score you assigned to our paper. \n\n\n**Reference**:\n\n[1] Ingraham, John, et al. \"Generative models for graph-based protein design.\" In Advances in neural information processing systems (2019).\n\n[2] Gao, Zhangyang, Cheng Tan, and Stan Z. Li. \"PiFold: Toward effective and efficient protein inverse folding.\" In International Conference on Learning Representations (2022).\n\n[3] Zheng, Zaixiang, et al. \"Structure-informed Language Models Are Protein Designers.\" In International Conference of Machine Learning (2023).\n"}]

Author rebuttal:
We thank all the reviewers for their detailed comments and insightful suggestions. We incorporated additional experiments and analyses as per recommendations. Here, we present a concise overview of the major enhancements that have been universally implemented, focusing on aspects such as diversity (Figure 1, Table 1), structural comparison (Table 2), sampling speed (Table 3), and foldability (Table 4 & 5). The visual representation in the form of the figure and tables has been included within the attached PDF, with corresponding references provided in each of our responses. We believe that a brief overview of these additional results will provide a clear context for comprehending the significance of the updates we have made.
 
 **Figure 1. t-SNE of the generated sequences in three different recovery levels.**
 
In response to the task of aligning recovery rate levels, we have introduced an intermediary plot positioned between the two subfigures in Figure 4. This supplementary plot effectively illustrates the intricate relationship between recovery rates and diversity across the three methods on a finer scale. Notably, the visualization highlights the rapid contraction of the sampling space for both PiFold and ProteinMPNN. In contrast, the samples generated by GraDe-IF exhibit a broader distribution that encompasses the wild type.
 
**Table 1. Comparison of diversity and recovery rate at three different levels**
 
The recovery rate and diversity of the generation results were compared by controlling temperature = {0.5, 0.1, 0.0001} for PiFold and ProteinMPNN and sample step={20, 100, 250} for GraDe-IF. The three levels are divided roughly into low, medium, and high recovery rates in the table below. The results, quantified as average diversity and recovery rate, are depicted in the table provided. For clarity, the metric "diversity" is quantitatively defined as `diversity = 1-sequence identity`, where the average sequence identity is computed pairwise for generated sequences. For instance, in the first cell of the table, where the method is PiFold, the recovery level is low, and the metric is diversity, the diversity value is indicated as 0.3796. 
 
**Table 2. Average RMSD and pLDDT across 42 protein structures (folded by AlphaFold2) with model-generated sequences.**
 
We expanded the validation of RMSD and pLDDT across a wider range of 42 proteins. For each protein backbone, a sequence is generated per model, which is then folded by AlphaFold2 to calculate the average pLDDT and RMSD with respect to the template wild-type protein structure. Notably, GraDe-IF exhibited the highest average pLDDT and lowest RMSD, highlighting its effectiveness over baseline methods.

**Table 3. Sampling speed of GraDe-IF (DDIM) at varying step sizes and baseline methods.**
 
The sampling speed of DDIM is influenced by the choice of skip steps (as illustrated in Figure 5). To illustrate, we present the generation of 3FKF-A and provide a comparative analysis of inference times along with baseline methods, considering skip steps of 100, 20, and 1 for GraDe-IF, where the skip step of 1 reverts to the original DDPM sampling algorithm. The inference time of GraDe-IF decreases significantly with increasing step size, and it becomes faster than all baseline methods when step size=100.
 
**Table 4. TM score comparison. 10 sequences were generated for each protein backbone.**
 
We computed the TM score to measure the foldability associated with the sequences generated by gauging the structural similarity between the structure of the generated sequences and the native structures. Generally speaking, a higher TM score demonstrates a better chance for the structure to be folded, and a protein with a TM score lower than 0.5 is basically considered unfoldable. The table below summarizes the average TM scores (along with the standard deviations) between the structures predicted by AlphaFold2 for generated sequences and their associated wild-type structures. Our GraDe-IF constantly achieves the highest TM score with the smallest variance.
 
**Table 5. TM score comparison on 42 protein backbones, with each backbone generating 1 sample. A sequence is considered foldable if its TM score > 0.5.**
 
The foldability of the generated protein sequences was also evaluated on a wider range of 42 proteins (Note that the reason we not investigating a larger volume of proteins is because that AlphaFold failed to fold more structures within the limited time for rebuttal). We define the number of successfully folded proteins by whether their TM score is larger than 0.5. Again, GraDe-IF delivers the largest amount of foldable proteins with the highest average TM score.


Meta-review: There was agreement among the reviewers that the authors method is novel, solves an important problem, and achieves very competitive performance. The reviewers mostly suggested a variety of new experiments, and the authors provided these results extensively, which they should incorporate into the camera ready version of the paper. Overall, the authors' performance improvements over even very recent inverse folding methods like PiFold are impressive, and the approach taken seems novel.
Decision: Accept (poster)