PiFold: Toward effective and efficient protein inverse folding
Authors: Zhangyang Gao
Cheng Tan
Stan Z. Li
Venue: ICLR 2023 notable top 25% (2023)

Abstract: How can we design protein sequences folding into the desired structures effectively and efficiently? AI methods for structure-based protein design have attracted increasing attention in recent years; however, few methods can simultaneously improve the accuracy and efficiency due to the lack of expressive features and autoregressive sequence decoder. To address these issues, we propose PiFold, which contains a novel residue featurizer and PiGNN layers to generate protein sequences in a one-shot way with improved recovery. Experiments show that PiFold could achieve 51.66\% recovery on CATH 4.2, while the inference speed is 70 times faster than the autoregressive competitors. In addition, PiFold achieves 58.72\% and 60.42\% recovery scores on TS50 and TS500, respectively. We conduct comprehensive ablation studies to reveal the role of different types of protein features and model designs, inspiring further simplification and improvement. The PyTorch code is available at \href{https://github.com/A4Bio/PiFold}{GitHub}.

Published as a conference paper at ICLR 2023
PIFOLD: TOWARD EFFECTIVE AND EFFICIENT PROTEIN INVERSE FOLDING
Zhangyang Gaoi, Cheng Tan", Stan Z. Li' AI Lab, Research Center for Industries the Future, Westlake University (gaozhangyang, tancheng, Stan. 2Q @ wake edu.en
ABSTRACT
How can we design protein sequences folding into the desired structures effec- tively and efficiently? AI methods for anueture-based protein design have at- tracted increasing attention in recent years; however, few methods can simultane- ously improve the accuracy and efficiency due to the lack of expressive features and autoregressive sequence decoder. To address these issues, we propose PiFold, which contains novel residue featurizer and PiGNN layers generate protein sequences in a one-shot way with improved recovery. Experiments show that Pi- Fold could achieve 51.66% recovery on CATH 4.2, while the inference speed is 70 times faster than the autoregressive competitors. In addition, PiFold achieves 58.72% 78% and 60.42% recovery scores on TS50 and TS500, respectively. We con- duct comprehensive ablation studies reveal the role of different types of protein features and model designs, inspiring further simplification and improvement. The PyTorch code is available at GitHub
Higher Recovery
$1.66
PiFold
45,96
Recovery
ProtcinMPNN
41.31
GVP
40
AlphaDesign
GCA
39.47
38.3
37.64
StructGNN
35.91
ESM-IF
35.82
StructTrans
Better Efficiency
30
Inference time!ss
Figure Performance comparison with other meaphbbased protein design methods. The recovery scores, the inference time costs, and the perplexities are shown in the Y-axis direction, the X-axis direction, and the circle size, respectively. Note that the recovery and perplexity results in the CATH dataset are reported without using any other training data. The inference time averaged 100 long protein sequences of average length 1632 on an NVIDIA V100.
INTRODUCTION
Proteins are linear chains of amino acids that fold into 3D structures to control cellular processes, such as transcription, translation, signaling, and cell cycle control. Creating novel proteins for hu- man purposes could deepen understanding of living systems and facilitate the fight against the
Equal Contribution, "Corresponding Author.

---

Published as a conference paper at ICLR 2023
disease. One of the crucial problems is to design protein sequences that fold to desired structures,
namely structure-
protein design (Pabo 1983). Recently, many deep learning models have
been proposed to solve this problem (Li et al. [014]] Wu et al. 2021 |earce & Zhang [02T]] Ovehin-
nikov & Huang
2021 Ding et al. 2022] Gao et al. 2020; 2022a| Dauparas et al. 2022] Ingraham
et al.
2019;
al.
2020 Tan et al. [022cc Hsu al. 2022; O'Connell et al. 20188] Wang al.
2018
Qi Zhang
22200] Strokach et al. 2020 Chen et Zhan et al. 2020aa] Huang et al. Anand al.
2017
2022; Strokach Kim [022]]
Li Kochl, [0144] Greener et al.
20118 Karimi
et al.
2020; Anishchenko et al. 2021 Co et al. 2021, Liu et al. 2022 McPartion et al. 2022
Huang et al.
2022] Dumortier et al. 2022] Li et al., 2022a Maguire al. 2021 Li al. 2022b),
among which graph-based models have made significant progress. However, there is still room to improve the model's accuracy and efficiency. For example, most graph models could not achieve 50+% sequence recovery in CATH dataset due to the lack of expressive residue representations. Moreover, they also suffer from the autoregressive generation, resulting in low inference speed. We aim to simultaneously improve the accuracy and efficiency with simple model containing few redundancies as possible.
For years, graph- models have struggled to learn expressive residue representations through better feature engineering, more elaborate models, and a larger training dataset. For example, Al- phaDesign (Gao et al., 2022a) and ProteinMPNN (Dauparas al. (022) point out that additional angle and distance features could significantly improve the representation quality. GraphTTram (In- graham et al. 2019). GVP (Jing et al. 2020), and GCA (Tan al., 2022c) introduce new modules considering graph-based message passing, equivariant vectors, and global attention to learn geo- metric representations from residue interactions. ESM-IF (Hsu al. 2022) incorporates additional training data to capture more structural learning bias. Although they have made significant progress, two problems remain to be solved under the same data set setting: (1) Is there a better way con- struct effective features to facilitate learning residual representations? (2) How can we improve the model to allow to learn better representation from residue interactions?
Most graph models (Dauparas et al. 2022 Ingraham et al. 2019 Jing et al.) 2020 Hsu et al. 2022
Tan et al. 2022c; Hsu et al. 2022) adopt the autoregressive decoding scheme to generate amino acids, dramatically slowing down the inference process. Interestingly, few studies have attempted to improve the model efficiency, perhaps because the efficiency gain requires sacrificing some accuracy
(Bahdanau et al. 2015 Vaswani al.] 2017 2017) Ghazvininejad al., [019:Geng et al.. [021 Xie et al.)
2020|Wangg et al. [2019] Gu et |l.1 18) while the latter is more important than effiiency in protein design. To address this dilemma, AlphaDesign (Gao et al. (022a)) proposes a parallel self-correcting module to speed up inference while almost maintaining the recovery. Nevertheless, it still causes some performance degradation and requires two iterations for prediction. Can we generate protein sequences in a one-shot way without loss of accuracy?
We propose PiFold (protein inverse folding) to address the problems above, containing novel residue featurizer and stacked PiGNNs. As to the featurizer, for each residue, we construct more comprehensive features and introduce learnable virtual atoms to capture information overlooked by real atoms. The PiGNN considers feature dependencies at the node, edge, and global levels to learn from multi-scale residue interactions. addition, we could completely remove the autoregres- sive decoder by stacking more PiGNN layers without sacrificing accuracy. Experiments show that PiFold can achieve of-te--f-hh-arr recoveries on several real-world datasets, i.c., 51.66% on CATH 4.2, 58.72 TS50, and 60. 40.42% on TS500. PiFold is the first graph model that could exceed 55% re- covery on TS50 and 60% recovery on TS500. In addition, PiFold's inference efficiency in designing long proteins is improved by 70+ times compared to autoregressive competitors. More importantly, we conduct extensive ablation studies to reveal the role of each module, which helps deepen the reader's understanding of PiFold and may provide further inspiration for subsequent research. In summary, our contributions include:
1. We propose novel residue featurizer to construct comprehensive residue features and learn virtual atoms to capture complementary information with real atoms.
2. We propose a PiGNN layer to learn representations from multi-scale residue interactions.
3. We suggest removing the autoregressive decoder to speed up without sacrificing accuracy.
4. We comprehensively compare advanced graph models in real-world datasets, e.g., CATH, TS50, and TS500, and demonstrate the potential of designing different protein chains.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORKS
Recently, Al algorithms have evolved rapidly in many fields (Gao ct al. 2022d Cao ct al. 2022 Tan
et al. 2022a Li et al. 2022c He et al. 2020 Stärk al. 2022), where the protein folding problem
(Jumper et al. 2021] Wu et al. 2022 Lin et al. [022] Mirdita et al.) [2022] Wang et al. 2022 Li
al. 2022d) that has troubled humans for decades has been nearly solved. Its inverse problem-
structure-based protein design - is receiving increasing attention.
Problem definition The structure-based protein design aims to find the amino acids sequence S = {s. : S i S n folding into the desired structure X' {a; R S S n), where n is the number of residues and the natural proteins are composed by 20 types of amino acids, i.c. 1 s < 20 and 8; E N Formally, that is to learn function Fe:
Fg x
Because homologous proteins always share similar structures (Pearson Sierk 2005). the problem itself is underdetermined, i.e., the valid amino acid sequence may not be unique (Gao et al.) 2020). To consider both sequential and structural dependencies, recent works (Hsu et al., 2022) suggest combining the 3D structural encoder and ID sequence decoder, where the protein sequences are generated in an autoregressive way:
PS) = II p(s))s<a,X:0))
(2)
Based on different types of structural encoders. existing works can be divided into MLP-based, CNN-based, and GNN-based ones (Gao et al,, 2002a).
MLP-based models MLP is used to predict the probability of 20 amino acids for each residue, and various methods are mainly difficult in feature construction. These methods are commonly evaluated on the TS50, which contains 50 native structures. For example, SPIN (Li et al.) 2014) achieves 30% recovery on TS50 by using torsion angles (á and r²), sequence profiles, and energy profiles. Through adding backbone angles (0 and r), local contact number, and neighborhood dis- tance, SPIN2 (O'Connell al. 2018) improves the recovery to 34%. Wang's model (Wang al., 2018) suggests using backbone dihedrals (ø. sb and w), the solvent accessible surface area of back- bone atoms (Ca, ., C, and O) secondary structure types (helix, sheet, loop), C. Ca distance and unit direction vectors of C. Ca, C. N and C. C' and achieves 33% recovery. The MLP method enjoys a high inference speed, but suffers from low recovery rate because the structural information is not sufficiently considered.
These methods use 2D CNN or 3d CNN to extract protein features (Torg & Altman 2017, Boomsma & Frellsen 2017 Weiler et al.. 2018 Zhang et al. 2020al Qi & Zhang 2020 Chen el al.) 2019) and are commonly evaluated on the TS50 and TS500. SPROF (Chen al. 2019) adopts 2D CNN learn residue representations from the distance matrix and achieves
CNN-based models
a 40.25% recovery on TS500. 3D CNN-based methods, such as ProDCoNN (Zhang et al. 2020a) and DenseCPD (Qi & Zhang 2020), extract residue features from the atom distribution
three- dimensional grid box. For each residue, after being translated and rotated to standard position, the atomic distribution is fed to the model to learn translation- and rotation- -avaiinnnnvariant features. ProD- CoNN (Zhang et al. (2020a) designs nine-layer 3D CNN with multi-scale convolution kernels and achieves 42.2% recovery on TS500, DenseCPD (Qi Zhang (020) uses the DensetNet architec- ture (Huang et al. 2017) to boost the recovery 55.53% TS500. Recent works (Anand et al.) 2022) have also explored the potential of deep models to generalize to de novo proteins. Despite the improved recovery achieved by the 3D CNN models, their inference is slow, probably because they require separate preprocessing and prediction for each residue.
Graph-based models These methods use k--N graph to represent the 3D structure and employ graph neural networks (Defferrard et al.] 2016 Kipf & Welling, 2016 Veličković et al. 2017 Zhou ctt al., 2020 Zhang et al. [220bb] Gao et al. 2022b Tan et al., 2022b Gao et al. 2022c) to extract residue features while considering structural constraints. The protein graph encodes residue infor- mation and pairwise interactions as the node and edge features, respectively. GraphTrans (Ingraham et al. 2019) uses the graph attention encoder and autoregressive decoder for protein design. GVP (Jing et al. 2020) proposes geometric vector perceptrons to learn from both scalar and vector fea- tures. GCA (Tan et al. 2022c) introduces global graph attention for learning contextual features. In
3

---

Published as a conference paper at ICLR 2023
addition, ProteinSolver (Strokach al. 2020) is developed for scenarios where partial sequences
are known while does not report results on standard benchmarks. Recently, AephaDesign (ao et al,
2022a), ProteinMPPN (Dauparas et al. 2022) and Inverse Folding (Hsu et al. (022) achieve dra- matic improvements. Compared to CNN methods, graph models do not require rotating each residue separately in CNN, thus improving the training efficiency. Compared to MLP methods, the well- exploited structural information helps GNN obtain higher recovery. In summary, GNN can achieve a good balance between efficiency and accuracy.
3 METHOD
3.1 OVERALL FRAMEWORK
We show the overall PiFold framework in Figure where the inputs are protein structures, and outputs are protein sequences expected to fold into input structures. We propose a novel residue featurizer and PiGNN layer to learn expressive residue representations. Specifically, the residue featurizer constructs comprehensive residue features and creates learnable virtual atoms to capture information complementary real atoms. The PiGNN considers multi-scale residue interactions in node, edge, and global context levels. PiFold could generate protein sequences in one-shot manner with higher recovery than previous autoregressive (Ingraham et al.) 2019 Ding et al., 2020 Tan
et al. 2022c
Dauparas et al. 2022
Hsu et al. (2022) or iterative models (Gao et al. 2022a).
Panial feotures
predicior
Confidence
Generator
NNN
NNO
Resion 7
Partial Featurizer
Encoder
Decoder
NAT Desader
Previous
Our
Reeldue structures
Distance
PNDA
Angle
AN0N
NV09
NN00
NN02
Direction
Pifold Featurizer
PiFold Model
Figure overall framework PiFold. The featurizer module extracts comprehensive residue features from the fixed real atoms and learnable virtual atoms. We stack ten layers of PiGNNs, considering node-level, leve- and global-level residue interactions, to learn expressive residue representations. PiFold generates protein sequences in one-shot manner, without autoregressive iterative decoding schemas.
3.2 FEATURIZE MODULE
This section describes how to construct protein graphs and residue features. We recommend readers view Fig 3 for intuitive understanding.
Graph structure We represent the protein as k-NN graph derived from residues to consider the 3D dependencies, where & defaults to 30. The protein graph G(A, X, E) consists the adjacency matrix A E {0, j node features X E IRn."=, and edge features E IR.. Note that and m are the numbers of nodes and edges, and we construct fn node features and fe edge features con- sidering the inductive bias of residues" stable structure, order, and coordinates. Since each residue consists of Ca, C, N and O, we could construct local coordinate system Q: = [b,, n,, b, x n: for residue i, where u, Ca. N₁, these coordinate systems, we could construct rotation- and translation -nivariann features for single or pairs of residues, including distance, angle, and direction; refer to Fig 3
C Ca, b, s. u.-vi || and n; | Based
Distance features Given atom pairs A and B, the distance feature is RBF( B where RBF a radial basis function. Note that A E {C,, Ca.: . N₁, O,), B E (Co, Co,, N,, O,} when computing
features for residue while A {Ci, Ca,, N., 0₂} and B E (C,, Cay, N₁. 02} when computing

---

Published as a conference paper at ICLR 2023
features for residue pair (2. Careful readers should notice that the number of features is propor- tional to the square of the number of atoms, which will be further analyzed and simplified in Ta- 6 Beyond the distances between real atoms, also introduce virtual atoms {V, V22 V ....... for residue i. The additional node and edge features are generated by sampling A and B from VV V ,..... respectively.
(V, V22, , To ensure that the virtual atoms are translation- and rotation- equivariant to the protein structure, the model required learn their relative position in each local coordinate system. The relative position vectors are shared across all residues, thus forcing the model to learn consensus knowledge to extend to general scenarios. We normalize the relative coordinates by limiting their length unit vector for stable training. Formally, denoting the relative position of the k-th virtual atom is (It y/ke =k), we compute the absolute coordinates of V. by
..... (V,1, V,2, , ;... and (V V . }
V = Ne 2x (b n:)] + Ca. x + v + 2 1,
(3)
where 2h. yk 2k are learnable parameters.
residue
21
M
||
1 uxxu
Q. n,,b Local coordinate system
8 1
++1
Node i
Edge (5,J)
Distance
||C,, Cil 1Cau Nill
1CC, Cyl
1[0
ag-Angge N.
-i-Derda angle N.
Angle
g(R,)) =(Q
Bi-Angle for C---NN-C. Cm Ar-Aggl Ca--Cr-Nes
v Dinodral angic C,-Ca,
Dihedral angle C.
Direction
Nr Cu of
O₂ Co.
Nj Co. of Oj C.. 10,, Call
[ Call
100 Ca.l
||, Ca,ll
Figure 3: PiFold featurizer. We construct distance, angle, and direction features for single or paired residues, all invariant to rotation and translation. Learnable virtual atoms are used for additional distance features.
Angle features As shown in Figure 3 we use the bond angles (a,, 3,, 7R) and torsion angles (dx, tha, w)) as node features considering the backbone order and local geometry between adjacent residues. For residue pairs (i, the angular features represent the quaternions of relative rotation between their local coordinate systems, written diş q(Q) Q2), where R is the quaternion encoding function.
Direction features For single residue i, we use the directions of inner atoms to C.N as node fea- N.-Ca,
tures. For example, the direction feature of N; is Qf' [[N-C., Similarly, for pairs of residues (i, Ny the directions include all atoms of residue j to the Ca,, and the direction of Nj is QT [[N, [ Note that the relative direction concerning the local coordinate system Q, make the fea- tures rotationally invariant.
3.3 PIGNN
We propose the PiGNN layer to learn geometric residue representations considering multi-scale residue interactions in node, edge, and global context levels.
Local node interactions We use the simplified graph transformer (Gao et al. 2022a) to update node representations, where the model leams multi-headed attention weights via MLP instead of
5

---

Published as a conference paper at ICLR 2023
separately computing Q and K. Formally, we take h and ej, the output feature vectors of node i and edge j + in layer Before the first GNN layer, we use MLPs to project all input features to the d-dimensional space, thus hς E E and e): R When considering the attention mechanisms centered in node i, the attention weight Oji at the + layer is calculated by:
Wjji P (14))
exp Mn5
(4)
ajs
ELEN; exp whe
where N₁ is the neighborhood system of node and || means the concatenation operation. The node feature f is updated by:
2 = NodeMLP(e (h/))
h EjEN; agtt)
Local edge interactions The protein graph is an attributed graph containing both node and edge features; however, the simplified graph transformer is a node-centered network, which does not iteratively update edge features. We find that this neglect will lead to suboptimal representations, and introducing the edge updating layer could improve the model capability:
H+1 Ed
(6)
Global context attention While the local interactions play a crucial role in learning residue rep- resentations (Ingraham et al. 2019 Jing et al. 2020 Gao et al. 2022a), the global information (Tan) ctt al. 2022c) is also proved to be valuable for improving protein design. However, the time complex- ity of global attention across the whole protein is proportional to the square of the protein length, which significantly increases the computational overhead. To simultaneously enjoy the improved recovery and with good efficiency, we suggest learning a global context vector for each protein and using it to apply gated attention for node representations:
c, = Mean( (h (h(+)) )KEB,)
h 1+ = h @G(GetMMLP(c)
where B; is the index set of residues belonging to the same protein residue i, @ is element wise product operation, and (-) is the sigmoid function. The computational cost of the global context attention is linear to the residue numbers.
EXPERIMENTS
We comprehensively evaluate PiFold on different datasets and show its superior accuracy and effi- ciency. Detailed ablation studies are applied to deepen the understanding of PiFold. summary, will answer the following questions:
Performance (Q1): Could PiFold achieve SOTA accuracy and efficiency on CATH dataset?
Ablation (Q2): How much gain can be obtained for each improvement?
Benchmarking (Q3): Could PiFold be extended to more benchmarks, such as TS50 and TS500?
4.1 PERFORMANCE (Q1)
Objective & Setting Could PiFold achieve SOTA recovery and efficiency on real-world datasets?
Does the efficiency improvement require sacrificing the generative quality? To answer these ques- tions, we compare PiFold against recent strong baselines on the CATH (Orengo et al. 1997) dataset. We use the same data splitting as GraphTrans (Ingraham et al. 2019) and GVP (Jing et al. 2020), where proteins training. 608 proteins for validation, and 1120 proteins for testing. We stack ten layers PiGNN construct the PiFold model with hidden dimension 128. The model is trained up 100 epochs by default using the Adam optimizer on NVIDIA V100s. The batch size and learning rate used for training 8 and 0.001, respectively. For studying the generative quality, we report perplexity and median recovery scores on short-chain, single-chain, and all-chain settings. Regarding efficiency, we report the model inference time when generating 100 long chains with an average length of 1632.
partitioned by the CATH topology classification, resulting in 18024 proteins for

---

Published as a conference paper at ICLR 2023
Baselines We compare PiFold with recent graph models, including StructGNN, StructTrans In- grabam et al. 2019). GCA Tan et al., (022c). GVP (Jing et al. 2020). GVP-large AlphaDesign (Gao et al.] 2022a). ESM-IF (Hsu et al., 2022). ProteinMPNN (Dauparas et al. 2022), since most of them are open-source. To make a tair and reliable comparison, we reproduce StructGNNN Struct- Trans, GCA, GVP, AlphaDesign, and ProteinMPNN under the same data splitting CATH 4.2. The relevant results were generally consistent with their papers. Because GVP-large and ESM-IF do not provide training scripts and their code framework is different from other methods, we have difficulty reproducing them and therefore copy the results from their manuscripts. It should be re- minded that ESM-IF uses CATH 4.3 for training. which may lead incomparable recovery and perplexity due to the different data volumes. MLP and CNN-based methods are ignored here be- cause most of them not report results on CATH nor provide publicly available codes, and we will discuss them in Section For more baseline details, we suggest the reader refer to the Appendix.
Table 1: Results comparison on the CATH dataset. All baselines are reproduced under the same code frame- work, except ones marked with f. We copy results of GVP-large and ESM-IF from their manuscripts Hssu al. 2022 The best and suboptimal results are labeled with bold and underline.
Perplexity & Single-chain
Recovery % t Single-chain
CATH
version
Model
Short
All
Short
All
4.2
4.3
StructGNN
8.29
8.74
6.40
29.44
28.26
35.91
GraphTrans
8.39
8,83
6.63
28. 14
28.46
35.82
GCA
7.09
7.49
6.05
32.62
31.10
37.64
GVP
7.23
7.84
5.36
30.60
28.95
39.47
GVP-larget
7.68
6.12
6.17
32.6
39.4
39.2
AlphaDesign
7.32
7.63
6.30
34.16
32.66
41.31
ESM-IFI 8.18
6.33
6.44
31.3
38.5
38.3
ProteinMPNN
6.21
6.68
4.61
36.35
34.43
45.96
PiFold
6.04
6.31
4.55
39.84
38.53
51.66
Recovery We report the perplexity and recovery score in Table where two subsets of the full test set are also considered, following previous works (Ingraham al. 2019 Jing al. 2020 Tan et al. 2022c Hsu et al.] 2022). The "Short" set contains proteins up to length 100, and the "Single chain" contains proteins recorded single chains in the Protein Data Bank. We observe that the proposed PiFold could consistently improve the perplexity (lower is better) and recovery score (higher is better) across different test sets. On the "Short" and "All" datasets, PiFold achieves the best perplexities and recovery scores, where the recovery improvement are 3.49% and 1.94%, respectively. On the "Single-chain" dataset, PiFold is the best model when trained on CATH 4.2.
Perp(AlphaDesiggn) Rec (AphaDDsiign)
Perp(PoteinMPPN) Perp(PIFold)
Rec (ProteinMPNN) - Rec(PIFald)
9
Perplexity
Recovery
8
0.3
0.2
6
5
0.1
4
2
3
5
15
16
17
18
19
20
Epoch
Figure 4: Training PiFold for 20 epochs enough achieve SOTA performance.
Efficiency We already know that PiFold can achieve higher recovery; does this improvement come at the expense of efficiency? We further assess the training and inference time costs of PiFold and the competitive baselines (AlphaDesign and ProteinMPNN), PiFold could achieve state-of-the-.. perplexity and recovery during the training phase with fewer epochs. As shown in Figure 4 20 epochs are enough for PiFold. As to the inference speed, all baselines except AlphaDesign adopt an autoregressive decoder to generate residues one by one, and the computational complexity is O(L)

---

Published as a conference paper at ICLR 2023
to the protein length L. However, PiFold enjoys O(1) computational complexity due the one-shot generative schema. We benchmark the inference speed different graph models on 100 long chains with average length of 1632 and show the results in Fig[T] where PiFold is 70 times faster than autoregressive competitors, including ProteinMPNN, ESM-IF, GCA, StructGNN, StructTrans, GVP.
4.2 ABLATION (Q2)
Objective & Setting How much gain could be obtained from the introduced features and modules? Will removing the autoregressive decoder reduce performance? We conduct comprehensive ablation studies for the features and modules to answer these questions. All models are trained up to 20 epochs with learning rate of 0.001 using the Adam optimizer and OneCycle scheduler.
Table 2:
Ablation studies of feature types. The lower the ranking number, the higher the recovery rate and the
model performs better.
Model
PiPold
model
model
model
model
model
model
Distance
Node
Angle
Direction
Distance
Edge
Angle
/
Direction
V
/
V
Perplexity
4.55
4.59
4.62
4,53
4.80
4.54
4.65
Results
Recovery
51.66
51.49
51.05
51.61
49.61
51.51
50.86
Summary
Rank
Feature type ablation We determine the contribution rank of different types of features in Table We find that edge features are more important than node features, suggesting that amino acid types may be inferred primarily from residue interactions. Another discovery is that edge distances are the essential features; however, the number of distances is proportional to the square of the atom numbers, which may lead to feature explosion and redundancy. We further analyze the role of each edge distance in Table where the "C,-0" and "C-N" distances are the most significant ones. In addition, the ablation study in Table demonstrates that virtual atoms lead to performance gains, and the more virtual atoms, the higher recovery. One possible explanation is that virtual atoms can learn valuable information complementary backbone atoms, c.g., side chain direction.
Table 3: Ablation of PiGNN modules.
PiFold
model
model
model
model 4 model
GCN
GAT
Node
QKV
AttLPP
V
v
Edge
UpdateEdge
V
V
v
Global
ContextAnt
V
V
Perplexity
4.55
4.74
4.96
4.54
4.62
4.76
Results
Recovery
51.66
49.65
45.78
50.74
51.22
50.00
Summary
Change
41
4J4
4.
Model ablation We conduct systematic experiments to determine whether modules of PiGNN are effective and reveal their relative importance. At to the node message passing module, we compare the node message passing layer (AuMLP) against GCN (Kipf & Welling 2016). GAT (Veličković
et al. 2017) and QKV-based attention layer (Ingraham et al. 2019 Tan et al. 2022c Dauparas et al. 2022). We also investigate whether the edge updating operation and global context attention could contribute to the model performance. As shown in T
find that the AttMLP-based messaging mechanism can improve the recovery by at least 0.9%. In addition, the global context attention and edge updating could improve the recovery by 0.44% and 1.22%, respectively.
AT ablation Could PiFold remove the autoregressive mechanism and still ensure good perfor- mance? We further explore the effectiveness of the autoregressive decoder (Ingraham et al. 2019) and one-shot PiGNN encoder. As shown in Table 4 by gradually replacing the autoregressive de- coder with the one-shot PiGNN (model - model 5), the recovery will improve. At the same time, the inference time cost CATH will reduce. This phenomenon suggests that the expressive encoder is more important than the autoregressive decoder. The recovery score can be further improved by using a deeper encoder (model + model 6 + PiFold).
8

---

Published as a conference paper at ICLR 2023
Table 4: Autoregressive ablation. "Enc" and "AT" indicate the encoder's and autoregressive layers' numbers.
PiFold
model
model
model 3
model
model
model
Enc
10
3
6
8
AT
0
3
2
0
Perplexity 4
4.55
5.06
5.04
4.92
4.83
4.58
4.53
Recovery
51.66
49.30
49.37
49.60
50.41
50.96
51.22
Test Time
36s
527s
707s
522s
347s
30s
34s
4.3 GENERALIZATION (Q3)
Table 5: Results on TS50 and TS500. All baselines are reproduced under the same code framework, except ones marked with t. whose results are copied from their manuscripts. The best and suboptimal results are labeled with bold and underline.
TS50
TS500
Group Model
Perplexity
Recovery 1
Worst
Perplexity
Recovery
Worst
CNN MLP
SPIN
30.30
30.30
SPIN2
33.60
36.60
Wang's model
33.00
36.14
SPROF
40.25
ProDCoNN
42.20
DenseCPD
50.71
55.53
StructGNN
5.40
26.92
4.98
45.69
0.05
GraphTrans
5.60
29.22
44,66
0.03
Graph
GVP
4.71
44.14
33.73
4.20
49.14
0.09
GCA
5.09
47,02
28.87
4.72
47.74
0,03
AlphaDesign
5.25
48.36
32.31
4.93
49.23
0.03
ProteinMPNN
3.93
54.43
37.24
3.53
58.08
0.03
PiFold (our)
3.86
58.72
37.93
3.44
60.42
0.03
Objective & Results To present a comprehensive comparison and assess the generalizability of PiFold across different data sets, we report results on TS50 and TS500 in Table[5] We reproduce graph-based models and show the perplexity, median recovery score, and worst recovery. For MLP and CNN models, the results are copied from their manuscripts. We summarize that PiFold could achieve consistent improvements across TS50 and TS500. To our knowledge. PiFold is the first graph model that could exceed 55% recovery on TS50 and 60% on TS500. Finally, Figure. 5 we show the potential of PiFold in designing all-alpha, all-beta, and mixed proteins.
All-alpha structure (3tld) (Reoovery=56.22 25, RMSD=0.87)
All-beta structure (2gly) (Recavery=54.1i 19, RMSD=0.35)
Mixed structure (1mgr) (Recovery=47.co 42, RMSD=0.43)
Figure Visual examples. native structures, we color Helix, Sheet, Loop with cyan, magenta. and orange, respectively, We use AlphaFold2 to predict the structures of designed sequences, marked green. provide the recovery score and structural RMSD relative to the ground truth proteins.
CONCLUSION
PiFold, as AI design method for structure-based protein design, significantly improves the recov- ery and greatly increases the efficiency simultaneously. by using the proposed protein featurizer and PiGNN. It achieves 51.66%, 58. 38.72%, and 60.42% recovery scores CATH 4.2, TS50, and TS500 test sets, respectively, and is 70 times faster in inference speed than autoregressive competitors for designing long proteins. We also conduct comprehensive ablation to inspire future studies.

---

Published as a conference paper at ICLR 2023
6 ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their constructive and helpful reviews. This work was sup- ported by the National Key R&D Program of China (Project 20222D0115100) the National Natural Science Foundation of China (Project U21 A20427), the Research Center for Industries of the Future (Project WU2022C043), and the Westlake University Research Funds from the Center for Synthetic Biology and Integrated Bioengineering (Project WU2022A009).
REFERENCES
Namrata Anand, Raphael Eguchi, Irimpan Mathews, Carla P Perez, Alexander Derry, Russ B Altman, and Po-Ssu Huang. Protein sequence design with a learned potential. Nature communications, 13(1):1-11, 2022.
Ivan Anishchenko, Samuel Pellock, Tamuka M Chidyausiku, Theresa A Ramelot, Sergey Ovehin- nikov, Jingzhou Hao, Khushboo Bafna, Christoffer Norn, Alex Kang, Asim K Bera, et al. De novo protein design by deep network hallucination. Nature, 600(7889):547-552 2021.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference Learning Representations, ICLR 2015, 2015.
Wouter Boomsma and Jes Frellsen. Spherical convolutions and their application in molecular mod- elling. In I. Guyon, U. Von Luxburg, S. Bengio, Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. https: [11301a76fffcccllbb33555eee22222
http://arro...cpppp///////////// ccc/paper/2C17/Hile
neur neurips pdf
Hanqun Cao, Cheng Tan, Zhangyang Gao, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li. A survey generative diffusion model. arXiv preprint (ex.ix:220900644: 2022.
Yue Cao, Payel Das, Vijil Chenthamarakshan, Pin- Yu Chen, Igor Melnyk, and Yang Shen. Fold2seq: generative model for protein design. In International Conference on Machine Learning, pp. 1261-1271. PMLR, 2021.
A joint sequence (Id)-fold (3d) embedding-
Sheng Chen, Zhe Sun, Lihua Lin, Zifeng Liu, Xun Liu, Yutian Chong, Yutong Lu, Huiying Zhao, and Yuedong Yang. To improve protein sequence profile prediction through image captioning on pairwise residue distance map. Journal of chemical information and modeling, 60(1):391-399 2019.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning based protein sequence design using proteinmpan. bioRxiv, 2022.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29:3844-3852, 2016.
Wenze Ding, Kenta Nakai, and Haipeng Gong. Protein design via deep learning. Briefings in bioinformatics, 23(3):bbac10: 2022.
Baldwin Dumortier, Antoine Liutkus, Clément Carré, and Gabriel Krouk. Petribert: Augmenting bert with tridimensional encoding for inverse protein folding and design. bioRxiy, 2022.
Wenhao Gao, Sai Pooja Mahajan, Jeremias Sulam, and Jeffrey J Gray. Deep learning in protein structural modeling and design. Patterns, 100414 2020.
Zhangyang Gao, Cheng Tan, Stan Li, et al. Alphadesign: A graph protein design method and benchmark on alphafolddb. arXiv preprint arXiv:2202.01079, 2022a.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Cosp: Co-supervised pretraining of pocket and ligand. arXiv preprint XiX::2206. 12241, 2022b.
10

---

Published as a conference paper at ICLR 2023
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Semiretro: Semi-template framework boosts deep retrosynthesis prediction. arXiv preprint arXiv:2202.082005 2022c.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Li. Simvp: Simpler yet better video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 31 3170-3180, 2022d.
Xinwei Geng, Xiaocheng Feng, and Bing Qin. Learning to rewrite for non-autoregressive neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3297-3308, 2021.
Marjan Ghazvininejad, Omer Levy. Yinhan Liu. and Luke Zettlemoyer. Mask- Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNN.-IJCN.P),, pp. 61 12-21221, 2019.
Joe G Greener, Lewis Moffat, and David T Jones. Design of metalloproteins and novel protein folds using variational autoencoders. Scientific reports, 8(1):1-12, 2018.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregreessive
neural machine translation. In International Conference on Learning Representations, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEFE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexan- der Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022.
Bin Huang, Tingwe Fan, Kaiyue Wang, Haicang Zhang, Chungong Yu, Shuyu Nie, Yangshuo Qi, Wei-Mou Zheng, Jian Han, Zheng Fan, et al. Accurate and efficient protein sequence design through learning concise local environment of residues. bioRxiv, 2022.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4700 2017.
John Ingraham, Vikas K Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. 2019.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. arXiv preprint #rXiv:2009.01411 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583 2021.
Mostafa Karimi, Shaowen Zhu, Yue Cao, and Yang Shen. De novo protein design for novel folds using guided conditional wasserstein generative adversarial networks. Journal of chemical information and modeling, 60(12):5667-568) 2020.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv:166990099900 2016.
Alex J Li. Mindren Lu, Israel Tilahun Desta, Vikram Sundar, Gevorg Grigoryan, and Amy Keat- ing. Neural network-derived potts models for structure-based protein design using backbone atomic coordinates and tertiary motifs. bioRxiv, 2022a.
Alex J Li, Vikram Sundar, Gevorg Grigoryan, and Amy E Keating, Terminator: A neural framework for structure-based protein design using tertiary repeating motifs. arXiv preprint arXiv:2204. 13048, 2022b.
Jie Li and Patrice Kochl. 3d representations of amino acids applliaaippiiinnsssssssss to protein sequence com- parison and classification. Computational and structural biotechnology journal, 11(18):47-58, 2014.
11

---

Published as a conference paper at ICLR 2023
Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, and Stan Z Li. Openmixup: Open mixup toolbox and benchmark for visual representation learning. arXiv preprint EXXi:2209.048511, 2022c.
Zhixiu Li, Yuedong Yang, Eshel Faraggi, Jian Zhan, and Yaoqi Zhou. Direct prediction of profiles of sequences compatible with a protein structure by neural networks with fragment-based local and energy-based nonlocal profiles. Proteins: Structure, Function, and Bioinformatics, 82(10): 2565-2573, 2014.
Ziyao Li, Xuyang Liu, Weijie Chen, Fan Shen, Hangrui Bi, Guolin Ke, and Linfeng Zhang. Uni- fold: An open-source platform for developing protein folding models beyond alphafold. bioRxiv, 2022d.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos San- tos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Yufeng Liu, Lu Zhang, Weilun Wang, Min Zhu, Chenchen Wang, Fudong Li, Jiahai Zhang, Houqiang Li, Quan Chen, and Haiyan Liu. Rotamer-free protein sequence design based on deep learning and self-consistency. 2022.
Jack B Maguire, Daniele Grattarola, Vikram Khipple Mulligan, Eugene Klyshko, and Hans Melo. Xenet: Using a new graph convolution accelerate the timeline for protein design on quantum computers. PLoS computational biology, 17(9): 2021.
Matt McPartlon, Ben Lai, and Jinbo Xu. A deep (3)-quivvrriant model for learning inverse protein folding. bioRxiv, 2022.
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steinegger. Colabfold: making protein folding accessible all. Nature Methods, pp. 1-4, 2022.
James O'Connell, Zhixiu Li, Jack Hanson, Rhys Hefferman, James Lyons, Kuldip Paliwal, Abdollah Dehzangi, Yuedong Yang, and Yaoqi Zhou. Spin2: Predicting sequence profiles from protein structures using deep neural networks. Proteins: Structure, Function, and Bioinformatics, 86(6): 629-633, 2018.
Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thornton. Cath-a hierarchic classification of protein domain structures. Structure, 508) 1109, 1997.
Sergey Ovchinnikov and Po-Ssu Huang. crtucture-basede protein design with deep learning. Current opinion in chemical biology, 136-144, 2021.
Carl Pabo. Molecular technology: designing proteins and peptides. Nature, 301(5897):200-200, 1983.
Robin Pearce and Yang Zhang. Deep learning techniques have significantly impacted protein struc- ture prediction and protein design. Current opinion in structural biology, 68:194-207, 2021.
William R Pearson and Michael L Sierk. The limits of protein sequence comparison? Current opinion structural biology, 15(3):254-260, 2005. 2005.
Yifei Qi and John ZH Zhang. Denseepd: improving the accuracy of neural-network-based compu- tational protein sequence design with densenet. Journal of chemical information and modeling. 124-):1245 2020.
Hannes Stärk, Octavian Ganea, Lagnajit Pattanaik, Regina Barzilay, and Tommi Jaakkola. Equibind: Geometric deep learning for drug binding structure prediction. In International Conference on Machine Learning, pp. 10503-205211 PMLR, 2022.
Alexey Strokach and Philip M Kim. Deep generative modeling for protein design. Current opinion in structural biology, 72:226-236, 2022.
Alexey Strokach, David Becerra, Carles Corbi-Verge, Albert Perez-Riba, and Philip M Kim. Fast and flexible protein design using deep graph neural networks. Cell Systems, 1144):022441, 2020.
12

---

Published as a conference paper at ICLR 2023
Cheng Tan, Zhangyang Gao, and Stan Z Li. Simvp: Towards simple yet powerful spatiotemporal predictive learning. arXiv preprint arXiv:2211.12509, 2022a.
Cheng Tan, Zhangyang Gao, and Stan Li. Target-aware molecular graph generation. arXiv preprint arXiv:2202.04829, 2022b.
Cheng Tan, Zhangyang Gao, Jun Xia, and Stan Li. Generative de novo protein design with global context. aiviv preprint arXiv:2204. 106773,, 2022c.
Wen Torng and Russ B Altman. 3d deep convolutional neural networks for amino acid environment similarity analysis. BMC bioinformatics, 18(1):1-23,2011 2017
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10900 2017.
Guoxia Wang. Xiaomin Fang, Zhihua Wu, Yiqun Liu, Yang Xue, Yingfei Xiang, Dianhai Yu, Fan Wang, and Yanjun Ma. Helixfold: An efficient implementation of alphafold2 using paddlepaddle. arXiv preprint arXiv:2207.0544 2022.
Jingxue Wang, Huali Cao, John ZH Zhang, and Yifei Qi. Computational protein design with deep learning neural networks. Scientific reports, 8(1):1-9, 2018.
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie- Yan Liu. Non-autoregressive machine translation with auxiliary regularization. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 5377-5384, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL
Pww.saaaaaPape..aee pdf
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, 2022.
Zachary Wu, Kadina Johnston, Frances H Aroold, and Kevin Yang. Protein sequence design with deep generative models. Current opinion in chemical biology, 65:18-21 27, 2021.
Pan Xie, Zhi Cui, Xiuying Chen, XuaoHui Hu, Jianwei Cui, and Bin Wang. Infusing sequential in- formation into conditional masked translation model with self-review mechanism. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 15-25, 2020.
Yuan Zhang, Yang Chen, Chenran Wang, Chun-Chao Lo, Xiuwen Liu, Wei Wu, and Jinfeng Zhang- Prodconn: Protein design using a convolutional neural network. Proteins: Structure, Function, and Bioinformatics, 88(7):819-829 2020a.
Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning on graphs: survey. IEEE Transactions Knowledge and Data Engineering, 2020b.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changeheng Li, and Maosong Sun. Graph neural networks: A review of methods and applica- tions. AI Open, 1:57-81, 2020.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
Distance ablation We further analyze the role of each edge distance in Table where the Ca-O" and "C-N" distances are the most significant ones. Last but least, find that the distances between learnable virtual atoms significantly improve the recovery, suggesting that the model can automatically discover critical residual points.
Table 6: Ablation of edge distance features
Model
PiFold
10
Ca-Ca
Ca
C,-N
Real distances
Ca-O
C-C
d
C-N
C-O
N-N
N-O
0-0
0
Results Virtual atoms
2
3
/
/
/
Perplexity Recovery
4.55
4.55
4.53
4.53
4.54
4.54
4.52
4.55
4.59
4.54
4,53
51.66
51.52
51.63
51.74
51.49
51.70
51.89
51.50
51.02
51.10
51,51
Change
411
Baseline details ProteinMPNN supports the design of proteins when partial residues are known, while we focus on designing sequences from structures without any known residues. On CATH4.2, we note that ProteinMPNN directly assigns sequence labels to residues with missing coordinates, indicating these residues are treated as known ones. If allow this trick, ProteinMPNN could achieve 51% recovery on CATH4.2. To force methods to be compared under same setting. remove this operation and update the recovery to 45.96%. For ESM-IF, report their results without training by third-party data. They have proved that increased data volume could significantly improve model capability, thus affecting researchers' judgments about the models. We reproduced most of the baselines and allow them to be trained and evaluated under the same data splitting and code framework. We admire these works, and thank all the authors providing open-source codes.
14


Reviews:
[{"review_id": "rmEvfemXBwm", "reviewer": "Reviewer_4ek8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for structure-based protein design that utilizes graph neural networks to capture local and global interactions between atoms in the protein and outputs a sequence of amino acids with the desired input structure. Authors introduce a method to generate useful features from the input structure. These features derived from real atoms and learned virtual atoms are then processed through a stack of attention and MLP-based blocks that can directly generate the amino acid sequence. Through numerical experiments on benchmark datasets, authors demonstrate the performance and efficacy of their algorithm, yielding state-of-the-art recovery with low inference time.", "summary_of_the_review": "Overall, I am leaning towards borderline rejecting the paper, as the results are not reproducible and the original contribution compared to some previous work is somewhat unclear (see both under Weaknesses).", "strength_and_weaknesses": "Strengths:\n\n- The paper is well-written and organized in a logical manner. It is easy to follow in most parts.\n  \n- The algorithm shows good performance on the benchmarks. It consistently outperforms other methods in terms of recovery and perplexity in most cases and has the fastest inference time.\n  \n- The ablation studies are in-depth, authors put great effort into verifying their design choices.\n  \n\nWeaknesses:\n\n- The source code is missing, and therefore the results are not reproducible.\n  \n- The novelty compared to AlphaDesign and GCA is somewhat limited. One contribution the authors claim is the way features are generated. Here the novelty appears to be the introduction of virtual atoms, other elements can be found in AlphaDesign. The second contribution claimed by the authors is the PiGNN module, however this is based on SGT (in case of modeling local interactions) which has been introduced in AlphaDesign as well. Extracting global context has also been done in GCA in a similar manner. Could authors clarify the original contribution compared with AlphaDesign and GCA?\n  \n- Questions/concerns with respect to the experiments:\n  \n  - How similar are CATH 4.2 and 4.3? ESM-IF has only been evaluated on 4.3 and I am unsure how meaningful it is to include those results in Table 1 as a basis of comparison with the proposed method given they are evaluated on different datasets.\n    \n  - Table 4 shows that when the number of encoders increases from 6 to 8, test time increases by 4 seconds and recovery increases by 0.26%. However, when the number of encoders is increased from 8 to 10 (PiFold), test time only increases by 2 seconds. Furthermore, recovery increases by 0.44% which is almost doubled compared with the increase from 6 to 8 encoders. Can authors explain what causes the significant difference from linear scaling in test time? How does test time scale with the number of encoders? Furthermore, I would expect either linear or saturating behavior of recovery with respect to number of encoders, but this is not what we see in Table 4. Could authors discuss the performance scaling with # of encoders? Why exactly 10 encoders have been used in the final architecture?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MM-YO0JVErz", "reviewer": "Reviewer_DZpe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- PiFold is a system for fixed backbone sequence design with a new PiGNN layer that combine node and edge information.\n- By using a parallel decoder, the model is able to make predictions many times faster\n- Using more features in the encoder, PiFold is able to still obtain state of the art results.", "summary_of_the_review": "The paper is technically strong, the writing could improve slightly, but overall I would recommend an accept.", "strength_and_weaknesses": "Strengths:\n- PiFold is compared to a suite of recent protein design architectures\n- The training and testing splits are solid and makes comparison to previous works easy.\n- The model shows great sequence recovery performance and speed improvements.\n- The paper is written in a concise manner and understanding it is easy\n\nWeaknesses:\n- The ablations are interesting but slightly lacking - the \"leave one out\" approach seems imply no one change makes a big difference, PiFold is still state of the art despite ablating so many different changes. Maybe a better ablation is to show which change is making the difference here, and giving it such a large gap between PiFold and other competitors.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-S0Dz9G5yl", "reviewer": "Reviewer_cTW6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the protein inverse folding problem, i.e., predicting the most probable amino acid sequence given protein structure. To this end, they propose to improve two aspects of this task, one is more informative structural features for better representing protein structures, and the other one is a one-shot sequence decoder, PiGNN, for a more efficient generation. Experiments show that the proposed approach, the PiFold, achieves faster and more accurate performance for inverse folding. \n\n", "summary_of_the_review": "Overall, I think this paper is mostly well presented, and the contributions can be considered a bit novel. But the empirical evaluations remain poor and not sufficiently convincing. As a  result, I lean to consider this as a boardline work. Acceptance would be possible upon author responses and more comprehensive and fair experimental evidence. ", "strength_and_weaknesses": "Overall, this paper is well-motivated and easy to follow. Extensive experiments are also presented.\n\nHowever, my major concerns come from the experiments and comparisons, which are performed less comprehensively and are not convincing. Please refer to the questions.\n\n**Questions:**\n1. The respective contributions of the proposed featurizer and PiGNN Layer remain unclear. How would it be if using the proposed features upon other strong models, e.g., protein mpnn? Besides, can PiGNN layers also help improves GVP GNN if placing them upon GVP features and encoders?\n2. Ablation study wrt autoregressivity (Tab 4) seems somewhat unfair, where PiFold uses 10 layers while all the AT competitors use much fewer layers (#Enc + #AT in total, the maximum number of layers are 6 < 10). What’s more, model 4 consisting of 4 Enc and 1 AT decoder performs very closely to PiFold (~-1%). For more convincing conclusions, authors should conduct experiments with AT-based models that must integrate the same number of layers as PiFold, and ablate the proportion btw Enc and AT. \n\nIt would be great if these results would be included during the author response period. \n\nBesides, I also have some other questions:\n\n1. How about the scale of parameters of the proposed model compared to the previous ones? Plus, what would it be if scaling up the size of the parameters? How about scaling up data size (e.g., using alphafold-predicted data like in ESM-IF)? \n2. In most NLP studies, non-autoregressive models are known to lag behind their autoregressive counterparts considerably. However in this study, the proposed non-autoregressive models, though the comparisons are not that fair as I just pointed out, seem to do a really good job and even beat the autoregressive ones. What are the reasons for such differences in these two areas?\n3. Could you further provide results on CATH 4.3 so as to compare with the SOTA method ESM-IF (and their variant with AF2 data augmentation)\n\n**Minors:**\n\n- Sec. 1, page 2, “Experiments show that Pi- Fold can achieve state-of-the-art recoveries on several real-world datasets, i.g., 51.66% on CATH 4.2, 58.72 on TS 50, and 60.42% on TS 500.”: “i.g.” → “e.g.”\n- Symbols of quotes are misused across the manuscript. For instance, see paragraph Recovery in page 7, and many others.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%