AlphaDesign: A graph protein design method and benchmark on AlphaFold DB
Authors: Zhangyang Gao
Cheng Tan
Stan Z. Li
Venue: Submitted to ICLR 2023 (2023)

Abstract: While AlphaFold has remarkably advanced protein folding, the inverse problem, protein design, by which protein sequences are predicted from the corresponding 3D structures, still faces significant challenges. First of all, there lacks a large-scale benchmark covering the vast protein space for evaluating methods and models; secondly, existing methods are still low in prediction accuracy and time-inefficient inference. This paper establishes a new benchmark based on AlphaFold DB, one of the world's largest protein structure databases. Moreover, we propose a new baseline method called AlphaDesign, which achieves 5\% higher recovery than previous methods and about 70 times inference speed-up in designing long protein sequences. We also reveal AlphaDesign's potential for practical protein design tasks, where the designed proteins achieve good structural compatibility with native structures. The open-source code will be released. 

Under review as a conference paper at ICLR 2022
AEDESIGN: A GRAPH PROTEIN DESIGN METHOD AND BENCHMARK ON ALPHAFOLD DB
Anonymous authors
Paper under double-blind review
ABSTRACT
While AlphaFold has remarkably advanced protein folding, the inverse problem, protein design, by which protein sequences are predicted from the corresponding 3D structures, still faces significant challenges. First of all, there lacks large-scale benchmark covering the vast protein space for evaluating methods and models; secondly, existing methods are still low in prediction accuracy and time-inefficient inference. This paper establishes a new benchmark based on AlphaFold DB, one of the world's largest protein structure databases. Moreover, we propose a new baseline method called AEDesign, which achieves 5% higher recovery than previous methods and about 70 times inference speed-up up in designing long protein sequences. We also reveal AEDesign's 's potential for practical protein design tasks, where the designed proteins achieve good structural compatibility with native structures. The open-source code will be released.
INTRODUCTION
As "life machines", proteins play vital roles in almost all cellular processes, such as transcription, translation, signaling, and cell cycle control. Understanding the relationship between protein struc- tures and their sequences brings significant scientific impacts and social benefits in many fields, such as bioenergy, medicine, and agriculture (Huo et al.
2011 Williams al. 2019). While AlphaFold2
has tentatively solved protein folding (Jumper et al. 2021 [u et al. 2022 Lin et al. 2022] Mirdita al., [022] Li al. 2022c) from ID sequences 3D structures, reverse problem, i.c., protein design raised by (Paoo] 1983) that aims to predict amino acid sequences from known 3D structures, has fewer breakthroughs
the ML community. The main reasons hindering the research progress include: (1) The lack of large-scale standardized benchmarks; (2) The difficulty in improving protein design accuracy: (3) Many methods are neither efficient nor open-source. Therefore, we aim to benchmark protein design and develop effective and efficient open-source method.
Previous benchmarks may suffer from biased testing and unfair comparisons. Since SPIN (Li et al. 2014) introduced the TS500 (and TS50) consisting of 500 (and 50) native structures, it has served as a standard test set for evaluating diffcrent methods (O OO'Conelll et al. 2018 Wang et al. 2018
Chen et al. 2019 Jing et al. 2020 Zhang al. 2020a Qi & Zhang 2020 Strokach et al. 2020).
However, such few proteins do not cover the vast protein space and are more likely to lead to biased tests. Besides, there
canonical training and validation sets, which means that different methods may use various training sets. If the training data is inconsistent, how can we determine that the performance gain comes from different methods rather than biases of the data distribution? Especially when the test set is small, adding training samples that match the test set distribution could cause dramatic performance fluctuations. Considering these issues, we suggest establishing a large-scale standardized benchmark for fair and comprehensive comparisons.
Extracting expressive residue representations is key challenge for accurate protein design, where both sequential and structural properties must be considered. For general 3D points, structural features should be rotationally and translationally invariant in the classification task. Regarding proteins, we should consider amino acids' stable structure, number, and order. Previous studies (O' Connell al. 2018: Wang et al. 2018 Ingraahm et al. 2019 Jing et al. (020) may have overlooked some important protein features and data dependencies, i.c., bond angles; thus, few of them exceeds 50% recovery except DenseCPD (Qi & Zhang 2020). How can protein features and neural models be designed to learn expressive residue representations?
Improving the model efficiency is necessary for rapid iteration of research and applications. Current advanced methods have severe speed defects due the sequential prediction paradigm. For example, GraphTrans (Ingraham et al.) 2019) and GVP (Jing et al., 2020) predict residues one by one during inference rather than in parallel, which means that it calls the model multiple times get the entire

---

Under review as a conference paper at ICLR 2022
protein sequence. Moreover, DenseCPD (Qi & Zhang 2020) takes 7 minutes to predict a 120-length protein on their server How can we improve the model efficiency while ensuring accuracy?
To address these problems, establish new protein design benchmark and develop graph model called AEDesign (Accurate and Efficient Protein Design) to achieve SOTA accuracy and efficiency. Firstly, we compare various graph models on consistent training, validation, and testing sets, where all these datasets come from the AlphaFold Protein Structure Database (Varadi et al.] (021). In contrast previous studies (Ingraham et al. 2019; Jing et al. 2020) that use limited-length proteins, we extend the experimental setups to the case of arbitrary protein length. Secondly, we improve the model accuracy by introducing protein angles as new features and introducing simplified graph transformer encoder (SGT). Thirdly, we improve model efficiency by proposing confidence-awe aware protein decoder (CPD) to replace the auto-regressive decoder. Experiments show that AFDesign significantly outperforms previous methods in accuracy (+5%) and efficiency (70+ times faster than before). We also reveal AEDesign's potential for practical protein design tasks, where the designed proteins achieve good structural compatibility with native structures.
RELATED WORK
We focus on structure- based protein design (Gao ct al. 2020 Pearce & Zhang 2021 ; Wu et al. 2021 I
Ovchinnikov & Huang 2021 Ding et al. [022]] Strokach & Kim 2022 Li & Koehl| [204]] Greener
ct al.
2018
et al. 2022 | Karimi et al. 2020 Cao et al. [202] Liu et al. [002]] McPartion et al.
2022 Huang et al. 2022; Dumortier et al. 2022 Li et al. 2022a Maguire et al. [202 Anishchenko
et al.
2021 Li et al.) (022b), and the approaches can be categorized into MLP-based, CNN-based,
and GNN-based ones. Some terms need be explained in advance: we refer amino acids as residues, and accuracy indicates the degree of prediction of the residues, i.c., recovery.
Problem definition The grructure-basd protein design aims to find the amino acids sequence S = {(s: S i S n that folds into known 3D structure X
Table 1: Statistics of structure- based protein design methods.
Neraen is the number of training samples. TS500 and TS500 are the test sets containing 500 and 50 proteins, respectively. All results are copied from their manuscripts or related papers.
(z; E R3 1 < i S n), where n is the num- ber of residues and the natural proteins are composed by 20 types of amino acids, i.e.,
Method
Nevain
TS500
TS50
Code
1 S 8 S 20, and 8; E N+ Formally, that is to learn function Fe X + S. Be- cause homologous proteins always share similar structures (Pearson & Sierk] 2005), the problem itself is underdetermined, i.e.,
CNN MLP
SPIN
1,532
30.30%
30.30%
SPIN2
1,532
36.60%
33.60%
Wang's 's model
10,173
36.14%
33.00%
SPROF
7,134
40.25%
39.16% 19.16%
Py ToTcch
ProDCoNN
17.044
42,20%
40.69%
RO
DenseCPD
<10,727
55.53%
50.71%
no
GraphTrans
18,024
Py Torch
the valid amino acid sequence may not be unique. In addition, the need to consider both ID sequential and 3D structural infor- mation further increases the difficulty of algorithm design.
GVP
18,024
44.90%
Torch
GNN
GCA
18,024
43.00%
Py Torch
AEDesign
18,024
49.23%
48.36%
Py Torch
ESM-IF
> >I
Py Torch
ProteinMPNN
18,024
Py Torch
MLP-based models These methods use multi-layer perceptron (MLP) to predict the type of each residue. The MLP outputs the probability of 20 amino acids for each residue, and the input feature construction is the main difference between various methods. SPIN (Li al. 2014 integrates torsion angles (ó and e fragment-derived sequence profiles, and structure-derived energy profiles to predict protein sequences. SPIN2 (O Connell al. 2018) adds backbone angles (9 and T). local contact number, and neighborhood distance to improve the accuracy from 30% to 34%. (Wang et al., 2018) uses backbone dihedrals (ø, 1/2 and w), the solvent accessible surface area of backbone atoms (Ca, C, and O), secondary structure types (helix, sheet, loop). Ca- C. distance and unit direction vectors of C. - Ca. C₂ N and C₂ - C, which achieves 33.0% accuracy on 50 test proteins. The MLP methods have a high inference speed, but their accuracy is relatively low due to the partial consideration of structural information. These methods require complex feature engineering using multiple databases and computational tools, limiting their widespread usage.
CNN methods extract protein features directly from the 3D structure (Torgg & Altman 2017 Boomsma & Frellsen| 2017 Weiler et al.. 2018) Zhang et al., [020a] Huang et al. 2017 Chen et 2019), which can be further classified as 2D CNN-based and 3D CNN-based.
CNN-based models
http://protein.. org. www...n//eeee wwt..//eeeddddd..eeemee/
2

---

Under review as a conference paper at ICLR 2022
The 2D CNN- SPROF (Chen et al., 2019) extracts structural features from the distance matrix and improves the accuracy to 39.8%. In contrast, 3D CNN-based methods extract residue features from the atom distribution in three-dimensional grid box. For each residue, the atomic density distribution is computed after being translated and rotated to a standard position so that the model learn translation and rotation invariant features. ProDCoNN (Zhang et al.. (220a) designs nine-layer 3D CNN to predict the corresponding residues at each position, which uses multi-scale convolution kernels and achieves 40.69% recovery on TS50. DenseCPD (Qi & Zhang 2020) further uses the DensetNet architecture (Huang et al., 2017) to boost the accuracy to 50.7 %. Although 3DCNN-based models improve accuracy, their inference is slow, probably because they require separate pre-processing and prediction for each residue.
Graph methods represent the 3D structure as a k-NN graph, then use graph neural networks (Defferrard al., 2016, Kipf & Welling 2016 Veličković et al. 2017 Zhou et al. 2020 Zhang et al. 20000 Gao et al. 2022 et al. 2021a) to learn residue representations considering structural constraints. The protein graph encodes the residue information in node vectors and constructs edges and edge features between neighboring residues. GraphTrans (Ingraham et al. 2019) combines graph encoder and autoregressive decoder to generate protein sequences. GVP (Jing et al. 2020) increases the accuracy to 44.9% by proposing the geometric vector perceptron, which learns both scalar and vector features in an equivariant and invariant manner concerning rotations and reflections. GCA (Tan et al. 2022) further improves recovery to 47.02% by introducing global graph attention. Another related work ProteinSolver (Strokach et al., 2020), but was mainly developed for scenarios where partial sequences are known and do not report results on standard datasets. In parallel with our work, ProteinMPNN (Dauparas et al., 2022) and ESM-IF (Hsu et al. 2022) achieve dramatic improvements, while they do not provide the training code and will be benchmarked in the future.
Graph-based models
3 METHODS
3.1
OVERVIEW
We present the overall framework of AEDesign in Fig. I We suggest using AEDesign as a future baseline model because it is more accurate, straightforward, and efficient than previous methods. The methodological innovations include:
Expressive features: We add new proteins angles (a, 8, 7) steadily improve the model accuracy.
Simplified graph encoder: We use the simplified graph transformer (SGT) to extract more expressive representations.
We propose constraint-awarc protein decoder (CPD) to speed up inference by replacing the autoregressive generator.
Fast sequence decoder:
3.2
GRAPH FEATURE AND ENCODER
The protein structure can be viewed as a particular 3D point cloud in which the order of residues is known. For ordinary 3D points, there are ways get rotation and translation invariant features: Using particular network architectures (Fuchs et al.) 2020) Satorras et al.] 2021: Jing et al. 2020 Shuaibi et al.) 2021) that take 3D points input using handicraft invariant features. For proteins, general 3D point cloud approaches cannot consider their particularity, including the regular structure and order of amino acids. Therefore, we prefer learning from the hand-designed, invariant, and protein-specific features. How to create invariant features and learn expressive representations from them is subject to further research.
Graph We represent the protein as k-NN graph derived from residues to consider the 3D dependencies, where the default matrix A E {0, 1)m.", node features X E Ig and edge features E E gm.223, Note that n and m are the numbers of nodes and edges, and we create these features by residues' stable structure, order, and coordinates.
30. The protein graph G(A, X, E) consists of the adjacency
Node features As shown in Fig. 2 we consider two kinds of angles, i.c., the angles a;, B,, 7/ formed by adjacent edges and the dihedral angles Φi, 1i, wi formed by adjacent surfaces. where a,, 8,, 74 are new features we introduced. For better understanding dihedral angles, it is worth stating that 6;-1 is the angle between plane C N₂ Ca.-, and N, Cay-1 C,-1, whose intersection N; Cou-r- Finally, there are 12 node features derived from (sin, cos} } x fa,, Bi, Yi Or. the w
3

---

Under review as a conference paper at ICLR 2022
Data&Results
Structures
Fealures
Model
Sequences
Train
True
Alphafold
AQYEDOKaY
DB
Causs mask
Prdd cache
Quality
Distence Drection
TraphTrans GVP) Encoder
GraphTrans GVP) Decander
GraphTrand
GVP) Decrzder
Clstening
Baselines
AEDesign
Proccessed
Train&Test
AOYEDOKON
DB
Simplited Gruph TupllTiaans
Confidencoe sware Protein Decoder
Benchmark
Encoder
Dlection
mocel
XXX
Figure 1: Overview of AEDesign. Compared with GraphTrans, StructGNN (Igraham et al. 2019 and GVP Jing et 2020), add new protein features, simplify the graph transformer, and propose a confidence-aware protein decoder improve accuracy and efficiency.
new features,
Angle N.
Di--eerrral N.-Co,,
Node
Angle Cr.
th Dihe-ial angle C,-Ca,
Angle for Co.-Cr-Ne41
aig Dihedral C.N
Q
Qit
mall Viil
Edge
Ry Q2 Ty r(| sdll). My Q
2j 2; a0 zil
residue
n, 6 Local coordinate system
++1
Ca
Figure 2: Angles of the protein backbone.
Edge features For edge j +i,, i, we use relative rotation Rys distance rÿú, and relative direction uja
as edge features, seeing Fig. 2 At the i-th residue's alpha carbon, we establish a local coordinate
system Q2 (b,, n., b, n. The relative rotation between Q; and Q2 is Ry; Q Q2 can be represented by an quaternion q( (Rz.). We use radial basis r(.) to encode the distance between C.. 2, 23-0; and Cay. The relative direction of Ca, respective to Ca, is calculated by Ujz Q. ||=, ||=;- In summary, there are 23 edge features: (quaternion) 16 (radial basis) (relative direction).
Simplified graph transformer Denote h and e/ as the output feature vectors node i and edge j -> in layer 7. We use MLP to project input node and edge features into d-dimensional space, thus Λς Rd and e, Rd When considering the attention mechanisms centered in node i, the attention weight az at the + layer is calculated by:
MLP((())
h
K MLPs(h,lles)
Cj
A
Simplified attention
Wiss = MLP (h (Ie).II)
Figure 3: Simplified graph transformer.
exp M031
(1)
0gji ΣNEN; exp Wis
where N₁ the neighborhood system of node and means the concatenation operation. Here, we simplify GraphTrans (Ingraham al. 2019) by using single MLP to learn multi-headed attention weights instead of using separate MI.Ps leam Q and K, seeing Fig. 3 The updated h,+1 is:
4

---

Under review as a conference paper at ICLR 2022
By M
h EjEN, ajs
By stacking multiple simplified graph transformer (SGT) layers, we can obtaín expressive protein representations considering 3D structural constraints by message passing.
3.3 SEQUENCE DECODER
To generate more accurate protein sequences, previous researches (Ingraham et al. 2019 Jing et al.
2020 prefer the autoregressive mechanism. However, this technique also significantly slows down the inference process because the residuals must be predicted one by one. Can we parallelize the predictor while maintaining the accuracy?
Confidence Predictor
Loghts
Con!
Com1
Cane
CanviD
Lineer
n
Confidence
Conf-aware Protein Predictor
Convi0
ConvTo
Corv1D
Linear
n
Zyn
20
Logits
L.
Figure 4: CPD: The confidence-aween aware protein decoder. We use two ID CNN networks to learn confidence scores and make final predictions based the input graph node features.
Context-aware Previously, we have considered the 3D constraints through graph networks but ignored ID inductive bias. As shown Fig. 4 the input features are Zynn {=1, ... F.v } where is the feature vector of node i extracted by the encoder. In the generation phase, we use ID CNNs to capture the local sequential dependencies based on the 3D context-aware graph node features, where the convolution kernel can be viewed as the sliding window.
Confidence-aware Given the 3D structure X' {zx 1 < S N and protein sequence S= [s, i N), the vanilla autoregressive prediction indicates p(SIX) = II, ((s sez). where residues must be predicted one-by-one We replace autoregressive connections with parallelly estimated confidence score c, written
Con
c= f(a)
p(S(X) = [[,
where Conf(-) is the model containing graph encoder and CNN decoder (called "Confidence predictor" in Figure 4 and outputs logit scores a E Rm.20 f(-) is the function computing confidence score E N The confidence score contains knowledge captured by the previous prediction and serves as hint message to help the network correct previous predictions. Let M = ColumnMax(a)) E R B=,1 as the first and secondary largest logits
and ColumnSubMax((0
of a, the M Let R 1,20 and noting i and as the indexes of the
confidence score is defined largest and sub-largest values of a, then M ag m = aj. and f(a) = 1 where [.] indicates the
floor function. By extending a E IR 1.20 as m, Eq. 3 shows the vectorized version of f-).. We encode the confidence score as learnable embeddings C 1 concatenate them with graph features, and feed these features into another CNN decoder to get revised predictions. Note that all CNN decoders for estimating confidence and final predictions use the same CE loss:
-=
L
()) log(pi.))
L5
where P.,j is the predicted probability that residue i's type is j. yi is the true label and 1 ( (.) is indicator function.
5

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
We conduct systematic experiments to establish large-scale benchmark and evaluate the proposed AEDesign method. Specifically, we aim to answer:
Q1: What is the difference between the new bechmark and the old one?
Q2: Can AEDesign achieve SOTA accuracy and efficiency on the new benchmark?
Q3: What is important for achieving SOTA performance?
4.1 BENCHIMARK COMPARATION (Q1)
Metrie Following (Li et al. 2014 O' C'Cnnell et al. 2018 Wang et al.] 20188] Ingraham et al. 2019 Jing al. 2020), we use sequence recovery evaluate different protein design methods. Compared with other metrics, such as perplexity, recovery is more intuitive and clear, and its value is equal to the average accuracy of predicted amino acids in single protein sequence. By default, we report the median recovery score across the entire test set.
Previous benchmark In Table. [ we show the old benchmark collected from previous studies, including MLP (Li et al. [2144] O Connell et al.] 2018:] Wang al.) [0018), CNN (Chen al.) 2019
Zhang et al. 2020a Qi & Zhang Huang et al. 2017 and GNN (Ingraham et al. 2019 Jing
ct al. 2020 Strokach et al. (l.20)20)) Dauparas al. [002]] Hsu al. 2022) models. Most approaches
report results on the common test set TS50 (or TS500), consisting of 50 (or 500) native structures
(Li et al., 2014). We also provide the results of AEDesign under the same experimental protocols as Graph Trans and GVP. Although the TS50, TS500 test sets have contributed significantly to establishing benchmarks, they still do not cover a vast protein space, and do not reveal how the model data. Besides, there are canonical training and validation sets, which means that different methods may use various training sets.
performs
New dataset We use the AlphaFold Protein Structure Database (until 2021.2.1) Varadi et al. 2021) to benchmark graph-based protein design methods. As shown in Table. [6 (Appendix), there over 360,000 predicted structures by AlphaFold2 (Jumper et al. 2021) across 21 model-organism proteomes. This dataset has several advantages:
This dataset provides well-organized species-specific data for different species, which is helpful to develop specialized models for each species.
Species-specific:
More structures: This dataset provides more than 360,000 structures, while Protein Data Bank (PDB) (Burley et al. 2021) holds just over 180,000 structures for over 55,000 distinct proteins.
High quality: The median predictive score of AlphaFold2 reaches 92.4%, comparable ex- perimental techniques (Callaway 2020). Nicholas (Fowler & Williamson 2022) found that AlphaFold tends be more accurate than NMR ensembles. In 2020, the CASP14 benchmark recognized AlphaFold2 as a solution to the protein-folding problem (Pereira et al. (021).
Missing value: There are no missing values protein structures provided by AlphaFold DB.
Widespread usage: AlphaFold DB has been used in many frontier works ( (Varadi et al. 2022
Morreale et al. 2022 Luyten et al. | 2022 Alderson et al. 2002 /hhng et al.) [ (Fowler &
Williamson, 2022 Shaban et al. 2022 Brems et al.) 2022 Hsu et al. (2022), and we believe that
investigating AlphaFold DB could yield more discoveries for protein design.
Dataset Preprocessing should be noted that the AlphaFold2 DB data itself may have model bias. Similar to ESM-IF (Hsu et al. (022), we address data quality and partitioning issues through data pre-processing for each species-specife subset. As suggested by (Baek Kepp, 2022; Callaway 2020), the MAE between the predicted and experimentally generated structures does depend on PLDDT. Thus, we filter low-quality structures whose confidence score (pLDDT) is less than 70. To prevent potential information leakage, for each species-specifer subset, cluster protein sequences their sequence similarities (Qi & Zhang 22020; Steeinegeer & Söding (017) are higher than 30% (Qi & Zhang 2020) and split the dataset by these clusters. As result, the proteins belonging to the same cluster must be in one of the training, validation, and test sets. By default, we keep the
https://alphaffll

---

Under review as a conference paper at ICLR 2022
ratio of the training set and test set at about 9:1 and choose 100 proteins belonging to randomly selected cluster for validation. If the randomly selected cluster has less than 100 proteins, then all of its proteins are used as the validation set.
4.2 AEDESIGN BENCHMARK (Q2)
Overall settings We extend the experimental setups to arbitrary length and species-specific, while most previous studies (Ingraham et al.) 2019; Jing et al. 2020 Tan et al. 2022) do not explore such vast protein space. Arbitrary length means that the protein length may be arbitrary to generalize the model to broader situations; otherwise, the protein must be between 30 and 500 in length. Species-specifte indicates that develop specific model for each organism's proteome to learn domain-specific knowledge. In summary, there are two settings, i.e., species-specific dataset with limited length (SL) and species-specific dataset with arbitrary length (SA). Denote the total amount of structures as Nat; and the i-th species has N₂ structures, we have Nau = Di=1 N, As shown in Table. [6] if the length is limited, Naur = 254, 636; otherwise, Nau = 365, 198. For each species- specific subset, we develop 21 models based on datasets with N1. N2 ... _N21 structures. We report the median recovery scores across the test set. All baseline results were obtained by running their official code with the same dataset. In parallel with our work, ProteinMPNN (Dauparas et al., 2022) also proposed, but they are not provided with the full training code and will be benchmarked in the future. Due to space constraints, we introduce abbreviations such as GTrans for GraphTrans and SGNN for Struct GNN.
and ESM-IF (Hsu et al. (2022)
Hyper-pant parameeesssssess AEDesign's 's encoder contains ten layers of SGT, and the decoder contains three layers CNN, where the hidden dimension is 128. We use Adam optimizer and OneCycleLR scheduler to train all models up to 100 epochs with early stop patience 20 and learning rate 0.001. In the SL setting, we set the batch size as 16 for GraphTrans, StructGNN, GCA, and AEDesign, and the max node number 2000 for GVP, which indicates the maximum number of residuals per batch. In the SA setting, we change GVP's max node parameter to 3000 to make it applicable to all data. GraphTrans, StructGNN, and GCA take more GPU memories because they must pad data according to the longest chain in each batch, and we adjust the batch size to 8 to avoid memory overflow.
Table 2: SL benchmark. The length of the protein must be between 30 and 500. We highlight the best (or next best) results in bold (or underline).
Table 3: SA benehmark. There is no constraint on the protein length. We highlight the best (or next best) results in bold (or underline).
GTrans
SGNN
GVP
GCA
Our(SL)
GTrans
SGNN
GVP
GCA
Our(SA)
Gain
DANRE
44.93
55.82
62.16
63.21
70.70
7.47
DANRE
48.82
57.24
66.18
65.14
69.65
3.47
CANAL
47.81
50.62
52.55
58.43
61.36
CANAL
51.14
55.17
61.82
60.88
68.45
6.63
MOUSE
49.53
54.04
59.23
62.00
68.34
6.34
MOUSE
47.79
57.40
64,23
64.50
74.07
9.57
ECOLI
46.95
52.62
59.79
60.71
ECOLI
50.60
54.22
59.11
60.71
66.22
DROME
42.39
$2.94
58.92
60.52
66.67
6.15
DROME
50.56
55.91
63.13
61.67
71.55
8.42
METIA
42.68
52.09
51.59
58.54
62.24
3.7
METIA
$1.03
$2.47
53.13
PLAF7
46.32
50.00
49.52
58.74
61.16
2.42
PLAF7
48.11
53.92
57.26
59.75
66.01
6.26
MYCTU
54.02
56.32
58.50
65.24
68.68
3.44
MYCTU
48.25
58.04
61.93
64.93
70.57
5.64
CABEL
47.44
54,89
62.32
63.00
70.23
7.23
CAFEL
46.82
54.05
64.48
64.03
73.17
8.69
DICDI
42.15
53.14
58.41
61.54
67.21
5.67
DICDI
55.58
56.82
64.17
62.91
71.66
7.49
TRYCC
51.13
56.18
62.35
61.33
71.08
8.73
TRYCC
48.56
59.57
64.52
66.06
75.18
9.12
YEAST
36.18
49.23
52.14
58.30
62.50
4,2
YEAST
50.00
54,00
60.41
58.45
66.71
6.3
SCHPO
44.56
49.34
$1,90
58.21
60.64
2.43
SCHPO
47.62
53.91
60.26
60.21
66.64
6.38
RAT
48.23
60.41
67.01
65.87
75.50
8.49
RAT
47.97
$7.83
65.99
63.89
73.54
7.55
HUMAN
51.15
54.83
61.19
63.10
HUMAN
55.28
$6.29
67.43
ARATH
45.78
55.56
63.08
63.83
71.15
7.32
ARATH
48.39
57.59
66.55
64.66
73.84
7.29
MAIZE
47.06
55.42
64.86
65.00
74.12
MAIZE
56.76
58.57
67.78
65.83
76.00
8.22
LEHIN
49.48
$1.33
55.33
60.22
64.71
4.49
LEHNN
54.58
56.85
63.92
62.81
63.54
.0.38
STAAS
45.57
48.31
47.52
57.14
60.00
STAAS
49.36
$1.43
53.85
58.46
63.85
5.39
SOYBN
47.73
56.52
64.84
64.10 10
73.96
9.12
SOYBN
49.94
58.44
66.25
69.23
76.51
7.28
ORYSI
46.29
54.62
63.64
63.64
72.98
9.34
ORYSJ
48.04
58.41
66.78
64.00
75.15
8.37
Averago
46.34
53.53
58.20
61.31
67.32
5.81
Average
50.24
56.10
62.70
62.91
69.97
7.06
SL&SA results We present the SL and SA benchmark in Table. and Table. 3 respectively. The order of model accuracy from high to low is AEDesign > GCA > GVP > SGNN > GTrans. Under subsets and exceeds previous
SL setting, AEDesign achieves the best recovery on all species-specinss methods by 5.81% 18% average. Under SA setting, AEDesign is still the most accurate model, and the increased training data further improves the model, i.e., the average accuracy of AEDesign (SA) is 69.97% while that of AEDesign (SI.) is 62.91 %. For species-specific subsets, the larger the data
7

---

Under review as a conference paper at ICLR 2022
volume, the better model performance; refer to T For example, the recovery METI is 58.54% when the number structures is 1605, which will increase to 73.96% on SOYBN when the data volume increases to 41,048.
4.3 EFFICIENCY COMPARATION (Q2)
A good algorithm should have excellent computational efficiency in addition to high accuracy. We compare the inference time cost of different approaches, especially when designing long proteins commonly found AF2DB.
Setting We evaluate various models' inference time costs under different scenarios, considering 100 proteins with short (L<500), medium (500<L< 1000), and long (1000<LL) lengths As to long sequence design, further investigate the time costs of encoder, decoder, and encoder+decoder. All experiments are conducted on an NVIDIA-V100.
2016.23 2026.53
377.53 378.75
345.98 346.63
435.43 452.71
100
17.28
10.3
4.57
5.1
1.22
0.65
0.53 AEDesign
GraphTrans
StructGNN
GCA
GVP
0.1
Encoder Decoder Enc
Figure 5: Inference time cost when designing long proteins. We report the total inference time of different methods on 100 long proteins, which are longer than 1000. The time costs of the encoder, decoder and encoder4decnder are reported.
Table 4: Inference time of different methods. #Number the number of proteins for time evaluation.
#Avg is is average protein length.
Short 100
Medium
Long 100
#Number
100
#Avg L
286
724
1632
GraphTrans
47.58
262.87
378.76
StructGNN
43.63
213.87
346.63
GCA
44.99
230.64
452.71
GVP
208.85
969.51
2026.53
AEDesign 1.61
2.77
5.09
Results We show the inference time costs for designing proteins of different lengths in Table 4 When designing short proteins (L 500), AEDesign is 25+ times faster than baselines; and the dominance extends to about 70 times for designing long proteins (L. 2 500). In Fig. 5] we observe that the time costs of baselines mainly come from the autoregressive decoder, and the proposed CPD module significantly speeds up the decoding process.
4.4 ABLATION STUDY (Q3)
While AEDesign has shown remarkable performance, we are more interested in where the improve- ments come from. As mentioned before, add new protein features, simplify the graph transformer, and propose confidence-awee aware protein decoder. Whether these modifications could improve model performance?
8

---

Under review as a conference paper at ICLR 2022
ADesign
w/o SGT
w/o CPD
w/o new feat
Setting We conduct ablation studies un-
DANRE
70.70
68.27
71.04
62.18
der the SL setting. Specifically, we may re-
CANAL
61.36
59.00
62.53
53.45
place the simplified attention module with
MOUSE
68.34
65.88
68.70
60.00
the original Graph Trans (w/o SGT), re-
ECOLI
60.71
58.27
61.05
52.97
DROME
64.22
67.11
54.45
place the CPD module with the autoregres- sive decoder of GraphTrans (w/ CPD), or remove the newly introduced angle features (w/o new feat). All experimental settings keep the same as previous SL settings.
METJA
62.24
59.23
63.10
50.61
PLAF7
61.16
57.64
MYCTU
65.21
69.57
56.02
CABEL
70.23
70.69
DICDI
63.79
67.48
54.94
TRYCC
71.08
71.70
59.58
YEAST
62.50
63.23
49.32
Results and analysis The ablation re- sults are shown in Table. 5 We conclude that: (1) The SGT module and the new features can improve the recovery rate by 2.51% and 10.85%, respectively. Most of performance improvement comes from the angular features, consistent with the recent ProteinMPNN, while they focus distance features.
SCHPO
60.64
58.14
62.05
48.67
RAT
75.50
72.91
75.54
64.50
HUMAN
69.69
67.28
70.21
58.55
ARATH
71.15
69.30
71.82
61.62
MAIZE
74.12
72.17
74.42
64.26
LEIIN
64.71
61.98
65.29
$2.52
STAA8
60.00
$7.87
60.33
47.31
SOYBN
73.96
72.06
73.58
63.89
ORYSJ
72.98
71.26
73.23
64.11
(2) If place the CPD module with the autoregres- sive decoder, the recovery rate will improve by 0.55%. However, the recovery improve- ment brought by the autoregressive decoder is marginal compared the SGT module and the new features. Therefore, we conclude CPD module dramatically improves the evaluation speed while maintaining good recovery. (3) we remove the introduced angular features, AEDesign as accurate as GCA and GVP, but the improvement in efficiency is still significant.
Average
67.32
64.81
67.87
56.47
Gain
-2.51
0.55
-10.85
Table Ablation study under SI. setting.
4.5 VISUAL EXAMPLES
We show the potential of AEDesign in real applications, i.e., designing all-alpha, all-beta, and mixed native proteins. We ensemble multiple models by selecting the sequence with the lowest perplexity as the final solution. We use AlphaFold2 to predict structures designed sequences and compare them with the reference ones. Visual examples provided in Figure.
All-alpha structure (3tld) (Recovery=50.00 RMSD=1.18)
All-beta structure (2giy) (Recovery=53.00 RMSD=0.54)
Mixed structure (1mgr) (Recovery=46.33) RMSD=0.47)
Figure 6: Visual examples. For native structures, we color Helix, Sheet, and Loop with cyan, magenta, and orange, respectively. Green structures are protein chains designed by our algorithm. We provide the recovery score and structural RMSD relative to the ground truth proteins.
5 CONCLUSION
This paper establishes new benchmark and proposes a new method (AEDesign) for Al-guided protein design. By introducing new protein features, simplifying the graph transformer, and proposing a confidence-ave aware protein decoder, AEDesign achieves state-of-the-ar accuracy and efficiency. We hope this work will help standardize comparisons and provide inspiration for subsequent research.

---

Under review as a conference paper at ICLR 2022
REFERENCES
T Reid Alderson, Iva Pritišanac, Alan M Moses, and Julie D Forman-Kay. Systematic identification of conditionally folded intrinsically disordered regions by alphafold2. bioRxiv, 2022.
Namrata Anand, Raphael Eguchi, Irimpan Mathews, Carla P Perez, Alexander Derry, Russ B Altman, and Po-Ssu Huang. Protein sequence design with a learned potential. Nature communications, 13 (1):1-11, 2022.
Ivan Anishchenko, Samuel Pellock, Tamuka M Chidyausiku, Theresa A Ramelot, Sergey Ovehin- nikov, Jingzhou Hao, Khushboo Bafna, Christoffer Norn, Alex Kang. Asim K Bera, al. De novo protein design by deep network hallucination. Nature, 600(7889):547-552 2021.
Kristoffer Torbjoern Baek and Kasper Planeta Kepp. Assessment of alphafold2 residue conformations for human proteins. bioRxiv, 2022.
Wouter Boomsma and Jes Frellsen. Spherical convolutions and their application in molecular mod- elling. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL htRpsttts////rpceiiiing ieurri http://pppp..c/ppppp///////// (LSSaaafffffeaaDDDo0003e44446663
nwwrips
Maarten A Brems, Robert Runkel, Todd O Yeates, and Peter Vimaau. Alphafold predicts the most complex protein knot and composite protein knots. Protein Science 31(8):4338, 2022.
Stephen K Burley, Charmi Bhikadiya, Chunxiao Bi, Sebastian Bittrich, Li Chen, Gregg V Crichlow, Cole H Christie, Kenneth Dalenberg, Luigi Di Costanzo, Jose Duarte, et al. Resh protein data bank: powerful new tools for exploring 3d structures of biological macromolecules for basic and applied research and education in fundamental biology, biomedicine, biotechnology. bioengineering and energy sciences. Nucleic acids research, 49(D1):D477-D451 2021.
Ewen Callaway. 'it will change everything': Deepmind's ai makes gigantic leap in solving protein structures. Nature, pp. 203-204, 2020.
Yue Cao, Payel Das, Vijil Chenthamarakshaan Pin- Yu Chen, Igor Melnyk, and Yang Shen. Fold2seq: A joint sequence (ld)-fold (3d) embedding- generative model for protein design. In International Conference on Machine Learning, pp. 1261 1171.. PMLR, 2021.
Sheng Chen, Zhe Sun, Lihua Lin, Zifeng Liu, Xun Liu, Yutian Chong, Yutong Lu, Huiying Zhao, and Yuedong Yang. To improve protein sequence profile prediction through image captioning on pairwise residue distance map. Journal of chemical information and modeling, 60(1):391-399 2019.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning based protein sequence design using proteinmpnn. bioRxiv, 2022.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 19-3844-3852. 2016.
Wenze Ding, Kenta Nakai, and Haipeng Gong. Protein design via deep learning. Briefings in bioinformaties. 23(3):bbac10) 102, 2022.
Baldwin Dumortier, Antoine Liutkus, Clément Carré, and Gabriel Krouk. Petribert: Augmenting bert with tridimensional encoding for inverse protein folding and design. bioRxiv, 2022.
Nicholas J Fowler and Mike P Williamson. The accuracy of protein structures in solution determined by alphafold and nmr. Structure, 2022.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3) transforrmers: 3d iio--translation equivariant attention networks. arXiv preprint ar0iv:2006. 10503, 2020.
10

---

Under review as a conference paper at ICLR 2022
Wenhao Gao, Sai Pooja Mahajan, Jeremias Sulam, and Jeffrey J Gray. Deep learning in protein structural modeling and design. Patterns, 1(9):100142, 2020.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Semiretro: Semi-template framework boosts deep retrosynthesis prediction. arXiv preprint arXiv:220.008205. 2022.
Joe Greener, Lewis Moffat, and David T Jones. Design of metalloproteins and novel protein folds using variational autoencoders. Scientific reports, 8(1):1-12, 2018. 2018.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022.
Bin Huang, Tingwe Fan, Kaiyue Wang, Haicang Zhang, Chungong Yu, Shuyu Nie, Yangshuo Qi, Accurate and efficient protein sequence design
Wei-Mou Zheng, Jian Han, Zheng Fan, through learning concise local environment of residues. bioRxiv. 2022.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Yi-Xin Huo, Kwang Myung Cho, Jimmy G Lafontaine Rivera, Emma Monte, Claire R Shen, Yajun Yan, and James C Liao. Conversion of proteins into biofuels by engineering nitrogen flux. Nature biotechnology, 2944):346-35,, 2011.
John Ingraham. Vikas K Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. 2019.
Bowen Jing, Stephan Eismann. Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. arXiv preprint arXiv:200900141, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583
2021.
Mostafa Karimi, Shaowen Zhu, Yue Cao, and Yang Shen. De novo protein design for novel folds using guided conditional wasserstein generative adversarial networks. Journal of chemical information and modeling, 60(12)55667-6611 2020.
Thomas Kipf and Max Welling. Semi cumi--uperriied classification with graph convolutional networks. arXiv preprint arXiv: 1 6099002977 2016.
Alex J Li. Mindren Lu, Israel Tilahun Desta, Vikram Sundar, Gevorg Grigoryan, and Amy Keating. Neural network-derived potts models for structure-basee protein design using backbone atomic coordinates and tertiary motifs. bioRxiv, 2022a.
Alex J Li, Vikram Sundar, Gevorg Grigoryan, and Amy E Keating. Terminator: A neural framework for structure-basee protein design using tertiary repeating motifs. arXiv preprint arXiv:2204.1300 2022b.
Jianshu Li, Jian Zhao, Congyan Lang, Yidong Li. Yunchao Wei, Guodong Guo. Terence Sim, Shuicheng Yan, and Jiashi Feng. Multi-human parsing with a graph-based generative adversar- ial model. ACM Transactions Multimedia Computing, Communications, and Applications (TOMM), 17(1): 2021a.
Jie Li and Patrice Koehl. 3d representations of amino acids applications protein sequence comparison and classification. Computational and structural biotechnology journal, 1 11(88)47-58, 2014.
Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature aug- mentation for domain generalization. In Proceedings the JEEE/CVF International Conference Computer Vision, pp. 8886-8895, 202lb.
11

---

Under review as a conference paper at ICLR 2022
Zhixiu Li, Yuedong Yang, Eshel Faraggi, Jian Zhan, and Yaoqi Zhou. Direct prediction of profiles of sequences compatible with a protein structure by neural networks with fragment- based local and energy-based nonlocal profiles. Proteins: Structure, Function, and Bioinformatics, 82(10): 2565-2573, 2014.
Ziyao Li, Xuyang Liu, Weijie Chen, Fan Shen, Hangrui Bi, Guolin Ke. and Linfeng Zhang. Uni-fold: An open-source platform for developing protein folding models beyond alphafold. bioRxiv, 2022c.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Yufeng Liu, Lu Zhang, Weilun Wang, Min Zhu, Chenchen Wang, Fudong Li, Jiahai Zhang, Houqiang Li, Quan Chen, and Haiyan Liu. Rotamer-fiee free protein sequence design based on deep learning and self-consistency. 2022.
Yvette Luyten, Deanna E Hausman, Juliana C Young, Lindsey Doyle, Kerilyn M Higashi, Natalia C Rodria--odri,uez, Abigail R Lambert, Corina S Arroyo, Kevin J Forsberg, Richard D Morgan, et al. Identification and characterization the wyl brxr protein and its gene as separable regulatory elements of a brex phage restriction system. Nucleic acids research, 50(9):517--5190, 2022.
Jack B Maguire, Daniele Grattarola, Vikram Khipple Mulligan, Eugene Klyshko, and Hans Melo. Xenet: Using a new graph convolution accelerate the timeline for protein design on quantum computers. PLoS computational biology, 17(9); 1009037, 2021.
Matt McPartlon. Ben Lai, and Jinbo Xu. A deep equivvvarrt model for learning inverse protein folding. bioRxiv, 2022.
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki. Lim Heo, Sergey Ovchinnikov, and Martin Steinegger. Colabfold: making protein folding accessible all. Nature Methods, pp. 1-4, 2022.
Francesca E. Morreale, Stefan Kleine, Julia Leodolter, Sabryna Junker, David Hoi, Stepan Ovehin- nikov, Anastasia Okun, Juliane Kley, Robert Kurzbauer, Lukas Junk, et al. Bacprotacs mediate targeted protein degradation in bacteria. Cell, 2022.
James O'Connell, Zhixiu Li, Jack Hanson, Rhys Hefferman, James Lyons, Kuldip Paliwal, Abdollah Dehzangi, Yuedong Yang, and Yaoqi Zhou. Spin2: Predicting sequence profiles from protein structures using deep neural networks. Proteins: Structure, Function, and Bioinformatics, 86(6): 629-633, 2018
Sergey Ovchinnikov and Po-Ssu Huang. Structure- based protein design with deep learning. Current opinion chemical biology, 65:136-144, 2021.
Carl Pabo. Molecular technology: designing proteins and peptides. Nature, (01 1983.
Robin Pearce and Yang Zhang. Deep learning techniques have significantly impacted protein structure prediction and protein design. Current opinion in structural biology, 68: 68:194 194-207, 2021.
William R Pearson and Michael I. Sierk. The limits of protein sequence comparison? Current
opinion in structural biology, 15(3):254-260, 2005.
Joana Pereira, Adam J Simpkin, Marcus D Hartmann, Daniel J Rigden, Ronan M Keegan, and Andrei N Lupas. High-aceuracy protein structure prediction in casp14. Proteins: Structure. Function, and Bioinformatics. 19(12):1687-169, 2001.
Yifei Qi and John ZH Zhang. Denseepd: improving the accuracy of noural-networkk based computa- tional protein sequence design with densenet. Journal of chemical information and modeling, 60 12): 2020.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling, (n) equivariant graph neural networks. arXiv preprint ar arXiv:2102.098.4 2021.
12

---

Under review as a conference paper at ICLR 2022
Nadine M Shaban, Rui Yan, Ke Shi, Sofia N Moraes, Adam Z Cheng, Michael A Carpenter, Jason S McLellan, Zhiheng Yu, and Reuben S Harris. Cryo-em structure of the ebv ribonucleotide reductase borf2 and mechanism of apobec3b inhibition. Science advances, 8(17)xcabm2827, 2022.
Muhammed Shuaibi, Adeesh Kolluru, Abhishek Das, Aditya Grover, Anuroop Sriram, Zachary Ulissi, and C Lawrence Zitnick. Rotation invariant graph neural networks using spin convolutions. arXiv preprint arXiv:2100.09575, 2021.
Martin Steinegger and Johannes Söding, Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology.
2017.
Alexey Strokach and Philip M Kim. Deep generative modeling for protein design. Current opinion in structural biology, 72:226-236, 2022.
Alexey Strokach, David Becerra, Carles Corbi-Verge Verge, Albert Perez-Riba, and Philip M Kim. Fast and flexible protein design using deep graph neural networks. Cell Systems, 11(4):02--411 2020.
Cheng Tan, Zhangyang Gao, Jun Xia, and Stan Z Li. Generative de novo protein design with global context. aiviv preprint arXiv:2204. 10673, 2022.
Wen Torng and Russ B Altman. 3d deep convolutional neural networks for amino acid environment similarity analysis. BMC bioinformatics, 18(1):1-23, 2017.
Michel van Kempen, Stephamic Kim, Charlotte Tumescheit, Milot Mirdita, Johannes Söding, and Martin Steinegger. Foldseek: fast and accurate protein structure search. bioRxiv, 2022.
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: Massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic acids research, 2021.
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequene space with high-accuracy models. Nucleic acids research, 50(D1): 2022.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1710 2017.
Jingxue Wang, Huali Cao, John ZH Zhang, and Yifei Qi. Computational protein design with deep learning neural networks. Scientific reports, 8(1):1-9, 2018
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman. N. Cesa-Bianchi, and R. Garnet (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL https: httppss://rooccedddnneee www.nngg....uuureee newn cttppp.cc..peeeeeeeccppeeoppppop wtcc/paper/20 1022000/100 8/file/ [40eee10550000e0065553ccdlld
Stephen A Williams, Mika Kivimaki, Claudia I Langenberg, Aroon D Hingorani, JP Casas, Claude Bouchard, Christian Jonasson, Mark A Sarzynski, Martin J Shipley, Leigh Alexander, al. Plasma protein patterns as comprehensive indicators of health. Nature medicine, 25(12):1851-1853 2019,
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, 2022.
Zachary Wu, Kadina E Johnston, Frances H Arnold, and Kevin K Yang. Protein sequence design with deep generative models. Current opinion in chemical biology, 65:18-27, 2021.
Yuan Zhang, Yang Chen, Chenran Wang, Chun-Chao Lo. Xiuwen Liu, Wei Wu, and Jinfeng Zhang Prodconn: Protein design using a convolutional neural network. Proteins: Structure, Function, and Bioinformaties, (1(7):819-82) 2020a.
13

---

Under review as a conference paper at ICLR 2022
Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning graphs: A survey. IEEE Transactions Knowledge and Data Engineering, 2020b.
Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. arXiv preprint arXiv:2203.6613 2022.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang. Changcheng Li, and Maosong Sun. Graph neural networks: review of methods and applications. AI Open, 1:57-81, 2020.
14

---

Under review as a conference paper at ICLR 2022
A DATA STATISTICS
Protein length Table we count the protein each species dataset by length. AlphaFold DB contains some extra-long proteins, so it is necessary to improve the algorithm efficiency.
ID
Name
Structures
30
(30,500)
500,, 1000]
1000
DANRE
24,664
27
15,460
6,573
2,604
2
CANAL
5,974
3,656
1,874
444
MOUSE
21,615
55
13.767
5,668
2,125
4
ECOLI
4,363
63
3,719
526
55
DROME
13,458
28
8,509
3,563
1,358
6
METJA
1,773
1,605
144
24
PLAF7
5,187
2,832
1,313
1,041
8
MYCTU
3,988
4
3,365
536
83
CAEEL
19,694
15,073
3,592
977
10
DICDI
12,622
7,663
3,367
1,589
11
TRYCC
19,036
12,622
5,007
1,401
12
YEAST
6,040
24
3,789
1,715
512
13
SCHPO
5,128
3,385
1,386
352
14
RAT
21,272
12
13,884
5,370
2,006
15
HUMAN
23,391
49
12,399
5,653
5,290
16
ARATH
27,434
42
19,885
6,389
1,118
17
MAIZE
39,299
83
29,145
8,360
1,711
18
LEIIN
7,924
4,276
2,505
1,143
19
STAA8
2,888
22
2,567
267
32
20
SOYBN
55,799
17
41,048
12,353
2,381
21
ORYSJ
43,649
78
35,987
6,738
846
Sum
365,198
571
254,636
82,899
27,092
Table 6: AlphaFold DB: show the total number of proteins Nau, and the number of proteins whose length within (0, 30), (30, 500), (500, 1000) and (1000, +oo/. The statistics of all species-specific subsets are also presented.
15

---

Under review as a conference paper at ICLR 2022
B PREPROCESSING
Filter low quality data by PLDDT Similar to ESM-IF (Hsu et al. 2022 we address data quality and partitioning issues through data pre-processing for each species-specific subset. As suggested by (Baek & Keppl 2022 Callaway| 2020), the MAE between the predicted and experimentally generated structures does depend on pLDDT. Thus, filter low-quality structures whose confidence score (pLDDT) is less than 70.
Filter test data by sequence identity To prevent potential information leakage, for each species- specific subset, we cluster protein sequences if their sequence similarities (Qi & Zhang 2020 Steinegger Söding 2017 are higher than 30% (Qi & Zhang 2020) and split the dataset by these clusters. As result, the proteins belonging to the same cluster must be in one of the training, validation, and test sets. By default, we keep the ratio of training set test set around 9:1 and select 100 proteins from randomly selected cluster for validation. If the randomly selected cluster has less than 100 proteins, then all of its proteins are used as the validation set. In Table 7 we show the number of training data (#Train), validation data (# (#Valid), and test data (#Test) under the partition based on 30% sequence identity.
Filter test data TS-score In addition to the sequence identity clustering, further filter the test sets by structural similarity using Foldseek (van Kempen et al. 2022 to exclude any structures with TM-score larger than 0.5 from those in the training set. Thus, the training and test sets strictly different at both the sequence and structure levels. In Table we show the number of test data (#TMTest) after filtering by 30% sequence identity and 0.5 TM-score.
SL setting
SA setting
Name
#Train
#Valid
#Test
#TMTest
#Train
#Valid
#Test
#TMTest
DANRE
13914
100
1446
493
22197
100
2367
808
CANAL
3290
99
267
168
5377
99
498
240
MOUSE
12417
73
1277
412
19524
2063
862
ECOLI
3347
100
272
114
3926
100
337
129
DROME
7672
86
751
369
12112
100
1246
610
METJA
1444
80
81
42
1595
88
90
47
PLAF7
2561
87
184
105
4668
100
419
267
MYCTU
3028
100
237
98
3589
100
299
CAEEL
13565
95
522
17724
1874
10
DICDI
6896
100
667
346
11359
100
1163
586
11
TRYCC
11360
98
1164
739
17132
86
1818
964
12
YEAST
3410
100
279
169
5436
100
504
263
SCHPO
3046
100)
239
118
4616
413
200
RAT
12495
45
1344
56
19145
2028
15
HUMAN
11160
99
1140
452
21051
100
2240
955
16
ARATH
17897
99
1889
750
24690
94
2650
910
17
MAIZE
26232
98
2815
1220
35369
100
3830
1420
LEIIN
3848
100
328
201
7131
693
19
STAA8
2310
100
157
76
2599
100
189
20
SOYBN
36944
99
4005
1732
50226
92
5481
2152
21
ORYSJ
32388
97
3502
2047
39289
95
4265
2273
Sum
229224
1955
23457
10229
328755
1976
34467
14851
Table 7: Dataset splits. We show the number of training data (#Train), validation data (# (HVad,,) and test data (#Test) under the partition based on 30% sequence identity. We also exhibit the number of test data (#TMTest) after filtering by 30% sequence identity and 0.5 TM-score.
16

---

Under review as a conference paper at ICLR 2022
C RESULTS AFTER TM-SCORE FILTERING
TM-score filtered Results We evaluated all models on test sets filtered by both 30% sequence identity and 0.5 TM-score, and provide the TM-score filtered benchmarks on Table 8 and Table 9 Compared the results without using the T'-scorre filter, the relative performance gain of our model is slightly reduced. The average model performance on this TM score- score-based test set remains nearly the same as on the sequence identity-basee test set. We note that ESM-IF also finds that "the model performance overall remains the same the TM score-based test set test set." These facts show that more filters do not make the model predictions as difficult as we expected. This means that the model may not rely the so-called homologous information leakage for prediction, but actually learns the patterns of the data.
the CATH topology split
Table 8: TM-score filtered SL. benchmark. The length of the protcin must be between 30 and 500. The se- quence identity and TM-score between training and lesting proteins are less than 30% and 0.5, respectively.
Table 9: TM-score filtered SA benchmark. There is no constraint the protein length. The sequence identity and -core between training and testing proteins are less than 30% and 0.5, respectively.
GVP
SurlSL.)
GTrans
SGNN
GVP
GCA
OurlSA]
Gain
DANRE
42,86
53.57
60.17
61.96
69.18
7.22
51.56
52.52
59.20
62.22
CANAL
$1.59
54.84
62.02
61.67
69.26
7.24
MOUSE
48.55
53.74
58.60
62.61
68.65
6.04
MOUSE
47.06
56.99
63.64
64.77
74,06
9.29
ECOLI
39.32
ECOLI
50.41
54.02
59.43
61.23
66.11
4.88
DROME
43.20
53.98
60.00
62.07
68.91
6.84
DROME
50.97
56.46
63.55
62.93
72.88
9.33
METIA
PLAF7
48.00
50.49
$1.59
61.29
64.10
2.81
PLAFF
48.30
54.39
58.32
60.35
67.04
6.69
MYCTU
53.88
56.42
58.08
66.45
69.20
2.75
MYCTU
CABEL
48.42
56.05
63.65
66.67
74.24
7.57
CAEEL
46.95
53.73
64, 40
66.34
74.59
8.25
DICDI
41.18
53.81
59.68
64.07
69.85
5.78
DICDI
56.85
58.10
64.86
64,82
73.51
8.65
TRYCC
54.13
59.04
64.24
64.68
74.55
9.87
TRYCC
50,26
61.41
66.36
69.08
77.73
YEAST
34,86
49.28
$1,95
58.82
62.07
3.25
YEAST
49.62
53.90
$9.92
59.01
67.69
7.77
SCHPO
46.35
49.32
52.17
59.77
62,43
SCHPO
47.31
59.38
60.26
RAT
44.14
54.43
59,52
63.05
70.96
7.91
RAT
47.14
56.52
64.
64.00
73.21
9.05
HUMAN
50.29
54.48
61.06
63.83
70.69
HUMAN
ARATH
44.83
55.68
62.86
65.05
72.26
7.21
ARATH
47.03
56.38
65.04
64.70
73.82
8:78
MAIZE
56.46
58.05
66.04
66.67
76.00
9.33
LEHIN
50.50
$2.96
$6.25
61.36
66.52
5.16
LEINN
55.64
58.00
64.55
65.03
64.62
-0.41
47.54
STAAS
48.68
51.35
53.85
58.95
63.16
4.21
SOYBN
46.93
55.74
63.16
6-143
73.68
9.25
SOYBN
48.98
57.52
64.00
69.85
76,78
ORYSJ
73.77
ORYSJ
46.38
57.14
64.81
65.12
75.36
10.14
Average
45.94
53.48
57.94
62.45
68.15
5.70
Average
50.03
55,88
62.37
70,09
17

---

Under review as a conference paper at ICLR 2022
D BENCHMARK ON CATH4.2
How we re-run baselines We tume hyperparameters of all models on CATH4.2. For baseline models, we prefer to choose the hyperparameters recommended in their original paper to ensure that the results produced by our code are consistent with those reported, as shown Table To For AEDesign, we also tune the hidden dimensions (128) and the number of layers (10 layers of GNN + 3 layers of CNN) in CATH4.2 to investigate whether could achieve competitive results. When adapting to the new dataset, such as AlphaFold DB, we fix hyperparameters of all models, including AEDesign, to the same as those on CATH4.2. For each model, we use the same batch size in the reproduction phase and adjust the learning rate in 0.001, 0.0001] to ensure that the model is converged stably.
Results on CATH4.2 We provide results where models are trained on CATH4.2 in Table 10 for investigating the potential of AEDesign model when designing native proteins. We use the same data splitting as GraphTrans (Ingraham et al. 2019) and GVP (ing et al. 2020), where proteins are partitioned by the CATH topology classification, resulting 18024 proteins for training, 608 proteins for validation, and 1120 proteins for testing. We observe that the con-autoregressive AEDesign outperforms its competitors in recovery while the autoregressive GVP and GCA achieve lower perplexity. Since recovery is the primary measure, conclude that AEDesign is competitive in designing native proteins. More importantly, the performance order of models consistent with the results AlphaFold DB: AEDesign GCA GVP StructGNN GraphTrans.
Table 10: Results comparison on the CATH dataset. All baselines are reproduced under the same code framework, where perplexity (lower is better) and recovery (higher is beller) reported. The best and next best results labeled with bold and underline.
Perplexity 4 Single-chain
Recovery % Single-chain
Model
Short
All
Short
All
StructGNN
8.33
8.86
6.55
GraphTrans,
8.54
9.03
6.85
27.6
Reported
GVP
7.10
7.44
5.29
32.1
32.0
40.2
GCA
7.68
8.09
6.44
33.25
33.04
36.11
StructGNN
8.29
8.74
6.40
29.44
28.26
35.91
GraphTrans
8.39
8.83
6.63
28.14
28.46
35.82
Reproduced
GVP
7.23
7.84
5.36
30.60
28.95
39.47
GCA
7.09
7.49
6.05
32.62
31.10
37.64
AEDesign
7.32
7.63
6.30
34.16
32,66
41.31
18

---

Under review as a conference paper at ICLR 2022
E DOMAIN GENERALIZATION
This work does not study the domain generalization problem, which could be another research direction. However, it will be good for readers to know the difference between proteins created by AlphaFold2 and native ones.
Discussion about Domain shifts We study the domain shifts between proteins created by Al- phaFold2 and native proteins. Taking all the testing sets AF2DB and CATH4.2 as examples, we statistics the distribution of angle features to study whether there are significant differences, as shown in Table [T] We also compare angle distributions of AF2DB and CATH4.2 in Figure 7 We observe that the angle distributions are quite similar but not the same between AF2DB and CATH4.2. The similar distribution means that the knowledge learned from AF2DB could be transferred into CATH4.2, while the difference may lead to performance degradation when transferring to different domains.
3.0
2000
25-
175
150
20
125
15
100
10
075
2
050
05
0.5
-3
-2
-1
1
-3
-2
-3
-2
0
1
2
CATHA.2
AF208
4-
0.0
0.5
1.0
15
2.0
25
3.
0.00
025
050
075
1000
125
1:50
0.0
05
1.0
15
26
25
30
Figure 7: Comparing the angle distributions between CATH4.2 and AF2DB, structural noise 0.00.
Noise std
s
Ø
CATH4.2
(.34(0.89)
-0.66(1.57)
-0.40(.04)
1,01(0009)
1.2000.05)
1.11(0.08)
AF2DB
1.20(0.96)
-0.78(1.44)
-1.53(2.55)
0.98(0.07)
1.19(0.05)
1.09(0.07)
KL
1.38
3.10
17.12
175.64
11.76
52.46
CATH4.2
1.34(0.89)
-0.66(1.57)
-0.33(3.05)
(.0
1.20(0.06)
1.11(0.09)
0.02
AF2DB
1.19(0.96)
-0.78(1.44)
-1.44(2.60)
0.98(0.08)
1.19(0.06)
1.09(0.08)
2.30
2.97
12.71
73.24
5.33
25.54
CATH422
1.34(0.90)
-0.65(1.57)
0,2((3003)
1.0((..12)
1.21(0.09)
1.11(0.11)
0.05
AF2DB
1.19(0.97)
-0.78(1.44)
1.16(2.772
0.98(0.11)
1.19(0.09)
1.09(0.10)
KL
1.07
2.61
9.25
18.21
1.87
3.58
CATH422
1.33(0.92)
-0.63(1.59)
-0.1((299)
1.03(0.18)
1.21(0.15)
1.12(0.18)
0.1
AF2DB
1.19(0.99)
-0.77(1.46
0.83(2280)
1.20(0.15) 0.99(0.18)
1.10(0.17)
KL
1.26
2.07
5.21
4.11
0.81
0.49
Table 11: Statistics angle features. We count angle distributions for the testing sets of AF2DB (SL setting) and CATH4.2, The mean and standard deviation are provided, where the standard deviation is marked brackets. We also provide KI. divergence between the angle distributions of AF2DB and CATH4.2. Gaussian noise could he added the input structures, where the noise std is listed in the left.
Address the domain shift issue How to eliminate domain differences? Inspired by Li at.el et al. 2021b Dauparas et al. 2022 Hsu et al., 2022), we find that perturbing input structures with Gaussian noise during training leads improved gemainngeneraaizztton performance. As shown in Table IT] the KL divergence between features of AF2DB and CATH4.2 decreases when the the standard deviation of Gaussian noise increases. When training AEDesign on SOYBN (a subset of
19

---

Under review as a conference paper at ICLR 2022
AF2DB) and evaluating it the testing set of CATH4.2, the generalization be enhanced by adding noise, as shown in Table I2 However, this does not mean that higher noise levels are better, too much noise may obscure all useful information.
Noise std
0.00
0.001
0.02
0.05
0.08
0.10
0.20
0.30
CATH4.2
20.74
29.21
32.62
34.51
34.24
33.63
31.90
29.26
Table 12: Results under the cross doss-domain setting. We train AEDesign on the training set of SOYBN, and evaluate the testing set of CATH4.2. We reveal how the noise level helps the domain generalization.
We have discussed the domain shift issue between AF2DB and CATH4.2, and verified that the model trained on AF2DB by simply adding Gaussian noise is effective for designing native proteins. This discovery is consistent with the results of ESM-IF and ProteinMPNN. However, we provide new insight into why adding noise is effective. We believe that better domain generalization methods could further improve the model's 's performance, but we do not intend to study it in depth in this work. Moreover, ESM-IF has verified that: training AF2DB with CATH4.2 should further improve the model performance from 38.3% to 51.6% on the CATH dataset, and this paper does not repeat their innovation. From our perspective, ESM-IF's data augmentation actually another domain generalization approach.
3.0
2:00
25
175
150
20
125
15
100
10
075
2
0.0
-3
-2
-1
2
-3
-2
11
2
-3
-2
CATH4
AF208
4-
0.0
15
25
1.5
Figure 8: Comparing the angle distributions between CATH4.2 and AF2DB, structural noise 0.02.
20

---

Under review as a conference paper at ICLR 2022
30
200
25
175
150
20
125
15
100
10
075
-3
-2
-1
-3
22
1
2
33
-2
CATH4.2 2
AF208
4
0.
15
0.00
0.25
150
15
30
Figure 9: Comparing the angle distributions between CATH4,2 and AF2DB, structural noise 0.05.
30
200
25
175
150
20
125
15
10
75
22
22
-1
1
-2
CATHA.2
AF208
4-
0.0
10
15
255
0.
0.5
1.0
15
25
30
Figure 10: Comparing the angle distributions between CATH4.2 and AF2DB, structural noise 0.10.
21


Reviews:
[{"review_id": "mXcLeI2U-h", "reviewer": "Reviewer_MXiS", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- The paper establishes a new benchmark on fixed backbone protein design using AlphaFold DB\n- Authors introduce a new model based on a simplified graph transformer encoder and a constraint aware decoder\n- This model achieves good results in comparison to a large number of baseline models using GVP or graph transformer.", "summary_of_the_review": "The paper does not use current best practices for data splits, and also trains/tests on a synthetic database without showing that there's no domain shift issues. Additionally, many ideas have been implemented in concurrent work. Therefore, I recommend a reject rating for this work.", "strength_and_weaknesses": "Strengths:\n- AlphaDesign's model is interesting, and does seem much faster than decoding autoregressively, especially on long sequences.\n- The paper comprehensively tests a variety of different models, showing that AlphaDesign produces good designs across species.\n\nWeaknesses:\n- The new dataset is unsound, it presumes that AlphaFoldDB is producing ground truth structures that can be used as a validation set. I recommend authors make an argument that switching to alphafold structures has no impact on design of natural proteins, otherwise it would be akin to testing on a synthetic dataset that might not transfer to real protein data.\n- It seems unreasonable not to include _any_ natural proteins in this work.\n- There's no validation set used, which means models may be overfitting on the test set, especially if the authors did more tuning on their model than the baselines.\n- The dataset is not held out in a structural manner. In Hsu et al, 2022; Ingraham et al., 2019; Jing et al., 2021b; Strokach et al., 2020, they all use CATH based topology splits to ensure that the result holds up across new topologies. One way to do this might be to use TMscore to do clustering of training / test sets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gQTQlyEUS3", "reviewer": "Reviewer_QZ9p", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new neural network architecture for the inverse folding problem (protein structure -> sequence) and develops a new benchmarking setup that the authors recommend for future work in the field. The new benchmark improves on prior practices by formalizing a train-validation-test split and by providing more diversity of proteins. The empirical performance of the proposed model is strong relative to prior work.", "summary_of_the_review": "I appreciate that the paper works to introduce a new benchmarking setup. This kind of work can be under-appreciated and is important to drive progress in the field. I also appreciate that the performance of the proposed method is strong.\n\nI have some key questions below:\n\nThe benchmark is based on a fairly old version of AlphaFoldDB. The current database contains ~200M structures. Using this would allow you to perhaps have a more diverse test set and to guarantee a bigger distance between the train and test sets. Why did you use an old version of the database?\n\nI don't understand why the results are stratified by organism. How does this improve the analysis? Homologous proteins will be fairly similar across different organisms. If you were to stratify, why not stratify by type of molecular function? \n\nAlso, when you constructed the clustering-based train-test split, did you cluster proteins from multiple organisms, or did you do clustering independently for each organism? \n\nI'm concerned that the paper does not benchmark against recently published papers on this topic. The paper cites the ESM-IF paper, for example, but explains that no benchmarking was performed because the work was in parallel and code has not been released. However, the paper was accepted at a prior ML conference (ICML 2022) and the code was released [1] about 4 months ago. Can you elaborate on why you do not consider the method? Is the challenge that ESM-IF would need to be retrained because you use a different train-test split?\n[https://github.com/facebookresearch/esm/tree/main/examples/inverse_folding]\n\nOne challenge of introducing a new benchmark dataset is that you have to re-run baseline methods. What protocols did you use, for example, when adapting GraphTrans, etc. to the new data? Did you re-tune any hyper-parameters?\n\n\n==Update after authors' response==\nI have raised my score to weak accept. My concern around technical correctness (the per-organism train-test split) has been adequately clarified. I wish the paper used the full AlphaFoldDB, but it would not be fair to penalize this paper when the AlphaFoldDB was updated not long before the ICLR deadline. ", "strength_and_weaknesses": "=Strengths=\nInteresting/relevant application\nStrong empirical results\nEstablishes a benchmarking dataset to be used in future papers.\nUseful ablations to identify the impact of various aspects of the proposed method.\n\n=Weakness=\nI have a few key hesitations around the comparison to recent work, how the dataset was constructed, etc. See my extended comments below.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LitXLxTrXg", "reviewer": "Reviewer_dT5A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper establishes a new benchmark based on AlphaFold DB, one of the world’s largest protein structure databases. Moreover, the authors propose a new baseline method called AlphaDesign, which achieves 5% higher recovery than previous methods and about 70 times inference speed-up in designing long protein sequences. The authors also reveal AlphaDesign’s potential for practical protein design tasks, where the designed proteins achieve good structural compatibility with native structures.", "summary_of_the_review": "See Strength And Weaknesses.", "strength_and_weaknesses": "Pros: \n1. The motivation is clear. \n2. The paper is well-written and organized. \nCons: \n1. The main contributions are not clear. \n2. Some related works are missing, e.g., Multi-Human Parsing With a Graph-based Generative Adversarial Model.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7K0PO_OUR2T", "reviewer": "Reviewer_QTBS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript introduces a new method for protein design (inverse folding) and a benchmark consisting of a curated set of structures from the AlphaFold Database. The authors demonstrate a substantial improvement compared to a selection of earlier methods.", "summary_of_the_review": "The paper presents a new method for inverse folding.  The main strength of the paper lies in the reported results, which look very competitive - and in the fact than the authors provide a new benchmark to the community. The manuscript does not contain a substantial methodological contribution, and the part that is new is currently not described very clearly in the paper. However, based on the results I expect the paper could have an impact on the community. If the authors improve the clarity of the paper by addressing the concerns described in the detailed comments above, I would therefore be willing to increase my score.\n\n---UPDATE---\nThe authors have improved the clarity of the paper as requested, and I have therefore increased my score to 6: marginally above the acceptance threshold.\n", "strength_and_weaknesses": "The paper makes good use of the recently released AlphaFold Database of protein structures, both by training a new model on these available structures, and by curating the database into a dataset which can be used as a standard for future work. Method-wise, the contribution is limited - the authors build on components from earlier work, adding three new angle features and a new decoder. However, the results are convincing and the method is therefore likely to have an impact.\n\nOverall, the paper is clearly structured. However, parts of the paper seem hastily written and in particular parts of the Methods section are not clearly explained. See detailed comments below.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject