ProteinInvBench: Benchmarking Protein Inverse Folding on Diverse Tasks, Models, and Metrics
Authors: Zhangyang Gao
Cheng Tan
Yijie Zhang
Xingran Chen
Lirong Wu
Stan Z. Li
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: protein design

Abstract: Protein inverse folding has attracted increasing attention in recent years. However, we observe that current methods are usually limited to the CATH dataset and the recovery metric. The lack of a unified framework for ensembling and comparing different methods hinders the comprehensive investigation. In this paper, we propose ProteinBench, a new benchmark for protein design, which comprises extended protein design tasks, integrated models, and diverse evaluation metrics. We broaden the application of methods originally designed for single-chain protein design to new scenarios of multi-chain and \textit{de novo} protein design. Recent impressive methods, including GraphTrans, StructGNN, GVP, GCA, AlphaDesign, ProteinMPNN, PiFold and KWDesign are integrated into our framework. In addition to the recovery, we also evaluate the confidence, diversity, sc-TM, efficiency, and robustness to thoroughly revisit current protein design approaches and inspire future work. As a result, we establish the first comprehensive benchmark for protein design, which is publicly available at \url{https://github.com/A4Bio/OpenCPD}.

ProteinInvBench. Benchmarking Protein Inverse Folding on Diverse Tasks, Models, and Metrics
Zhangyang Gao 12, Cheng Tan 1,2f, Yijie Zhang Xingran Chen 4, Lirong Wu 12, Stan Z. Li 2,
Zhejiang University
2 AI Lab, Research Center for Industries of the Future, Westlake University
McGill University, University of Michigan
Abstract
Protein inverse folding has attracted increasing attention in recent years. However, we observe that current methods are usually limited to the CATH dataset and the recovery metric. The lack of unified framework for ensembling and compar- ing different methods hinders the comprehensive investigation. In this paper, we propose ProteinInyBench. new benchmark for protein design, which comprises extended protein design tasks, integrated models, and diverse evaluation metrics. We broaden the application of methods originally designed for single-chain pro- tein design to new scenarios of multi-chain and de novo protein design. Recent impressive methods, including GraphTrans, StructGNN, GVP, GCA, AlphaDesign, ProteinMPNN, PiFold and WWDesig are integrated into our framework. In addi- tion to the recovery, we also evaluate the confidence, diversity, sc-TM, efficiency, and robustness to thoroughly revisit current protein design approaches and inspire future work. As result, we establish the first comprehensive benchmark for protein design, which is publicly available at httth:t///// htte::///iitie gigghh..bb.... com/ /0mm//////00p//
1 Introduction
Protein inverse folding a fundamental problem biology and has many applications in medicine, agriculture, and bioenergy [T4]. It has thereby attracted increasing attention in both the machine learning and biology communities [5] (6) Traditional physical-inspired methods suffer from the problem of expensive computation and unsatisfactory accuracy. Recently, deep learning methods have shown great potential in science areas 37- including protein inverse folding, by simplifying
the process and improving accuracy [12 31 62]. Among them, we observe that graph-based methods achieve statc-of-the ohe-the-art performance. However, previous methods are usually limited to the CATH dataset and the recovery metric. We emphasize that recovery is not the only important metric for protein design. Other metrics such as confidence, diversity, TM-score, efficiency, and robustness are also important for comprehensively revisiting current approaches. Also, the evaluation dataset should be further extended from CATH dataset to broader or more difficult cases to facilitate practical applications. All these challenges motivate us to establish ProteinIny Bench, unified benchmark for protein inverse folding, in which multiple tasks, models, and metrics are introduced and integrated.
ProteinInyBende extends the task of protein design from single-chain to multi-chain and de novo protein design. To our knowledge. many computational protein design methods [41] 46) 47 [50 63 have only been evaluated on the outdated single-chain structure dataset CATH4.2. Furthermore, few studies (48 56 have investigated protein design performance in multi-chain tasks, and even fewer have evaluated the performance comparison on de novo protein structures. To fill this knowledge gap.
EEuual Contribation, "Corresponding Author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

we first benchmark open-sourced graph-based models on the latest CATH4.3 dataset and extend them the case of multi-chain protein design. We then collect de novo protein structures with little or no similarity to existing structures from the CASP15 competition. Evaluating models on the CASPI5 dataset allows us gain better understanding of the potential of Al models designing de novo proteins and reveals that different models exhibit non-trivial differences in generalizability. We hope that the more complex and challenging tasks ProteinInv Bench will facilitate the development of protein design methods for practical applications.
ProteinlnvBench also provides range of metrics comprehensively understand the strengths and weaknesses of each method. Beyond the recovery score that measures the percentage of exactly matched residues, metrics will be introduced in detail in the Sec Notably, we use novel metrics such as confidence and se- sc-TM to measure the quality of designed sequences in an unsupervised and unbiased manner, respectively. With additional metrics, we encourage researchers to develop more robust, efficient models that can generate diverse proteins for higher success rates in wet experiments.
also evaluate the confidence, diversity, sc- sc-TM, efficiency, and robustness. These
Tasks: Designing proteins over single-chain, multi-chain and de-novo structures
CATH422
CATH43
PDB
CASPIS
Single-chain
Multi-chain
De-movo
Models: Recent strong baselines that provide open-source training code
Graph Trans
StructGNN
GVP
GCA
AlphaDesign
ProtciaMPNN
PiFold
KWDesign
Metries: The most comprehensive evaluation metries to date
Recovery
Confidence
Robustness
Diversity
Effieciency
Widely considered
Partially considered
New Contents
Figure 1: The framework of the proposed benchmark. The benchmark is organized incrementally from tasks to models, metries. We color contents in green and blue that widely and partially considered by previous studies, respectively. Newly introduced contents colored in pink.
Based on the constructed benchmark, we conduct extensive experiments to extend baseline models to new tasks and evaluate them on diverse metrics. All models are reproduced, integrated, and extended under unified code framework. ProteinInvBench not only reproduces the reported results of baselines but also provides new insights into the detailed strengths and weaknesses of each method under different scenarios. To summarize, our contributions are as follows:
1. Tasks:
We extend recent impressive models from single-chain protein design to the scenarios of multi-chain and de novo protein design.
Models: We integrate recent impressive models into a unified framework for efficiently reproduc- ing and extending them to custom tasks.
We incorporate new metries such confidence, sc-TM, and diversity for protein design, and integrate metrics including recovery, robustness, and efficiency to formulate a comprehensive evaluation system.
Metrics:
Benchmark: We establish the first comprehensive benchmark of protein design, providing insights into the strengths and weaknesses of different methods.

---

2 Overall Framework and Problem Definition
Overall Framework We present the overall framework in Figure II which contains three compo- nents: (1) Multiple datasets for the task of single-chain, multi-chain, and de-novo protein design. (2) Strong baselines are integrated into our unified framework, including GraphTrans, GVP, GCA, AiphaDesign, ProteinMPNN, PiFold, and KWDesign. (3) Diverse metrics are used to evaluate the designed proteins in a quantitative and comprehensive manner.
Problem Definition The protein inverse folding problem [64] aims to find the amino acids sequence
S {sy 1 i } folding into the desired structure X' = {X, E R : 1 S S n).. where m is the maximum number of points belonging to the i-th residue, n. is the number of residues and the natural proteins are composed by 20 types of amino acids, i.e., S 8 S 20 and 8; N+ Therefore, the protein inverse folding problem is usually formulated as a prrueture-to-sequenee learning problem, where the goal is to learn function Fg:
Fo: 10:20 x' 1 S.
The function Fø usually parameterized by a neural network, and the parameters are learned by minimizing the cross-entropy loss, i.e., ((Fo) = Ei= log s:p($:1X
3
Datasets
CATH The CATH (Class, Architecture, Topology, Homology) [65 database is comprehensive resource for protein structure classífication that hierarchical group proteins based on their structural features. The database defines classes based topological similarities, architectures based on the arrangement of secondary structure elements, topologies based on the connectivity of secondary structure elements, and homologous domains based on sequence similarity. Previous works such as GraphTrans (50), GVP [41], and GCA [46] have used the CATH4.2 version of the database as a benchmark for protein design, which splits the dataset by CATH topology classification and includes 18,024 proteins for training, 608 proteins for validation, and 1,120 proteins for testing. However, CATH4.2 an outdated version and may not represent the current protein structure space. To address this, we use the newer version, CATH4.3 for benchmarking protein design and follow the data splitting protocol of ESMIF [53]. This results in a training set of 16.153 structures, validation set of 1,457 structures, and a test set of 1,797 structures. Note that the curated CATH dataset contains only single-chain structures and does not consider the case of designing multi-chain proteins.
PDB The Protein Data Bank (PDB) 66 is comprehensive database of 3D structural data for biological molecules. To study multi-chain protein design, we utilized a ProteinMPNN dataset derived from PDB assemblies with high resolution and less than 10,000 residues. 'he dataset was preprocessed by clustering sequences at 30% identity, resulting in 25,361 clusters. Following ProteinMPNN'ss setup, we divided the clusters randomly into training (23,358), validation (1,464), and test sets (1,539), ensuring that none of the chains from the target chain or biounits of the target chain were present in the other two sets. During each training epoch, we cycled through the sequence clusters and randomly selected a sequence member from each cluster. This dataset was used for the task of multi-chain protein design, expanding the comparison of computational protein design methods, as many previous methods were omitted in this task.
TS45 In addition designing single- and multi-chain proteins, we also include set of de novo proteins collected from the CASPI5 competition to provide a more realistic assessment [67 68) The Critical Assessment of Protein Structure Prediction (CASP15), which took place from May through August 2022, was held after the release dates of CATH4.3 (July 1, 2019) and PDB (August 2, 2021). In CASP15, diverse protein targets are introduced, including FM (Free Modeling), TBM (Template-Based Modeling). TBM-case and TBM-hard proteins. There are 18 FM, 25+2 TBM (including 20 TBM-cazy, 5 TMB-hard, 2 FM/TBM). The FM targets have no homology to any known protein structure, making them particularly suitable for de novo protein design. The TBM targets have some homology to known protein structures, while the TBM-easy targets are relatively easy TBM targets. The TBM-hard targets are more difficult TBM targets, with lower levels of sequence identity to known structures. We download the public TS-domains structures from CASP15 which consists of 45 structures, namely TS45. We use TS45 as a benchmark for de nove protein design, as the structures are less similar to known structures and were not determined prior to the construction of the training sets.
3

---

Baseline models
We ensemble recent strong protein design baselines under the unified framework. Currently, we support open-sourced graph methods, such GraphTrans, StructGNN [50] GVP 41], GCA [46], AlphaDesign 477, ProteinMPNN [48), PiFold [63 and KWDesign [69], that we can access their
codes and training scripts. StructGNN and GraphTrans (50 employ C-alpha for geometric features, while GCA [46] adds global attention. GVP [1]] introduces a novel GNN layer for invariant and equivariant features. However, these methods suffer from poor inference efficiency due to autoregressive decoders. overcome this, AlphaDesign [47 replaces the decoder with an iterative ID CNN. Recent advancements include ProteinMPNN 48), which incorporates additional structural information, and PiFold [63]. a combination of AlphaDesign and ProteinMPNN, KWDesign [69 an ensemble model that utilizes PiFold to create prompt template. It refines the template using pre-trained knowledge, including sequence pretraining (ESM-650M (70) and structure pretraining
models (ESMIF's encoder (53]
New baselines [56] 71 will continue to be added in the future. We have not included ESMIF in our benchmark since it lacks training script and is challenging for us train. We plan to add it to the benchmark once we successfully train the model. According to the generation scheme, these baselines can be categorized as autoregressive, iterative, and one-shot models.
Autoregressive models consider both sequential and structural dependencies by combining the structural encoder and autoregressive sequence decoder, such as GraphTrans [50], GVP [41), GCA [46], and ProteinMPNN [48]. The protein sequences are generated by:
p( II p(8,láct,
(2)
Autoregressive models have been criticized for being slow generating long proteins 633
Iterative models
generate residues in parallel and iteratively refine the generated sequence (Al- phaDesign 47 and KWDesign [69]:
((() p(S)x; S, p(S) ;x; xx;
where indicates the refinement step, affecting the inference time costs. 0 is learnable parameter.
One-shot models generate the protein sequence in parallel, e.g., PiFold 63 which is quite efficient in generating long proteins, written
$ S~p(S)x;;0)
Metrics
In this section, introduce metrics that will be used for protein design evaluation, including recovery, confidence, diversity, and sc-TM. Previous researches [4] 46
50
63) mainly focus
on improving recovery, while ignoring other metrics. However, argue that recovery is not the only important metrie for protein design. Other metrics introduced follows are also crucial for comprehensively revisiting current approaches, such confidence, diversity, sc-TM, efficiency, and robustness.
Recovery Recovery is the primary metric of the ability of the designed protein to recover its original residues, defined as:
Rec
T2 Σ 1(s = st)
(5)
where 1(-) is the indicator function, 3; the designed residue at the i-th position, and s; is the corresponding reference residue. A high recovery rate indicates that the designed protein sequence similar the reference sequence, and it is therefore expected that the folded structure will also be similar to the reference structure. Since measuring structural similarity is computationally expensive, previous protein design methods have placed great deal of emphasis on improving recovery. However, it is important to note that recovery itself is proxy metric for measuring structural similarity. other words, a higher recovery rate does not necessarily ensure a higher level of structural similarity. Moreover, high recovery rate may result in low diversity.

---

Confidence Calculating recovery requires access to the reference sequence, which is not always available practice. When the ground-truth sequence unknown, measuring and ranking the quality of the designed sequence becomes more challenging. We introduce the confidence metric to address this problem, which is the average predictive probability of designed amino acids, defined as:
- 71
Σ
"
Conf
p(S.)
(6)
Diversity To improve the success rate of protein design, is important to explore a set of protein sequences rather than placing bet single sequence. In this case, generating diverse sequences is crucial for exploring the reasonable protein sequence space. We define the pairwise diversity [72] as
Diy -m- tnddddd where ri, indicates the 1-th residue of the i-th designed sequence. The overall diversity score is
Div Σ Dij
m
where i.j E f1,2,3.... m and m is the number of totally designed sequences. By default, we set = 10. However, measuring diversity alone without combining it with other metrics may be misleading. For example, a high diversity indicates a low recovery rate, more likely to result a low structural similarity.
se-TM The structural similarity is the ultimate standard for measuring the quality of the designed sequence. However, the structures of designed protein sequences needed to be predicted by other algorithms, such as AlphaFold [731, Rose ITAFold (74), OmegaFold [75] and ESMFold [70]. The protein folding algorithm itself has certain inductive bias and will cause some prediction errors, which will affect the evaluation. To overcome the inductive bias, we introduce the self-consistent TM-score (sc-TM) metric:
SC-TM TMScore
where / is the protein folding algorithm and TMscore(:; :) is widely used metric (76) for measuring protein structure similarity. Since the structures of the designed sequence and reference sequence are predicted by the same protein folding algorithm, the model's 's inductive bias is expected to be canceled when calculating the TM-score. This approach results in a more robust metric, called the sc-TM, that is less affected by the inductive bias of the protein folding algorithm.
Robustness Robustness measures an algorithm's 's ability to maintain its original performance under geometric perturbations. It is useful metric for assessing the stability and generalizability of an algorithm. We define robustness as:
Rob Rec'
Rec
where Rec and Rec' are the recovery after and before applying small Gaussian perturbations to the Cartesian coordinates of the structure, correspondingly. As the template protein structures may be perfect, more robust methods are expected to be more suitable in real-world applications.
Efficiency Efficiency measures the computational resources and time required design set of proteins. This study reports the training time, evaluation time, and model parameters of different methods over the standard benchmarks. While efficiency may not be crucial problem compared to the recovery and sc-TM, it a useful metric for assessing the model's 's scalability and practicality.
Benchmarking Protein Design
In this section, we retrain baselines on the newly introduced datasets and evaluate them using diverse metrics, resulting in a comprchensive benchmark. The experiments are organized as follows:
1. Establish a basic benchmark within recovery and confidence. As emphasized by previous studies, the recovery rate and predictive confidence are the most important and straightforward metrics. We benchmark baselines over these metrics CATH4.2, CATH4.3, PDB, and TS45 for the task of single-chain, multi-chain, and de novo protein design, respectively. These results could serve the basic benchmark for future studies.
5

---

2. Measuring diversity and sc- TM for practical challenging tasks in protein design. We further extend the evaluation metries diversity and sc-TM. The diversity is opposite of recovery and is meaningless if we measure it alone. By examining the sequence diversity and structural sc-TM together, we could have more comprehensive understanding of the designable protein space.
Assessing the robustness when input structures are not perfect. Although the model performs well on natural proteins, it may fail when the artificially designed structure is noisy. In this case, the robustness of the model crucial for practical applications. We evaluate the robustness of different methods by applying geometric perturbations to the template protein structures during the evaluation phase.
Comparing the efficiency. Towards designing efficient, scalable, and generalizable algorithms, we evaluate the efficiency in terms of training time, evaluation time, and model parameters to facilitate the development of more efficient protein design methods.
6.1 Recovery and Confidence
In this section, benchmark the recovery rate and confidence of different methods the CATH4.2, CATH4.3, PDB, and TS45 datasets to address the problems of single-chain, multi chain, and de protein design. By extending from CATH4.2 to the newer CATH4.3 and from single-chain to multi-chain to de novo, we have constructed the most comprehensive benchmark to date for protein design. All models retrained and evaluated under the same code framework for fair comparison. The hyperparameters used for training models are provided in the Appendix.
Table 1: Single-chain results. The best and suboptimal results are labeled with bold and underlined.
Model
Confidence
Recovery S f
length
4 < 100
100 S L
300
300
S L 500)
Full
L < 100
100
L < 300
300 S 500
Full
SirucGGNN
0.31
0.45
0.45
0.43
0.26
0.36
0.36
0.35
GraphTrans
0.31
0,43
0.43
0.43
0.25
0.35
0.35
0.34
GCA
0.34
0.46
0.47
0.45
0,27
0.38
0.38
0.37
GVP
CATH44
AlphaDesign
0.36
0.49
0.49
0.47
0,33
0.43
0.44
0.42
ProteinMPNN
0.38
0.51
0.52
0.50
0.32
0.47
0.45
Pifold
0.44
0.58
0.60
0.57
0.39
0.53
0.56
0.52
KWDesign
0,50
0.68
0.72
0.67
0.44
0.62
0.66
0.61
SinucGGNN
0.35
0,41
0.47
0.41
0.30
0.34
0.40
0.34
GraphTrans
0.37
0.42
0,48
0.42
0.29
0.34
0.39
0.34
GCA
0.38
0.49
CATH43
GVP
0.45
0.51
0.55
0.50
0.33
0.38
01.45
0.38
AlphaDesign
ProteinMPNN
0.42
0.49
0.57
0.49
0.38
0.44
0.52
0.44
Pifuld
0.47
0.56
0.64
0.55
0.43
0.52
0.59
0.51
KWDesiga
0.58
0.68
0.76
0.67
0.51
0.61
0.69
0.60
Single-chain Results The results of single-chain protein design are shown in Tab. 1 where both CATH4.2 and CATH4.3 datasets are included. We present metrics for proteins of different sequence lengths. From Tab. D it could be concluded that:
KWDesign and PiFold are the best and second-best models. They consistently outperform all other models in terms of both confidence and recovery across all protein lengths in both CATH4.2 and CATH4.3 datasets. This highlights their effectiveness towards protein inverse folding.
Models perform better on longer proteins. This could be due to the increased complexity and information available for longer proteins, allowing the models to make more confident predictions.
3. CATH4.2 and CATH4.3 datasets show the same performance trend and very similar results, thereby validating the performance consistency of the different models. However, it also informs that they unable provide complementary information for analyzing protein design methods. More diverse, complex, and challenging datasets are needed for further investigation.
The unsupervised confidence is highly correlated to supervised recovery. This discovery suggests that researchers can rank the quality of designed proteins based confidence alone, without needing to access the ground truth.
Multi-chain Results To remedy the problem that CATH4.2 and CATH4.3 are highly consistent and not bring complementary information, we extend the experiment a multi-chain dataset. The corresponding results are presented in Table showing that:

---

Table 2: Multi-chain results. The best and suboptimal results labeled with bold and underlined.
Model
Confidence
Recovery &
length
I. < 100
100
I. < 500
500
I.
1000
Full
I.
100
100
S
500
500 S I. 1000
Full
StruciGNN
0.49
0.49
0.50
0.49
0.41
0,41
0.42
0.41
GraphTrans
0.48
0.47
0.48
0.48
0.40
0.39
0.40
0.40
GCA
0.45
0.45
0.46
0.45
0.41
0.41
0.42
0.41
PDB
AlphaDesign
GVP
0.51
0.52
0.53
0.53
0,55
0.54
0.54
0.53
0.48
0,49
0.42
0.50
0.45
0.43
0.49
ProteinMPNN
0.54
0.56
0,58
0.57
0.52
0.53
0.55
0.53
PiFold
0.56
0.60
0.63
0.61
0.54
0.58
0.60
0.58
KWDesign
1. KWDesign achieves the best performance across all the models for proteins of all lengths, followed by PiFold and ProteinMPNN.
The longer the protein sequence, the higher the recovery. Like the single-chain case, confidence and recovery generally increase with the length the protein chain. As the length of multi-chain protein could be up 1000, models perform better the PDB than the CATH dataset.
De novo Protein Design
To investigate the models' potential in designing novel proteins, we evaluate pre-trained models TS45. We present the de novo protein design results in Tab. 3 considering four subsets of TS45: FM (Free Modeling), TBM (Template-Bare Based Modeling), TBM- easy, and TBM-hard. The results show that:
1. For models pre-trained on CATH4.2 and CATH4.3, KWDesign outperforms others by large margin. The PiFold model consistently performs as the second-best model after KWDesign.
For models pre- pre-trained on PDB, PiFold achieves the best performance, while ProteinMPNN pro- vides very competitive recoveries. Switching from the CATH to the PDB dataset, ProteinMPNN achieves a more significant performance gain than PiFold.
3.
The consistent performance trend across different protein subsets suggests that the difficulty level of the protein design task depends on the nature of the protein subset. For instance, models tend perform better TBM-casy proteins than on TBM-hard proteins. more challenging subset of proteins may help reveal the shortcomings of current protein design algorithms.
Al methods have demonstrated great potential in de novo protein design, with all models (except StructGNN and GraphTrans) achieving recoveries of approximately 40% or higher. However, there is still slight performance degradation compared to the results on the original test set.
Table 3: Results of de novo protein design. The best and suboptimal results are labeled with bold and underlined.
Training Model
Confidence TBM-eazy
Recovery TBM-
FM
TBM
TBM-han hard
Full
FM
TBM
TBM- hard
Full
StrikGGNN
0.41
0.43
0.48
0.43
0.45
0.35
0.33
0.38
0.35
0.35
GraphTrans
0.39
0.43
0,46
0.42
0.44
0.33
0.30
0.37
0.36
0.36
GCA
0.48
0.43
0.53
0.48
0.50
0.39
0.37
0.41
0.38
0.40
GVP
0.42
0.39
0.39
CATH4.2
AlphaDesign
0.44
0.41
0.50
0.46
0.48
0.41
0.36
0.46
0.41
0.42
ProteinMPNN
0.49
0.48
0.53
0.51
0.44
0.41
0.46
0.40
0.44
PiFold
0.52
0.46
0.59
0.53
0.47
0.38
0.50
0.47
0.47
KWDesign
0.55
0.52
0.70
0.62
0.64
0.49
0.40
0.59
0.55
0.54
StrucGGN
0.40
0.40
0.45
0.43
0.35
0.33
0.38
0.37
0.36
GraphTrams
0.39
0.42
0.46
0.43
0.45
0.35
0.32
0.37
0.35
0.35
GCA
0.46
0.42
0.51
0.44
0.48
0.37
0.33
0.43
0.40
0.41
GVP
0.47
0.50
0.31
0.39
CATH4,3
AlphaDesign
0.44
0.40
0.50
0.47
0.48
0.40
0.36
0.44
0.44
0.42
ProteinMPNN
0.49
0.48
0.53
0.49
0.44
0.34
0.48
0.43
0.46
PiFold
0.54
0.45
0.56
0.51
0.54
0.47
0.38
0.52
0.49
0.49
KWDesign
0.59
0.50
0.70
0.63
0.65
0.50
0.43
0.59
0.60
0.56
StrcIGNN
0.46
0.41
0.53
0.47
0.39
0.34
0.42
0.41
0.41
Trank
0.43
0.42
0.51
0.45
0.48
0.38
0.33
0.44
0.40
0.41
GCA
0.45
0.41
0.49
0.45
0.47
0.40
0.33
0.44
0.43
0.43
GVP
0.51
0.46
0.55
0.53
0.43
PDB
AlphaDesign
0.49
0.43
0.54
0.50
0.51
0.43
0.39
0.48
0.46
0.46
ProteinMPNN
0.56
0.49
0.58
0.55
0.52
0.39
0.55
0.51
0.52
PiFold
0.55
0.48
0.59
0.53
0.57
0.52
0.45
0.53
0.52
0.53
KWDesign
0.60
0.67
0.69
0.65
0.66
0.56
0.59
0.60
0.62
0.59

---

6.2 Diversity and sc-TM
Diversity We benchmark the diversity on TS45 dataset using models pre-trained on CATH4.3. As dis- covered by previous research 48 53 the sampling temperature affects diversity. Denote the temperature as T, the predicted probability vector is we sample new sequences from the distribution of Multinomiall (sof aatomiialsoffaaartt (softmax(p//)), We vary the tempera-
0.9
Diverity
08
0.7
m
Recuver
0.6
Stucishli
0.5
03
Graphilanss
6CA
04
ture from 0.0 to 0.5 and plot the trends of recovery and diversity in Fig. 2 Under the same sampling tempera- ture, high recovery leads to decreased diversity. How- at the same level of recovery, stronger models have
DVP
03
Apphateeiggn
02
HWDeedgg
0.0
0.1
0.2
0.5
higher diversity.
Figure 2: The trends of recovery and diversity. sc-TM While generating diverse protein sequences is appealing. it would be meaningless if these sequences could not fold to structures with topologies similar the reference one. With this in mind, we investigate the recovery and sc-TM metrics as the temperature increases. To compute sc-TM, we utilize AlphaFold2 [X] to predict protein structures from sequences. According to Fig S and Fig 2 we observe that slight increase in temperature from O to 0.1 is beneficial in significantly enhancing diversity while maintaining good recovery and sc-TM. However, increasing the temperature 0.5 renders the designed sequences meaningless in recovery and sc-TM, despite the higher diversity. Carefully tuning the temperature would be beneficial practical applications.
Tempenteroo0d
Temporatumee0
Figure 3: The statistics recovery and sc-TM with increasing temperature.
6.3 Robustness and Efficiency
Robustness We further investigate whether the models robust structure perturbations, the artificially designed structures may not be perfect, and the atom position may deviate slightly due to thermodynamic vibrations errors in experimental measurements. We add different Gaussian noise to the input structure, i.e., X' + + I). Note that the Gaussian noise (in Angstrom) is added in both training and evaluation structures, where the noise scale is chosen from [0.02, 0.2, 0.5, 1.0]. As we have shown that models perform consistently on CATH and PDB, we benchmark the robustness on CATH4.3. The experimental results shown in Tab from which observe that:
1. Weaker models tend to exhibit greater robustness than stronger models. For example, with 6 1, the recoveries of StructGNN and GraphTrans decrease by only 14%, while AlphaDesign, ProteinMPNN, PiFold, and K KWDesign decrease by at least 20%. This is natural outcome, as weaker models may be the first reach the performance floors of the dataset.
KWDesign achieves the highest recovery across noise scales, followed by PiFold. StructGNN, GraphTrans, GCA, and GVP degrade quickly and reach similar lower bounds. AlphaDesign is more affected by noise compared to GCA and GVP, likely due to its reliance on angular features, which are more sensitive noise than distance features.
3. All models show decrease in performance the Gaussian noise scale increases. Developing protein design methods with higher robustness remains challenging.

---

Table 4: Results of robustness. We calculate the difference in terms of model performance the noisy and clean structures and show it parentheses. smaller absolute value of this difference indicates a more robust model. The best and suboptimal robust models are labeled with bold and underlined.
Model
Cunfidence
Recovery
length
100 + L. 2001
300 S
Full
100 L. 310
300
StraciGNNN
0.57 (40.31)
(.2001))11)
0,49 (40.02)
(.D001)))))
0.28 (-1,22)
(-0.01)
0.38 (-0.22
(-001)
Graphitrrans
0.37 (+0.00)
0.41 (4.01)
0.48 (+0.00)
0.41 (40.01)
0.28 (-0.1)
(40.22
0.37 (-0.0)
0.32 (-0.22)
GCA
0.36 (-8.02)
0:20
0.47 (-0.02)
0.4 (-0.02
0.29 (-0.85)
(40055
0.39 (-0.02)
(-0.03)
CATH4433 3
GVP
(-.001
0.48 (0.05)
0.54((0.01)
0.51 (+0.01)
0.29 (-0.44)
0.34 (0.06)
0.43 (-0.22)
0.36 0.002
(e 0.02)
AlphaDesigg
(4200)))))
0.50 (+0.02)
0.56 (40.03
0.49 (+0.2))
0.33 (-0.04)
0.39 (-0.0)
0.41 (40.4)
(0000)
ProciiMPNN
0.55
(-0.06
Pileld
(-0.6)
(0005)
(-004)
(0.05)
0.45
KWDesign
(+B.88)
(40.55)
0.72 (-0.44
0.61 (40.06
0.43 (-0.05)
0.56 (40.05)
0.05 (-0.04)
(-0.06
StractGNN
0.34 (.0.0)
(-0.05)
0.41 (-0.96)
0.36 (-8.5)
0.25 (-1,05
(-0.06)
0.32 (-0.08
0.28
GraphiTrans
0.53 (4.044
0.36 (40.66)
0 (10,0))
0.56 (0.06)
0.25 (-1.84)
0.27 (40.07)
0.31 (40.09)
0.27 (-007)
GCA
0.33 (-0005
0.35 (4006)
0.39 (0.390)
0.35 0.08)
0.25 (-0.077
0.25 (4006)
0.31 1-0.00
0.25 (.00))
CATHH4 1
GVP
0.39 (-0.05)
0,43 (0006)
0.45 (00.10)
0.42 0.08)
0.2 (-0.08)
0.25 ( (.0.00
0.241-0.11 (..113
0.29( 0.09)
(c 0.2)
AighaDesign
(-0.00)
0-40 (-0.08)
0.43 (-0.10)
0.39 (-0.08)
0.29 (-1..55
0.33 (-0,
0.36 (-0.11)
0.33 (-0.99)
ProteiMPPNN
0.37 (-0.05)
0:41 (-0.6)
0.47 (-0.10)
0.41 (0.08)
0.29 (-0.09)
0.35 (-0.09)
0.41 (-0.11)
0.35
PiPeld
(-0.12)
0.48 (-0.36)
0.41
(.0.13)
0.45 (-0.34)
0.39 (-0.1)
KWDesign
(-0.15)
(50.1))
(-0.1-)
0.38 (-0.15)
StraciGNN
(-0.05)
0.34
0.31 (-0.10) 10)
0.24(-0.10)
0.26
0.24 (-0.10)
GraphTrans
(-300))
(-0.11)
0.33 (-0.15)
0.31 (40.11)
122 [-1.077
(-0.11
0.24
GCA
(.30540:183
0:31 (40125
0 (4.15))1
0.31 (40. 12)
0.22 (-1.10)
(20.15
0.26 1-0.15)
0.24 (-0.12)
GVP
0.32 (-0.13)
0.34 (.0.17)
0.37 (4.15)
(.0.1))
0.22 (-0.13)
0.25 (.0.15)
0.26 (-0.19
0.25 (-2.5-00133 (30.13)
(c 0.5)
AlphaiDesgn
(.30(-0:11)
0.33 (33.110.15)
0.35 (-0.15)
0.35 14)
0.26 (-0.11)
0:28 (-0.
0.30 (-0.17
0.28 (-0 14)
ProteiMMPNN
(-0.08)
(0..13
0.39 (-0.18)
0.37 0.12)
0.26 (-0.12)
0.29 (-0.15)
0.33 (-0.19)
0,30 (1)
PiFold
0.52 (-0.15)
(40.00)
0.40
0.35
0.30
(40.18)
0.37
0.33 18)
KWDesign
(-0.20)
(10.1)
(-0.22)
(-0.11))
(40,
19)
StracIGNN
(-0.08)
0.28
(-0.14)
0.31
0.20
GraphiTrans
(-0.11)
(0.16)
0.27 (-0.21)
(...6)
(-0.10)
0.19 (o.15)
0.20
0.20
GCA
0.25 (-0.13)
0.25 (0.18)
0.21
0.165
0.19 (-0.13)
0.1970.177
0220
0.19
C
GVP
0.29 (-0.16)
0.29 (40.22)
(-0.25)
0.16( (-.23)
0.20 (-0.13)
0.20
0.21 (-0.24)
0.
(6 1.0)
AighaDesign
0.16
0.16 (-0.22)
0.15 (-0.35)
0.18 (-0.19)
0.15 (-0.25)
0.38 1-0.29)
(-12()0024)
ProcciMPNN
(-B.11
0.30 (-0.11)
0.33 (-0.25)
0.31 (40.18))
0.22 (-0.16)
0.23 (-0.21)
0.25 (-0.27)
0.25 (-0.21)
FiFold
(-0.19))
(40,27)
0.29
(40.34)
0.29
(-0
KWDesign
(-0.25)
(4006)
0.45 (-0.31)
(-407)
(-0.25)
(4004)
(-0.28)
(-022))
Efficiency To encourage efficient and scalable models, also benchmark the training cost, evaluation cost, the number of trainable parameters, and training epochs in Tab We conclude that:
1. Training these models over single epoch is generally fast, except for KWDesign (w/o memory). Fortunately, with the memory retrieval technique, KWDesign can achieve a similar speed as the other models. It is worth noting that PiFold and KWDesign require only up to 20 epochs to achieve competitive performance.
In terms of evaluation efficiency, iterative and one-shot models like AlphaDesign and PiFold perform exceptionally well, as they do not require autoregressive generation. On the other hand, KWDesign is relatively slower in this category as it needs to make multiple calls to large pre-trained models to generate higher-quality sequences.
3. Stronger models are associated with a higher number of trainable parameters. Among these models, GVP shows superior efficiency in utilizing model parameters. KWDesigg achieves the best performance with the most parameters.
Table 5: Efficiency comparison.
Training Cost
Evaluation Cost
Others Trainahle Params
Model
CATH4.2
CATH4.3
PDB
CATH4.2
CATH4.3
PDB
epochs
StrucIGNN
112s
600s
378s
662x
1068s
L.8MB
100
GraphTrans
130g
123s
583s
438s
737s
1252s
1.51BB
100
GCA
660s
1308s
37%s
688s
2.09MB
100
GVP
402s
380s
840s
1874s
3193s
3753s
0.93MB
100
AlphaDesign
290s
267s
546a
31s
50s
75s
6.62MB
100
ProtcinMPNN
165s
1548
540s
347s
570s
889s
166MB
100
PiFold
410s
364s
780s
39s
69s
162s
5.79MB
20
KWDesign(wfo memory)
3820s
36244
451s
752x
54,49MB
KWDesigntw memory)
453s
4578
20
7 Conclusion
Protein inverse folding has received significant attention in recent years. However, the lack of thorough comparisons across multiple tasks and metrics hinders the progress toward practical applications. To address this issue, we propose ProteinlnvBench, which consists of diverse tasks, models, and metrics and provides a comprehensive view of computational protein design. We plan to update ProteinIny Bench when the CATH dataset (every 12 months) is updated.

---

Acknowledgement
This work was supported by the National Key R&D Program of China 10022D00111000 1 the National Natural Science Foundation of China (U21A20427), the Competitive Research Fund (WU2022A009) from the Westlake Center for Synthetic Biology and Integrated Bioengineering.
References
[1] James W Bryson, Stephen F Betz, Helen S Lu, Daniel Suich, Hongxing X Zhou, Karyn T O'Neil, and William F DeGrado. Protein design: hierarchic approach. Science, 270(5238): 935-941, 941, 1995.
[2] Bassil Dahiyat and Stephen Mayo. De novo protein design: fully automated sequence selection. Science, 278(5335):82-88 87, 1997.
[3] Yi-Xin Huo, Kwang Myung Cho. Jimmy G Lafontaine Rivera, Emma Monte, Claire Shen, Yajun Yan, and James C Liao. Conversion of proteins into biofuels by engineering nitrogen flux. Nature biotechnology, 2944)3346335, 2011.
[4] Stephen A Williams, Mika Kivimaki, Claudia Langenberg, Aroon D Hingorani, JP Casas, Claude Bouchard, Christian Jonasson, Mark Sarzynski, Martin Shipley, Leigh Alexander, et al. Plasma protein patterns as comprehensive indicators of health. Nature medicine, 25(12): 1851-1857, 2019.
[5] Kevin K Yang, Zachary Wu, and Frances H Arnold. baccine-learning-guiee directed evolution for protein enginecring. Nature methods, 16(8):687-694, 2019.
[6] Cheng Tan, Yijie Zhang, Zhangyang Gao, Hanqun Cao, and Stan Z. Li. Hierarchical data- efficient representation learning for tertiary structure-based ma design, 2023.
[7] Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan Z Li. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering, 2021.
[8] Lirong Wu, Jun Xia, Zhangyang Gao, Haitao Lin, Cheng Tan, and Stan Z Li. Graphmixup: Improving class-imbalanced node classification by reinforcement mixup and self-supervised con- text prediction. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 519 535. Springer, 2022.
[9] Lirong Wu, Haitao Lin, Yufei Huang, Tianyu Fan, and Stan Z Li. Extracting low- gigg-freeeency knowledge from graph neural networks and injecting it into mips: An effective gnn-to-mip distillation framework. arXiv preprint arXiv:2305. 10758, 2023.
[10] Lirong Wu, Haitao Lin, Yufei Huang, and Stan Z Li. Knowledge distillation improves graph structure augmentation for graph neural networks. Advances in Neural Information Processing Systems, 35:11815-11821 2022.
[11] Lirong Wu, Haitao Lin, Yufei Huang, and Stan Zi. Quantifying the knowledge in gnns for reliable distillation into mlps. arXiv preprint arXiv:2306.05628. 2023.
[12] Hanqun Cao, Cheng Tan, Zhangyang Gao, Guangyong Chen, Pheng- Ann Heng, and Stan Z Li. survey on generative diffusion model. arXiv preprint arXiv:2209.02646 2022.
[13] Zhangyang Gao. Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet better video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3170-3180, 2022.
[14] Cheng Tan, Zhangyang Gao, Lirong Wu, Yongjie Xu, Jun Xia, Siyuan Li, and Stan Z Li. Temporal attention unit: Towards efficient spatiotemporal predictive learning. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 18770-18782, 2023.
10

---

[15] Cheng Tan, Zhangyang Gao, and Stan Li. Rfold: Towards simple yet effective ma secondary structure prediction. arXiv preprint MtXiv:2212.14041. 2022.
[16] Zhangyang Gao, Yuqi Hu, Cheng Tan, and Stan Z Li. Prefixmol: Target-and chemistry-aware molecule design via prefix embedding. arXiv preprint arXiv:2302.071 120, 2023.
[17] Cheng Tan, Zhangyang Gao, and Stan Z Li. Target-aware molecular graph generation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 410-427. Springer, 2023.
[18] Cheng Tan, Zhangyang Gao, and Stan Z Li. Generative tertiary structure-based ma design. arXiv preprint arXiv:2301.10774, 2023.
[19] Cheng Tan, Zhangyang Gao, Siyuan Li, and Stan Li. Simvp: Towards simple yet powerful spatiotemporal predictive learning. arXiv preprint arXiv:2211.12500 2022.
[20] Cheng Tan, Siyuan Li, Zhangyang Gao, Wenfei Guan, Zedong Wang. Zicheng Liu, Lirong Wu, and Stan Z Li. Openstl: A comprehensive benchmark of spatio-temporal predictive learning. arXiv preprint arXiv:2306. 1249, 2023.
[21] Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Cosp: Co-supervised pretraining of pocket and ligand. arXiv preprint arXiv:2206. 12241, 2022.
[22] Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Semiretro: Semi-template framework boosts deep retrosynthesis prediction. arXiv preprint arXiv:2202.08205, 2022.
[23] Cheng Tan, Zhangyang Gao, and Stan Z Li. Protein complex invariant embedding with cross- gate mlp is a one-shot antibody designer. arXiv preprint arXiv:2305.0944 099400, 2023.
[24] Zhangyang Gao, Cheng Tan, and Stan Z Li. Vqpl: Vector quantized protein language. arXiv preprint arXiv:2310.04985, 2023.
[25] Lu Yi, Zhangyang Gao, Zhehui Shen, Haitao Lin, Zicheng Liu, Siqi Ma, Stan Z Li, and Ling Li. Precipitation measurement based on satellite data and machine learning. Technical report, Copernicus Meetings, 2023.
[26] Shunzhi Wang, Zhe Li, Isaac Lutz, Christoffer Norn, Una Nattermann, and David Baker. Accurate computational design of programmable 3d protein crystals and capsids. In 2023 AIChE Annual Meeting. AIChE, 2023.
[27] Zhe Li, Shunzhi Wang, Una Nattermann, Asim K Bera, Andrew Borst, Muammer Y Yaman, Matthew J Bick, Erin C Yang, William Sheffler, Byeongdu Lee, al. Accurate computational design of dimenssona protein crystals. Nature Materials, pages 1-8, 2023.
[28] Sanaa Mansoor, Minkyung Back, David Juergens, Joseph L Watson, and David Baker. Zero-shot mutation effect prediction on protein stability and function using rosettafold. Protein Science, page e4780, 2023.
[29] Nathaniel R Bennett, Brian Coventry, Inna Goreshnik, Buwei Huang, Aza Allen, Dionne Vafeados, Ying Po Peng, Justas Dauparas, Minkyung Baek, Lance Stewart, et al. Improving de novo protein binder design with deep learning. Nature Communications, 14(1):2625, 2023.
[30] Isaac D Lutz, Shunzhi Wang, Christoffer Norn, Alexis Courbet, Andrew Borst, Yan Ting Zhao, Annie Dosey, Longxing Cao, Jinwei Xu, Elizabeth M Leaf, et al. Top-down design of protein architectures with reinforcement learning. Science, 38066642)226622222 2023.
[31] Jie Li and Patrice Koehl. 3d representations of amino acids applicaopios protein sequence comparison and classification. Computational and structural biotechnology journal. 11(18): 47-58, 2014.
[32] Zhixiu Li, Yuedong Yang, Eshel Faraggi, Jian Zhan, and Yaoqi Zhou. Direct prediction of pro- files of sequences compatible with protein structure by neural networks with fragment-based local and energy-based nonlocal profiles. Proteins: Structure, Function, and Bioinformatics, 82 (10):2565-2577 -5733, 2014.
11

---

[33] James O'Connell, Zhixiu Li, Jack Hanson, Rhys Heffernan, James Lyons, Kuldip Paliwal, Abdol- lah Dehzangi, Yuedong Yang. and Yaoqi Zhou. Spin2: Predicting sequence profiles from protein structures using deep neural networks. Proteins: Structure, Function, and Bioinformatics, 86 (6):629-633, 633, 2018.
[34] Jingxue Wang, Huali Cao, John ZH Zhang, and Yifei Qi. Computational protein design with deep learning neural networks. Scientific reports, 8(1): 2018.
[35] Sheng Chen, Zhe Sun, Lihua Lin, Zifeng Liu, Xun Liu, Yutian Chong, Yutong Lu, Huiying Zhao, and Yuedong Yang. To improve protein sequence profile prediction through image captioning on pairwise residue distance map. Journal of chemical information and modeling, 60(1):391-399, 2019.
[36] Raulia Syrlybaeva and Eva-Maria Strauch. Deep learning of protein sequence design of protein- protein interactions. Bioinformatics, 39(1):btac733, 2023.
[37] Raghav Shroff, Austin W Cole, Barrett R Morrow, Daniel J Diaz, Isaac Donnell, Jimmy Gollihar, Andrew D Ellington, and Ross Thyer. A structure-based deep learning framework for protein engineering. bioRxiv, page 833905, 2019.
[38] Yuan Zhang, Yang Chen, Chenran Wang, Chun-Chao Lo, Xiuwen Liu, Wei Wu, and Jinfeng Zhang. Prodconn: Protein design using convolutional neural network. Proteins: Structure, Function, and Bioinformatics. 88(7))819-82) 2020.
[39] Namrata Anand, Raphael Eguchi, Irimpan I Mathews, Carla P Perez, Alexander Derry, Russ B Altman, and Po-Ssu Huang. Protein sequence design with a learned potential. Nature communications, 13(1):746, 2022.
[40] Cheng Tan, Zhangyang Gao, and Stan Z Li. Cross-gat goss--gatedoe mip with protein complex invariant embedding a one-shot antibody designer. arXiv e- e-prints, pages arXiv-23300 2023.
[41] Bowen Jing. Stephan Eismann, Patricia Suriana, Raphael JL. Townshend, and Ron Dror. Learn- ing from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.0411, 2020.
[42] Moshe Eliasof, Tue Boesen, Eldad Haber, Chen Keasar, and Eran Treister. Mimetic neural networks: A unified framework for protein design and folding, Frontiers in Bioinformatics, 2: 39, 2022.
[43] Alex Li, Vikram Sundar, Gevorg Grigoryan, and Amy E Keating. Terminator: a neural framework for structure-based protein design using tertiary repeating motifs. arXiv preprint arXiv:2204.1 13048, 2022.
[44] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Antibody-antigen docking and design via hierarchical equivariant refinement. arXiv preprint arXiv:2207.06611 2022.
[45] Alex J Li, Mindren Lu, Israel Desta, Vikram Sundar, Gevorg Grigoryan, and Amy E Keating. Neural network-derived potts models for structure-based protein design using backbone atomic coordinates and tertiary motifs. Protein Science, 32(2):e4554, 2023.
[46] Cheng Tan, Zhangyang Gao, Jun Xia, and Stan Z Li. Generative de novo protein design with global context. arXiv preprint arXiv:2204. 10673, 2022.
[47] Zhangyang Gao, Cheng Tan, Stan Li, et al. Alphadesign: A graph protein design method and benchmark on alphafolddb. arXiv preprint arXiv:2202.01079 2022.
[48] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai. Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, al. Robust deep learning based protein sequence design using proteinmpnn. bioRxiv, 2022.
[49] Deqin Liu, Sheng Chen, Shuangjia Zheng, Sen Zhang, and Yuedong Yang. Se (3) equivalent graph attention network as an energy-based model for protein side chain conformation. bioRxiv, pages 2022-09, 2022.
12

---

[50] John Ingraham, Vikas K Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-basen protein design. 2019.
[51] Yue Cao, Payel Das, Vijil Chenthamarakshan, Pin- Yu Chen, Igor Melnyk, and Yang Shen. Fold2seq: A joint sequence (Id)-fold (3d) embedding-basee generative model for protein design. In International Conference on Machine Learning, pages 1261-1271, PMLR, 2021.
[52] Yufeng Liu, Lu Zhang, Weilun Wang, Min Zhu, Chenchen Wang, Fudong Li, Jiahai Zhang, Houqiang Li, Quan Chen, and Haiyan Liu. Rotamer-free protein sequence design based on deep learning and self-consisteney. Nature Computational Science, 2(7):451-462, 2022.
[53] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022.
[54] Baldwin Dumortier, Antoine Liutkus, Clément Carré, and Gabriel Krouk. Petribert: Augmenting bert with tridimensional encoding for inverse protein folding and design. bioRxiv, pages 2022- 08, 2022.
[55] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, al. Evolutionary--cale prediction of atomic-level protein structure with a language model. Science, 379(6637):
2023.
[56] Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei Ye, and Quanquan Gu. Structure- informed language models are protein designers. bioRxiv, pages 2023-02, 2023.
[57] Shengchao Liu, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nic, Anthony Gitter, Chaowei Xiao, Jian Tang, Hongyu Guo, and Anima Anandkumar. A text-guided protein design framework. arXiv preprint arXiv:2302.04611 2023.
(58) Xuezhi Xie and Philip M Kim. Helixgan: A bidirectional generative adversarial network with search in latent space for generation under constraints, 2021.
(59) Mostafa Karimi, Shaowen Zhu, Yue Cao, and Yang Shen. De novo protein design for novel folds using guided conditional wasserstein generative adversarial networks. Journal of chemical information and modeling, 60(12):5667-568, 2020.
[60] Yufeng Liu, Linghui Chen, and Haiyan Liu. De novo protein backbone generation based on diffusion with structured priors and adversarial training. bioRxiv, pages 2022-12, 2022.
[61] Bo Ni, David L Kaplan, and Markus J Buehler. Generative design of de novo proteins based on secondary-structun constraints using an attention-based diffusion model. Chem. 2023.
[62] Yifei Qi and John ZH Zhang. Denseepd:
improving the accuracy of www--nnetwwooeeo computational protein sequence design with densenet. Journal of chemical information and 2020.
modeling,
[63] Zhangyang Gao, Cheng Tan, and Stan Z. Li. Pifold: Toward effective and efficient protein inverse folding. In International Conference on Learning Representations, 2023. URL https / wtt.://orrumm..ddoaallltt...
[64] Carl Pabo. Molecular technology: designing proteins and peptides. Nature, 30:63)9)0 1983.
[65] Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thornoon, Cath-a hierarchic classification of protein domain structures. Structure, (8):1093-1109 1997.
[66] Stephen K Burley, Charmi Bhikadiya, Chunxiao Bi, Sebastian Bittrich, Li Chen, Gregg V Crichlow, Cole H Christie, Kenneth Dalenberg, Luigi Di Costanzo, Jose M Duarte, et al. Resb protein data bank: powerful new tools for exploring 3d structures of biological macromolecules for basic and applied research and education in fundamental biology. biomedicine, biotech- nology, bioengineering and energy sciences. Nucleie acids research, 49(D1):D477-D451, 2021.
13

---

[67] Andrew W Senior, Richard Evans, John Jumper. James Kirkpatrick, Laurent Sifre. Tim Green, Chongli Qin, Augustin Zidek. Alexander WR Nelson, Alex Bridgland, et al. Protein structure prediction using multiple deep neural networks in the 13th critical assessment of protein structure prediction (caspl3). Proteins: Structure, Function, and Bioinformatics, 87(12):1141- 1148, 2019.
[68] Lisa N Kinch, R Dustin Schaeffer, Andriy Kryshtafovych, and Nick Grishin. Target clas- sification in the 14th round of the critical assessment of protein structure prediction (casp14). Proteins:
Structure, Function, and Bioinformatics. 89(12).1618-1632 2021.
[69] Zhangyang Gao, Cheng Tan, and Stan Z Li. Knowledge-design:: Pushing the limit of protein deign via knowledge refinement, arXiv preprint arXiv:2305.15151, 2023.
[70] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022.
[71] Weian Mao, Muzhi Zhu, Hao Chen, and Chunhua Shen. Modeling protein structure using geometric vector field networks. bioRxiv, pages 2023-05, 2023.
[72] Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid Rector-Brooks. Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning, pages
PMLR, 2022.
[73] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58)) 2021.
[74] Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871-876 2021.
[75] Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolutonn de novo structure prediction from primary sequence. BioRxiv, pages 2022-00 07, 2022.
[76] Yang Zhang and Jeffrey Skolnick. Tm-align: protein structure alignment algorithm based on the tm-score. Nucleic acids research, 33(7):2302-230)) 2005.
14


Reviews:
[{"review_id": "DHnkZGhGCg", "reviewer": "Reviewer_wq41", "summary": "", "questions": "", "limitations": "see above.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Including (to my knowledge) all or most of the SOTA protein design tools is a robust benchmarking result for the broader community. Further, extending the benchmarks from single-chain to multi-chain prediction tasks also steps the work up a notch, as multi-chain applications are a challenging yet impactful area of development. The inclusion of benchmarks on multiple evaluation metrics is one of the paper's strongest contributions, as is the use of a broader number of datasets compared to previous work.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "tycWXBZNQC", "reviewer": "Reviewer_xYTD", "summary": "", "questions": "", "limitations": "There are no obvious limitations of this work. ", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper tests many structure-based models that are all cutting edge research tools. The evaluation criteria is clear and thorough. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "lbgMtEjwxX", "reviewer": "Reviewer_oPQ8", "summary": "", "questions": "", "limitations": "The authors have discussed potential limitations adequately.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The work is well-motivated. Deep learning bears the promise of solving protein inverse folding. However, it lacks comprehensive benchmarks to evaluate the performance of current and future models. \n2. The proposed metrics, especially sc-TM, provide an important evaluation of inverse folding other than recovery rate alone. \n3. ProteinBench covers a wide range of deep learning based inverse folding models as baselines. \n4. The work extends the current database (i.e., CATH4.2) to include more diverse and challenging tasks including CATH4.3, PDB, and CASP15. \n5. The authors provide a well-organized codebase to reproduce ProteinBench.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "c5PmEz7W1d", "reviewer": "Reviewer_HGkS", "summary": "", "questions": "", "limitations": "-\tThe confidence evaluation metric uses the average of prediction probabilities of the residues of a designed protein sequence to evaluate its quality. However, this metric can be method-dependent and may not be always applicable for method comparison. Some additional processing on the metric may be needed.\n-\tSome cutting-edge protein design methods such as RF Diffusion are not discussed and included.\n", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ Three datasets for protein design (CATH 4.3, PDB, and CASP15 de novo proteins) are provided. \n+ Multiple open-source protein design models are included into the benchmark framework.\n+ New/complementary metrics to evaluate protein design are added. \n+ scTM and robustness can be a useful metric to evaluate the quality of designed protein sequences. \n+ Several protein design methods are included in this benchmark. They were retrained and evaluated. \n+ The evaluation results in this work can serve as a baseline for the development of new methods. \n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "HQXtQ7uViq", "reviewer": "Reviewer_Gd38", "summary": "", "questions": "", "limitations": "The manuscript provides an appendix detailing potential limitations of the provided benchmark.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "## Significance and relevance to the broader research community\nPredicting amino acid identities from structure is an important problem, arising both in protein design and in recent advances in generative modelling of proteins. A large number of new methods have come out in recent years, but it has been difficult to track exactly how significant the progress has been. In my opinion, this is therefore a useful resource - and I expect it will have an impact on the Bio-ML subcommunity at NeurIPS, and more broadly on the many downstream users of these methods.\n\n## Quality of the research\nThe authors choose a number of datasets which are composed according to slightly different criteria. This makes it possible to contrast the performance of the different SOTA methods in interesting ways. Generally, I think these design choices are sound, and the results reveal meaningful differences between the methods.\n\n## Clarity of paper\nThe paper is clearly structured, and easy to follow.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The reviewers are enthusiastic about this work and consider it is a useful resource for users to train and test protein design methods, a solid evaluation framework for structure-based protein design, and a valuable benchmark for inverse protein folding. The review comments are also well addressed.
Decision: Accept (Poster)