SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. KWDesign achieves the best performance across all the models for proteins of all lengths, followed by PiFold and ProteinMPNN.
2. The longer the protein sequence, the higher the recovery. Like the single-chain case, confidence and recovery generally increase with the length of the protein chain. As the length of multi-chain protein could be up to 1000, models perform better on the PDB than on the CATH dataset.
De novo Protein Design To investigate the models' potential in designing novel proteins, we evaluate pre-trained models on TS45. We present the de novo protein design results in Tab. 3, considering four subsets of TS45: FM (Free Modeling), TBM (Template-Based Modeling), TBM-easy, and TBM-hard. The results show that:
1. For models pre-trained on CATH4.2 and CATH4.3, KWDesign outperforms others by a large margin. The PiFold model consistently performs as the second-best model after KWDesign.
2. For models pre-trained on PDB, PiFold achieves the best performance, while ProteinMPNN provides very competitive recoveries. Switching from the CATH to the PDB dataset, ProteinMPNN achieves a more significant performance gain than PiFold.
3. The consistent performance trend across different protein subsets suggests that the difficulty level of the protein design task depends on the nature of the protein subset. For instance, models tend to perform better on TBM-easy proteins than on TBM-hard proteins. A more challenging subset of proteins may help reveal the shortcomings of current protein design algorithms.
4. AI methods have demonstrated great potential in de novo protein design, with all models (except StructGNN and GraphTrans) achieving recoveries of approximately  $40\%$  or higher. However, there is still slight performance degradation compared to the results on the original test set.
Table 3: Results of de novo protein design. The best and suboptimal results are labeled with bold and underlined.
Training set	Model	Confidence ↑	Recovery % ↑
FM	TBM	TBM-eazy	TBM-hard	Full	FM	TBM	TBM-eazy	TBM-hard	Full
CATH4.2	StructGNN	0.41	0.43	0.48	0.43	0.45	0.35	0.33	0.38	0.35	0.35
GraphTrans	0.39	0.43	0.46	0.42	0.44	0.33	0.30	0.37	0.36	0.36
GCA	0.48	0.43	0.53	0.48	0.50	0.39	0.37	0.41	0.38	0.40
GVP	0.48	0.49	0.50	0.50	0.49	0.37	0.33	0.42	0.39	0.39
AlphaDesign	0.44	0.41	0.50	0.46	0.48	0.41	0.36	0.46	0.41	0.42
ProteinMPNN	0.49	0.48	0.53	0.51	0.52	0.44	0.41	0.46	0.40	0.44
PiFold	0.52	0.46	0.59	0.53	0.55	0.47	0.38	0.50	0.47	0.47
KWDesign	0.55	0.52	0.70	0.62	0.64	0.49	0.40	0.59	0.55	0.54
CATH4.3	StructGNN	0.40	0.40	0.45	0.43	0.44	0.35	0.33	0.38	0.37	0.36
GraphTrans	0.39	0.42	0.46	0.43	0.45	0.35	0.32	0.37	0.35	0.35
GCA	0.46	0.42	0.51	0.44	0.48	0.37	0.33	0.43	0.40	0.41
GVP	0.47	0.45	0.50	0.48	0.49	0.37	0.31	0.41	0.38	0.39
AlphaDesign	0.44	0.40	0.50	0.47	0.48	0.40	0.36	0.44	0.44	0.42
ProteinMPNN	0.49	0.48	0.53	0.49	0.52	0.44	0.34	0.48	0.43	0.46
PiFold	0.54	0.45	0.56	0.51	0.54	0.47	0.38	0.52	0.49	0.49
KWDesign	0.59	0.50	0.70	0.63	0.65	0.50	0.43	0.59	0.60	0.56
PDB	StructGNN	0.46	0.41	0.53	0.47	0.48	0.39	0.34	0.42	0.41	0.41
GraphTrans	0.43	0.42	0.51	0.45	0.48	0.38	0.33	0.44	0.40	0.41
GCA	0.45	0.41	0.49	0.45	0.47	0.40	0.33	0.44	0.43	0.43
GVP	0.51	0.46	0.55	0.53	0.53	0.40	0.32	0.47	0.43	0.43
AlphaDesign	0.49	0.43	0.54	0.50	0.51	0.43	0.39	0.48	0.46	0.46
ProteinMPNN	0.56	0.49	0.58	0.55	0.55	0.52	0.39	0.55	0.51	0.52
PiFold	0.55	0.48	0.59	0.53	0.57	0.52	0.45	0.53	0.52	0.53
KWDesign	0.60	0.67	0.69	0.65	0.66	0.56	0.59	0.60	0.62	0.59
6.2 Diversity and sc-TM
Diversity We benchmark the diversity on TS45 dataset using models pre-trained on CATH4.3. As discovered by previous research [48, 53], the sampling temperature affects diversity. Denote the temperature as  $T$ , the predicted probability vector is  $p \in \mathbb{R}^{n,20}$ , we sample new sequences from the distribution of Multinomial(softmax  $(p / T)$ ). We vary the temperature from 0.0 to 0.5 and plot the trends of recovery and diversity in Fig. 2. Under the same sampling temperature, high recovery leads to decreased diversity. However, at the same level of recovery, stronger models have higher diversity.
Figure 2: The trends of recovery and diversity.
sc-TM While generating diverse protein sequences is appealing, it would be meaningless if these sequences could not fold to structures with topologies similar to the reference one. With this in mind, we investigate the recovery and sc-TM metrics as the temperature increases. To compute sc-TM, we utilize AlphaFold2 [73] to predict protein structures from sequences. According to Fig.3 and Fig.2, we observe that a slight increase in temperature from 0 to 0.1 is beneficial in significantly enhancing diversity while maintaining good recovery and sc-TM. However, increasing the temperature to 0.5 renders the designed sequences meaningless in recovery and sc-TM, despite the higher diversity. Carefully tuning the temperature would be beneficial in practical applications.
Figure 3: The statistics of recovery and sc-TM with increasing temperature.
6.3 Robustness and Efficiency
Robustness We further investigate whether the models are robust to structure perturbations, as the artificially designed structures may not be perfect, and the atom position may deviate slightly due to thermodynamic vibrations or errors in experimental measurements. We add different Gaussian noise to the input structure, i.e.,  $\mathcal{X} \gets \mathcal{X} + \epsilon \mathcal{N}(0, I)$ . Note that the Gaussian noise (in Angstrom) is added in both training and evaluation structures, where the noise scale  $\epsilon$  is chosen from [0.02, 0.2, 0.5, 1.0]. As we have shown that models perform consistently on CATH and PDB, we benchmark the robustness on CATH4.3. The experimental results are shown in Tab.4, from which we observe that:
1. Weaker models tend to exhibit greater robustness than stronger models. For example, with  $\epsilon = 1$ , the recoveries of StructGNN and GraphTrans decrease by only  $14\%$ , while AlphaDesign, ProteinMPNN, PiFold, and KWDesign decrease by at least  $20\%$ . This is a natural outcome, as weaker models may be the first to reach the performance floors of the dataset.
2. KWDesign achieves the highest recovery across noise scales, followed by PiFold. StructGNN, GraphTrans, GCA, and GVP degrade quickly and reach similar lower bounds. AlphaDesign is more affected by noise compared to GCA and GVP, likely due to its reliance on angular features, which are more sensitive to noise than distance features.
3. All models show a decrease in performance as the Gaussian noise scale increases. Developing protein design methods with higher robustness remains challenging.
Efficiency To encourage efficient and scalable models, we also benchmark the training cost, evaluation cost, the number of trainable parameters, and training epochs in Tab.5. We conclude that:
1. Training these models over a single epoch is generally fast, except for KWDesign (w/o memory). Fortunately, with the memory retrieval technique, KWDesign can achieve a similar speed as the other models. It is worth noting that PiFold and KWDesign require only up to 20 epochs to achieve competitive performance.
2. In terms of evaluation efficiency, iterative and one-shot models like AlphaDesign and PiFold perform exceptionally well, as they do not require autoregressive generation. On the other hand, KWDesign is relatively slower in this category as it needs to make multiple calls to large pre-trained models to generate higher-quality sequences.
3. Stronger models are associated with a higher number of trainable parameters. Among these models, GVP shows superior efficiency in utilizing model parameters. KWDesign achieves the best performance with the most parameters.
Table 5: Efficiency comparison.
Model	Training Cost	Evaluation Cost	Others
CATH4.2	CATH4.3	PDB	CATH4.2	CATH4.3	PDB	Trainable Params	# epochs
StructGNN	120s	112s	600s	378s	662s	1068s	1.38MB	100
GraphTrans	130s	123s	583s	438s	737s	1232s	1.53MB	100
GCA	660s	604s	1308s	378s	688s	1020s	2.09MB	100
GVP	402s	380s	840s	1874s	3193s	3753s	0.93MB	100
AlphaDesign	290s	267s	546s	31s	50s	75s	6.62MB	100
ProteinMPNN	165s	154s	540s	347s	570s	889s	1.66MB	100
PiFold	410s	364s	780s	39s	69s	162s	5.79MB	20
KWDesign(w/o memory)	3820s	3624s	-	451s	752s	-	54.49MB	20
KWDesign(w memory)	453s	437s	-	-	-	-	20
Protein inverse folding has received significant attention in recent years. However, the lack of thorough comparisons across multiple tasks and metrics hinders the progress toward practical applications. To address this issue, we propose ProteinInvBench, which consists of diverse tasks, models, and metrics and provides a comprehensive view of computational protein design. We plan to update ProteinInvBench when the CATH dataset (every 12 months) is updated.
[1] James W Bryson, Stephen F Betz, Helen S Lu, Daniel J Suich, Hongxing X Zhou, Karyn T O'Neil, and William F DeGrado. Protein design: a hierarchic approach. Science, 270(5238): 935-941, 1995.
[2] Bassil I Dahiyat and Stephen L Mayo. De novo protein design: fully automated sequence selection. Science, 278(5335):82-87, 1997.
[3] Yi-Xin Huo, Kwang Myung Cho, Jimmy G Lafontaine Rivera, Emma Monte, Claire R Shen, Yajun Yan, and James C Liao. Conversion of proteins into biofuels by engineering nitrogen flux. Nature biotechnology, 29(4):346-351, 2011.
[4] Stephen A Williams, Mika Kivimaki, Claudia Langenberg, Aroon D Hingorani, JP Casas, Claude Bouchard, Christian Jonasson, Mark A Sarzynski, Martin J Shipley, Leigh Alexander, et al. Plasma protein patterns as comprehensive indicators of health. Nature medicine, 25(12): 1851-1857, 2019.
[5] Kevin K Yang, Zachary Wu, and Frances H Arnold. Machine-learning-guided directed evolution for protein engineering. Nature methods, 16(8):687-694, 2019.
[6] Cheng Tan, Yijie Zhang, Zhangyang Gao, Hanqun Cao, and Stan Z. Li. Hierarchical data-efficient representation learning for tertiary structure-basedrna design, 2023.
[7] Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan Z Li. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering, 2021.
[8] Lirong Wu, Jun Xia, Zhangyang Gao, Haitao Lin, Cheng Tan, and Stan Z Li. Graphmixup: Improving class-imbalanced node classification by reinforcement mixup and self-supervised context prediction. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 519-535. Springer, 2022.
[9] Lirong Wu, Haitao Lin, Yufei Huang, Tianyu Fan, and Stan Z Li. Extracting low- high-frequency knowledge from graph neural networks and injecting it into mlps: An effective gnn-to-mlp distillation framework. arXiv preprint arXiv:2305.10758, 2023.
[10] Lirong Wu, Haitao Lin, Yufei Huang, and Stan Z Li. Knowledge distillation improves graph structure augmentation for graph neural networks. Advances in Neural Information Processing Systems, 35:11815-11827, 2022.
[11] Lirong Wu, Haitao Lin, Yufei Huang, and Stan Z Li. Quantifying the knowledge in gnns for reliable distillation into mlps. arXiv preprint arXiv:2306.05628, 2023.
[12] Hanqun Cao, Cheng Tan, Zhangyang Gao, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li. A survey on generative diffusion model. arXiv preprint arXiv:2209.02646, 2022.
[13] Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet better video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3170-3180, 2022.
[14] Cheng Tan, Zhangyang Gao, Lirong Wu, Yongjie Xu, Jun Xia, Siyuan Li, and Stan Z Li. Temporal attention unit: Towards efficient spatiotemporal predictive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18770-18782, 2023.
[67] Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Žídek, Alexander WR Nelson, Alex Bridgland, et al. Protein structure prediction using multiple deep neural networks in the 13th critical assessment of protein structure prediction (casp13). Proteins: Structure, Function, and Bioinformatics, 87(12):1141-1148, 2019.
[68] Lisa N Kinch, R Dustin Schaeffer, Andriy Kryshtafovych, and Nick V Grishin. Target classification in the 14th round of the critical assessment of protein structure prediction (casp14). Proteins: Structure, Function, and Bioinformatics, 89(12):1618-1632, 2021.
[69] Zhangyang Gao, Cheng Tan, and Stan Z Li. Knowledge-design: Pushing the limit of protein design via knowledge refinement. arXiv preprint arXiv:2305.15151, 2023.
[70] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022.
[71] Weian Mao, Muzhi Zhu, Hao Chen, and Chunhua Shen. Modeling protein structure using geometric vector field networks. bioRxiv, pages 2023-05, 2023.
[72] Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning, pages 9786-9801. PMLR, 2022.
[73] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589, 2021.
[74] Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using a three-track neural network. Science, 373(6557):871-876, 2021.
[75] Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pages 2022-07, 2022.
[76] Yang Zhang and Jeffrey Skolnick. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic acids research, 33(7):2302-2309, 2005.