ProteinBench: A Holistic Evaluation of Protein Foundation Models
Authors: name: Fei Ye; name: Zaixiang Zheng; name: Dongyu Xue; name: Yuning Shen; name: Lihao Wang; name: Yiming Ma; name: Yan Wang; name: Xinyou Wang; name: Xiangxin Zhou; name: Quanquan Gu
Published: 2024-09-10 06:52:33

Abstract: Recent years have witnessed a surge in the development of protein foundation
models, significantly improving performance in protein prediction and
generative tasks ranging from 3D structure prediction and protein design to
conformational dynamics. However, the capabilities and limitations associated
with these models remain poorly understood due to the absence of a unified
evaluation framework. To fill this gap, we introduce ProteinBench, a holistic
evaluation framework designed to enhance the transparency of protein foundation
models. Our approach consists of three key components: (i) A taxonomic
classification of tasks that broadly encompass the main challenges in the
protein domain, based on the relationships between different protein
modalities; (ii) A multi-metric evaluation approach that assesses performance
across four key dimensions: quality, novelty, diversity, and robustness; and
(iii) In-depth analyses from various user objectives, providing a holistic view
of model performance. Our comprehensive evaluation of protein foundation models
reveals several key findings that shed light on their current capabilities and
limitations. To promote transparency and facilitate further research, we
release the evaluation dataset, code, and a public leaderboard publicly for
further analysis and a general modular toolkit. We intend for ProteinBench to
be a living benchmark for establishing a standardized, in-depth evaluation
framework for protein foundation models, driving their development and
application while fostering collaboration within the field.