ProteinBench: A Holistic Evaluation of Protein Foundation Models
Authors: Fei YE
Zaixiang Zheng
Dongyu Xue
Yuning Shen
Lihao Wang
Yiming Ma
Yan Wang
Xinyou Wang
Xiangxin Zhou
Quanquan Gu
Venue: ICLR 2025 Poster (2025)
Keywords: Protein foundation model
benchmark
protein design
protein conformation prediction
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: This work provides a holistic evaluation of protein foundation models.

Abstract: Recent years have witnessed a surge in the development of protein foundation models, significantly improving performance in protein prediction and generative tasks ranging from 3D structure prediction and protein design to conformational dynamics. However, the capabilities and limitations associated with these models remain poorly understood due to the absence of a unified evaluation framework. To fill this gap, we introduce ProteinBench, a holistic evaluation framework designed to enhance the transparency of protein foundation models. Our approach consists of three key components: (i) A taxonomic classification of tasks that broadly encompass the main challenges in the protein domain, based on the relationships between different protein modalities; (ii) A multi-metric evaluation approach that assesses performance across four key dimensions: quality, novelty, diversity, and robustness; and (iii) In-depth analyses from various user objectives, providing a holistic view of model performance. Our comprehensive evaluation of protein foundation models reveals several key findings that shed light on their current capabilities and limitations. To promote transparency and facilitate further research, we release the evaluation dataset, code, and a public leaderboard publicly for further analysis and a general modular toolkit. We intend for ProteinBench to be a living benchmark for establishing a standardized, in-depth evaluation framework for protein foundation models, driving their development and application while fostering collaboration within the field.

Published as a conference paper at ICLR 2025
PROTEINBENCH: A HOLISTIC EVALUATION OF PRO- TEIN FOUNDATION MODELS
Fei Ye, Zaixiang Zheng", Dongyu Xue", Yuning Shen', Lihao Wang' Yiming Ma, Yan Wang, Xinyou Wang, Xiangxin Zhou and Quanquan Gu ByteDance Research
{yefei. joyce, pooeee.caarraaan.ccuyeetteeeeeee
Project page:
ABSTRACT
Recent years have witnessed a surge in the development of protein foundation models, significantly improving performance in protein prediction and generative tasks ranging from 3D structure prediction and protein design to conformational dynamics. However, the capabilities and limitations associated with these models remain poorly understood due to the absence of a unified evaluation framework. To fill this gap, we introduce ProteinBench, a holistic evaluation framework de- signed to enhance the transparency of protein foundation models. Our approach consists of three key components: (i) A taxonomic classification of tasks that broadly encompass the main challenges in the protein domain, based on the re- lationships between different protein modalities; (ii) A multi-metric evaluation approach that assesses performance across four key dimensions: quality, novelty, diversity, and robustness; and (iii) In-depth analyses from various user objectives, providing holistic view of model performance. Our comprehensive evaluation of protein foundation models reveals several key findings that shed light on their current capabilities and limitations. To promote transparency and facilitate further research, we release the evaluation dataset, code, and leaderboard publicly for further analysis and general modular toolkit. We intend for ProteinBench be a living benchmark for establishing a standardized, in-depth evaluation framework for protein foundation models, driving their development and application while fostering collaboration within the field.
INTRODUCTION
Proteins are fundamental molecules playing pivotal roles in vast array of biological processes, from enzymatic catalysis and signal transduction to structural support and immune response. Their functions determined by their amino acid sequences, often mediated through folding into spe- cific three-dimensional structures. Understanding the complex interplay between protein sequence, structure, and function is crucial for advancing science and engineering spanning pharmaceuticals, agriculture, specialty chemicals, and biofuels (Kuhlman & Bradley) 2019)).
recent years, there has been a surge in the development of protein foundation model aimed at understanding fundamental biological processes by capturing the intricate mechanisms of pro- teins (Jumper et al.]|2021 [2021] Abramson et al. 2024 Lin et al.) [023a]] Watson et al.] [202bb] Ingraham
al. 2023 Krishna et al. 2024 et al. 2021 Madani et al. [ Alley et al. [219 Wang al.
2024b Hayes et al. 2024; Hie et al. 2024). These models, leveraging advanced deep-learning and
generative Al techniques, have demonstrated remarkable capabilities and mark significant shift from traditional, task-specific approaches to more generalizable frameworks capable of learning complex patterns and relationships within vast protein datasets. For instance, AlphaFold3 (Abram- son et al. 2024), which is based on diffusion models, has achieved unprecedented accuracy in full atom structure prediction for all biomolecules, while others like the ESM series (Rives al. 2021
Equal contribution. Corresponndig author.
In this study, we broaden the definition of protein foundation models to include any generative model aimed addressing foundational problems in protein science.

---

Published as a conference paper at ICLR 2025
folding conditional structure generation"
3D structure and Conformation dynamics
Amino acid sequence
sequence generation
structure generation
soquence-structure Co co-generation
inverse folding
conditional sequence generation
Figure 1: Comprehensive overview of funddaeenall protein modeling tasks in ProteinBench. ProteinBench incorporates a spectrum of protein modeling challenges. Tasks marked with blue stars highlight domains where standardized performance benchmarks were previously unavailable.
Hsu al. 2022 Lin et al. 2023a; Verkuil et al., 2022 Hayes al. 2024) and DPLM (Wang al. 2024b) have shown impressive representation capability in protein language modeling benefiting di- verse downstream tasks. Furthermore, these foundation models are not limited single modalities. Multi-modal models that jointly consider sequence, structure, and function are emerging, offering comprehensive understanding of protein behavior (Hayes al. 2024, Liu et al. 2023). addition to sequence and structural modeling, recent work protein conformational dynamics further ex- tends model capabilities, offering deeper insights into the connections from sequence to structural dynamics (and ultimately to function) through the lens of generative AI (Jing et al. 2023 Zheng
ct al. 2024 Jing et al. 2024 Wang et al. 2024c; Lu et al. 2024).
However, the rapid progress of protein foundation models has also led to an urgent need for a unified framework to holistically evaluate their performance across a diverse set of tasks, datasets, and met- rics, as shown in Appendix [A] The current landscape of protein foundation models is characterized by non-unified modeling approaches, and task-specific model-specifie evaluation criteria. This heterogeneity in evaluation methods makes challenging to draw meaningful comparisons between different models and to fully understand their relative strengths and limitations.
Through systematic evaluations of datasets spanning diverse biological domains, with particular emphasis on protein design and conformational dynamics, we aim to provide comprehensive anal- ysis of model architecture and performance protein foundation models. This approach allows us to dissect the impact various model components and data characteristics on different aspects of protein modeling. Comparing the capabilities of these models standardized benchmarks is crucial for guiding future research directions, informing model selection for practical applications, and driving the advancement of the field whole.
In this study, shown in Figure D present ProteinBench. the first benchmark designed to provide a comprehensive evaluation of protein foundation models through four key components:
(1) A taxonomic classification tasks encompassing the main generative challenges in the pro- tein domain. ProteinBench covers wide range of generative tasks, including protein design (span- ning structure design, sequence design, cenueturessqqeeeee co-design, and an application-spee -specfffc task of antibody design) and protein conformation prediction (single-state, multiple-state and con- formational distribution prediction). These tasks, addressing different protein modalities, enable a nuanced analysis of the interplay between model architecture and modal characteristics perfor- mance. We utilize diverse and carefully curated datasets to capture the complexity and diversity of the protein universe, ensuring a thorough evaluation of model capabilities.
(2) A multi-metric evaluation approach assessing performance across four key dimensions: quality, novelty, diversity, and robustness. Current evaluations of protein generative models of- ten suffer from con-uni metrics and incomplete assessments, typically focusing on only or two aspects. However, protein scientific problems encompass a complex and systematic array of challenges. Downstream tasks in protein modeling and design involve intricate interplays between sequence, structure, and function. ProteinBench addresses this limitation by providing a comprehen- sive measurement of model's ability based on four critical dimensions: quality, novelty, diversity, and robustness. This multi-faccted approach offers a more holistic view of model performance and capabilities.
2

---

Published as a conference paper at ICLR 2025
Table I: Overview of ProteinBench, which summarizes the dimensions, metries, and methods used in ProteinBench. We use 'italics' for highlighting, a method that has not yet been evaluated in ProteinBench but will be assessed in the future. Details regarding task selection, metric design principles, implementation specifics, and extended discussions are provided the corresponding sections.
Tasks
Dimension
Metries
Methods
Protete Devign
Sequence reoowey
AAR
ProteinMPNN, ESMIFI,
Inverse Felding (Section
Refoldability
(AF2))
LM Desiga, ESM3 Pifvdd, CurbenDDsigg
Subility
PLDDT
Robusiness
Raclbon: Desien (Section 0.12 B.11
Quality
Novelly
Max. TM score to PDB danisuse (Foldseek)
scRMSDD (PoscinMPNN ESMIok))
Rediftusion, Francfloww Chroma,
Framedift, Foldflow, Genie, Proteus
Divenuty
Pairwsee TM, Max Cluster (Foldseek)
foldingdijf
Sequence Desion (Section B.11 B313
Novelly
Quality
pl BLDDT (AF2)
Max. TM to PDB dasabuse (Foldseek)
DPLM, ESMB
ProCie2, EvaDiff
Diversity
Pairoise TM, Max Cluste (Poldicek)
Stract-spa Cooiesign (Section
Novelly
Quality
Max. TM score PDB danabase (Foldseek)
cTM, saRMSDD (ESMFad)
ProsinCienenttoe ProtPardelie, Multiflw,, ESMS, CarbonNovo
Diversty
(Poldseok)
Motif Seafisldigg (Section B.1.5
Quality
Moti Scafold RMSD
FrameFlow, Ridiffusion, EveDiff.
Accuracy
RMSD,
Antibads Desion
(Section 6.16
Rationality
Specificity
Functionality
CN Score, Clashes, Seq Naturaleess
Binding Eisergy (Rosetta)
Similanity PHR
DINAA, ANDPO
dvMEAN,
Tonal Eseigy (Rosctia), xRMSD (IFFlld)
Prorete Conformattin Prediction
Singlostate (olding)
Accency
TM-aoon, RMSD, GDT. IDDT
AlyiaFaa22, OpenFold, FSMFiM. RoseTTAfoMd2. Eigen
(Section B.2.1
Quality
CA clash, Peptide bond break
Multiple-ttat (Section
Ensemible TM-soxe/RMMD clash, Peptide bond
DigenFold. MSA subsampling. St.25n. Aigtu
Quality
Distribution Prodiction (Section B.2.3
Accumcy
Divensty
Pairwise RMSD, RMSP
Flexibility securacy, Distrbutional similarity Ensemble observables ConfDiff, Dismitoted grophomser
Quality
CA clash, Peptide bord break
(3) In-depth analyses from various user objectives, providing a holistic view of model per- formance. Recognizing that different users may have varying objectives when applying protein foundation models, we conduct in-depth analyses from multiple perspectives. For instance, in pro- tein design, some users may prioritize models that fit natural evolutionary distributions, while others may seek models capable of generating novel proteins outside the training set distribution. By an- alyzing model capabilities from these different objectives, ProteinBench provides insights that are beneficial for a wide range of practical applications.
(4) Leaderboard and code framework. To facilitate fair comparisons and support the development of new methods, we provide unified experimental framework. This includes public leaderboard and open-source code, enabling researchers to easily benchmark their models against existing ones and contribute to the ongoing advancement of the field.
By incorporating these four components, ProteinBench aims to establish standardized, compre- hensive, and user-centric evaluation framework for protein foundation models. This approach not only illuminates the current state-of-the-ar but also guides future research directions and accelerates progress the field of protein modeling and design.
2 PROTEIN BENCH
In this section, provide ProteinBench, holistic evaluation framework for protein foundation models as shown in T with a particular focus on two key generative tasks: protein design and conformation prediction. These two areas are further divided into eight subtasks. More details about the task definitions, evaluations, and discussions can be found in Appendix
2.1 PROTEIN DESIGN
We evaluate protein foundation models six design tasks using standardized metrics, enabling cross-task comparisons and assessment of different modeling approaches.
2.1.1
INVERSE FOLDING
Inverse folding is fundamental task aiming to design amino acid sequences that can fold into predetermined structures. Evaluations were conducted on different datasets targeting two distinct objectives of structure-based sequence design: evolutionary distribution capturing and de novo pro- tein design. For more detailed information, please refer to Section B.I.I Evaluation results are presented in Table 2 Our analysis of native distribution fitness reveals that language model-based
3

---

Published as a conference paper at ICLR 2025
Table 2: Performance of structure-based sequence design models on inverse folding tasks. The reported results are the median of repetitive experiments. 'N/A' stands for not applicable. ESMIFI and ESM3 use all native structures and sequences for model training, therefore, they not measured in the cvolution distribution fitting objective.
Objectives Fitting Evolution Distribution
De novo backbones based sequence design
CASP
CAMEO
length 100
length
longth 300
length
longth 500
AAR
AAR
scTM
PI.DDT
scTM
pL.DDTT 7 scTM 7 pl.DDT
Tc pL.DIIT 7 scTM
pI.DDT 1
ProteisMPNN
0.450
0.468
0.962
94.14
0,945
89,34
0.962
90.28
83.76
0.568
ESM-IFI
NIA
NIA
0.810
88.83
0.635
69.67
0.336
74.36
0.449
64,59
0.462
58.97
LM-DESIGN
0.516
0.570
0.834
78.45
0:373
58.41
0,481
86
59.87
0.397
ESM3
NIA
N/A
0.942
86.60
0.486
60.69
0.632
70.78
0.564
62.63
0,452
59.37
method LM-DESIGN (Zheng et al. 2023) achieve high sequence recovery rates for native protein
structure-based sequence design. This suggests that these models effectively learn and replicate the intricate patterns of amino acid selection that have emerged through evolutionary processes. While sequence design. Conversely,
its performance decreases when applied to de novo backbone-base ProteinMPNN (Dauparas et al. 2022a), a method specifically developed for de novo design and trained using coordinates perturbed with 0.2À added noise, consistently demonstrates superior per- formance in de novo design tasks. However, ProteinMPNN's performance shows a decline when evaluated on the objective of fitting native evolution. This finding suggests no single model cur- rently excels across all inverse folding objectives. The choice of model should be carefully aligned with the intended applications.
ESM-IF1 (Hsu et al. 2022), trained AlphaFoldDB (Varadi et al. 2022) using GVP (Jing al., 2020 and Transformer architectures with O. IÅ noise, showed suboptimal performance in de novo backbone sequence design. While the model excels functional mutation prediction as demon- strated in ProteinGY'M (Notin et al.] 2024). these tasks were not included in our study. The recently released ESM3 (Hayes et al.,, 2024) performs similarly to ESM-IFI, with advantages at specific sequence lengths (100, 300, and 400 residues). PiFold (Gao et al. 2022) and CarbonDesign (Ren) al. 2024) are not currently included in ProteinBench.
2.1.2 STRUCTURE DESIGN
Protein backbone design focuses on generating new protein folds to expand the repertoire of protein structures beyond those found in nature. Further details are provided in Section B.1.2 Evaluation
results are presented in Table and Table [1] Based on the quality metrics of McMM-e and
scRMSD, RFdiffusion (Watson al.) 2023b) demonstrates exceptional performance in backbone design for chain lengths ranging from 50 to 300 amino acids. FrameFlow (Yim et al. 2023) achieves the second-best performance in this range. However, we observe significant performance decrease across all models for longer chains (500 amino acids), with scTM scores dropping by more than 20%. This decline suggests that developing methods for long-chain backbone design remains an important challenge for future research. Proteus (Wang et al., (024a) shows a superior design qual- ity for long-chain backbones (500 residues). However, we observed significant decline in structure diversity for long chains. Novelty is an equally important metric, as it gauges method's 's capacity to explore new structural space beyond known protein folds. Under moderate quality constraints (scTM score >0.5). FoldFlow (Bose et al. 2023) and Genie (Lin & AlQuraishi] 2023) exhibit good performance in generating novel structures. When we increase the quality threshold (scTM score >0.8). Chroma (Ingraham et al. 2023) generally shows the best performance across chain lengths from 50 to 500 amino acids. For structural diversity, Chroma shows commendable performance across the tested chain lengths. It is important to note that we used the released FoldFlow model trained on a smaller training set with shorter sequences. This limitation may lead to an unfair com- parison of the model architecture other methods trained on the entire PDB database, particularly for longer chain lengths. Foldingdiff (Wu et al. 2024a) is not featured in ProteinBench.
2.1.3 SEQUENCE DESIGN
Protein sequence design aims to generate amino acid sequences with desired properties. See Section B.1.3 for details. In this section, we assess the performance of various protein sequence generative models. The evaluation metrics include AlphaFold2 (AF2) predicted pLDDT scores for structural plausibility (quality), maximum TM-score and maximum cluster values for structural diversity, and maximum TM-score to PDB structures for structural novelty. We choose representative methods of distinct modeling foundations for evaluation. Among the methods evaluated, ProGen2 (Nijkamp) et al. 2023) is an autoregressive protein language model (AR-LM), while EvoDiff (Alamdari al.
4

---

Published as a conference paper at ICLR 2025
Table 3: Performance of backbone design models evaluated using various lengths ranging from 50 to 500. The reported results are the median of repetitive experiments. We highlight the best performance in bold and the second-best with the underline. For the novelty and diversity metries, we only highlight results with the corresponding scTM score higher than 0.5. 'N/A' stands for not applicable.
length
length
Quality
Novelly
Diversity
Quality
Novelty
Diversity
scTM
scRMSD
Max TM & pairwise TM
Max Clust.
scTM
scRMSD
Max IM 1 pairnise TM
Mux Clust.
Native PDBs
0
0774
N/A
0:29:0033
0.66
0.96::
0.67:1161
N/A
0.30=0002
0.77
RFdiffusion
0.95100.1
0.45111.71
0.6510.16
0.58.10.05
0.67
0.9811..12
0.48
0.76110.01
0.412:0.33
0.32
FrameFlow
0,9120009
0.5820.51
0.75:0.01
0.68:00100
0.39
0.96::0.08
0.7020070
0.72-:0.01
0.55=:0.08
0.49
Caroma
0.85100.15
1.05&1499
0,5910.06
0.29110.01
0.48
0.89110.13
12711.85
0.70.10.01
0.352:0.03
FrameDidf(latet)
0.851:0.13
1.0
0.67:0.01
0.352:0.02
0.61
0.90100.0
1.23:11.02
0.71:10.08
0.52200.05
0.11
FoldFlow ((sta))
0.90=:0.10
0.67+00.88
0.68:+0.03
0.63:200.07
0.48
0.87=+0.11
1.34::142
0.65:200001
0.49::0.08
0.83
FoldFlow l(base)
0.7910014
1.1911,27
0.6610.02
0.53:10.08
0.76
0.8110.15
1.70
0.62:10.01
0.481:0.07
0.83
FoldFlow (oe)
0.83+0.16
1.10+1153
0.66++0.02
0.53+000M
9.22
0.83+0
1.00+1.95
0.64+#000
0.48+0006
Genie
0.57::0.15
3.124:2.07
0.57
0.02
0.90
0.69
3.38::3.04
0.59:10.01
0.31::0.02
0.96
Proceus
0:94+0.99
0.57+1.115
0.78+0.10
0.96-+0.17
0.15
0:9400112
0.84+00.52
0,73++0.10
0.57=00.88
0.5
length
length 500
Quality
Novelty
Diversity
Quality
Novelty
Diversity
xIM
scRMSD
Max TM L pairwise IM
Max Clust.
scTM
sRRMSD
Max IM 1 pairmise TMJ
Max Clust.1
Native PDRs
0.97+0.10
0.82426677
N/A
0:28+00:22
0.77
0.97+0011 17
10744596
N/A
0.29=003
REdiffusion
0.96:20.15
103::3.14
0.0100011
0,36::0.03
0.65
0.79+ 19
5.6015.66
0.62=00.044
0.33=0002
0.89
Frow
0.92::0155
1.95+22766
D655+0.
0.43=:0.07
0.88
0.61 19
7:92-14.08
0.61-10011
0.40:00066
0.92
Caroma
0.8710.13
2.47.3..33
0.6611.00
0.3620.04
0.67
0.7211..15
6.7115.76
0.60:1001
0.29110011
FraneDii((ates))
0.87:00122
2.73:22.69
0.69:0000
0/48:00004
0.21
0.63:20.24
9.52:118.19
0.58:10.03
0:40::0.06
0.52
FoldFlow ((sm))
0.4510.111
9.04.22.52
0.54.10.01
0.39:10.04
1.00
0.37.00005
13.04
0.53-10.01
0.37::0.03
1.00
FoldFlow ((base)
0,4310009
9.56.12.42
0.51100.11
0.391:0:05
0.98
0.35100005
23.20.12.29
0.52:10.01
0.391:0.05
1.00
FoldFlow ((e))
0.54:0.12
8.21++2.38
0.58::0.00
0.1::00066
0.94
0.37+0.06
12.48:4-2.00
0.51::0:11
0.35=-0.03
1.00
Genie
0.27::0.02
20.37711.70
0.30110011
0.23:10.01
1.00
0.2510.01
26.08.11.58
0.22:10.02
0.23-00004
1.00
Proceus
0.94+0.05
1.46+1008
0.78+00.05
0.62:40.09
0.34
0.90=+0.13 2.76+:3.57
0,72+40.02
0.62=00.99
0.34
Table 4: Performance of protein sequence generative models/language models on sequence genera- tion tasks. The reported results are the average of repetitive experiments with the standard derivation. The PLDDT score is the output of AlphaFold2. Max TM is an abbreviation for Maximum TM-score to PDB database. 'N/A' stands for not applicable. We highlight the best performance in bold.
length 100
leagth 200
Quality
Diversity
Novelty
Quality
Diversity
Novelty
ppl
PLDDT
pairwise IM
Max Clast.
Max TM
ppl
PLDDI1
pairuise TM J Max Clust.
Max TM
Sogs
68.46116650
0.551:0:19
0.75
0.0910.10
0.78
NIA
Prugen (700M)
8.28.13..8
61.001.21.26
0.42.10110
0.91
0.641:0.8
5.66
69.9111923
0.1010.13
0.91
0.69.0.05
EvoDiff
16.8 1.04
10.27
0.43:00.05
0.98
0.69:10.03
17. 17.2811.6
10.66:116.38
0.36:10.04
1.00
0.71:00022
DPL M (650M)
6.21:33.10
15.38:14.20
0.50-20.00
0.80
0,7400100
2.6+
93.54+3.73
0.54:0.34
0,70
001-00004
ESMB (1.4B)
14.79.2.90
51.26115.35
0.1510.15
0.90
0.6810.07
12.9612.38
58.4511.40
0,35100007
1,00
0.80.0.01
length 300
length 500
Quality
Diversity
Novelty
Quality
Diversity
Novelty
ppl
PLDDT'
paiewise IM.
Max Clast.
Max TM
ppl.
PLDDI!
pairwise TM J Max Clust. 1 Max TM L
Native
61.49414.47
0:51+0:13
0.85
N/A
62,95+12660
D.51+0.111
0:78
NIA
Proges (700M)
6.25+ 4.02
65.69+20.93
0.42+0.16
0.66=0006
4.743.60
61.45+200117
0.52+0.11
0.95
EvoDif
17.13:2.00
45.1411995
0.31:0003
1,000
0.68:00.22
16.51
43.141.5.16
0.3110033
1,00
0.69:0002
DPI M (650M)
3447:1.44
93.07-55.77
0:57::025
0.63
0.11:0011
3.38-188
17.78+11.61
0.43+00.18 18
0.85
088-:004
ESM3 (1.4B)
14.99=2:97
48.134+11.34
0.32+0.03
1.00
0.75+0.02
11. 11.0022.26 52,17+10.52
0,30++0.55
1,00
0.54-0.03
2023 is designed as an arder-agnostic autoregressive diffusion model (OADM). DPLM (Wagg et al. 2024b) and ESM3 (Hayes et al. (024) share probabilistic foundation absorbing discrete diffu- sion models or generative masked language models. Notably, ESM3 is multimodal model that eeeeeneeeueeee
advances beyond other sequence-only methods by jointly learning protein sequences, structures and functions through tokenization. For each model and sequence length, we sample 50 sequences to evaluate their performance.
As shown in Table 4 DPLM consistently shows the highest quality scores, indicating superior ac- curacy in sequence generation. However, it has relatively lower diversity metrics, suggesting less variation in its generated sequences. EvoDiff, while demonstrating lower PLDDT scores, excels in diversity, particularly in producing highly diverse sequence clusters. Surprisingly, ESM3, a multi- modal protein LM, displays lower pLDDT in sequence generation, while maintaining competitive diversity, especially in generating novel sequences. ProGen2 strikes a balance between quality and diversity, offering moderate PLDDT scores and satisfactory diversity and novelty. This model is effective for generating sequences that are both diverse and close to known structures, depending on specific application needs. Regarding different chain lengths, all the models generally exhibit consistent trends in their performance metrics. As the chain length increases, there is slight de- cline in the quality of sequences generated by some models, particularly for Diff and ESM3.
5

---

Published as a conference paper at ICLR 2025
Table 5: Performance of protein co-design models on structure-sequence co-generation tasks. The reported results are the average of repetitive experiments with the standard derivation. We highlight the best performance in bold and the second-best with the underline. *: We have tried our best to reproduce all models according to the instructions in their respective codebases, using publicly available model weights. However, some results may differ from those reported in the original studies. We welcome any feedback and corrections to help us make timely updates in the future.
length 100
length 200
Quality
Diversity
Novelty
Quality
Diversity
Novelty
scTM
scRMSD
Max Clust.
Max TM 4
scTM
seRMSD
Max Clust.
Max TM
Native PDBs
0.91::0.11
2.98:13.49
0.75
N/A
0.88:+0.09
3.24:13.77
0.77
N/A
ProteinGenerator
0.91=:0.08
3.75
0.24
0.73
0,88:10.09
6,2444410
0.25
0.72
ProtPurdelle
0.91:10.09
1.07
0.73
0.16
0.92:1:.04
2.36:t1.19
0.09
0.75
Multiflow
0.96++0.04
1.10+:0.1
0.33
0.71
0,95++0.04
1.61+1773
0.71
Carbonnovo
0.91=:0.14
1.16+1.003
0.71
0.69
0.94:10009
1.18::1.47
0.50
0.71
ESM3*
0.72::0.19
13.80 10.51
0.64
0.41
0.63:+0.20
11.18:::6.19
0.63
0.61
length 300
length 500
Quality
Diversity
Novelty
Quality
Diversity
Novelty
scTM
scRMSD 1
Max Clust. t Max TM
scTM
seRMSD
Max Clust.
Max TM
Native PDBs
0.92++0.12
3.94+44.5
0.75
N/A
0.90
9.64
0.80
N/A
ProteinGienerator
0.81:10.14
9.26:1:.13
0.22
0.71
0.41410.9
33.91::15.10
0.18
0.73
ProtPardelle
0.94:10.02
2.07=:0.33
0.05
0.73
0.41+00.10
41.10=8.85
0.65
Multiflow
0.96:10.06
2.14::3:24
0.58
0.71
0.83::0.15
8.48:17.02
0.67
0.68
Carbonnovo
0.95:100008
1.53
0.31
0.74
0.85:10.15
4.07:14.114
0.67
0.68
ESM3
0.59:50.21
25.5:120.68
0.52
0.73
0.54::0.20
33.70:121.08
0.37
0.77
This indicates a challenge in maintaining high sequence quality as the chain length grows. Among them, DPLM demonstrates robust performance across all lengths, maintaining high pLDDDT even for longer sequences. Overall, DPLM is good at highly structural protein sequence generation, while EvoDiff and ESM3 are preferable for better diversity and novelty, with ProGen2 offering balanced performance across metrics.
2.1.4 STRUCTURE AND SEQUENCE CO-DESIGN
Protein structure-sequence co-design represents challenge in protein engineering that involves the simultaneous generation of both backbone structure and amino acid sequence. We evaluate the struc- tural quality, novelty, and diversity similarly to backbone design. See Section 1.1.4 for details. in Table 5 We inspect the performance of ProteinGenerator (Lisanza et al. 2023). ProtPardelle (Chu et al. 2024), Multiflow (Campbell et )..). CarbonNovo (Ren et al. 2024) and ESM3 (Hayes et al., 2024 for different lengths. The performance is assessed using metrics similar to those applied in backbone generation. Note that, however, the quality here is about structure- -euucuur-sequueeeee sequence compatibility measuring how well the designed sequence can fold into the corresponding designed structure, using seTM and scRMSD. The key difference is that co-design models are tasked with simultaneously gen- erating both the sequence and structure, while backbone design models require an additional inverse folding model, such as ProteinMPNNN to design the sequence. Other metrics used for evaluation include diversity (max cluster) and novelty (max TM-score to PDB).
As shown in Table ProteinGennerator, ProtPardelle, CarbonNovo, and Multiflow consistently show strong performance of pruucure-sequenee compatibility when length less than 300, with high scTM scores (up to 0.96+0.06) and relatively low scRMSD values, indicating superior structural qual- ity in generated sequences. ProtPardelle and ProteinGenerator particularly excel at shorter lengths while failing at long proteins. CarbonNovo and Multiflow maintain high performance even as se- quence length increases, demonstrating their robustness with consistently high scTM scores and lower scRMSD values, which indicates their capability to generate high-quality structures. ESM3, on the other hand, shows suboptimal performance by its low scTM scores and very high scRMSI) values, suggesting that it struggles with unconditional generation. Overall, these findings suggest that CarbonNovo and Multiflow are particularly robust and superior as a co-design protein generative model across all tested lengths.
2.1.5 MOTIF SCAFFOLDING
Motif scaffolding represents specialized challenge in protein design that focuses creating pro- tein structures incorporating specific functional motifs or binding sites. See Section B.1 for more details. In this section, we evaluate the performance of various motif-scaffolding methods across dif-

---

Published as a conference paper at ICLR 2025
Table 6:
Performance of antibody design methods on 55 antibody-antigen complexes from the RAbD dataset. For methods that can generate multiple antibodies (marked with *), the standard deviations between different antibodies generated against the same antigen are also reported.
Accuracy
Functionality Binding Energy
Specificiity
AAR
RMSD
TM-seore
|SegSim-outer
SeqSim-Inner
PHR
RABD (natural)
100.00%
0,00
1,00
15.33
0.26
N/A
45.78%
HERN
33.17%
9.86
0.16
1242.77
0.41
N/A
39.83%
MEAN
33.47%
1.82
0.25
263.90
0.65
N/A
40.74%
2.36
0.36
889.28
N/A
42.04%
MyMEAN-FiXFR
40.05% 1.06
2.37400.33
0.35=0.01
612.75=56.03
0.60
0.96
43.75%-+22.44
"DilAb
8..0
2.53+00.00
0.37=0.06
489 489.42-499.76
0.37
0.45
40.68%
AADDP
71.29%
2.79:33.01
0.35=0.06
116.06 186.06
0.38
0.60
69, 89.699--+88. 8,
*AbDPO++
36.25%
2.48:00599
0.35=0.06
223.73=281.7
0.39
0,54
44.51%:29.55
Rationality
CN-score
Clashes-outer Clashes-inner
SeqNatf
Total Energy
scRMSD
RAbD (natural)
50,19
0.07
0.00
-1.74
-16.76
1.77
HERN
0.04
0.04
3.25
-1.47
5408.74
9.89
MEAN
1.33
11.65
0.29
-1,83
1077.32
2.77
dyMEAN
1.49
0.47
-1.79
1642.65
2.11
"LMEANNFFAFR
114.11.71
8.88
0.48 12
-1,82:10.10
1239 19
2.48-0.24
"DiffAb
2.02+22.83
1.8
0.19=0.31
-1.88=0.47
495, 6300099
2.57=0.77
*ADDPO
1.33:12.31
4.141:184
0.10=0.24
-1.99210.34
270.12::11744
2.79=3.25
*ANDPO++
2.34.13.20
1.66
0.20
.1.78.10.33
338. 14 266.48
2.50....75
ferent scaffolds used in Watson et al. in designing scaffold structures. The primary objective of this evaluation is compare the efficacy
(2023b) and Yim et al. (002)), focusing on their effectiveness
of structure-based and sequence-based approaches in generating designable scaffolds. For purely
sequence-based methods, e.g., EvoDiff (Alamdari al. 2023) and DPLM (Wang et al., 2024b), we
use ESMFold to predict the structures of their designed motif-scafool sequences.
Figure 2: Performance of motif-scaafolldng of structure-basse and sequence-based methods the benchmark used in Watson et al. (2023b). Results of FrameFlow, RFDiffusion and TDS are quoted from Yim et al. 2024
Figure 2 reveals wide range of performance levels among the tested methods, each exhibiting dis- tinct strengths and weaknesses depending the specific scaffold context. Notably, structure-based
methods such as REdiffusion (Watson et al. 2023b), TDS (Wu et al. 2024b) and FrameFlow (Yim) scenarios, with RFdifuusion showing particular
et al. 2024) consistently perform well across robustness in generating high number of designable scaffolds. This suggests that structure- basedd methods highly effective at capturing the intricate structural details necessary for successful scaffold design. In contrast, sequence-based methods like EvoDiff and DPLM display variable per- formance, excelling in certain scaffolds that primarily governed by evolutionary constraints, but underperforming in others with more complex structural motifs. This variability may reflect their limitations in recognizing and adapting to specific structural features.
Interestingly, find that ESM3 (Hayes al. 2024). recent multimodal language model that handles both sequence and structure, performs similarly to advanced structure-based models and even succeeds in cases where those models fail. This shows that multimodal models like ESM3 can process both types of information within a unified framework, making them useful for conditional design. However, ESM3 still doesn't 't outperform structure-based methods in most cases. Our results highlight that no single model is best for all scaffolds. Future research should focus on combining these methods to leverage their strengths for more effective protein design.
2.6 ANTIBODY DESIGN
Antibody designing aims to generate antibodics that specifically bind to target antigens (details refer to B.1.6 Task Definition]). In this section, we selected five antigen-specific antibody design meth-
scaf folding

---

Published as a conference paper at ICLR 2025
2022). MEAN (Kong al. 2022), dyMEAN (Kong et al. 2023). DiffAb Luo al. 2022). A (hou et al. 2024))am two of their variants (dyMEAN-FixPP) and AbDPO++) to evaluate their performance in CDR-H3 generation towards the given antigens. All methods were trained on the same dataset with the configurations reported in the corresponding papers and tested al. 2018) (further details of model implementation and data construction are provided in Appendix B. [Model Implemen- tations]) The final results of the evaluation are shown in Table 6 For each evaluation metric, we highlighted the best performance in bold and the second-best with the underline, the detailed con- cept and implementation of each metric can be seen in Appendix |...6 | (Evaluation Metrics].
ods (HERN (Jin et al.
on unified set of 55 test cases from the RAbD dataset Adoof-Brytogle
In the Accuracy evaluation, dyMEAN, and MEAN achieved the best performance in terms of quence and structure (highest AAR and lowest RMSD), while DiffAb performed best in TM-Score.
In the Functionality evaluation, all methods produced antibodies with binding energies to the given antigens significantly higher than those of natural antibodies. AbDPO and AbDPO++ achieved the best performance among all the methods.
In the Specificity evaluation, we mainly observed the sequence similarity between antibodies against different antigens (SegSim-outer) and the proportion of hydrophobic residues in the generated an- tibodies (PHR). The former metric indicates whether the method can design antibodies specific to given antigen and DiffAb, AbDPO, and AbDPO++ achieved better performance, while the latter reflects the potential nonspecific binding due to high hydrophobicity and HERN performed best.
In the Rationality evaluation, we mainly observed: structural rationality, sequence rationality, and joint structural and sequence rationality. AbDPO++ performed best in structural rationality with fewer clashes and reasonable peptide bond length. HERN performed best in sequence rationality with the highest naturality. AbDPO and dyMEAN perform best in joint structural and sequence rationality from two perspectives. Details the Specificity and Rationality evaluation refer to Ap- pendix [. Explanations and Discussion Model Performance].
In general, evaluating antibody design methods encompasses various aspects, and using only a few metries will seriously mislead researchers' understanding model performance. No single method outperformed all others across the board, and all methods showed substantial gaps compared to natu- ral antibodies. However, AbDPO++, by utilizing synthetic data and aligning with various properties, achieved one of the best performances in almost all metrics among all methods.
2.2 PROTEIN CONFORMATION PREDICTION
Protein conformation prediction infers the 3D conformations of proteins from their sequences, eval- uating models based on their understanding of structure, dynamics, and ultimately functions. We begin by benchmarking common folding models, as they play a crucial role in the development of conformation prediction models. We then compare five recent studies that explore different strate- gies to extend folding to conformation prediction, focusing on multiple-state and distribution pre- diction tasks. A detailed discussion of task definition, evaluation, and results can be found in Section
B.2
2.2.1 PROTEIN FOLDING: UNGLE-STATE PREDICTION
Protein folding task predicts the 3D structure of a protein based on its sequence. This sequence-to- structure prediction is critical measure of model's understanding of these modalities. See Section B for more details. As shown in Table 7 Multiple Sequence Alignment folding models (AlphaFold2., OpenFold. TTsFoo))) outperform protein language model-based approaches (ESMFold, EigenFold). While the predicted structure quality is comparable among all- atom resolution models, AlphaFold2 and its reproduction, OpenFold, achieve the best performance across all accuracy metrics, offering strong foundation for conformation prediction.
2.2.2 MULTIPLE-STANE STATE PREDICTION
Multiple-state prediction builds upon single-state prediction by aiming to accurately generate two or more distinct conformational states of protein, typically associated with functional conformational changes. Further details can be found in Section B.2.2 Regarding evaluation, we first investigate the results on BPTI with the best model of each study highlighted in Table g and the complete eval- uation provided in Table B The classifier-fres guidance in ConfDiff (Wang et al. 2024c) achieved performance comparable to fine-tuning on MD conformation data and, when combined with force guidance, delivered the best overall accuracy (RMSDens).. This suggests that incorporating struc-
8

---

Published as a conference paper at ICLR 2025
Table 7: Performance of protein folding on CAMEO2022. Results are reported as mean/median over 183 proteins. The best performance is highlighted in bold, and the second-best is underlined.
EigenFold only predicts CA coordinates and PepBond break % is not available (shown in "N/A").
Accuracy
Quality
TM-score
RMSD
GDI-TS
IDDT
CA clash (%) J PepBond break (5)
AlphaFood2
0.871/0.952
3.21/1.64
0.860/0.921
0.904/0.933
0.3/0.0
4.8/4.1
OpenFold
0.870/0.447
3.21/1.59 0.836/0.913
0.899/0.933
0.4/0.0
2.0/1.7
RoseTTAFold2
0.859/0.941
3.52/1.75
0.845/0.903 0,892/09926
0.3.0.0
5,5/4.0
ESMFold
0.847/0.929
3.98/2.10 0.826/08881 0,870/09007
0.3/0.0
4.7/3.4
EigenFold
0.743/0.823
7.65/3.73 0,703/0.781 0.737/0.810
8.04.6
N/A
Table 8: Performance of multiple-state prediction on BPTI. Accuracy metrics (RMSDens, RMSD
Cluster 3) are reported as the mean and standard deviation from 20 bootstrap samples at different sample sizes. Diversity and quality are evaluated based on 1,000 conformations.
RMSDens (A).
RMSD (Á) Cluster 3
Diversity
Quality
Pairwise RMSD
N-100
N-1000
N-10
N-100
N-1000
clush%
PepBond break S..
FigenFoold
1.56-0.02
L50+0.01
146++000
2.54+0.03
2.48+0001
2.46+00.11
0.85
144
N/A
MSA-deptha2
1.66=0003
1 1.544004
1.4
243:00.6
2.19:0.16
185=0.05
2.14
0.6
10.6
Sw22tr-ODF (Tio 0.15)
2.40=0012
2 2:20+0.05
2.09:0011
3.00++.20
2.73++0.12
2.58+0.05
1886
0.0
13.9
ESMFlow MDD
147.10044
1.39-0.03
244.10.11
2.27:0.10
2.10:0022
1.17
SerfDiffEESM-Frce
1.58=0.04
1.43:10033 0.36=0001 2:44+0.06 2:35:0005 2.24:0.06
1.76
0.1
8,9
tural exploration and physical constraints can enhance the sampling of high-accuracy conforma- tions. However, structural exploration alone may be error-prone as Str2Str (Lu al.] 2024), the structure-only models, showed low accuracy even with small perturbations. Other strategies to en- hance conformation sampling also showed improved performance: MSA subsampling (Del Alamo et al.] (202)) with reduced MSA depth excelled at sampling Cluster 3, the most difficult state to cap- ture, and ESMFlow (Ting et al. 2024) fine-tuned on the MD dataset showed improved diversity and accuracy compared to the PDB-
base model. However, these approaches also experienced a decline in quality, with increased CA clashing or peptide bond breaking. Lastly, for most meth- ods, increasing the sample depth (N) significantly improved expected accuracy, suggesting that sufficient sample size is essential for thorough evaluation.
Apo-holo is a more challenging dataset where models are required to predict both the unbound (apo) and bound (holo) conformations induced by ligand binding. As shown in Table 9 and Ta- ble 14 strategies to enhance conformation diversity did not improve the TMens score, and the best-perforning models closely resemble folding models. In comparison, a baseline model that con- sistently predicts the perfect apo structure achieved a higher TMens score. These findings suggest these models struggle to accurately sample apo-holo conformation changes, and higher accuracy may stem from using a stronger folding model.
In summary, strategies such as MSA subsampling, guidance during diffusion, or training on MD conformation data can improve sample diversity and enhance ensemble accuracy for certain local dynamics (as in BPTI). However, they may not be sufficient to capture the complex dynamics in- volved in processes like apo-holo conformational changes.
2.2.3 DISTRIBUTION PREDICTION
Distribution prediction assesses model's ability to generate distributions that closely resemble a target distribution, such as the empirical distribution obtained from molecular dynamics (MD) simulations. See (.2tion[B2..311 for details. The results are summarized in Table Table [5] We include reference performances for (1) MD iid: independent samples from reference MD trajectories and (2) MD 2.5 ns: consecutive samples from the trajectories corresponding to 2.5 of simulation.
Overall, generative models trained to sample protein conformations (AlphaFlow, ConiDiff) sig- nificantly outperform perturbation-bas perturbation levels. Using strong folding model like AlphaFold2 generally results in higher accu- racy. Classifier-free guidance in ConfDiff improved distribution sampling but was less effective than direct fine-tuning on MD data, highlighting the importance of large-scale conformational data for future models. Additionally, the trade-offs between diversity, prediction performance, and sample quality persist: fine-tuning on MD data improves sample diversity and performance for AlphaFlow but decreases sample quality.
methods (MSA subsampling and Str2Str), regardless of
AlphaFlow and ConfDiff models fine-tuned on MD data have shown promise in capturing the con- formational distributions, achieving performance comparable that of 2.5 ns MD simulations on

---

Published as a conference paper at ICLR 2025
Table 9: Performance of multiple-state prediction on apo-holo. apolholo-TM represents the maxi- mum TM-score of the samples relative to the reference apolholo structure. 20 conformations were sampled for each protein, and the results are reported as mean across 91 proteins.
Accuracy
Diversity
Quality
opo-TM 1 holo-TM MM TMess
Pairwise TM CA clash %
PepBond break %
apo model
1.000
0.790
0.895
N/A
N/A
N/A
EigenFold
0.831
0.864
0.847
0,907
3.6
N/A
MSA MSA-depth256
0.845
0.889
0.867
0.788
0.2
4.6
Str2Sm-ODE (Trae
0.3)
0.766
0.781
0.774
0.872
0.2
14.7
Alphaflow PDB
0.847
0.886
0.867
0.909
0.5
5.5
Performance on distribution prediction for ATLAS, 250 conformations were sampled for each protein and the median values across 82 proteins are reported. *These metrics are not available for models that lack side-chain or full backbone information.
Table 10:
Diversity
Flexibility: Pearson r
Distributional accuracy
Pairwise RMSD
RRMSF
Pairwise
Global
"Per target
RRWWD
MD PCA
Joint
PC sim
MD iid
2,76
1663
0.96
0.97
0.99
0.67
0.73
0.71
93.9
1.54
36.6
EigenFool
5,96
N/A
-0.03
N/A
N/A
N/A
2.31
7.96
12.2
MSA-depih256
St22t=ODE (Ti.. = 0.14)
1,66
N/A
0.13
N/A
N/A
N/A
2.14
4,39
6.1
AlphaFlow-MD>
2,87
1,63
0.66
0.85
2.64
1.55
2.29
39,0
ConfDiff Open MD
3.43
2,21
0.59
0.67
0.83
2.75
1.41
2.27
35.4
Ensemble observables
Quality
Weak contacts J
Transient contacts J residue
"Exposed
"Exposed MI
CA clash
"PepBond break %
matrix p
sid
0.80
0.56
MD 2.50s
0.62
0.45
0.64
0.25
0.0
3.4
EigenFold
0.36
0.19
N/A
5.6
N/A
MSA des-depp
0.30
0.29
0.36
0.06
0.0
Su2Sii-DE (Tim 0.14)
0.42
0.18
NIA
N/A
0.0
12.1
AlpbaFlow-MD
0.62
0.41
0.35
22.2
ConfDiff-Open-MD
0.63
0.39
0.65
0.33
0.5
65
some metrics. However, a key goal for these models is to achieve i.i.d. sampling equivalent to
long-term MD simulations, and benchmark results reveal a remaining gap in reaching this objective.
3 CONCLUSIONS AND FUTURE WORK
In summary, we present the first comprehensive study evaluating the capabilities of various protein foundation models across eight distinct tasks, with a particular focus on protein design and con- formation dynamics. We have developed unified, multi-metric evaluation framework, which is essential for unbiased assessment of protein foundation models from multiple facets. Based on the performance results, provide insights and considerations for the development and effective use of protein foundation models, offering guidance for future research.
With the detailed discussion available in the Appendix C we highlight the key observations from our holistic evaluation as follows: (1) Valid evaluation of protein foundation models requires accu- rate and comprehensive evaluation metrics; (2) No single model currently excels across all protein design objectives. The choice of model should be carefully aligned with the intended applications; (3) While generative models extended from classic folding models have shown the ability to sample protein conformations, challenges remain in both multiple-state prediction and distribution predic- tion.
LIMITATIONS FUTURE WORK
We acknowledge several limitations and opportunities for enhancement in our current benchmark: (1) The selection foundation models may not be exhaustive. Future iterations should incorporate additional foundation models to provide more comprehensive comparison. (2) Inconsistencies in training data across models currently hinder direct comparisons of different model architectures. Future work could address this by standardizing datasets, allowing for more accurate comparisons of architectural performance. (3) The benchmark could be expanded to include a wider range of tasks, further broadening its scope and utility. We committed to continually refining and expanding ProteinBench. Our vision is for to evolve into dynamic, growing benchmark that accelerates progress in the field of protein modeling and design.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Abstract book of the 15th critical assessment of structure prediction. 2022. URL https://
predictiioncenter.r org/
wwt.eerrrrggaappallppp casp15/ 5/
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Jared Adylf-Br Oleks Kalyuzhniy, Michael Kubitz, Brian D Weitzner, Xiaozhen Hu, Yumiko Adachi, William R Schief, and Roland L Dunbrack Jr. Rosettaantioddydesign (rabd): A general framework for computational antibody design. PLoS computational biology, 14(4):e1006112, 2018.
Gustaf Ahdritz, Nazim Bouatta, Christina Floristean, Sachin Kadyan, Qinghui Xia, William Gerecke, Timothy O' DDnnelll, Daniel Berenberg, Ian Fisk, Niccolò Zanichelli, Bo Zhang, Arka- diusz Nowaczynski, Bei Wang. Marta M Stepniewska- Skepniewskaa-zitubinkkz Shang Zhang, Adegoke Ojew- ole, Murat Efe Guney, Stella Biderman, Andrew M Watkins, Stephen Ra, Pablo Ribalta Lorenzo, Lucas Nivon, Brian Weitzner, Yih-En Andrew Ban, Peter K Sorger, Emad Mostaque, Zhao Zhang, Richard Bonneau, and Mohammed AlQuraishi. OpenFold: Retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalization. bioRxiv, 2022. doi: 10. 10.1101//022.11.200 1120 2022. 11. 20. 017210
URL https: //t:://wwwww..c www biorxiv..ivvv...........i..i.w..
Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex Lu, Nicolo Fusi, Ava P Amini, and Kevin K Yang. Protein generation with evolutionary diffusion: sequence is all you need. In Machine Learning for Structural Biology Workshop. NeurIPS 2023, 2023.
Rebecca F Alford, Andrew Leaver-Fay, Jeliazko R Jeliazkov, Matthew J O' Meara, Frank P DiMaio, Hahnbeom Park, Maxim V Shapovalov, P Douglas Renfrew, Vikram K Mulligan, Kalli Kappel, et al. The rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation, 13(6):3031-300) -3048, 2017.
Ethan C. Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M. Church. Unified rational protein engineering with sequence-based deep representation learning. Nature Methods, 16:1315-1322 12 2019. ISSN 1548-7091. doi: 10. 10.10388441522-19-05599..
Minkyung Baek, Ivan Anishchenko, Ian R Humphreys, Qian Cong, David Baker, and Frank DiMaio. Efficient and accurate prediction of protein structure using rosettafold2. BioRxiv, pp. 2023-05, 2023.
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip Bourne. The protein data bank. Nucleic acids research, 28(1), 235-242, 2000.
Avishek Joey Bose, Tara Akhound-Sadegh, Kilian Fatras, Guillaume Huguet, Jarrid Rector-Brooks, Cheng- Liu, Andrei Cristian Nica, Maksym Korablyov, Michael Bronstein, and Alexan- der Tong. Se (3)-stochastic flow matching for protein backbone generation. arXiv preprint (xXiv:231000299) 2023.
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In Forty-first International Conference on Machine Learning.
Alexander E Chu, Jinho Kim, Lucy Cheng, Gina El Nesr, Minkai Xu, Richard W Shuai, and Po-Ssu Huang. An all-atom protein generative model. Proceedings of the National Academy of Sciences, x21(27)ee231 1003311500000 1501221, 2024.
Peter JA Cock, Tiago Antao, Jeffrey Chang, Brad A Chapman, Cymon Cox, Andrew Dalke, Iddo Friedberg, Thomas Hamelryck, Frank Kauff, Bartek Wilezynski, et al. Biopython: freely available python tools for computational molecular biology and bioinformatics. Bioinformatics, 25(11):1422) 2009.
11

---

Published as a conference paper at ICLR 2025
J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel, P. J. Y. Leung, T. F. Huddy, S. Pellock, D. Tischer, F. Chan, B. Koepnick, H. Nguyen, A. Kang, B. Sankaran, A. K. Bera, N. P. King, and D. Baker. Robust deep learning-based protein sequence design using proteinmpnn. Science, 378:49-56, 10 2022a. ISSN 0036-8075. doi: 10.1 2dd8ssc1eence.dd2187
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas, Neville Bethel, et al. Robust deep learning- based protein sequence design using proteinmpnn. Science, 378(6615):49-555 2022b.
Diego Del Alamo, Davide Sala, Hassane Mchaourab, and Jens Meiler. Sampling alternative conformational states of transporters and receptors with alphafold2. Elife, 11:e75751,
James Dunbar, Konrad Krawezyk, Jinwoo Leem, Terry Baker, Angelika Fuchs, Guy Georges, Jiye Shi, and Charlotte M. Deane. SAbDab: the structural antibody database. Nucleic Acids Research, 42001):D1140-D1146 11 2013. ISSN 0305-1048.
doi: 10. 10.1093/nar/gkt1044 URL https:
//doi. 1 0993 na r //kt
Zhangyang Gao, Cheng Tan, Pablo Chacón, and Stan Z Li. Pifold: Toward effective and efficient protein inverse folding. arXiv preprint arXiv:2209. 12643, 2022.
Zhangyang Gao, Cheng Tan, Yijie Zhang. Xingran Chen, Lirong Wu, and Stan Z Li. Proteinin- vbench: Benchmarking protein inverse folding on diverse tasks, models, and metrics. Advances in Neural Information Processing Systems, 36, 2024.
Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with a language model. bioRxiv, pp. 2024-07, 2024.
Brian L. Hie, Varun R. Shanker, Duo Xu. Theodora U. J. Bruun, Payton A. Weidenbacher, Shaogeng Tang. Wesley Wu, John E. Pak, and Peter S. Kim. Efficient evolution of human antibodies from general protein language models. Nature Biotechnology, 42:275-2888 283, 2024. ISSN 1087-0156.
doi: 10. 00.1008//41587-023--1111
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, om Sercu, Adam Lerer, and Alexan- der Rives. Learning inverse folding from millions of predicted structures. Proceedings of the 39th International Conference on Machine Learning, 162, 2022. doi: 10.1101/2022.04.10022 14.047777..
John B Ingraham, Max Baranov, 7ak Costello, Karl W Barber, Wujie Wang, Ahmed Ismail, Vincent Frappier, Dana M Lord, Christopher Mi-Thow-HHing Erik R Van Vlack, et al. Illuminating protein space with a programmable generative model. Nature, 623(7989):1070-1000
2023.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Antibody Antibody-antigen docking and design via hierarchical structure refinement. In International Conference on Machine Learning, pp. 10217- 10227. PMLR, 2022.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2020.
Bowen Jing, Ezra Erives, Peter Pao-Huang, Gabriele Corso, Bonnie Berger, and Tommi S Jaakkola. Eigenfold: Generative protein structure prediction with diffusion models. In ICLR 2023-Machine Learning for Drug Discovery workshop. 2023.
Bowen Jing, Bonnie Berger, and Tommi Jaakkola. Alphafold meets flow matching for generating protein ensembles. In Forty-first International Conference on Machine Learning, 2024.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Se- bastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596:5596555555 589, 2021. ISSN 0028-0836. doi: 10. -56-000--111111---22
12

---

Published as a conference paper at ICLR 2025
Xiangzhe Kong, Wenbing Huang, and Yang Liu. Conditional antibody design as 3d equivariant graph translation. In The Eleventh International Conference on Learning Representations. ICLR, 2022.
Xiangzhe Kong, Wenbing Huang, and Yang Liu. End-to-end full-atom antibody design. In Proceed- ings of the 40th International Conference on Machine Learning, pp. 17409-17429, 2023.
Daniel Kovtun, Mehmet Akdel, Alexander Goncearenco, Guoqing Zhou, Graham Holt, David Baugher, Dejun Lin, Yusuf Adeshina, Thomas Castiglione, Xiaoyun Wang, et al. Pinder: The protein interaction dataset and evaluation resource. bioRxiv, pp. 2024-07, 2024.
Rohith Krishna, Jue Wang. Woody Ahern Pascal Sturmfels, Preetham Venkatesh, Indrek Kalvet, Gyu Rie Lee, Felix S. Morey-Burrows, Ivan Anishchenko, Ian R. Humphreys, Ryan McHugh, Dionne Vafeados, Xinting Li, George A. Sutherland, Andrew Hitchcoek, C. Neil Hunter, Alex Kang, Evans Brackenbrough, Asim K. Bera, Minkyung Back, Frank DiMaio, and David Baker. Generalized biomolecular modeling and design with rosettafold all-atom. Science, 384, 2024. ISSN 0036-8075. doi: 10. 16t.1//ccccccc.al222222..
Brian Kuhlman and Philip Bradley. Advances in protein structure prediction and design. Na- ture Reviews Molecular Cell Biology, 20:681-697, 11 2019. ISSN 1471-0072. doi: 10.1038/
s41580-019-0163-x
Marie-Paule Lefranc, Veronique Giudicelli, Chantal Ginestoux, Joumana Michualod, Geral- dine Folch, Fatena Bellahcene, Yan Wu, Elodie Gemrot, Xavier Brochet, Jero'me 'e Lane, et al. Imgt@) the international immunogenetics information system@. Nucleic acids research, 37 (suppL.)::D100--D100) 2009.
Yeqing Lin and Mohammed AlQuraishi. Generating novel, designable, and diverse protein struc- tures by equivariantly diffusing oriented residue clouds. arXiv preprint arXiv:2301. 230.v:2300..24855, 2023.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scabe prediction of atomic-level protein structure with language model. Science, 379:1 11300 3 2023a. ISSN 0036-8075. doi: 10.1 122//ciienc..ae25555
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 12996667):1123-111 2023b.
Sidney Lyayuga Lisanza, Jake Merle Gershon, Sam Tipps, Lucas Amoldt, Samuel Hendel, Jeremiah Nelson Sims, Xinting Li. and David Baker. Joint generation of protein sequence and structure with rosettafold sequence space diffusion. bioRxiv, pp. 2023-05, 2023.
Haiyan Liu, Yufeng Liu, and Linghui Chen. Diffusion in a quantized vector space generates non- idealized protein structures and predicts conformational distributions. bioRxiv, pp. 2023-11, 2023.
Jiarui Lu, Bozitao Zhong, Zuobai Zhang, and Jian Tang. Str2str: A score-based framework for zero-shot protein conformation sampling. In The Twelfth International Conference on Learning Representations, 2024.
Shitong Luo, Yufeng Su, Xingang Peng. Sheng Wang, Jian Peng. and Jianzhu Ma. Antigen-specific antibody design and optimization with iiffusion-based generative models for protein structures. Advances in Neural Information Processing Systems, 35:9754-9767, 2022.
Ali Madani, Ben Krause, Eric R. Greene, Subu Subramanian, Benjamin P. Mohr, James M. Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z. Sun, Richard Socher, James S. Fraser, and Nikhil Naik. Large language models generate functional protein sequences across diverse families. Na- ture Biotechnology, 41:1099-1106, 8 2023. ISSN 1087-0156. doi: 10. 10.10388441577-02-00111222
Valerio Mariani, Marco Biasini, Alessandro Barbato, and Torsten Schwede. Iddt: local superposition-free score for comparing protein structures and models using distance difference tests. Bioinformatics, 29(21):2722-2725 20133..
13

---

Published as a conference paper at ICLR 2025
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steinegger. Colabfold: making protein folding accessible to all. Nature methods, 19(6):679-688 2022.
Erik Nijkamp. Jeffrey A. Ruffolo, Eli N. Weinstein, Nikhil Naik, and Ali Madani. Progen2: Ex- ploring the boundaries of protein language models. Cell Systems, 14:968-978.e3, 11 2023, ISSN 24054712. doi: 10.10166j.cels.2223 10.002.
Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, et al. Proteingym: Large-scale benchmarks for protein fitness prediction and design. Advances in Neural Information Processing Systems, 36, 2024.
Milong Ren, Chungong Yu, Dongho Bu, and Haicang Zhang. Accurate and robust protein sequence design with carbondesign. Nature Machine Intelligence, 6(5):566 547, 2024.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118, 2021. ISSN 0027-8424. doi: 10. 10. (0.1073/paas.22162223
Xavier Robin, Juergen Haas, Rafal Gumienny, Anna Smolinski. Gerardo Tauriello, and Torsten Schwede. Continuous automated model evaluation (cameo)-perspecttin) on the future of fully automated evaluation of structure prediction methods. Proteins: Structure, Function, and Bioin- formatics, 19:1977-1986, 12 2021. ISSN 0887-3585. doi: 10. 26.10
Jeffrey A Ruffolo, Jeffrey J Gray, and Jeremias Sulam. Deciphering antibody affinity maturation with language models and weakly supervised learning. In Machine Learning for Structural Biol- ogy Workshop, NeurIPS 2021., 2021.
Jeffrey A Ruffolo, Lee-Shin Chu, Sai Pooja Mahajan, and Jeffrey J Gray, Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies. Nature communica- tions, 14(1):2389 2023.
Tadeo Saldaño, Nahuel Escobedo, Julia Marchetti, Diego Javier Zea, Juan Mac Donagh, Ana Julia Velez Rueda, Eduardo Gonik, Agustina García Melani, Julieta Novomisky Nechcoff, Martín N Salas. et al. Impact of protein conformational diversity on alphafold predictions. Bioinformatics, 38((1)):242--2 2748, 2022.
David E Shaw, Paul Maragakis, Kresten Lindorff-Larsen, Stefano Piana, Ron O Dror, Michael P Eastwood, Joseph A Bank. John M Jumper, John K Salmon, Yibing Shan, et al. Atomic-level characterization of the structural dynamics of proteins. Science. 330(6002):341-346, 2010.
Jung-Bun Shin, Adam Riesselman, Aaron W. Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C. Kruse, and Debora S. Marks. Protein design and variant prediction using autoregressive generative models. Nature Communications, 12:2403, 4 2021. ISSN 2041-1723. doi: 10. 10.10088/41166--02-22233333
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, 35(1 10:006-008 2017.
Kiera H. Sumida, Reyes Núñez-Franco, Indrek Kalvet, Samuel J. Pellock, Basile I. M. Wicky, Lukas F. Milles, Justas Dauparas, Jue Wang, Yakov Kipnis, Noel Jameson, Alex Kang, Josh- myn De La Cruz, Banumathi Sankaran, Asim K. Bera, Gonzalo Jiménez-Osés, and David Baker. Improving protein expression, stability, and function with proteinmpnn. Journal of the American Chemical Society, 146:2054-2011, 2024. ISSN 0002-7863. doi: 10. /et////s..3119
Baris E. Suzek, Yuqi Wang, Hongzhan Huang, Peter B. McGarvey, and Cathy H. Wu. Uniref clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 31 31:926-932, 3 2015, ISSN 1367-4811. doi: 10.1003//oinfommatt///b333333
Brian L Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. arXiv preprint arXiv:2206.041 19, 2022.
14

---

Published as a conference paper at ICLR 2025
Michel van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Cameron LM Gilchrist, Johannes Söding. and Martin Steinegger. Foldseek: fast and accurate protein structure search. Biorxiv, pp. 2022-02, 02, 2022.
Yann Vander Meersche, Gabriel Cretin, Aria Gheeraert, Jean-Christophe Gelly, and Tatiana Ga- lochkina. Atlas: protein flexibility description from atomistic molecular dynamics simulations. Nucleic acids research, 52(DI):D384-D53 2024.
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with 2022.
high-aceuracy models. Nucleic acids research, 50(D1):D439-D44
Robert Verkuil, Ori Kabeli, Yilun Du, Basile IM Wieky, Lukas F Milles, Justas Dauparas, David Baker, Sergey Ovchinnikov, Tom Sercu, and Alexander Rives. Language models generalize be- yond natural proteins. BioRxiv, pp. 2022-12, 2022.
Chentong Wang, Yannan Qu, Zhangzhi Peng, Yukai Wang, Hongli Zhu, Dachuan Chen, and Longx- ing Cao. Proteus: exploring protein structure generation for enhanced designability and efficiency. bioRxiv, pp. 2024-02, 2024a.
Chuanrui Wang. Bozitao Zhong, Zuobai Zhang, Narendra Chaudhary. Sanchit Misra, and Jian Tang. Pdb-struct: A comprehensive benchmark for structure-based protein design. arXiv preprint arXiv:2312.000000, 2023.
Xinyou Wang. Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners. In International conference machine learning, 2024b.
Yan Wang. Lihao Wang, Yuning Shen, Yiqun Wang, Huizhuo Yuan, Yue Wu, and Quanquan Gu. Protein conformation generation via force-guided se (3) diffusion models. In Forty-first Interna- tional Conference on Machine Learning, 2024c.
Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile M. Wicky, Nikita Hanikel. Samuel Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Sergey Ovchinnikov, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung Baek, and David Baker. De novo design of protein structure and function with rfdiffusion. Na- ture, 120:1009-1100, 8 2023a. ISSN 0028-0836. doi: 10.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eise- nach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rídiffusion. Nature, (20(7976):1089-11
2023b.
Hannah Wayment-Steele Adedolapo Ojoawo, Renee Otten, Julia M Apitz, Warintra Pitsawong, Mare Hömberger, Sergey Ovchinnikov, Lucy Colwell, and Dorothee Kern. Predicting multiple conformations via sequence clustering and alphafold2. Nature, 625(7996):832-833 2024.
Kevin E Wu, Kevin K Yang, Rianne den Berg, Sarah Alamdari, James Y Zou, Alex X Lu, and Ava P Amini. Protein structure generation via folding diffusion. Nature communications, 15(1): 1059, 2024a.
Luhuan Wu, Brian Trippe, Christian Naesseth, David Blei, and John P Cunningham. Practical and asymptotically exact conditional sampling in diffusion models. Advances in Neural Information Processing Systems, 36, 2024b.
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pp. 2022-07, 2022.
Jason Yim, Andrew Campbell, Andrew YK Foong, Michael Gastegger, José Jiménez-Luna, Sarah Lewis, Victor Garcia Satorras, Bastiaan Veeling, Regina Barzilay, Tommi Jaakkola, et al. Fast protein backhone generation with se (3) flow matching. arXiv preprint arXiv:2310.05297 2023.
15

---

Published as a conference paper at ICLR 2025
Jason Yim, Andrew Campbell, Emile Mathieu, Andrew YK Foong, Michael Gastegger, José Jiménez-Luna Sarah Lewis, Victor Garcia Satorras, Bastiaan S Veeling, Frank Noé, et al. Im- proved motif-scaffolding with (3) flow matching. ArXiv, 2024.
Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics, 57(4):702-710 2004.
Yang Zhang and Jeffrey Skolnick. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic acids research, 33(7):2302-2399, 2005.
Shuxin Zheng, Jiyan He, Chang Liu, Yu Shi, Ziheng Lu, Weitao Feng, Fusong Ju, Jiaxi Wang, Jianwei Zhu, Yaosen Min, et al. Predicting equilibrium distributions for molecular systems with deep learning. Nature Machine Intelligence, pp. 1-10, 2024.
Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei Ye, and Quanquan Gu. Structure-informedd language models are protein designers. In International conference on machine learning, pp. 42317-42338. PMLR, 2023.
Xiangxin Zhou, Dongyu Xue, Ruizhe Chen, Zaixiang Zheng, Liang Wang. and Quanquan Gu. Aptigen-specific antibody design via direct energy- energy-based preference optimization. In (CML2004 Workshop A14Science, 2024.
A OVERVIEW OF PROTEIN FOUNDATION MODEL BENCHMARKS
A.1
GENERAL BENCHMARK DESIGN RATIONALE
The field protein three- dimensimmnsional structure prediction has witnessed remarkable progress, ex- emplified by established benchmarks like CASP and CAMEO, and breakthrough methodologies including AlphaFold series, RosettaFold, ESMFold, and OmegaFold. While structure prediction fo- cuses on determining protein structures from known sequences, protein design addresses the inverse challenge: creating sequences that will fold into desired structures or achieve specific functions. Despite growing interest in protein design, the field has been hampered by the absence of a compre- hensive benchmark, with existing evaluations primarily targeting specialized tasks, as documented in Appendix Table 1. A similar limitation exists in conformational dynamics research. Our work addresses these gaps by introducing the first comprehensive benchmark focusing protein design and conformation prediction.
In our benchmark, protein design categorized into five distinct areas, following the natural seguuence-structur--funneee
hierarchy. This begins with sequence design, focusing on optimizing amino acid sequences for stable folding, and progresses to backbone design, which involves en- gineering the overall protein architecture. 'he more complex sequence-structure co-design task requires simultaneous optimization of both sequence and structure. At the functional level, mo- tif scaffolding involves incorporating functional motif's into stable scaffolds, while antibody design represents a specialized application focusing on engineering antibody structures and sequences for antigen binding, particularly erucial for therapeutic development.
The conformation prediction component is similarly structured into three distinct categories, reflect- ing increasing levels of complexity in protein dynamics. Single conformation prediction focuses on identifying the lowest energy state among possible conformations. Multiple conformation pre- diction addresses the more complicated challenge of predicting discrete conformational states. The most sophisticated category, conformational distribution prediction, tackles the complex task of pre- dicting probability distributions of conformations, essential for understanding proteins with dynamic structural ensembles.
A.2 OVERVIEW OF EXISTING BENCHMARKS
In this section, we provide a comprehensive overview of existing benchmarks for protein foundation models. Table II illustrates the current landscape of these benchmarks, revealing significant limi- tations in the scope and applicability. The majority of existing benchmarks are narrowly focused, primarily addressing task-specific evaluations rather than offering a holistic assessment of protein foundation models.
Notably, our proposed ProteinBench stands out by offering the most comprehensive coverage across various tasks. It encompasses wide range of evaluations, including inverse folding, backbone
16

---

Published as a conference paper at ICLR 2025
design, sequence design, structuressqueenc co-design, and antibody design in the protein design domain, as well as single-state folding, and multiple-ste state prediction in the conformational dynamics domain.
Table 11: comparison of benchmarks for protein fundamental tasks.
Benchmark
Protein Design
Protein Conformation Prediction
Inverse Backbome Sequence Struc-Seq Folding
Moiil
Antibady Design
Folding (single-state)
Multiple State Distribution
Design
Design
Codesign scaflolding
Prediction
Prediction
PDB-Struct Wann et 2023
X
X
X
Proteininvee
2024
x
RFDiftusion Walson et al 20256
x
x
X
X
CASP cas
X
CAMEO Robin et al. 2021
x x V
PINDER Koviin
X
ProteinBeich
B DETAILS ON BENCHMARKING EVALUATIONS
In this section, we provide detailed discussions for each task addressed by various protein foundation models, as shown in Table T Our focus will on the following aspects:
[Task Definition] A detailed description of the task, including its objectives and relevance to protein science. Specification of the input data format and expected output for each task. The impact of the task for protein provided.
[Evaluation Metries] Justification and description of the metrics used to assess model performance, including quality, novelty, diversity, and robustness measures. For each specific task, provided detailed thought process behind the metric selection and detailed implementation information.
[Datasets] Overview of the datasets used for each task, including their size, diversity, and any pre pre-processing steps applied. Detailed considerations datasets, such as dataset impacts, are provided.
[Model Implementations] We place the detailed implementation information for the evaluated methods in this part.
[Extended Explanations and Discussion on Model Performance] Due to space limitations the main text, we provide an additional explanation and discussion of the performance of specific methods here.
B.1
PROTEIN DESIGN
B.1.1 INVERSE FOLDING
[Task Definition] Inverse folding is a fundamental task in protein engineering aimed at designing amino acid sequences that can fold into predetermined structural configurations. This task is essential for various applications, including capturing evolutionary distribution of protein se- quences (Zheng et al.) (202)). facilitating de novo protein design (Dauparas al. [2022),, and optimizing protein stability for therapeutic and industrial purposes (Sumida al. 2024).
[Evaluation Metries] key factor inverse folding the diverse range of downstream objectives, each requiring specific datasets and evaluation approaches. To provide comprehensive assessment, we evaluate model performance separately for these distinct objectives. For assessing evolutionary distribution capture, we employ datasets comprising native protein structures. To evaluate sequence design capabilities for novel structures, we utilize datasets of protein backbones generated by RFd- iffusson (Watson et al., 2023). Additionally, we incorporate the pLDDT metric to evaluate the pre- dicted folding stability of designed sequences, providing insights into their structural reliability.
For the inverse folding task, metrics for evaluation have evolved over time. ProteinMPNN initially employed amino acid recovery (AAR) as the primary evaluation metric(Dauparas et al., 2022a) More recent studies have introduced structure-based self-consistency TM-score (scTM) as an addi- tional metric for this task (Gao et al. 2024 Ren al. 2024). Based on these previous studies, recognize that both metrics provide valuable and complementary insights into model performance. AAR measures sequence-level accuracy while se TM assesses the structural validity of the designed sequences. Performance in protein sequence design is assessed using multiple complementary met- rics.
17

---

Published as a conference paper at ICLR 2025
Sequence Recovery Amino Acid Recovery Rate (AAR), measures the sequence recovery rate of designed proteins and quantifies how well the design method can recapitulate evolutionarily conserved sequence patterns associated with specific structural motifs. While it highly efficient for evolutionary design evaluation, AAR is limited to cases with known ground truth sequences and cannot assess the refoldability of de novo designed sequences.
Refoldability self-consistent TM-score (scTMM, assesses the structural refoldability of designed sequences and has been widely adopted in the field for structural validation. This measure evaluates the structural similarity between the target backbone and the predicted structure of the designed sequence. scTM can be applied to both native and de novo designed sequences, making it more versatile. The prediction is performed using AlphaFold2 (Jumper et al. (02I). We used the Co- labFold implementation MMidiita et al. (2022) for AlphaFold2 inference, with input MSAs obtained using the Colab pipeline. Specifically, the designed sequences are input into AlphaFold2 in the MSA alignment mode, For each prediction round, five structures are generated, and the candidate structure with the highest PL.DDT' confidence score is selected for comparison with the targeted structure. Similarity is quantified using self-consistency template modeling score (scTM) (Trippe) ct al. 2022) and self-consistency root-mean anean-square deviation (scRMSD), providing insight into how well the designed sequence would fold into the intended structure.
Stability Stability is assessed using the predicted local distance difference test (pLDDT), which is also calculated by AlphaFold2 in the same MSA alignment mode used for the scTM score. The PLDDT score serves as a proxy for the predicted stability of the designed protein, as utilized in mnDapparas et al. (2022a). We have observed that removing the MSA alignment may yield optimal results for PLDDT, and we plan to update our findings accordingly in future analyses.
[Datasets] Typically, inverse folding methods are trained using CATH or PDB datasets. To prevent data leakage, we utilized newly released PDB structures collected from CASP and CAMEO, as well as de novo designed backbones that have not been included in any training sets. Evaluations were conducted on different datasets targeting two distinct objectives of structure-based sequence design.
Capture the native evolutionary distribution We evaluated two independent datasets contain- ing newly released experimentally well-determined PDB structures: CASPI5 (as 2022) and CAMEO (Robin et al. 2021). We collected new structures from the ongoing CAMEO assessment between January and July 2024, resulting in a total of 332 complex structures. Additionally, 32 protein structures were collected from CASP15, which includes only protein entities, excluding nucleic acids or ligands. These datasets mainly comprise native single-chain protein structures and sequences, making optimal for evaluating evolutionary information capture and enabling assessment of natural sequence recovery.
De novo protein design REdiffusion (Watson et al. 2023a) was used to unconditionally generate backbones of varying lengths: specifically, 100, 200, 300, 400, and 500 residues. For each length, 10 different structures were randomly sampled, using a sampling temperature of 0. for all meth- ods. The designability of these sequences and DLDDT metrics serving as the primary assessment criteria. Existing benchmarks for inverse folding, such PDB-Struct (Wang et al. 2023) and Proteininvbench (Gao al. 2024), provide standardized protein structure sets for eva evaluating inverse folding methods. While these bench- marks have significantly contributed to the field's advancement, there growing need for more comprehensive evaluation frameworks. These expanded evaluations should align more closely with diverse user objectives in protein design, encompassing aspects like accuracy in capturing natural evolutionary distributions and robustness in de novo backbone-basse sequence design.
evaluated using AlphaFold2, with the scTM score
Model implementations A sampling temperature of 0.1 was used for each method to generate sequences. While this value balances sequence diversity and quality, optimal temperatures may vary across inverse folding methods. For each structure, one generated sequence was predicted for performance evaluation.
ProteinMPNN (Dauparas al. 2022a): inference, with the sampling temperature set to 0.1. The default model weight v_48_020. pt is used.
We follow the official repository and instructions for
ESM-IF1 (Hsu et al. 2022): We use the public ESM repository for inference.
LM-DESIGN (Zheng et al. 2023): We used the official repository with the model
1m design gn esm2 650m.
18

---

Published as a conference paper at ICLR 2025
ESM3 (Hayes al. 2024): We follow the official repository with model esm3-open-small which has 1.4B parameters.
[Extended Explanations and Discussion on Model Performance]
Dataset Characterístics Impact Performance Our evaluation spans two dataset types: high- quality experimental structures (CASP/CAMEO) and computational novo structures containing inherent noise. Models performing well on the more challenging de novo structures demonstrate superior robustness, as they must overcome structural uncertainties while maintaining design ac- curacy.
Training Strategy Influences Robustness ProteinMPNN's approach of incorporating backbone noise during training proves highly effective. Our results confirm their findings that increased training noise correlates with improved model robustness. This evidenced by ProteinMPNN's superior performance in de novo backbone- based sequence design, validating backbone noise augmentation as an effective strategy for enhancing model resilience.
B.1.2 PROTEIN BACKBONE DESIGN
[Task Definition] Protein backbone design is classical protein design problem, centered on developing new protein folds to meet de novo design objectives. It is widely accepted that novel structures can approximate new functions. As such, the backbone design task is crucial for expanding the repertoire of protein structures beyond those found in nature, offering significant applications in areas such drug discovery, biomaterials, and therapeuties. With de novo designed protein backbones, inverse folding methods can be employed to generate the corresponding sequences.
[Evaluation Metries] The rapid development of protein backbone design methods in recent years has been accompanied by inconsistent evaluation practices across different studies. While some researchers report quality in terms of designability, others utilize scTM scores, making it difficult draw meaningful comparisons between methods. To address this fragmentation the field, our benchmark implements a comprehensive evaluation framework with standardized metries. This unified approach enables both fair comparisons between methods and thorough assessment of their performance across multiple aspects of backbone design quality.
Building on previous studies Trripe et al. et al.) 2023b), we evaluate backbone design through multiple criteria that assess the quality, diversity, and novelty of generated structures.
2022 Lin AlQuraishi 2023 Yim et al. 2023 Watson
Quality We utilize ProteinMPNN (Dauparas et al. 2022a) to generate eight different sequences for each backbone structure, which are unconditionally sampled from various methods. The struc- tures of these eight sequences are predicted using FSMFoll (Lin et al. (023a). We employ self- consistency TM-score (scTM) and self-consistency RMSD (scRMSD) to measure structural re- foldability. For each backbone, select the sequence with the maximum scTM score. For each method, we unconditionally sample 100 different backbones and report the median scTM and scRMSD.
Novelty Equally important are novelty metrics, which gauge the method's capacity to explore new structural space beyond known protein folds. The metrics measuring the novelty of generated structures were introduced in recent studies (Yim et al. 2023; Campbell et al.). This aspect is evaluated using two key metries: The maximum TM-score obtained when comparing designed structures to existing entries in the RCSB Protein Data Bank (PDB) (Berman et al.) 2000). This
comparison is performed using Foldseek with threshold TM-score of less than 0.5 (van Kempen
et al. 2022).
Diversity Two metrics were used to evaluate diversity: (a) We calculate the average pairwise maximum TM-scores among the designed structures. A lower TM-score indicates better diversity. (b) The number of distinct structural clusters identified within the set of designed backbones was also determined using Foldseek (van Kempen et al., 2022) with TM-score threshold less than 0.5. More clusters stand for higher diversity. These diversity metrics help quantify the range of
19

---

Published as a conference paper at ICLR 2025
Table 12: Performance of backbone design models evaluated using 200, and 400 amino acids in lengths. The reported results are the medium of repetitive experiments. We use bold text to highlight the best and suboptimal results for cach metric. For the novelty and diversity metrics, only highlight results with the corresponding scTM score higher than 0.5.
length 200
length 400
Quality
Novelty
Diversity
Quality
Novelty
Diversity
scTM 1 scRMSD Max TM pairwise TM | Max Clast. 1 scTM 1 scRMSD 1 Max TM pairwise TM
Max Clust.T
Native PDBs
0.974
0.674
NIA
0.278
0.790
0.970
1.085
N/A
0.261
0.840
REdidfusion
0,982
0.617
0.638
0.363
0.64
0,927
2.12
0.634
0.356
0.720
FrameFlow
0,953
1.02
0.648
0.458
0,800
0.805
4.46
0.620
0,41%6
0.920
Chroma
0.892
1.776
0.674
0.346
0.620
0.761
4,891
0.626
0.304
0.95
FramcDiff (latest)
1.789
0.464
0.800
4.324
0.467
FoldFlow (base)
0.529
7.108
0.579
0.430
0.950
0.415
11.743
0.525
0.357
1,00
FoldFlow (sfm)
0.619
5,270
0.586
0.433
0,980
0.598
11.135
0.534
0.372
0.99
FoldFowl (oc)
6.877
0.392
10.78
Genie
0.367
13,699
0.431
0.264
1.00
0.251
24.453
0.238
0.229
1.00
unique structures the design method can produce, ensuring that it's not simply recreating known folds but generating varied repertoire of protein backbones.
[Datasets] Protein backbone design is a generative task, whose primary objective is to map the overall distribution of the training set accurately. To conduct comprehensive performance analysis of protein structure generation, we compare the models' performance against the distribution real PDB structure data. Specifically, we randomly sampled 100 high-resolution experimental structures from the Protein Data Bank (PDB) to serve as references for the data distribution. To ensure diversity, we iteratively removed structures with the highest TM-score compared to others until arrived at a final set of 100 distinct structures. This approach provides a representative snapshot of the single-chain structural distribution within the PDB, serving as benchmark for evaluating the performance of generative models in capturing the true distribution of protein structures.
[Model implementations] To ensure comprehensive evaluation, we test each method's ability to perform unconditional monomer generation across diverse protein sizes. We follow the official repository and instructions for inference. The generation process is constrained only by specified target lengths, which we set at 50, 100, 200, 300, 400, and 500 residues. This range allows us to assess method performance across both small proteins and larger, more complex structures.
[Extended Explanations and Discussion on Model Performance] A notable observation across different backhone design methods reveals inverse relationship between structural quality and diversity: as methods generate more diverse and novel structures, the quality of the generated back- bones tends to decrease. We emphasize that structural quality should be considered the primary metric, as diversity and novelty become meaningful only when the generated structures maintain sufficient quality. Without adequate structural quality, high diversity or novelty scores may simply reflect the generation of unrealistic or physically implausible conformations.
In the revised manuscript, we have expanded our evaluation to include Proteus's performance across multiple protein lengths (100, 200, 300, and 500 residues). Our analysis reveals that Proteus demonstrates superior design quality for long-chain backbone design (500 residues), achieving a scTM score of 0.90 compared to REdiffusion's 0.79. However, we observed a significant decline in structural diversity for Proteus when designing longer chains. At 300 residues, Proteus shows diversity score of 0.34 vs. REdiffusion of 0.65. At 500 residues, Proteus shows a diversity score of 0.34 vs. RFdiffusion of 0.89. Case analysis revealed that Proteus tends to generate structures limited to three categories, predominantly characterized by helical tandem repeats, confirming our diversity metric findings.
[Additional results for backbone design]
In this section, we provide more detailed evaluation results of protein backbone design across addi- tional lengths 200 and 400. The results are shown in Table 12
1.1.3 PROTEIN SEQUENCE DESIGN
[Task Definition] Protein sequence design aims to generate amino acid sequences with desired properties, including quality, diversity, and novelty. The task encompasses both sequence-level
20

---

Published as a conference paper at ICLR 2025
evaluation and assessment of structural characteristics of the generated sequences, making fundamental for various applications in protein engineering and therapeutic development.
[Evaluation Metries] The evaluation of protein sequence design requires a multi-faceted approach that considers both sequence-level properties and structural characteristics. Sequence-leve as- sessment should verify that generated sequences follow natural protein patterns, while structural evaluation ensures the designs are likely to fold into stable, well-defined conformations. This comprehensive evaluation strategy helps validate both the theoretical and practical utility of designed sequences.
The evaluation of protein sequence design requires multi-faceted approach that considers both sequence-leve properties and structural characteristics. Sequence-lever assessment should verify that generated sequences follow natural protein patterns, while structural evaluation ensures the designs are likely to fold into stable, well-defined conformations. This comprehensive evaluation strategy helps validate both the theoretical and practical utility of designed sequences.
We employ multiple complementary metrics to assess different aspects of sequence design:
Sequence Naturalness We utilize perplexity scores from ProGen2 (Nijkamp et al., 2023). an autoregressive protein language model, to quantify how well the generated sequences align with natural sequence distributions. Lower perplexity indicates sequences that better match patterss observed in natural proteins.
Structural Stability For structure-bassd evaluation, we single-sequence folding model ESM- Fold (Lin et al. 2023b) to predict the structure of the generated sequences, and then measure the structural quality by DLDDT the proxy of structural stability of the sequence.
Structural Properties We evaluate structural diversity and novelty using the same protocols es- tablished in backbone design evaluation, ensuring a comprehensive assessment of the structural characteristics of generated sequences.
[Datasets] For the training dataset, UniRef50 (Suzek et al., 2015) is the commonly used dataset for training protein sequence generative models and language models. Similar to backbone design, conduct comprehensive performance analysis protein sequence generation, we compare the models' performance against the distribution of real sequence data. Specifically, we randomly sam- pled 50 protein sequences from UniRef50 serve as references for the data distribution.
[Mddel implementations]
To ensure comprehensive evaluation, test each method's ability to perform unconditional monomer generation across diverse protein sizes. We follow the official repository and instruc- tions for inference. The generation process is constrained only by specified target lengths, which set at 100, 200, 300, 400, and 500 residues.
1.4 STRUCTURE AND SEQUENCE CO-DESIGN
[Task Definition] Protein gructure-scquent co-design represents an advanced challenge in protein engineering that involves the simultancous optimization of both backbone structure and amino acid sequence. This integrated approach aims to achieve desired properties or functions while maintain- ing structural stability and sequence compatibility. Unlike isolated sequence structure design, co-design explores a significantly larger solution space, making it both more challenging and poten- tially more powerful for creating novel functional proteins.
[Evaluation Metrics] The evaluation of co-designed proteins requires comprehensive framework that addresses both structural and sequence aspects simultaneously. This dual consideration essen- tial because success in either domain alone is insufficient the designed sequence must be compat- ible with its intended structure, and both must contribute to the desired functional properties. The interdependence of sequence and structure necessitates metrics that can capture this relationship effectively.
We implement a multi-faceted evaluation approach that combines metrics from both sequence and structure design domains:
21

---

Published as a conference paper at ICLR 2025
Quality We similar self-consistency- based designability in backbone design for the design task, but with a key difference. The quality for simultancously co-gen oo-generated structure and sequence is about prructure-squeene compatibility by measuring how well the designed sequence can fold into the corresponding designed structure, using scT' and scRMSD, whereas backbone design models require additional inverse folding model, such as ProteinMPNN, to design the sequence.
Novelty Similar to backbone design, we measure the structural novelty compared to known pro- teins, ensuring that designs represent meaningful additions to the protein design space while main- taining realistic properties.
Diversity Similar to backbone design, we measure the structural diversity by maximum pair-wise TM-scores and the number of distinct structural clusters using Foldseek.
[Dataset]] High-resolution protein structures from the Protcin Data Bank (PDB) are the commonly used datasets for this task, with careful consideration given to remove redundancy.
[Model implementations] To ensure comprehensive evaluation, we test each method's ss ability to perform unconditional monomer generation across diverse protein sizes. We follow the official repository and instructions for inference. The generation process is constrained only by specified target lengths, which we set at 100, 200, 300, 400, and 500 residues. This range allows to assess method performance across both small proteins and larger, more complex structures.
B. MOTIF SCAFFOLDING
[Task Definition] Motif scaffolding represents a specialized challenge in protein design that focuses on creating protein structures incorporating specific functional motifs or binding sites. The objective is to engineer a stable protein framework that precisely positions the desired motif while maintaining its functional geometry. This task crucial for developing proteins with targeted functionalities, including enzyme design, therapeutic proteins, and biomolecular recognition systems.
[Evaluation Metries] Evaluating motif scaffolding designs requires a careful balance between maintaining the precise geometry of the functional motif and ensuring the overall stability of the scaffold structure. The assessment must consider both the structural accuracy of the motif placement and the broader protein context that supports it, making this a multi-scale evaluation challenge.
We measure motif scaffolding in terms of both motif accuracy and overall designability. For motif accuracy, we calculate RMSD between the input motif structure and the corresponding region of the designed protein to assess whether the motif structure is preserved (motifRMSD < 1.0)) As for the overall designability, we use scTM score >0,8 as being designable. We have accordingly elaborated on motif-scaffolding evaluation in the appendix.
[Datasets] Datasets typically include libraries of known functional motifs (e.g., catalytic sites, binding interfaces) and diverse scaffold structures that can potentially accommodate these motifs. The Protein Data Bank is primary source, but curated datasets of functional sites like the Catalytic Site Atlas are also valuable.
[Related benchmarks] Enzyme Design Challenge provides relevant test cases. However, given the specificity of motif scaffolding tasks, benchmarks often need to be tailored to the particular class of motifs functions being targeted. Currently, there exists no comprehensive benchmark for this task in the field. A widely used benchmark containing 17 (25) motif-scaffolding problems was used in RFDiffusion WWatson et 2023b).
[Evaluation Metries] Following Yim et al. (2024). we implement several key metrics.
Motif Accuracy We measure the structural retention of motif placement using RMSD calcula- tions, focusing on the geometric alignment of critical functional elements.
Overall Designability We assess the overall stability and structural integrity of the designed pro- tein framework using metrics using self-consistency TM Score
22

---

Published as a conference paper at ICLR 2025
Figure 3: scaffolding.
Sequence-bar based, paacture-based, and co-generation evaluation pipeline motif-
sequence-based
prediction seqpredi StruCCpred
motif-preserving RMSD(ESMFoLd (se stroici,ssruutiiiiiiii (mo
designability
PLODT seqpre)) )=70
structure-based
prediction
seqpredi x
structpredi V
motif-preserving
RMSD ESMFold PNPNN (M (moii)) soft
designability
TNScore ( structpra)>0088
co-generation
prediction
seqpreat //
StruCCpred! //
motif-presene motif-preservvng
RMSD (seeol [motif], struct (mot)))))
designability
(NSooreleS)Fooll(sseerrn)) structpre(>00.8
[Model implementations] Structure- models (FrameFlow. RFDiffusion), sequence-basee models (DPLM, EvoDiff), and multimodal models (ESM3) require different ways to take in- put the motif information and generate the scaffolds. For example, structure-based methods require extra sequence design model to predict sequences of the designed proteins, while sequence- methods cannot directly read motif structure and also require an extra folding model to predict the structure of the designed proteins. Multimodal approaches, structure and sequence simultaneously by themselves. Hence, comparing them in completely iden- tical settings challenging, and we resort slightly different criteria to evaluate these approaches.
the other hand, can read and predict
We focus on two aspects assess the success of motif scaffolding: overall designability and motif- preservation. The overall illustration is shown in Figure 3 Specifically, (1) For the sequence-based method, we only take the generated sequence and utilize ESMFold to obtain the predicted structure, and the PLDDT score provided by ESMFold is used to assess overall quality. (2) For the structure- based method, we only take the generated structure and then leverage ProteinMPNN predict the sequence, followed by ESMFold to predict the structure, where overall quality is assessed by scT'M. (3) For the so-generation method, we take both the generated structure and sequence and predict the structure given the generated sequence with ESMFold, where scTM is calculated between the generated structure and ESMFold predicted structure to evaluate overall quality. Considering that the ground truth motif structure is given, we only utilize the ESMFold predicted structure to calculate motif-RM Rotif-RMMSD
1. ANTIBODY DESIGN
[Task Definition] The goal of antibody design is to generate antibodies that can specifically bind a given antigen. Since the papplmmentarty--eeemmnnnng Regions (CDRs) of antibodies are highly variable and primarily responsible for antigen binding, antibody design could be simplified the design of CDR regions and further reduced the design of the third CDR in heavy chain (CDR-H3). Given the crucial role that protein structure plays in interactions. antibody design usually involves the simultaneous design of the sequence and the structure when binding to the antigen.
[Evaluation Metries] As mentioned in the main text, antibody design can ultimately be simplified to the design of CDR-H3. Therefore, in this study, we evaluate the performance of different anti- body design methods by evaluating the CDR-H3 sequences generated by these methods. Given the primary objective of this study is to assess the relative performance of various design models rather than the in vivo/vitro functionality of the antibodies they generate, we opted to directly evaluate the designed antibodies using their predicted structures. This approach is grounded in several consid- erations: firstly, it ensures a clear focus on evaluating the design methodology itself, independent of experimental constraints. Secondly, the significant time and resources required for extensive perimental validations, as well as the limitations of methods that can accurately simulate the real binding structure of antibodies, render in vivo/vitro assessments impractical. Direct evaluation of
23

---

Published as a conference paper at ICLR 2025
the designed structures presents feasible and efficient strategy that aligns with the study's goals and resource constraints while still providing valuable theoretical benchmarks for subsequent exper- imental investigations.
For methods capable of generating multiple antibodies for the same antigen, we generated 64 CDR-H3 sequences per antigen using each method and calculated the average performance across these different generated samples. Additionally, we also calculated the standard deviation of the performance among different samples generated for a single antigen.
As highly goal-oriented functional protein design task, the evaluation of antibody design is straightforward. namely the Functionality (binding capability to the target antigen) and Specificity of the designed antibody. Additionally, the Rationality of the designed antibodies sequence and structure needs to be evaluated for filtering out invalid designs. However, it remains a challenging task to accurately simulate the performance of antibodies in real wet-lab experiments, including functionality and specificity, using computational methods, and there is a lack of reliable approaches. Therefore, we focus on evaluating antibody design models. Superior models can often provide a better understanding of interactions hetween antibodies and antigens, potentially leading to the production of improved antibodies. Although both the evaluation of design models and antibodies involve assessing antibodies, their approaches differ. The former directly evaluates the structures and sequences generated by the models without modifications, whereas the latter focuses on the actual performance of antibodies in experiments, meaning that their structures may change and differ from the design. Existing studies also evaluate the Accuracy of designed antibodies by measuring their similarity to natural antibodies natural ones are confirmed to be effective. However, using accuracy as an evaluation metric is inadequate in many cases, we demonstrate the misleadingness of AAR and RMSD in 2.1.6
Accuracy:
AAR is the accuracy evaluation of generated sequences compared to refereence/natuaal se- quences. For the calculation of AAR (Amino Acid Recovery Rate), similar to existing work, we calculated the number of residues in the generated CDR-H3 sequences that match the natural antibody.
AAR:
RMSD: RMSD the consistency evaluation of generated structures compared to reference/natural structures. In the calculation of RMSD (Root Mean Square Deviation), we measured the RMSD of the generated and natural antibodies in the CA coordinates of the CDR-H3 region. For methods other than dyMEAN, since their task setting provides the true binding pose of the antibody FR region and antigen, there is no need to align the generated structure with the natural structure when calculating RMSD. For dyMEAN, we aligned the 2 FR residues at each end of CDR-H3 with the corresponding residues in the natural antibody, applied the obtained transformation to CDR-H3, and then calculated the RMSD.
TM-score: TM-score is also the consistency evaluation of generated structures compared refer- ence/natural structures. We calculated the TM-score only for the CDR-H3 region. To this end, we saved the generated CDR-H3 part as a .dd file and used TMalign (Zhang Skolnick) 2005) to calculate the TM-score between the generated CDR-H3 and the natural CDR-H3,
Functionality:
Binding Energy indicates the strength of antibody-antigen binding with the gen- erated structures and we use Rosetta to calculate the Binding Energy. The calculation of binding energy requires the all-atom structure of the protein, while most methods only generate the back- bone atom structure. Therefore, we first used Rosetta to pack the missing side-chain atoms. Sub- sequently, we optimized the side-chains the CDR-H3 region using Rosetta minimization while keeping the backbone structure unchanged to ensure that the CDR-H3 generated by the model reaches the minimum energy state in the binding environment with the antigen. During minimiza- tion, we set the step to 100 (we tried using more steps and repeats, although the energy did further decrease, the reduction was very limited and much smaller than the energy difference between different methods; however, the time consumption significantly increased). After minimization, we calculated the energy on the all-atom structure. Finally, we used the Int anterfaceAnalyzer in Rosetta to calculate the binding energy between CDR-H3 and the antigen.
Binding Energy:
24

---

Published as a conference paper at ICLR 2025
Specificity:
SeqSim: We use SeqSim to detect the mode collapse in sequence generation (or sequence speci- ficity towards different antigens), which indicates that the generated antibodies lost the specificity for specific antigens. SeqSim is defined as the average similarity between any sequence pairs among the generated sequences. First, we introduce the definition and implementation of similar- ity. The similarity between two sequences is defined as the percentage of matched amino acids over the aligned length after alignment (thus, this metric is affected by the length gap between the two sequences). Given that our goal is calculate the number of matches rather than the matching score and that the two ends of CDR-H3 are fixed to FR3 and FR4, we need an align- ment method that: (I) assigns score of for matches, and 0 for gaps and mismatches; (2) does not introduce gaps at the two ends of C3R-H3. We used the Pairwi mesirwiseAA seAl igne in Biopython (Cock et al., 2009) for sequence alignment, setting mat catcche score to 1, all other scores O, and the end.gap.scoo sceaeccoreeecc to -inf so that the alignment process meets our requirements. For methods that generate only one antibody per antigen, we directly calculate the average SeqSim among the 55 generated CDR-H3 sequences as Segerm-outer. For methods that generate multiple antibod- ies, we calculate the average SeqSim between two sets of sequences generated for two antigens SeqSim-outer and also calculate the average SeqSim within each set SeqSim-inner. The formulas for calculating SeqSim-outer and SeqSim-inner are as follows:
N
M M
SeqSim-outer
1 N * (N - 1) * M
Σ
Σ
Σ
Σ
Seq s8),
j=
11j
N
M
M
SeqSim-inner N M (M
1
Σ
SeqSim(s?,.8))
(2)
y 1ars
where N denotes the number of antigens in the test set (N=55 in this study), M denotes the number of samples generated for each antigen (M=64 in this study), and sf represents the x-th CDR-H3 sequence generated for the i-th antigen.
PHR: PHR is the proportion of hydrophobic residues in the generated CDR-H3 sequences, can also reflect the lack of specificity, as the binding caused by the interactions generated by these residues is generally considered to lack antigen specificity. Although both PHR and SeqSim are used to represent the specificity of antibody design methods, they focus on different aspects. Thus, the same method may exhibit different tendencies in these two metrics (SeqSim can be understood an evaluation of the method's specificity, while PHR is an evaluation of the specificity of the generated antibodies. When SeqSim performs poorly, the performance of PHR is of limited sig- nificance). For example, AbDPO achieves high SeqSSm-outer but does not perform well in PHR. This indicates that AbDPO can specifically design antibodies for different antigens, but these an- tibodies contain many hydrophobic residues, leading to potential nonspecific interactions with multiple proteins.
Rationality:
CN-Score: CN-Score is the evaluation of the rationality of the structure by scoring the distribution of generated peptide bond length. To evaluate the consistency of the peptide bond length of generated antibodies with that of natural antibodies, we fit a Kernel Density Estimation (KDE) function using the length of peptide bonds found within the CDR-H3 regions of natural antibodies.
The density of the generated peptide bond length, CN-Score, is used to represent the consisteney. For generated peptide bonds shorter than the minimum natural peptide bond length or longer than the maximum, the density is defined as O. The final CN-Score for a generated antibody is defined as the average density of the lengths of all its peptide bonds. is important to note that the length variation of peptide bonds is very small, which leads to very narrow distribution of natural peptide bond lengths. When the generated peptide bond length deviates slightly from the average length (1)3310), its density in the KDE function will sharply decrease, which explains why all methods show a significant difference in CN-Score compared to natural antibodies.
Clashes: Clashes is the assessment of the potential clashes. Although atomic clashes within pro- teins mainly occur between the side chains, most methods do not generate the side chains of residues. Using packing methods to complete side chains can always find a side-chain confor- mation with the fewest clashes through extensive searching. Therefore, we instead evaluate the
25

---

Published as a conference paper at ICLR 2025
potential clash level in the generated structures rather than the specific number of clashes. To do this, we calculate the CA distance between two residues; when the CA-CA distance between two residues not connected by covalent bond is less than the minimum CA-CA distance commonly found in covalently bonded residues (3.6574, derived from the CA-CA distance statistics in the CDR-H3 region of the RAbD dataset), consider these two residues to have potential clashes. We then calculate the number of residue pairs with distances below this threshold to measure the clash level in the generated structures. The difference between Clashes-inner and Clashes-outer is: Clashes-inner measures the clash level within the generated CDR-H3 structure, while Clashes- outer measures the clash level between the generated CDR-H3 structure and other components, including the antigen, the heavy chain FR region, and the light chain of the antibody.
SeqNat, Sequence naturalness, is the evaluation of the rationality of the generated se- quence. To measure how close the designed CDR-H3 sequences are to natural sequences, we used the pLL predicted by the AntiBERTy model. We input the entire heavy chain sequence into the model, which means that AutiBERTy makes predictions based on the entire heavy chain of the antibody, but unlike the standard procedure in AntiBERTy, the pL.. calculation area is only within the CDR-H region (the standard procedure calculates pLL over the entire input sequence).
SeqNat:
Total Energy: Total Energy is the evaluation of the joint rationality of the generated sequence and structure from the perspective of physical energy. Before calculating the total energy, we performed the same energy optimization process on the designed CDR-H3 regions as described in the Functionality section. We then used Rosetta's full atom score function with the default weights from REF15 (Alford et al.) 2017) to calculate the total energy of each residue in the CDR- H3 region. The Total Energy of the CDR-H3 region is defined as the sum of the total energy of all its residues.
scRMSD: We use scRMSD to evaluate the model's ability of structural modeling by calculating the difference between the designed structure and the simulated structure. In this metric, we used a two-stage method to predict the structure of the generated sequences. In the first stage, we used IgFold to predict the structure based on the sequence pair of the antibody's ' light and heavy chains (although the region we evaluate only exists in the heavy chain, and IgFold also supports single- chain input, we found that inputting two chains results in higher accuracy). The real structure of the non-CDR-H3 regions of the antibody was also provided as a template to obtain the initial predicted structure. We then used the Kabsch algorithm to align the non-CDR-H3 regions of the heavy chain with the real structure and applied the resulting transformation to the predicted CDR- H3 structure. This aligns the predicted CDR-H3 structure to its original complex. At this point, the CA-RMSD between the predicted CDR-H3 structure and the real structure in the RAbD dataset is 1.95. The structure predicted by IgFold is unrelated to the antigen, and since the antibody undergoes conformational changes in the binding interface after binding with the antigen, we used Rosetta to relax the predicted CDR-HL H3 in the presence of the antigen in the second stage. The relaxation involves changes in both the backbone and side-chain structures. Specifically, we repeated relaxation runs five times for each structure predicted by IgFold, with 200 steps each time, and selected the structure with the lowest energy as the final predicted structure. At this stage, the CA-RMSD with the real structure decreased to 1.77. We then calculated the RMSD of the CA coordinates between the predicted structure and the backbone CA coordinates generated by the model, which is referred to as ScR MSD.
The Structural Antibody Database (SAbDab Dunbar et al. (2013) is the commonly used dataset for antibody design. It contains structural data of the antibody-antig anttignn complex, but the data size is limited and contains numerous redundancies. Although SAbDab's official statistics indicate that the database includes over 8,000 entries of complexes containing antigens, only more than 3,000 entries remain after deduplication. Furthermore, researchers typically cluster the SAbDab data based on the sequence identity of CDR-H3, with a clustering threshold generally set at 40% iden- tity. Subsequently, within different clusters, the data is divided into training, validation, and test sets.
[Datasets]
[Model Implementations) We retrained all the methods with unified training data and the official training config for fair comparison and evaluated the methods with unified test data.
Training data:
To build the unified training data, we use antibody-antigen complex structural data from the SAbDab dataset under the IMGT scheme (Lefranc et al.] 2009) the training dataset. We collected
26

---

Published as a conference paper at ICLR 2025
aattignn-antibone complexes with both heavy and light chains and protein antigens. We then discarded duplicate data with the same CDR-L3 and CDR-H3 sequence. The remaining complexes used to cluster via MMseqs22 (Steinegger & Söding 2017) with 40% sequence similarity as the threshold hased on the CDR-H3 sequence of each complex. Finally, we select the clusters that do not contain complexes in the RAbl dataset and split the complexes into training and validation sets with a ratio of 9:1 (1786 and 193 complexes respectively).
Test data:
To build the unified test data, we extracted 55 The original RAbD dataset contains 60 antibody-antigen complexes. In this study, we hope that the evaluation of antibody design methods based on antibodies that contain both light and heavy chains, and simultaneously the antigen contains at least one protein chain. In practice, 2ghw and Buzq lack light chains, while 3h3b lack heavy chains. 5d96 is excluded because of the incorrect chain ID information in rabd commarr..jsnnff where heavy chain J and light chain I do not bind to antigen chain A. 4etq is excluded as HERN reported an error when running for this complex.
complexes from the RAbD dataset.
Model:
All the models are retrained with their default training config. It should be noted that [dyMEAN- FixFRJ is not an official variant of dyMEAN, and we implemented this variant for a fair comparison with other methods. Unlike other methods, which are designed accept the true structure of the antibody-antigen complex and generate the missing CDR-H3 region, dyMEAN is set up to accept only the structure of the antigen and the sequence of the non-CDR-H3 regions of the antibody. Therefore, the model needs to both generate the CDR-H3 region and predict the overall structure of the antibody as well as the binding pose between the antibody and antigen. Incorrect pose estimation can severely affect the interactions between CDR-H3 and the antigen, making direct comparison between dyMEAN and other methods unfair. To compare dyMEAN with other methods more fairly, made some modifications to dyMEAN by providing the true structure of the non-CDR- regions of the antibody and the binding pose, aligning dyMEAN with the other methods. In dyMEAN-FiXFR, consistent with other methods, to avoid the influence of the side chains generated by dyMEAN on the evaluation results. Additionally, we introduced some randomness the initialization of the structure, which allows dyMEAN-FixFR to generate multiple different antibodies for the same antigen.
also used Rosetta (Alford al., 2017 to repack the side chains,
[Extended Explanations and Discussion on Model Performance] Specificity:
SeqSim-outer, we noted that MEAN and dyMEAN generated highly similar sequences for dif- ferent antigens (the maximum SeqSim-outer in our test set was 0.79, indicating that all antibody differences came only from length variations). This suggests that their excellent AAR might stem from learning high-frequency patterns in antibody sequences, generating antibodies according to these patterns for different antigens. In contrast, DiffAb and AbDPO performed the best.
For methods that can generate different antibodies for the same antigen, we also measured the sequence similarity among different antibodies generated for the same antigen (SeqSim-inner We expect antibodies generated for the same antigen be more similar. In this aspect, dyMEAN- FixFR and AbDPO performed the best. However, the 0.96 SeqSim-inner of dyMEAN-FixPH indicates that despite introducing randomness during model initialization, the final sequence gen- eration showed almost no differences. Additionally, DiffAb, which performed best in SeqSim- outer, generated less similar antibodies for the same antigen, suggesting possible underfitting in sequence generation. Considering both types of SeqSim, AbDPO achieved the best performance.
In PHR, HERN and dyMEAN performed the best, but overall, almost all methods performed bet- ter than natural antibodies. Only AbDPO generated an excessive number of hydrophobic residues, reducing specificity. However, its variant, AbDPO++, controlled PHR well, closely matching nat- ural antibodies among all methods.
Rationality:
http///bbbbmmrrrraae susmmaumyyy ry.
json1
27

---

Published as a conference paper at ICLR 2025
In structural rationality, we focused the score for peptide bond lengths conforming to the natural peptide bonds length distribution (CN-score), the number of potential internal clashes in the gen- erated structure (Clashes-inner), and the clashes between the generated structure and other parts (Clashes-outer). It was evident that irrational structures were prevalent in generated antibodies, but overall, diffusion-hased methods performed better. AbDPO++ and DiffAb achieved the best performance among all methods. HERN and MEAN/dyMEAN exhibited different tendencies in inar/oeeinnerruuter,,, corresponding to our observations of the generated samples. HERN tends to generate large CDR-H3 structures, leading to fewer internal clashes but more clashes with the antigen, whereas MEAN/dyMEAN tends to generate smaller CDR-H3 structures.
In sequence rationality, we used the inverse perplexity of AntiBERTy (Ruffolo et al. 2021 to represent sequence naturalness, Seq showing that HERN performed the best, possibly due to HERN being the only auto-regressive model. AbDPO++ achieved the second-best performance and was closest to natural antibodies.
In the joint evaluation of structure and sequence, we mainly focused on the consistency between the generated structure and sequence from two perspectives: physical energy and structure predic- tion. In terms of physical energy, we calculated the total energy of the generated CDR-H3s (Total Energy), which would be severely affected by the clashes caused by sidechains and thus reflect the irrationality between the generated structure and sequence. In this energy-related metric, AbDPO and AbDPO++ performed best among all methods. From the perspective of structure prediction, we used IgFold (Ruffolo al.] (023) to predict the structure of the generated sequence, performed a post-optimization with the antigen as the condition, and calculated the CA-RMSD between the predicted structure and the generated structure (scRMSD). dyMEAN and dyMEAN-FixFR per- formed best in scRMSD. Although these two metrics both reflect the consistency between se- quence and structure, they focus on different aspects. Moreover, both energy calculations and structure predictions have inherent errors, so the performance of different methods may not be consistent across these two metrics.
B.2
PROTEIN CONFORMATION PREDICTION
B.2.1 PROTEIN FOLDING: SINGLE PREDICTION
[Task Definition] Protein folding task predicts the 3D structure of a protein based on its sequence. Folding models such as AlphaFold2 (Jumper et al. 2021) have achieved unprecedented accuracy in predicting protein structures at scale, complementing experimental characterizations and driv- ing advancements in biology and drug discovery (Varadi al. 2022 From modeling perspective, prediction a critical measure of model's 's understanding of these two modal- ities. The ability translate sequences into structures forms the foundation for recent progress in protein conformation prediction (Jing et al.] 2024) ZZhnn et al.) 20024) Wang et al. (202cc). As such, we recognize the necessity of including protein folding in this benchmark, viewing it as a specific instance of protein conformation prediction for single conformational state.
Evvlluttoon Metrics] The primary goal of evaluating protein folding models is to assess their accu- racy in predicting structures of unseen proteins. This is done by comparing the predicted structures to reference structures, such as experimentally determined ones available in the Protein Data Bank (PDB). To ensure an unbiased evaluation, time-based splits are commonly employed, using recently deposited structures of previously unseen proteins for benchmarking (cas) 2022; Robin et al.. 2021). Beyond accuracy, the ability predict protein structures with minimal structural violations pro- vides a reference-free measure of a model's capability to generate high-qualit protein conforma- tions. Unlike the design tasks discussed earlier, structural diversity not a focus in this evaluation. Detailed implementations are provided below:
Accuracy: Structural accuracy measured by the structural similarity with reference struc- tures. Specifically, global similarity metries including TM-score, RMSD and global distance test (GDT) are calculated using TMscore Zhang & Skolnick (2004) obtained from https: / //zhanggroup. org TM-score/ We -sec option to align sequences before structural alignment. Local distance difference test (IDDT) is an alignment free method to compare local structural similarity. We calculate the value using the original implementation (Mariani et al.
expasy......................a... 2013) from https::/t/// sttts:://swii sanodell sew.eeeel.............. wawaaay..org/// lddt/ down loads/
Quality: The structural quality of generated conformations are assessed by CA clash % and PepBond break %:
28

---

Published as a conference paper at ICLR 2025
CA clash % is the rate of potential residue-residue clashes based on the positions of alpha- carbon atoms. clash determined if the distance between a pair of alpha carbon atoms is less than 3.0 A, similar to Lu et al. (2024). And CA clash % is calculated as
number residues with clashes x 100%-
CA clash %
sequence length
- PepBond break % evaluates the potential peptide bond (C-N) break between connecting residues, providing more rigorous metric about inter-residue disconnection than CA level metrics used in Lu et al. (2024). We use a maximum peptide length threshold of 1.4 Á to de- Similarly, PepBond
termine a chain break, as suggested by the Biopython implementation f break % is calculated as
number of C-N bond break
PepBond break % sequence length - X 100%.
[Datasets] The folding models included in this benchmark are those that serve as base models for protein conformational predictions and were established prior 2022. We use CAME02022 from Jing et al. (202) for evaluation, which consists of 183 short-to-middlength single protein chains (< 750 amino acids) from the targets of CAMEO (a continuous benchmarking initiative for structure prediction of newly deposited protein structures) between Aug 1 and Oct 31, 2022. CAME02022
consists of 183 single protein chains collected from CAMEO targets between August and October 2022, with sequence lengths of less than 750 amino acids, following Jing et al. (2023). Protein sequences and structures were extracted from the mmCIF files available at the RCSB Protein Data Bank (https://ww wtt.w..w.ww.rg/////////oom.oo [Berman et al. (2000)). One of the proteins (PDB ID: 8AHP, chain A) has since been superseded by a new PDB entry 8QCW and we have replaced this chain with updated record.
[Model Implementations] Several folding models, including AlphaFold2. OpenFold, Rose TTAFold,, use Multiple Sequence Alignment (MSA) as sequence input. We standard- ize MSA curation using the querying pipeline and the online server provided by ColabFold (Mirdita ctt al. 2022). Templates are not provided in model inference. Additional model implementation details for each model are as follows:
AlphaFold2 (Jumper et al., 202T): We used the ColabFold implementation Mirdita et al. (2022) for AlphaFold2 inference. All five models (with PTM) were used to predict five candidate struc- tures, and the structure with the highest pLDDT confidence score selected for performance evaluation. All models were run with default settings.
OpenFold (Ahdritz with their pretrained OpenFold weights (with pTM). Since only one checkpoint (finetuning no no.templ.pp 1tm.1) corresponding to the model configuration model.3 3.pptm is available, we generated three structures using three random seeds and made a total of 5 predictions. The structure with the highest pLDDT score was selected for performance evaluation.
al.
2022): We used openfold v2.0.0 for inference
ESMFold (Lin et al. 2023a): We use the public ESM repository for inference with the model esm . wwwppprriieeee e esmfoll somfold.vi v1. Since EMSFold predictions are deterministic, generated only one structure per protein for performance evaluation.
RoseTTAFold2 (Back al. 2023): We follow their official repository and instructions for infer- ence. Only one structure per protein was predicted for performance evaluation.
FigenFold (Jing et al. 2023): We follow the official repository, weights, and the setups provided by the authors for inference. In the protein folding task, we sampled 5 structures for each protein and selected the one with the highest ELBO estimation for performance evaluation. Because EigenFold not predict sequences containing unknown amino acids (labeled 'X'), removed the 'X' in the input sequences, as done in the original implementation.
[Extended Explanations and Discussion on Model Performance]
Our benchmarking results (Taile with previous reports (Jing et al. (023). showing that MSA- based folding model generally outperforms protiinlanggaage-modee -awaagggggggmmdeabaeeeeeee based folding model. Eigen- Fold (Jing et al. 2023 one of the first diffusion generative models for both protein folding and
PDB. internal. laoordd..cooooo_ooaao htm² 48
PDB. www.internal_comm coordd. Chain
29

---

Published as a conference paper at ICLR 2025
conformation prediction, shows relatively weaker performance the folding task. Its performance could be limited by several design factors: it built on OmegaFold Wu et al. (2022), uses a coarse- grained representation with only alpha carbons, and has small model size of 572K trainable pa- rameters.
B.2.2 MULTIPLE-STATE PREDICTION
[Task Definition] Multiple-state prediction aims accurately generate more distinct formational states of protein. These states are typically associated with functional conformational changes, such as those induced by ligand binding, metastable states observed during molecular dynamics simulations. The ability to predict these "alternative" conformations, in addition to the folded structure, offers valuable insights into a model's capability to generate plausible stable con- formational states. This serves as an essential first step toward understanding protein conformational dynamics.
EEvllationn Metries] The accuracy and quality metrics for multiple-state prediction are naturally derived from those used in single-state prediction, with some modifications. Unlike single-state pre- diction, multiple-state prediction involves sampling an ensemble of conformations from the model. Given fixed sample size, the accuracy of recovering one state can be evaluated as the best accuracy among all samples compared to the reference structure. For direct model comparison, single accu- racy score is preferred represent the average performance across the recovery of different states. Additionally, transitioning from folding to conformation generation introduces the need to evaluate the diversity of the generated samples, reflecting the model's ability to capture range of plausible conformational states. Implementation details are as follows:
Accuracy:
The accuracy of predicting conformational state is determined by the best structural similarity among the samples to the reference structure, measured by TM-score (for apo-holo) or RMSD (for BPTI). Use RMSD an example:
Accuracy of state k min
RMSD(X,, xF
x, Esmpless
where x is the reference structure for state We then calculate the average accuracy across states in Jing al. (023). referred as "ensemble accuracy":
Ensemble RMSD = K kEK states
Σ
Accuracy of state k
Diversity: Diversity is evaluated by the average pairwise structural similarity among the generated samples for protein, measured using TM-score or RMSD. To reduce computation time, randomly sample 100 pairs of samples for estimation.
Quality: The structural quality is evaluated using CA clash and PepBond break, see single-state prediction for details.
[Datasets] We benchmark the models on two datasets reflect common scenarios in the study of pro- tein conformational changes; apo-holo captures the conformational changes related to specific pro- tein function (i.e., ligand-binde states discovered from long-timi time MD simulations (Shaw al. 2010). Specifically:
processes) (Saldaño et al. (0222) and BPTI captures the metastable
Apo-holo consists of 91 single chain proteins curated by Saldaño et (2022), each featuring pair of experimentally determined conformations: apo (unbound) and holo (bound), representing a two-state prediction task related of both apo and holo conformations were extracted using the same pipeline as in CAMEO2022. Consistent with Jing al. (202)), we use the sequences of the apo state as the primary sequence for model inference. A total of 20 conformations are sampled for each protein during evaluation.
baaund-indding The protein sequences and the structures
BPTI (Bovine Pancreatic Trypsin Inhibitor) is 58 58-amino-acid protein whose dynamics have been extensively studied through long-time MD simulations (Shaw al. 2010). We use the structures of the five cluster centers identified in the MD study as reference structures. This represents a five- state prediction task with Cluster being the most challenging to sample (Wang et al. 2024c). A total of 1,000 conformations are sampled for evaluation.
Model implementations]
30

---

Published as a conference paper at ICLR 2025
EigenFold (Jing al. 2023): The implementation is the same as in the folding task. See the section above for details.
MSA-subsampling (Del Alamo et al. 2022): We implemented MSA-subsampling us- ing the openfold v2.0. package by adjusting the two configuration parameters, max msa clusters s and max ext ra msa, following Del Alamo et al. (2022). Specifically, we refer to max_extra .msa the MSA depth and set ma x_msa_ct usserss to half that depth, while keeping other OpenFold settings at their default values. The original MSAs were obtained using the same ColabFold pipeline as in AlphaFold2.
Str2Str (Lu et al. 2024): We followed the official implementation of Str2Str and used OpenFold- predicted structure as the initial structures. Ensemble results were collected by uniformly sam- pling from & values. For BPTI, we used the author-recommentde
noising schedule with max- imum forward time of T Tim = 0.15 (t = 0.10, 0.15). For apo-holo and ATLAS datasets, we experimented with Tmax = 0.14 (t 0.06, 0.08, 0.10, 0.12, 0.14) and Tmax 0.3 (t 0.06, 0. 021 0.18, 0.24, 0.30) for both the SDE and ODE models.
AlphaFlow/ESMFlow (Jing et al. 2024): We used the official repository and released model weights for inference. The MSAs for AlphaFlow models were obtained through ColabFold's pipeline. We included models pretrained on PDB (-PDB) and fine-tuned on MD datasets (-MD).
ConfDiff (Wang al. 2024c): We followed the authors' implementation and used the released In this benchmark, we used recycle3 representations for both ConfDiff-
weights for inference. Open and ConfDiff-ESM models, with comparison between classifier-free guidance models ClsFree), PDB base models (-PDB), and MD data fine-tuned models (-MD). The energy and force guidance models are sptasct- and are only available for the BPTI dataset with ESMFold representations.
[Extended comparison on protein conformation models]
Recent works on protein conformational prediction have explored several strategies to extend folding models to generate multiple conformations. A common goal across these studies is to enhance sample diversity while ensuring that the generated conformations remain accurate and faithful to the protein, given the high-dimensional space of protein structure. Below, briefly highlight the key differences among the studies and strategies evaluated in this benchmark:
Perturbing the input of folding models. While AlphaFold2 is designed to predict single folded structure of a protein, several studies have proposed perturbing its MSA input to generate alter- native structures (as proxy to conformations) without re-training the model (Del Alamo et al, 2022 Wayment-Steele et al., 2024). In this benchmark, we assess MSA subsampling, a method that reduces the number of input MSAS (referred to as "depth") by subsampling the full MSA, enabling the prediction of different structures due to the depletion of the input information. The depth of MSA controls the trade-off between the sample diversity and how faithful the structure is to the protein.
Perturbing folded structures. Instead of perturbing the input to a folding model, Str2Str Lu et al. 2024) perturbs the structure predicted from a folding model. It uses structure-only dif- fusion model (i.e., a backbone design model) to perturb the input structure through a forward- backward diffusion process. The level of perturbation is controlled by the maximum diffusion time, Tmax- They also used ensembling by sampling at various diffusion times S Tmax-
Training generative models on large-scale structural data from experiments or simulations. A more direct approach involves training sequence-condition
generative models using diffusion or flow frameworks. EigenFold (Jing et al. 2023). AlphaFlow (Jing et al. 2024). and ContD- iff (Wang et al., 2024ch follow similar approaches by fine-tuning diffusion time t-dependent score or denoising model based on folding models, using structural data from PDB. Specifically, AlphaFlow finetunes all layers of AlphaFold2, while EigenFold and ConfDiff use pretrained rep- resentations from folding models and train a lightweight add-on module for score or denoising prediction. Despite adopting generative framework, models solely trained on PDB data are lim- ited in predicting conformational distributions. To address this, AlphaFlow and ConfDiff further fine-tuned their models on a recent MD dataset containing densely sampled conformations for proteins (see Atlas in the Datasets section).
Integrate physical priors in conformational training or sampling. Due to limited availability of large-scale protein conformation data from MD simulation, some models have explored inte-
31

---

Published as a conference paper at ICLR 2025
grating structural and physical priors during training. ConfDiff (Wang al. 2024c) introduced two guidance techniques to improve conformational sampling: (1) classifier free guidance, which combines cnsuuecc-conditione conformation model with an unconditional (structure-only) model to explore conformational space (ConfDiff-CIsFree)) and (2) energy/force guidance, which directs sampling toward regions with lower potential energy (ConfDiff- CConDDiffFFnergy/Fooc through auxiliary prediction modules for intermediate energy/force guidance. However, such physical prediction modules are dataset-specific and requires training additional modules.
(Extended Explanations and Discussion Model Performance]
The complete evaluation results for multiple-state prediction (BPTI and apo-holo) are shown in Table 13 and Table [4]
For BPTI, as discussed in the main text, certain conformation exploration techniques, such as MSA subsampling and guidance used in ConfDiff, have shown their ability to sample diverse structures while staying faithful to the protein.
contast, structure-only approaches like Str2Str perform poorly this task, likely because these models do not ensure that the perturbed structure remains faithful to the provided sequence. On the other hand, EigenFold shows limited diversity, as was trained solely PDB structures and does not incorperate conformation exploration strategies. This limits its effectiveness in sampling diverse samples. While AlphaFlow and ESMFlow demonstrate competitive performance, fine-tuning the MD dataset introduces trade-offs in quality, notably an increased incidence of peptide bond breaking between residues.
For apo-holo, strategies to improve sample diversity such as reducing MSA depth, applying structural perturbation, fine-tuning MD conformation data, using classifier-freee guidance generally do not improve (and sometimes even reduce) the TMens score. Interestingly, we found that the best-performing models are those that most closely resemble folding models (e.g., MSA-depth256, AlphaFlow-PDB; The included baseline apo, that always predicts the perfect apo structures, confirmed that higher TMens score can result from accurate prediction of one of the states. These findings suggest that using better folding model provide strong baseline performance for conformational sampling but none of the current models show clear evidence of effectively modeling conformational changes during complex biological processes, such as ligand binding.
Table 13: Complete performance on the multiple-state prediction of BPTL. Accuracy metrics (RMS- Dens, RMSD Cluster 3) are reported as the mean and standard deviations from 20 bootstrap samples with replacement, at different sample sizes (N = 10 ~ 1000). Diversity and Quality scores are evaluated based 1,000 conformations for each model. The best performance is highlighted in bold, and the second-best is underlined. "N/A" indicates not applicable due to model resolution.
RMSD is measured A.
RMSDens
RMSD Cluster 3
Diversity
Quulity
Pairwise RMSD
CA clash
PepBond break
N=10
N=100
N=1000
N=10
N=100
N=1000
EigenFold
1.56:0.02
1.5010.11
1.4620.00
2.5410.03
2.48=0.01
2.46:10.01
0.85
1.4
N/A
MSA-depth256
1.58:10.01
1.54+00.01
1.52:4001
2.51+:0.2
2,48=0.01
2.44::0.01
0.20
0.0
9.2
MSA-depth64
1.60:::.01
1.55++0.02
1.51:10.01 2.46:00033
2.41=0.04
2.34:1..03
0.55
0.0
7.9
MSA-depth32
1666::0.3
1.5410.04
1.4110.02
2.43::0.66
2.19=0.16
1.85:0.00
0.6
10.6
Str2Su-ODE (T....
0.15)
2,40:10.12
2.20=:0.05
2.09:10:01
3.00+0.20
2.73=0.12
2.58-10.05
1.86
0.0
13.9
Str2Str-SDF (Tim
0.15)
2.76+10.16
2.46++0.08
2.26:10.04
3.26-H0.25
2.86=:0.55
2.55:+0.16
3.60)
0.3
16.0
AlphaaFoow-PDB
1.53:10.03
1.46:0.01
1.41:10.11
2.8810.04
2.43=0001
2.40:10.01
0.0
AlphaFlow-MD
1.71=0008
1.51
1.43#0001
2.46:00.09
2.32=0.06
2.25+10.01
1.26
0.0
26.2
ESMFlow-PDD
1.59:10.04
1.49-0.022
1.4240001
2.49-10.03
2.41=00.33
2.34:10.01
0.74
0.0
6.0
ESMFlow-MD
1.68:1006
1.47+0.04
1.39-1.03
2.44::0111
2.27=0.10
18:00.0002
0,0
14.3
ConfDiff-Open.Wee
1.64++0.05
1.50+-0.02
1.43++0.02
2.50+H0005
2.38
2.31-H0.112
1.37
112
4.6
ConfDiff OpeneClFree
1.66:00066
1.50:0.04
1.3740.02
2.56:0.007
2.39=0.17
2.024-0.10
1.77
0.5
ConDDiff-ESM-MD
162410.04
1.7200.02
1.40:1.01
2.4510.09
2.32=0.05 2.25::0.22
1.42
0.1
ContDitf- ESM-ClsFree
1.57+40.44
1.45+00.22
1.40=#0.01
2.48:00044
2,40=0.03
2,34::0.22
1.80
0.5
7,5
ConiDiff ESM Energy
1.61:10.03
1.46
1.42:10011
2.51:10.05
2.44=00.3
240:10.01
1.22
0.1
7.5
1.58:10.04 1.3310.03 1.36:0.01 2.4410.06 2.35=0.05 2.24100066
1.76
0.1
B.2.3 DISTRIBUTION PREDICTION
[Task Definition] Distribution prediction challenges models to generate distributions that closely resemble a target distribution, such as the empirical distribution obtained from molecular dynamics (MD) simulations. Unlike previous two tasks, which focus recovering specific conformations, this task requires models to demonstrate an understanding of "physics and energy" to accurately predict the conformational landscape the distribution level. This approach further bridges the
32

---

Published as a conference paper at ICLR 2025
Table 14: Performance on the conformation prediction task for the apo-holo dataset. apotholo- TM represents the maximum TM-score of the samples relative to the reference apolholo structure. Twenty conformations were sampled for each protein, and the results are reported as mean/median across 91 proteins. The best performance is highlighted in bold, and the second-best is underlined. "N/A" indicates not applicable due to model resolution.
Accuracy
Diversity Pairwise TM CA clash %
Quality
apo-TM
holo-TM
TMens
PepBond break %
ape model
1,000)
0.790
0.895
N/A
N/A
NIA
EigenFold
0.831
0.864
0.847
0:907
3.6
N/
MSA-depth256
0,889
0.862
0.978
4.6
MSA-dopth664
0,844
0.883
0.363
0.950
0.2
5.7
MSA-depthi32
0.824
0.857
0,841
0.864
0.2
8,9
Str2Str-OER (Teus
0.14)
0.762
0.778
0.770
0.954
0.2
14.0
Str2Str ODE 17
0.3)
0.766
0.781
0.774
0.872
0.2
14.7
Str2St-SDE (fim = 0.14)
0.682
0.693
0.688
0.760
0.2
22.6
Str2St-SDE ( 0.3)
0.680
0.689
0.684
0.639
0.2
21.1
AlphaFlow-PDD
0.924
AlphaFlow-MD
0.857
0.863
0,860
0.894
0.2
20.8
ESMFlow-PD8
0.849
0,882
0,866
0.935
0.3
48
ESMFlow-Me MD)
0,851
0.864
0.858
0.897
0.1
10.9
CoalDiffOppen-PDD
0.847
0,886
0,867
0.909
0.5
5.5
ConfDiff-Open-Cheee
0.879
0.859
0.870
ConIDiff Open MD
0,839
0,874
0.857
0. 0.863
0.4
6.8
ConiDidE-ESM EEE--EM-PDDDD
ConDiif-ESM-CI--P
0.837
0.864
0.850
0.846
0.7
4.6
ConfDiff-ESM-M
0.836
0,862
0,849
0.846
4.1
gap between protein conformation prediction models and MD-based methods for studying protein dynamics and thermodynamic properties.
[Evaluation Metrics] The diversity and quality evaluation are the same as in the previous task. We also included the average RMSF (root mean square fluctuation) as an additional metrics for atom-level diversities. dynamics of proteins, extended the implementation from Jing al. (2024), with modification to explicitly align atom orders in mdtraj before comparing sample and reference structures. The accuracy evaluated from three sub-categories:
evaluate the accuracy of capturing the conformational distribution and
Flexibility: This metric assesses how accurately the generated samples reflect the protein's 's flexi- bility at both the protein and atom levels. It is quantified by the Pearson correlation coefficient (r) between the diversity measures, such as Pairwise RMSD for protein or RMSF for atoms, of the model-generated samples and those of the reference MD samples.
Distributional accuracy: This category evaluates the model's accuracy on recovering the target distributions. Wasserstein-2 distances are used measure the similarity between model-generated and reference distributions. RMWD is the root mean Wasserstein distance between the distribu- tions aligned coordinates, modeled as multivariate Gaussians. We also evaluate the W2 distance in the PCA projected subspace (PCA W2). Additionally, the cosine similarity between the first principal components from PCA analysis of model -ener-ggeeeated and reference conformations serves another indicator of how well the model captures the correct subspace.
Ensemble observables: Another objective of conformational sampling is to identify certain func- tionally relevant behaviors (so called observables). such as transient residue-residue contacts ob- served in molecular dynamics. The accuracy recovering such observables is assessed by com- paring those derived from the model generated conformations to reference conformations, using metrics like Jaccard similarity or Spearman correlation.
[Datasets] We evaluate performance using the ATLAS dataset (Vander Meersche et al., 2024), recent database of MD simulation results for diverse proteins. To avoid data leakage for models trained on portions the ATLAS dataset, we follow DallowJingggo et al. (2024) and benchmark on 82 proteins whose PDB entries were deposited after May 1, 2019 and not part of the training validation set. ATLAS is a recently published dataset containing triplicated 100 ns MD simulations for 1.390 diverse single-chain proteins. In this work, we were deposited after May 1, 2019, following Jing et al. 2024). "Protein-only" trajectories were
subset of 82 proteins whose PDB entries
ensemble .p
33

---

Published as conference paper at ICLR 2025
downloaded from the ATLAS database for evaluation. We sample 250 conformations for each protein for evaluation.
[Model Implementations] The models are implemented the same as the multiple-state prediction task. See the previous Section B.2.2 for details.
[Extended Explanations and Discussion on Model Performancel
The complete evaluation results for distribution prediction (Atlas) are shown in Table 13] When comparing the effects of structure exploration strategies, classifier guidance used in ConfDift stands out as the only approach that improves upon the base PDB model. In contrast, strategies such as MSA subsampling and structural perturbation in Str2Str negatively impact most accuracy metries. However, as discussed in the main text, fine-tuning models on protein conformational data from MD simulations proved to be the most effective strategy, offering significant improvements over the base models. This contrasts with the challenges observed in multiple-state prediction tasks. The likely reason is that capturing conformational distributions requires detailed physical insights, which are difficult to extract solely from PDB structural data.
Table 15: Performance on distribution prediction for the ATLAS test set. A total of 250 conforma- tions were sampled for each protein, and the median values across 82 proteins are reported. The best performance is highlighted in bold, and the second-best is underlined. *These metrics require all-atom backbone predictions; therefore, FogenFold and Str2Str do not have sufficient resolution for evaluation (indicated as "N/A").
Diversity
Flexibility: Pearson
Distributional accuracy
Pairwise RMSD
Pairwise RMSD
GGobbal RMSF
Per target RMSF
MD PCA
Joint
PC sim
*RMSF
'RMWD
W2
PCA W2
0.5 %
MD lid
2,76
1.3
0.96
0.97
0.99
0.67
0.73
0.71
91.9
EigenFold
5.96
N/A
-0.03
N/A
N/A
N/A
2.31
7.96
12.2
dSp-depph256
0.83
0.53
0.25
0.34
0.59
3.60
1.79
2.91
29.3
MSA-deptb64
2,03
1,51
0.30
0.57
4,00
1.94
3,34
18.3
MSA-depth32
5,70
7,96
0.08
0.17
0.53
6.09
2,56
5,70
17.1
(Tea 0.14)
NIA
Str2StrODE (Tua
0.3)
3.15
N/A
0.13
N/A
N/A
N/A
2.19
4,80
9.8
St2Sa-SDD (Tam
0.30
7.54
N/A
0.01
N/A
N/A
N/A
3.24
12.28
7.3
AlphaFlow-PDR
2.58
1.20
0.46
0,81
37.8
AlphaFloo-MM
2.87
1.63
0.53
0.66
0.85
2.64
1.55
2.29
39.0
ESMFlow-PDB
2,99
1668
0.14
0.27
0.71
4.15
1.87
3.61
28.0
ESMFlow-MD
3.33
2.13
0.30
0,76
3.61
1.66
25.6
CentDif-Open-Clssaae
3,68
2.12
0.39
0,54
0183
2911
1.4
2.46
46.3
ConfDiff Open.PDB
ConiDiff Open MD
3.43
2.21
0.39
0.67
0.85
2.75
141
2.27
35.4
ConfDiff SonfDif-FFSM-CO FS
4.04
2.84
0.43
3.78
1,73
37.8
ConfDiff ESM. PDB
3.42
2.06
0.29
0.40
0.80
3.62
1.68
3.13
34.1
CoolDir-ESMN ESM-MD
3,90
2.79
0.35
0.48
0.82
3.62
1.73
3.00
37.8
Ensemble observables
Quality
Weak contacts
Transient contacts J1
"Exposed residue J
"Exposed MI
CA clash
"PepBond break %
matrix p
MD sid
0.90
0.80
0.93
0.56
0.0
3.4
EigcnFold
0.36
0.19
NIA
N/A
5.6
N/A
MSA-depth256
0.30
0.29
0.36
0.06
0.0
5.5
MSA-deppth64
0.38
0.28
0.40
0.16
0.0
7.6
MSA-deptb32
0.40
0,24
0.4D
0.19
0.1
11.2
0.14)
Stz2St-ODE (Ti.
0.3)
0.42
0.17
NIA
N/A
0.0
13.2
Su2St=SDE
St/2at-SDB (T... 0.30
0.36
0.13
NIA
N/A
0.2
20.2
AlphaFlow-PDD
0.45
0.36
0.50
0.25
AlphaFlow-MD
0.62
0.41
0.35
ESMFlow-PDB
0.42
0.30
0.46
0.11
0.2
ESMFlow-MD
0.55
0.34
0.29
0.1
10.9
CoofDiff-Open SaeeeOppnn-Cspppee
0.58
0.36
0.60
0.28
0.8
5.7
ConIDiff Open.PDB PDB
ConiDiff-O Open MD
0.63
0.39
0.65
0.33
0.5
6.5
CoutDiff-ESMA ESM-ClsFre
ConfDiffEEMM --B
0.50
0.33
0.50
0.24
0.5
40
ConfDiff SonDiff-EMMMMI ESM-MD
0.61
0.36
0.61
0.31
43
htttps://ww/ diimb. inserm. fr/ATLA8/indees htm1
34

---

Published as a conference paper at ICLR 2025
C DETAILED DISCUSSION OF KEY OBSERVATIONS
Valid evaluation of protein foundation models requires accurate and comprehensive evalua- tion metries. The emergence of folding models like AlphaFold2 and ESMFold offers opportunities for precise assessment of quality, stability, and accuracy in protein generative tasks. However, cer- tain complex tasks may still lack sufficiently accurate evaluation methods. For example, within the realm of antibody design, researchers have at times been misled by reconstruction metrics like Amino Acid Recovery (AAR) and Root Mean Square Deviation (RMSD) related aceuracy, result- ing in overly optimistic conclusions. In this study, we intend to tackle this challenge by proposing an evaluation strategy integrating reconstitution and physical rationality metries. Also, we provide multifaceted evaluation strategy to capture various facets of protein structure and function, foster- ing a more holistic understanding of the performance of foundation models. Furthermore, metrics alone are insufficient. In the development of generative models for protein, the primary objective is to accurately fit the distribution of the training data. Our evaluation adopts a more comprehensive strategy that includes measuring the same metrics for the training data (which encompasses native proteins, antibodies, and molecular dynamics conformations in various lengths). This provides a high-resolution gold reference for protein generative targets.
No single model currently excels across all protein design objectives. The choice of model should be carefully aligned with the intended applications. In the field of protein foundation models, two primary approaches have emerged: language models and geometric models. Each ap- proach has its strengths and limitations, which are reflected in the performance of ProteinBench. We found language models show good performance in capturing nature evolution distributions, ev- idenced by their high accuracy in native sequence recovery (inverse-folding) and high quality in scaffolding wolution-conserved motifs. However, language models show limitations robustness when designing sequences for de novo backbones, and in generating novel sequences for sequence- based protein design. In contrast, structure- models exhibit greater robustness and tolerance for structural noises in de novo design task, and show greater potential for creating proteins with new folds or functions. These findings underscore the importance of carefully considering specific design objectives when researchers are selecting a model to use.
While generative models extended from classic folding models have shown an ability to sample protein conformations, challenges remain in both multiple-state prediction and distribution prediction. Protein conformation prediction is new but crucial assessment of the multi-modal capabilities and physical understanding of protein foundation models. While strategies proposed in current models may benefit certain tasks, they often provide limited improvement in others. For example, although fine-tuning models using the MD conformation dataset showed promising results on the ATLAS benchmark, little to no improvement was observed in the multi-state prediction of apo-holo conformations. Additionally, the common trade-off between diversity and quality in cur- rent models underscores the importance of consistent evaluation across the dimensions of accuracy, diversity, and quality in protein conformation prediction tasks.
35


Reviews:
[{"review_id": "5Ip40otUwC", "reviewer": "Reviewer_jXTi", "summary": "The authors introduce a standardized benchmarking framework to evaluate the performance of protein foundation models. The framework includes 1) task taxonomy, categorizing key tasks in protein modeling (protein design, confirmation prediction);  2) multi-metric evaluation across quality, novelty, diversity, and robustness dimensions. 3) In-depth analyses based on different user objectives. The study finds that thorough evaluation metrics are crucial for adequately validating protein models and that no single model is optimal across all protein design tasks, which underlines the need to match models to specific applications. The authors intend the framework to serve as a collaboratively developed comprehensive and transparent benchmark for evaluating protein foundation models.", "questions": "- The paper highlights four key dimensions (quality, novelty, diversity, and robustness), but the result tables, including Table 1, emphasize only the first three dimensions. Why are robustness metrics omitted from the tables? Additionally, some dimensions in Table 1, such as those in antibody design, do not align with these four key categories.\n-\tGiven that no model excels across all metrics, how should the models be ranked on the leaderboard, given the trade-offs across different metrics?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "-\tThe framework’s taxonomy of tasks within the domain of protein foundation models is insightful.  It makes it easier to evaluate where each model excels or falls short.\n-\tThe multi-dimensional metrics aims to capture various aspects of model performance which is appropriate given the complexity of the protein modeling. \n-\tThe authors conduct a large number of experiments, demonstrating the breadth of the evaluation and ensuring the results' validity across various models and tasks. \n-\tLeaderboard and open-source code can potentially facilitate more fair comparison and promote transparency.", "weaknesses": "-\tGiven that the authors have made an extensive amount of experimental study, some reorganization of the paper could strengthen the delivery of the contributions of the paper. Including clear and complete definitions, explanations, and relevance of the metrics would be helpful. The relevance and insights of the results could replace the explanations of the results. For example, Section 2.2.6 Antibody Design, instead of listing the outperforming models for evaluation, which is provided in Table 6, authors could discuss the relevance of these metrics along with the insights gained from the results similar to the one that they provided in the last paragraph.  \n-\tLack of consistency in the training data across models is a limitation that undercuts the one of the main promises of the proposed framework which is standardization of the evaluation of protein foundation evaluation. This may not be an issue in the future as the framework is further developed and more mature. \n-\tThe authors mention that no model performs optimally across all metrics. It does not, however, fully explore the causes and potential trade-offs. Insights into why certain models perform better for certain tasks would provide better guidance on choosing task-appropriate models.\nMinor Issues:\n-\tThe description of Table 12 does not align with the data presented in the table\n-\tItems (3) and (4) in the conclusion are the same.\n-\tFigure 2 is too small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xyfkcdLxdE", "reviewer": "Reviewer_bamz", "summary": "I am not sure how to review this manuscript. I have been in the field for over 20 years, so I am an experienced ML/Deep learning researcher, but I do not know much about deep learning for protein design/generation beyond having read blog posts about AlphaFold, ESM, and similar advances. I do know a fair bit about biology (probably more than most ML researchers), but I am still far from an expert in this area. I thus have some high-level opinions about the work, but cannot evaluate the vast majority of the content. I would like my high-level thoughts to be considered, but I cannot evaluate any of the details in the paper, so I have to defer to other expert reviewers to do that. Here are examples of the questions I cannot evaluate: Are the challenges in the benchmark the right ones? Are they comprehensive enough? Are the models evaluated the right ones? Are major ones missing? Were the evals done fairly and correctly? All of that I’ll have to leave for subject matter experts. \n\nThat said, here are my high-level thoughts:\n\nThe field benefits from benchmarks, and we should reward people who take the time to make them. I thus support publication because from what I can tell, this is a new, needed, helpful benchmark. However, I would not know if other similar benchmarks already exist. \nIt is also helpful to have a set of leading models tested on such benchmarks. Assuming the set of models is a good one and the evals are well done, that is another contribution worthy of sharing with the community via a publication. That said, I do not know how novel such a comparison is. \nThe paper does a poor job of explaining almost anything in a way that a non-domain expert could understand. The worst example of this is the challenges in the benchmark. They are extremely superficially described, with a reference to the Appendix presumably doing all the work of explaining what these challenges really are, how they are evaluated, why they matter, why they were chosen, etc. I think more of that belongs in the paper. I’d rather see a paper arguing for and introducing a benchmark do MUCH more of that motivation and explanation than a slog through the performance on the benchmark of a bunch of models. After all, why do we care how these models perform on problem x before we know what problem x is and why it matters? That said, perhaps domain experts already know these challenges so well that the current level of text is sufficient? I am not in a position to judge, but I think the paper would likely be higher-impact and better if it was more readable by non-insiders. I was hoping to learn a LOT about this area by reading this manuscript, and instead, sadly, I learned basically nothing (except there is a new benchmark). The reason I am still voting for publication despite that is I imagine this paper is quite valuable for insiders, but great papers do better at offering *something* to non-insiders. \n\nMinor: \nYou say ProteinBench provides a holistic view. That is too strong. It is virtually impossible to provide a holistic view of such a complex topic. Please switch such language to “more holistic” or similar. \nYou say benchmarks are crucial or critical for progress. This is a strong claim. I think they are helpful, but progress can be made without them, and they can actually hurt progress too (see Why Greatness Cannot Be Planned), so I recommend a more nuanced statement. \nLine 181: You say this finding has significant implications for the field, but it is drawn from 2 data points only! Please properly soften the claims. \nLine 192: “We have noticed that….” That’s a weird way to describe your own work/choices. You sound surprised you didn’t include more methods? I suggest finding clearer, less confusing language. \n\nNote: I am rating this a 6 and not an 8 simply because I do not know enough to evaluate the vast majority of the work. If the other reviewers think it is good, then this 6 should support publication. But I do not want to set a super high score that would override experts with doubts. If all experts agree the technical work and novelty are solid, I'd be happy with an 8.", "questions": "See main review.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "See main review.", "weaknesses": "See main review.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bHbBO6uiQk", "reviewer": "Reviewer_hosV", "summary": "The paper  introduces ProteinBench, an evaluation framework aimed at standardizing and broadening the assessment of protein foundation models. These models have gained prominence due to advancements in protein prediction and design, covering tasks from structural prediction to conformational dynamics. ProteinBench aims to address gaps in current evaluation practices by introducing a comprehensive, multi-dimensional approach that evaluates models based on quality, novelty, diversity, and robustness. The authors aim for ProteinBench to become a continually updated benchmark to guide research and collaboration in protein modeling.", "questions": "1. I am not very familiar with AI for Protein. Could you provide with the reason why you separate whole protein tasks to these 8 parts?\n\n2. Could you give some additional metric about tasks? While ProteinBench is a strong foundation, additional tasks and metrics (especially regarding dynamics and multi-modal integrations) could improve its scope, making it more universally applicable in protein science.\n\n3. Could you plan to implement controls to account for differences in training data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Novel Evaluation Framework: The paper proposes a well-structured framework that standardizes evaluation for protein foundation models, addressing a significant need in the field. By evaluating on multiple fronts—quality, novelty, diversity, and robustness—ProteinBench gives a well-rounded assessment of model performance.\n\nTask Diversity and Practical Relevance: ProteinBench is inclusive of various protein modeling tasks, including antibody design and multi-state prediction, which are highly relevant to real-world applications in pharmaceuticals and bioengineering.\n\nUser-centered Analysis: The framework is flexible, accommodating different user needs (e.g., evolutionary fidelity vs. novelty), which makes the tool versatile for diverse research goals. This feature improves the applicability of model results to specific scientific or engineering contexts.", "weaknesses": "Lack of Standardized Training Data: Differences in training datasets among models hinder direct comparison. Standardizing datasets would improve the ability to compare model architectures and may be essential for achieving fairer assessments within ProteinBench.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GVOKvrCnb1", "reviewer": "Reviewer_d7D6", "summary": "The paper introduces a unified protein benchmark to evaluate various methods for different tasks such as protein structure prediction, sequence design, structure design, sequence-structure design, and molecular dynamics. For each task, results are presented for various state-of-the-art methods and a discussion is followed on the strengths/weaknesses of these methods. The evaluation is performed using multiple metrics commonly used for these tasks, and, that sometimes incorporate different objectives in protein design. The benchmark is planned to be shared in an open-source manner for the community to compare methods for the presented tasks.", "questions": "Comments:\n\n1.\tMy main concern is related to how the paper is structured and the information it contains. From Page 3 the task results are presented with a small definition and comparison between methods. I see that the authors presented the creation of unified datasets as a current limitation and future work by the community, but the reviewer thinks that the manuscript should contain more information about the thought process about creating the benchmark, with the thought process about the metrics, the impact of methods using different datasets, tasks in which this is critical, etc. These are crucial for the community to adopt the benchmark and trust the results that are being presented and compared. In its current form, it is hard to understand these intrinsic details that are important for protein-related tasks.\n2.\tThe authors define the methods as “protein foundation models” and add their own explanation of how this definition is being used. From the reviewer's understanding, usually, foundation models are defined for methods that can be applied, e.g. using their latent space, for many different tasks.  Any additional reasoning for using this new definition of protein foundation models?\n3.\tMotif Scaffolding: For the Motif Scaffolding task, which evaluation metrics are being used? The reviewer is confused if RMSD metrics are being used or also designability metrics are being used.\n4.\tAntibody Design: Antibody metrics are usually very challenging to trust. As a benchmark, some of the metrics such as the scRMSD from structure prediction networks like IgFold can be misleading, as we are more interested in antigen-antibody complex structures. I understand current structure prediction networks accuracy for antibodies is limited, but it would be interesting to discuss and choose reliable metrics, even at a reduced number. As the evaluation metrics evolved, they could be added to the benchmark.\n\nMinor Comments:\n\n1.\tLine 81: “we aims”\n2.\tLines 193-195: “We have noticed…” and Lines 214-215: “We will soon…”: These sentences can be re-written to mention or list just current state-of-the-art methods that are currently not evaluated.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tA benchmark with multiple metrics is proposed for protein-related tasks. The metrics incorporate different objectives for different problems encountered in protein design and molecular dynamics tasks.\n2.\tThe paper tackles an important and complex problem in the protein community which is the creation of benchmarks and datasets for training AI methods.\n3.\tThe benchmark incorporates recent challenges like evaluating sequence-structure compatibility in recent co-design methods.", "weaknesses": "1.\tThe evaluation of methods for protein-related tasks is challenging and, usually, the decision of dataset curation and splits, metrics used for evaluation, and other small details are very important so the results can be trusted by the community. The reviewer thinks that the manuscript could have more information on all these details, instead of presenting results and discussing the performance of the methods. Some of these details are presented in the Appendix while others are missing.\n2.\tSome metrics used for the benchmark, even though used before by previous references, might need more discussion as they can be misleading for checking the quality of designs, e.g. scRMSD in antibody design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)