The datasets used for evaluating protein foundation models may exhibit several limitations and biases, including:
Sampling Bias: Datasets may over-represent certain protein families or structures while under-representing others, leading to models that perform well on familiar sequences but poorly on novel or less common proteins. To address this, ProteinBench could implement a more diverse dataset curation strategy, ensuring that datasets encompass a wide range of protein types, structures, and functions.
Data Quality: The quality of the data can vary significantly, with some entries in databases like PDB being of lower resolution or containing errors. ProteinBench should incorporate data quality assessments as part of its evaluation metrics, filtering out low-quality entries and ensuring that only high-resolution structures are used for training and evaluation.
Temporal Bias: As protein science evolves, older datasets may not reflect the latest advancements in protein modeling and design. Regular updates to the datasets used in ProteinBench, including the incorporation of newly published structures and sequences, would help mitigate this issue.
Functional Bias: Datasets may focus predominantly on structural data without adequately representing functional aspects of proteins. To address this, ProteinBench could include datasets that emphasize functional annotations, such as binding affinities or enzymatic activities, ensuring that models are evaluated not just on structural accuracy but also on their functional relevance.
By addressing these limitations and biases, ProteinBench can ensure a more representative and unbiased assessment of protein foundation models, ultimately leading to more reliable and generalizable predictions.