Title: Genomic Foundationless Models: Pretraining Does Not Promise Performance
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: ai4science, foundation models, genomics, biology, pretraining, deep learning

Abstract: The success of Large Language Models has inspired the development of Genomic Foundation Models (GFMs) through similar pretraining techniques. However, the relationship between pretraining performance and effectiveness in downstream genomic tasks remains unclear. Additionally, the high computational cost of pretraining raises questions about its cost-efficiency. To assess the usefulness of pretraining in genomics, we evaluated seven different GFMs across 52 diverse genomic tasks, comparing them to their counterparts with randomly initialized weights. Surprisingly, we found that randomly initialized models can match or even surpass the performance of pretrained GFMs in finetuning and feature extraction tasks. We also discovered that pretrained GFMs fail to capture clinically relevant genetic mutations, which are crucial for understanding genetic disorders and phenotypic traits. Our results indicate that most of the current pretrained GFMs lack a ``foundational'' understanding of genomics and provide minimal utility, even for basic tasks such as sequence classification. These findings collectively highlight the need to critically rethink the pretraining approaches for genomics.

TL;DR: Pretrained Genomic Foundation Models offer little to no advantage over randomly initialized models on many genomic tasks.