Towards Data-Efficient Pretraining for Atomic Property Prediction
Authors: name: Yasir Ghunaim, name: Hasan Abed Al Kader Hammoud, name: Bernard Ghanem

This paper challenges the recent paradigm in atomic property prediction that
links progress to growing dataset sizes and computational resources. We show
that pretraining on a carefully selected, task-relevant dataset can match or
even surpass large-scale pretraining, while using as little as 1/24th of the
computational cost. We introduce the Chemical Similarity Index (CSI), a novel
metric inspired by computer vision's Fr\'echet Inception Distance, for
molecular graphs which quantifies the alignment between upstream pretraining
datasets and downstream tasks. By selecting the most relevant dataset with
minimal CSI distance, we show that models pretrained on a smaller, focused
dataset consistently outperform those pretrained on massive, mixed datasets
such as JMP, even when those larger datasets include the relevant dataset.
Counterintuitively, we also find that indiscriminately adding more data can
degrade model performance when the additional data poorly aligns with the task
at hand. Our findings highlight that quality often outperforms quantity in
pretraining for atomic property prediction.