Title: Benchmarking Self-Supervised Vision Transformers in Astronomy
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: AI4Science, self-supervised pre-training, benchmark datasets, transfer learning

Abstract: This work does not describe a novel method. Instead, it aims to extend the success of self-supervised pre-training on natural images to astronomical data.
To address the lack of comprehensive benchmarks in astronomy, we first curate an unlabeled pre-training dataset and multiple datasets for typical astronomical tasks. 
Through extensive experiments, we demonstrate that our pre-training scheme has the following advantages.
Representation transferability: pre-training followed by fine-tuning can not only significantly boost performance but also reduce training epochs compared to from-scratch training on downstream tasks (e.g., improve 12\% accuracy and reduce 83\% epochs in galaxy classification), mirroring trends in natural image domains.
Cross-instrument generalization: our pre-trained model generalizes across telescope instruments and outperforms domain models.
Domain-specific pre-training data value: In-domain pre-training data further improves model performance and surpasses the results trained on general datasets such as ImageNet and other domain datasets. 
Furthermore, we explore Vision Transformers (ViTs) scaling in astronomy via parameter and data variation to offer insights and experiences for  vision foundation model development in astronomy.

TL;DR: This work aims to extend the success of self-supervised pre-training on natural images to astronomical data.