CosmoCLIP: Generalizing Large Vision-Language Models for Astronomical
  Imaging
Authors: name: Raza Imam, name: Mohammed Talha Alam, name: Umaima Rahman, name: Mohsen Guizani, name: Fakhri Karray

Existing vision-text contrastive learning models enhance representation
transferability and support zero-shot prediction by matching paired image and
caption embeddings while pushing unrelated pairs apart. However, astronomical
image-label datasets are significantly smaller compared to general image and
label datasets available from the internet. We introduce CosmoCLIP, an
astronomical image-text contrastive learning framework precisely fine-tuned on
the pre-trained CLIP model using SpaceNet and BLIP-based captions. SpaceNet,
attained via FLARE, constitutes ~13k optimally distributed images, while BLIP
acts as a rich knowledge extractor. The rich semantics derived from this
SpaceNet and BLIP descriptions, when learned contrastively, enable CosmoCLIP to
achieve superior generalization across various in-domain and out-of-domain
tasks. Our results demonstrate that CosmoCLIP is a straightforward yet powerful
framework, significantly outperforming CLIP in zero-shot classification and
image-text retrieval tasks.