CosmoCLIP: Generalizing Large Vision-Language Models for Astronomical
  Imaging
Authors: name: Raza Imam; name: Mohammed Talha Alam; name: Umaima Rahman; name: Mohsen Guizani; name: Fakhri Karray
Published: 2024-07-10 02:24:43

Abstract: Existing vision-text contrastive learning models enhance representation
transferability and support zero-shot prediction by matching paired image and
caption embeddings while pushing unrelated pairs apart. However, astronomical
image-label datasets are significantly smaller compared to general image and
label datasets available from the internet. We introduce CosmoCLIP, an
astronomical image-text contrastive learning framework precisely fine-tuned on
the pre-trained CLIP model using SpaceNet and BLIP-based captions. SpaceNet,
attained via FLARE, constitutes ~13k optimally distributed images, while BLIP
acts as a rich knowledge extractor. The rich semantics derived from this
SpaceNet and BLIP descriptions, when learned contrastively, enable CosmoCLIP to
achieve superior generalization across various in-domain and out-of-domain
tasks. Our results demonstrate that CosmoCLIP is a straightforward yet powerful
framework, significantly outperforming CLIP in zero-shot classification and
image-text retrieval tasks.