Jina CLIP: Your CLIP Model Is Also Your Text Retriever
Authors: name: Andreas Koukounas; name: Georgios Mastrapas; name: Michael Günther; name: Bo Wang; name: Scott Martens; name: Isabelle Mohr; name: Saba Sturua; name: Mohammad Kalim Akram; name: Joan Fontanals Martínez; name: Saahil Ognawala; name: Susana Guzman; name: Maximilian Werk; name: Nan Wang; name: Han Xiao
Published: 2024-05-30 16:07:54

Abstract: Contrastive Language-Image Pretraining (CLIP) is widely used to train models
to align images and texts in a common embedding space by mapping them to
fixed-sized vectors. These models are key to multimodal information retrieval
and related tasks. However, CLIP models generally underperform in text-only
tasks compared to specialized text models. This creates inefficiencies for
information retrieval systems that keep separate embeddings and models for
text-only and multimodal tasks. We propose a novel, multi-task contrastive
training method to address this issue, which we use to train the jina-clip-v1
model to achieve the state-of-the-art performance on both text-image and
text-text retrieval tasks.