Jina CLIP: Your CLIP Model Is Also Your Text Retriever
Authors: name: Andreas Koukounas, name: Georgios Mastrapas, name: Michael Günther, name: Bo Wang, name: Scott Martens, name: Isabelle Mohr, name: Saba Sturua, name: Mohammad Kalim Akram, name: Joan Fontanals Martínez, name: Saahil Ognawala, name: Susana Guzman, name: Maximilian Werk, name: Nan Wang, name: Han Xiao
Organization: Jina AI

Contrastive Language-Image Pretraining (CLIP) is widely used to train models
to align images and texts in a common embedding space by mapping them to
fixed-sized vectors. These models are key to multimodal information retrieval
and related tasks. However, CLIP models generally underperform in text-only
tasks compared to specialized text models. This creates inefficiencies for
information retrieval systems that keep separate embeddings and models for
text-only and multimodal tasks. We propose a novel, multi-task contrastive
training method to address this issue, which we use to train the jina-clip-v1
model to achieve the state-of-the-art performance on both text-image and
text-text retrieval tasks.