CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster
  Pre-training on Web-scale Image-Text Data
Authors: name: Sachin Mehta; name: Maxwell Horton; name: Fartash Faghri; name: Mohammad Hossein Sekhavat; name: Mahyar Najibi; name: Mehrdad Farajtabar; name: Oncel Tuzel; name: Mohammad Rastegari
Published: 2024-04-24 05:13:28

Abstract: Contrastive learning has emerged as a transformative method for learning
effective visual representations through the alignment of image and text
embeddings. However, pairwise similarity computation in contrastive loss
between image and text pairs poses computational challenges. This paper
presents a novel weakly supervised pre-training of vision models on web-scale
image-text data. The proposed method reframes pre-training on image-text data
as a classification task. Consequently, it eliminates the need for pairwise
similarity computations in contrastive loss, achieving a remarkable 2.7times
acceleration in training speed compared to contrastive learning on web-scale
data. Through extensive experiments spanning diverse vision tasks, including
detection and segmentation, we demonstrate that the proposed method maintains
high representation quality. Our source code along with pre-trained model
weights and training recipes is available at
https://github.com/apple/corenet.