Three Towers: Flexible Contrastive Learning with Pretrained Image Models
Authors: name: Jannik Kossen, name: Mark Collier, name: Basil Mustafa, name: Xiao Wang, name: Xiaohua Zhai, name: Lucas Beyer, name: Andreas Steiner, name: Jesse Berent, name: Rodolphe Jenatton, name: Efi Kokiopoulou

We introduce Three Towers (3T), a flexible method to improve the contrastive
learning of vision-language models by incorporating pretrained image
classifiers. While contrastive models are usually trained from scratch, LiT
(Zhai et al., 2022) has recently shown performance gains from using pretrained
classifier embeddings. However, LiT directly replaces the image tower with the
frozen embeddings, excluding any potential benefits of contrastively training
the image tower. With 3T, we propose a more flexible strategy that allows the
image tower to benefit from both pretrained embeddings and contrastive
training. To achieve this, we introduce a third tower that contains the frozen
pretrained embeddings, and we encourage alignment between this third tower and
the main image-text towers. Empirically, 3T consistently improves over LiT and
the CLIP-style from-scratch baseline for retrieval tasks. For classification,
3T reliably improves over the from-scratch baseline, and while it underperforms
relative to LiT for JFT-pretrained models, it outperforms LiT for ImageNet-21k
and Places365 pretraining.