Three Towers: Flexible Contrastive Learning with Pretrained Image Models
Authors: Jannik Kossen
Mark Collier
Basil Mustafa
Xiao Wang
Xiaohua Zhai
Lucas Beyer
Andreas Peter Steiner
Jesse Berent
Rodolphe Jenatton
Effrosyni Kokiopoulou
Venue: NeurIPS 2023 poster (2023)
Keywords: three towers
contrastive learning
transformers
vision transformers
pretrained models
representation learning
finetuning
CLIP
ALIGN
classification
zero-shot
few-shot
retrieval

Abstract: We introduce Three Towers (3T), a flexible method to improve the contrastive learning of vision-language models by incorporating pretrained image classifiers. While contrastive models are usually trained from scratch, LiT (Zhai et al., 2022) has recently shown performance gains from using pretrained classifier embeddings. However, LiT directly replaces the image tower with the frozen embeddings, excluding any potential benefits from training the image tower contrastively. With 3T, we propose a more flexible strategy that allows the image tower to benefit from both pretrained embeddings and contrastive training. To achieve this, we introduce a third tower that contains the frozen pretrained embeddings, and we encourage alignment between this third tower and the main image-text towers. Empirically, 3T consistently improves over LiT and the CLIP-style from-scratch baseline for retrieval tasks. For classification, 3T reliably improves over the from-scratch baseline, and while it underperforms relative to LiT for JFT-pretrained models, it outperforms LiT for ImageNet-21k and Places365 pretraining.

Three Towers: Flexible Contrastive Learning with Pretrained Image Models
Jannik Kossen VA
Mark Collier V
Basil Mustafa 3
Xiao Wang?
Xiaohua Zhai
Lucas Beyer?
Andreas Steiner
Jesse Berent
Rodolphe Jenatton a
Efi Kokiopoulou 27
OATML, Department of Computer Science, University of Oxford
2 Google Research
3 Google DeepMind
Abstract
We introduce Three Towers (3T), flexible method to improve the contrastive learning of vision-language models by incorporating pretrained image classifiers. While contrastive models are usually trained from scratch, LiT [85] has recently shown performance gains from using pretrained classifier embeddings. However, LiT directly replaces the image tower with the frozen embeddings, excluding any potential benefits from training the image tower contrastively. With 3T, we propose a more flexible strategy that allows the image tower to benefit from both pretrained embeddings and contrastive training. To achieve this, introduce third tower that contains the frozen pretrained embeddings, and we encourage alignment between this third tower and the main image-text towers. Empirically, 3T consistently improves over LiT and the CLIP-style from-scratch baseline for retrieval tasks. For classification, 3T reliably improves over the from-scratch baseline, and while it underperforms relative to LiT for JFT-pretrained models, it outperforms LiT for ImageNet-21k and Places365 pretraining.
Introduction
Approaches such as CLIP [58] and ALIGN [34] have popularized the contrastive learning of aligned image and text representations from large scale web-scraped datasets of image-caption pairs. Com- pared to image-only contrastive learning, e.g. [51, 9, 25], the bi-modal image-text objective allows these approaches to perform tasks that require language understanding, such retrieval zero-shot classification [42, 58, 34]. Compared to traditional transfer learning from supervised image represen- tations [52, 68, 44, 38], contrastive approaches can forego expensive labelling and instead collect much larger datasets via inexpensive web-scraping [58, 11, 64]. A growing body of work seeks to improve upon various aspects of contrastive languaee modelling, cf. related work in $5.
CLIP and ALIGN train the image and text towers from randomly initialized weights, i.e. 'from scratch'. However, strong pretrained models for either image text inputs often readily available, and one may benefit from their use in contrastive learning. Recently, Zhai et al. [85] have shown that pretrained classifiers can be used improve downstream task performance. They propose LiT, short for 'locked-image text tuning', which is a variation of the standard CLIP/ALIGN setup that uses frozen embeddings from a pretrained classifier the image tower. In other words, the text tower in LiT is contrastively trained from scratch to match locked and pretrained embeddings in the image tower. Incorporating knowledge from pretrained models into contrastive learning an important research direction, and LiT provides simple and effective recipe for doing so.
However, concern with LiT is that it may be overly reliant the pretrained model, completely missing out on any potential benefits the image tower might get from contrastive training. Zhai et al.
Equal contribution. Work done while interning at Google. Equal advising. Correspondence to jannik kossen www..o........eeooeooooooe ack.uuk and markcollicr@ gom.le.com.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

variety of classes CLIP/ ALIGN
LiT
Three Towers
T
T
Figure 1: CLIP and ALIGN do not make use of pretrained models, and LiT directly uses frozen pretrained model the image tower. With Three Towers (3T), we propose flexible strategy to improve contrastive learning with pretrained models: in addition to a pair of CLIP--tyle from-scratch image and text towers, introduce third tower which contains fixed pretrained image embeddings; extra loss terms align the main towers to the third tower. Unlike for CLIP/ALIGN and LiT, the image tower can benefit from both contrastive learning and pretrained classifier embeddings.
[85] themselves give example where LiT performs worse than standard contrastive training: when using models pretrained on Places365 [87]- a dataset relating images to the place they were taken-thh fixed embeddings do not generalize to downstream tasks such ImageNet-Ik (NN- [40, 61] or CIFAR-100 [40]. For their main results, Zhai et al. [83] therefore use models pretrained on datasets such as ImageNet-21k (IN-21K) [14, 61] and JFT [68, 84] which
and inputs. However, even then, we believe that constraining the image tower to fixed classifier embeddings is not ideal: later, we will show examples where LiT performs worse than standard contrastive learning due to labels or input examples not covered by IN-2Ik, cf. $4.2. Given the scale and variety of contrastive learning datasets, we believe it should be possible to improve the image lower by making use of both pretrained models and contrastive training.
In this work, we propose Three Towers (3T): flex- ible approach that improves the contrastive learning of vision-language models by effectively transferring knowledge from pretrained classifiers. Instead of lock- ing the main image tower, we introduce third tower that contains the embeddings model. The main image and text towers are trained from scratch and aligned to the third tower with addi- tional contrastive loss terms (ef. Fig. 1). Only the main two towers are used for downstream task applications such that no additional inference costs are incurred compared to LiT or CLIP/ALIGN baseline. This simple approach allows us to explicitly trade off the main contrastive learning objective against the transfer of prior knowledge from the pretrained model. Compared to LiT, the image tower in 3T can benefit from both contrastive training and the pretrained model. We highlight the following methodological and empirical contributions:
Table 1:
For retrieval, 3T improves on LiT and the CLIP-style baseline (top-1 recall 1). Models are g scale, using Text-Filtered We- bLI, and JFT pretraining for LiT/3T, cf. $4.1.
a frozen pretrained
Method
Basel.
LiT
3T
Flickr img2txt
85.0
83.9
87.3
Flickr txt2img
67.0
66.5
72.1
COCO img2txt
60.0
59.5
64.1
COCO txt2img
44.7
43.6
48.5
We propose and formalize the 3T method for flexible and effective transfer of pretrained classifiers into contrastive vision-language models ($3).
3T consistently improves over LiT and from-scrater baseline for retrieval tasks (e.g. Table 1, $4.1). For classification tasks, 3T outperforms LiT and the baseline with IN-21k pretrained models; for
JFT pretraining, 3T outperforms the baseline but not LiT ($4.2).
We extend the evaluation of Zhai et al. [85] to additional tasks and pretraining datasets, showing that 3T is significantly more robust than LiT to deficits in the pretrained model ($4.2 and $4.4).
We show that 3T benefits more from model size or training budget increases than LiT ($4.3).
We introduce simple post-hoe method that allows us further improve performance by combining
3T- and LiT-like prediction (54.5).
2 Background: Contrastive Learning of Vision-Language Models
Before introducing 3T, we recap contrastive learning of vision-language models as popularized by [58, 34, 86] and give a more formal introduction to LiT [85].
CLIP/ALIGN. We assume two parameterized models: an image tower J So with parameters 0 and text tower g = Io with parameters Φ). Each input sample (T, T:) consists of a pair of matching image I, E I and text T, T. and the contrastive loss is computed over batch {L....,N N)

---

of examples. The towers map the input modalities to a common D D-dimensional embedding space, f I + RD and g : T + RD. We further assume that and g produce embeddings that are normalized with respect to their 1.2 norm, (I [Tg(() ll2 = 1 for any T E I and T E T. For batch of input samples, the bi-directional contrastive loss [66, 77, 51, 9, 86] computed as
Geeg = 2 ( + Lg-ss); where
N
exp(I(1.) /T)
G =
log
(2)
N 1=1
Σn j=1 exp( ((TT))>))
1 N
exp(f(L)) ((T)//T
f--s=
log
(3)
N ΣE= exp(f(L) ((T)//T)
Here, T is a learned temperature parameter and f(I) E R are dot products. The two directional loss terms, Ly-zg and Lg-r,, have natural interpretation as standard cross-entropy objectives for classifying the correct matches in each batch. The parameters 0 and Φi of the two towers, fo and gø. are jointly updated with standard stochastic optimization based on Eq. (1).
Downstream Tasks, After training, f and g are treated fixed representation extractors. For retrieval, the dot product f(I)T E R ranks similarity between inputs. For few-shot image classification, linear classifier is trained atop the feature representations of f from few examples; g is not used. For zero-shot image classification, f embeds images and all possible class labels (see [58]). For each image, one predicts the label with the largest dot product similarity in embedding space.
LIT. Zhai et al. [85] initialize the parameters of the image tower from a pretrained classifier and then keep them frozen them during training. That is, only the parameters é of the text tower are optimized during contrastive training. As the image tower fe. LiT uses the pre-softmax embeddings of large scale classifiers, such vision transformers [17] trained on JFT-3B [84] or IN-21k. During contrastive training. the text tower is trained from scratch using the same objective Eq. (1).
Experimentallyy Zhai et al. [85] investigate all combinations for 'training from scratch', locking and finetuning a pretrained model for both towers on custom union of the CL.PP-subset of YFCC- 100M [69] and CC12M [7] (cf. Fig. 3 in [85]). For the image tower, locking gives significant lead on IN-1k over finetuning and training from scratch, and performs similarly to finetuning and better than training from scratch for retrieval. For the text tower, locked configuration performs badly, and finetuning gives small to negligible gains over training from scratch. Given these results, Zhai et al. [85] choose the 'locked image tower and from-scratch text tower' setup that they call LiT. At large scale, they show that a locked image tower outperforms from-scraten training and finetuming on zero-shot IN- 1k, ImageNet-v2 (IN-v2) [59], CIFAR- 100, and Oxford IIIT Pet [53] classification tasks. They further show LiT outperforms CLIP/ALIGN on IN-R [31], IN-A [32], and ObjectNet [3].
While Zhai et al. [85] show strong classification performance with LiT on wide range of datasets, locking the image tower is drastic measure that introduces a severe dependency on the pretrained model, prohibiting the image tower from improving during contrastive training. We will later show that, if the embeddings in the frozen image tower are not suited to a particular downstream tasks, LiT underperforms compared to approaches that train the image tower on the varied contrastive learning dataset, see, for example, $4.2 and $4.4. The 3T approach seeks to address these concerns.
3 Three Towers: Flexible Contrastive Learning with Pretrained Models
With Three Towers (3T), we propose simple and flexible approach to incorporate knowledge from pretrained models into contrastive learning. Instead of directly using the pretrained model locked as the main image tower, we instead add third tower, h. which contains the fixed pretrained embeddings. The main image and text towers are trained from scratch, and we transfer representations from the third tower to the main towers with additional contrastive losses. In this setup, the main image tower benefits from both pretraining knowledge and contrastive learning.
More formally, in addition to the standard image and text towers, fo and go. cf. $2, we now have access to fixed pretrained image embeddings p: I + RRP. Because P can different from the target di- mension D, we define the third tower as h([) = 1iiner(p([])) (p())), where linear B RD projects
3

---

(a)
p(D
(b)
Legend
Image I₃
h h1
(T)
Compare f(I)"
h(lI)
Linear Layer
Cyn tho
Text T2
T
Linear Layer & L2-Normalize
(R(
gh (T')
Cf+g
f(1)
g(T))
Contrastive Loss
Figure 2: Details of the 3T approach. (a) Linear adaptor heads (gray) align the representations between the main towers and the third tower. (b) For downstream tasks. 3T is used in the same way as CLIP/ALIGN and LiT. We discard the third tower, using only the main towers, f()) and g(T).
embeddings to the desired dimensionality. In principle, the 3T architecture is also compatible with pre- trained text models. However, like Zhai et al. [85], we do not observe benefits from using pretrained text models, cf. $A.3, and so our exposition and evaluation focuses on pretrained image classifiers.
When computing loss terms involving the third tower, we make use of learned linear projection heads. These heads afford the model a degree of flexibility when aligning representations between towers. First, we define (.) norm norm(linear() in)ar(-)). where norm(x) = x/ || normalizes with respect to L2 norm and, overloading notation, linear: RDD + RR now preserves dimensionality. We adapt the main image and text towers as (I) [L)) and gn (T) (L)) We project the third tower embeddings h to , = (L() and hgg (T) = (L(h(I)) for computation of the loss with the image and text towers respectively. The linear layers introduced for fir gh. h,, and hg are independently learned from scratch. Per input batch, the 3T approach then optimizes the following loss objective:
LaT - (Cosog + Lineahh + ggg,tshh..
Here. Lyeng is the original contrastive loss, cf. Eq. (1), and Li eshy and Lg + are additional contrastive losses between the image/text tower and the third tower projections. All loss terms share a global temperature r. We train both towers, f and g. and all linear layers from scratch by optimizing Eq. (4) over imput batches. Figure 2 (a) visualizes the adaptor heads and loss computation for 3T.
After training, the third tower is discarded and use only the main two towers, ef. Fig. 2 (b). Therefore, the inference cost of 3T is equal to the baseline methods. For training, the additional cost over the trom-scratch CLIP/ALIGN baseline is negligible, as frozen embeddings from the third tower can be pre-computed and then stored with the dataset, as also done in Zhai et al. [83].
Intuitions for the 3T Architecture. Intuitively, the additional losses align the representations of the main towers to the pretrained embeddings the third tower. In fact, Tian et al. [71] show that contrastive losses can be seen as distillation objectives that align representations between teacher and a student model. They demonstrate that contrastive losses are highly effective for representation transfer, outperforming alternative methods of distillation. Thus, the additional terms, Lir++h and Lgg, transfer representations from the pretrained model to the unlocked main towers, albeit without the usual capacity bottleneck between the student and teacher models. Of course, for 3T, also need to consider the original objective Litt between the unlocked text and image towers. In sum, the unlocked towers benefit both from the pretrained model and contrastive training.
Design Choices. $4.6, we ablate various design choices, such as our use of equal weights among the terms of Eq. (4), the contrastive loss for representation transfer, the shared global temperature the linear layers for in the third tower, as well as our design of the adaptor heads.
Discussion. The 3T approach includes pretrained knowledge without suffering from the inflexibility of directly using the pretrained model as the main image tower. For example, unlike LIT, 3T allows for architectural differences between the unlocked image tower and pretrained model. Further, it seems plausible that 3T should generally be more robust than LiT: as the image tower learns from the highly-diverse contrastive learning datasets, the chances of encountering 'blindspots' in downstream applications, e.g. due to labels or examples not included in the pretraining dataset, should be lower. In a similar vein, LiT is most appropriate for pretrained models capable that they need not adapt during contrastive training. For example, few-shot classification performance, which uses only the image tower, by design cannot improve at all during contrastive training with LiT. On the other hand, with 3T

---

we may be able to successfully incorporate knowledge from weaker models, too. Lastly, finetuning instead of locking the main image tower from a pretrained classifier often has most marginal positive effects over training from scratch after high number of training steps, cf. $2. In contrast, the additional terms in Eq. (4) consistently ensure the main towers align with the pretrained model.
4 Experiments
In this section, we compare 3T to LiT and to a standard CLIP/ALIGN baseline trained from scratch, which we will refer the 'baseline" for simplicity. Our experimental setup largely follows Zhai et al. [85]: for all methods, we use Vision Transformers (ViT) [17] for the image and text towers, replacing visual patching with SentencePiece encoding [41] for text inputs, further sharing optimization and implementation details with [85]. We rely on the recently proposed WebLI dataset [11]. a large-scale dataset of 10B image-caption pairs (Unfiltered WebLI). We also explore two higher-quality subsets derived from WebLI: Text- Filtered WebLI,, which uses text-based filters following Jia et al. [34], and Pair-Filtered WebLI (see $A.7), which retains about half of the examples with the highest image-text pair similarity. For image tower pretraining, we consider both proprietary JFT-3B [84] and the publicly available IN-2Ik checkpoints of Dosovitskiy et al. [17]. For IN-21K experiments, our largest model scale is L, with 16x 16 patch size for ViT, and for JFT pretraining we go up to g scale at patch size of 14x 14. Unless otherwise stated, we train for 5B examples seen at batch size of 14 336.
4.1 Retrieval
We study zero-shot retrieval perfor- mance on COCO [10] and Flickr [56]. Table 1 shows results for g scale mod- els trained
Table 2: 3T outperforms LiT and the baseline for retrieval (top-1 recall f, L scale models, Unfiltered WebLI, see $4.1).
Text-Filtered WebLI, with JFT pretraining for 3T and LiT. In Table 2, we report performance of L scale models trained on Unfiltered WebLI for JFT and IN-21k pretrain- ing. We give results for additional WebLI splits for IN-21k and JFT pretraining in Tables A.4 and A.6.
Pretraining Method
IN-21k LiT
JFT LiT
Basel.
3T
3T
Flickr img2txt
75.6
71.7
80.0
78.7
80.0
Flickr" tx12img
57.1
49.3
60.9
58.8
61.4
COCO img2txt
51.0
46.1
54.4
52.7
54.4
COCO tx22img
34.2
27.8
37.7
36.7
37.9
3T improves LiT and the baseline for retrieval tasks across scales, datasets, and for both JFT and IN-21k pretraining. A rare exception to this are the Unfiltered WebLI results Table A.6, where 3T beats LiT for retrieval on average and for txt2img, but not for img2Ixt. In general however, LiT underperforms for retrieval and regularly does not outperform the baseline: at L scale, LiT shows a strong dependence the pretrained model and can only outperform the baseline with JFT pretraining. In contrast, 3T obtains similar improvements over the baseline for both pretraining datasets. We will continue to this pattern in experiments: 3T consistently improves over the baseline, while LiT results can vary wildly and depend strongly on the pretraining dataset. We discuss our retrieval results in the context of SOTA performance in $A.7: the SOTA method CoCA [81] achieves better results but uses about 4 times more compute; increasing the compute budget for 3T would likely reduce the gap.
Intuitively, while the fixed classifier embeddings in LiT can categorize inputs into tens of thousands of labels, they may not be fine- fine-grained enough for retrieval applications. On the other hand, the contrastive training of the baseline is closely related to the retrieval task but misses out on knowledge from pretrained models. Only 3T is able to combine benefits from both for improved performance.
4.2 Few-Shot and Zero-Shot Image Classification
Next, we compare the approaches on few- and zero-shot image classification. See $D for citations of all the datasets that we use. For zero-shot classification, we follow the procedure described in $2. For few-shot tasks, we report 10-shot accuracy, more specifically, the accuracy of a linear classifier trained on top of fixed image representations, averaging over 3 seeds for the 10 random examples per class. As few-shot performance depends only on the image embeddings, LiT's sew-shot accuracy is precisely the same that of the pretrained model. Despite this, Zhai et al. [85] show that LiT outperforms the unlocked baseline on shw- IN-1k lk and CIFAR- 100 evaluations.
For historical reasons, Flickr" results do not use the 'Krppthy'' split [36]. However, they are available for all runs, and they are directly comparable. Our g scale runs do have Karpathy split results, denoted Flickr (no star).
5

---

For IN-21k pretraining, Table 3 reports the performance of L scale models trained on Unfiltered WebLI, and we give results on additional datasets in Table A.4. Here, 3T outperforms both LiT and the baseline. For JFT pretraining. Table A.5 gives results at I. scale on Unfiltered WebLI and Table A.6 presents results at g scale across WebL.I splits. In all JFT settings, LiT performs best on average for image classification tasks, despite few-shot performance being fixed for LiT. However, for both JFT and IN-21k pretraining, 3T improves over the baseline for almost all tasks. This is different for LiT, where performance heavily depends on the pretraining data.
Risk of Locking. There risk associated with LIT, both in a positive and negative sense. Using fixed classi- fier representations can result excellent performance if the downstream task distribution and pretraining dataset are well-aligned: for example, IN-21k contains hundreds of labels of bird species, and IN-21K-LiTT performs well on the Birds task, outperforming 3T by 18 %p. However, the IN-21k label set does not contain
Table 3: For IN-21k pretraining, 3T has the best average classification accuracy (1) (L scale models, Unfiltered WebLI).
Method
Basel.
LiT
3T
IN-IK
62.8
79.0
68.0
Few-Shot Classification
CIFAR-100
70.4
83.6
72.5
Caltech
91.0
88.4
92.3
single car brand, and thus, IN-2Ik-LiT does not perform well Stanford Cars, underperforming relative to the baseline and 3T by almost 40 %p. ObjectNet, IN-A,
Pets
85.9
89.2
86.5
DTD
70.3
69.2
73.3
and IN-R were created to be challenging for ImageNet models, and so IN-2Ik-LiT performs worse than the baseline and 3T here, too. For example, IN-R contains artistic renditions of objects that are challenging for IN-21k-based models they have mostly been trained on realistic images. IN-21k-LiT also struggles with more specialized tasks, performing 29 %p worse than the baseline on the remote sensing RESISC dataset.
UC Mereed
91.8
92.8
94.0
Cars
81.5
41.9
84.9
Col-Hist
71.7
86.4
76.6
Birds
53.4
83.4
65.0
IN-1k
69.5
76.0
71.7
CIFAR-100
73.5
82.9
73.4
Caltech
81.9
82.4
84.1
Zero-Shot Classification
Pets
84.2
87.1
87.0
The above results support our hypothesis that image em- beddings trained on the highly diverse contrastive learn- ing dataset will be more broadly applicable. Strikingly, even when using the same IN-21k model, 3T almost always improves over the baseline and never suffers the same failures as LiT. However, it seems that JFT- pretrained models can fix many of LiT's shortcomings for image classification. JFT-LiT performs remarkably well and almost never underperforms significantly com- pared to the baseline. A deviation from this pattern are the results for Eurosat and RESISC g scale for JFT pretraining in Table A.6, where, e.g. on the Text-Filtered WebLI split, LiT lacks behind 3T by 12 %p and %p.
DTD
58.6
51.8
60.3
IN-C
49.6
62.0
51.8
IN-A
53.0
45.6
54.3
IN-R
85.8
66.1
88.1
IN-v2
62.2
67.2
64.9
ObjectNet
56.2
41.9
58.3
EuroSat
32.7
27.6
42.8
Flowers
62.0
72.6
65.7
RESISC
58.0
29.0
57.9
Sun397
67.6
65.4
68.7
Average
68.4
68.3
71.4
4.3 Scaling Model Sizes and Training Duration
Since the publication of LiT, the scale of contrastive learning datasets, both public and proprietary, has increased, for example with the release of LAION-5B [64] or WebLI- [11]; we use the latter here. Given their cheap collection costs, it seems likely that this growth will continue to outpace that of more expensive classification datasets. However, locking the image tower ignores any potential benefits from the increased contrastive learning data for the image tower. Additionally, larger datasets often lead to increases in model scales to fully make use of the additional information [84]; unlike LiT, 3T can increase the scale of the main image tower independently of the pretrained model, cf. $4.4.
Here, we separately study the effects that model scale and dataset size have on 3T, LiT, and the baseline. First, we vary the scale of all involved towers, including the pretrained model. We study both IN-21k and JFT pretraining, always train contrastively for 5B examples seen, and the S, B, L, and g scales correspond to S/32, B/32, L/16, and g/14 for the ViT models. We compute averages separately across retrieval, few-shot, and zero-shot classification tasks, where the tasks are those from Tables 2 and
In Fig. 3, we observe that 3T's lead over LiT and the baseline in retrieval performance holds across scales and pretraining datasets. Further, across all tasks and scales, 3T maintains a consistent performance gain over the baseline. Also across tasks and pretraining datasets, observe that

---

Retrieval
Few-Shol Classification
Zero-Shut Classification
90
Average Performance
60
Baselliee
70
LiT
80
50
60
121k 3F
70
40
50
60
40
30
i2lk-
121k-B
121--L
JFT-B
JFT-L
JFT
121k-S
i21k-B
i2lk-L
JFT-B
JFT-L
JFT-g
121K-S
i21k-B
121k-L
JFT-B
JFT-L
JFT-g
Model Scale
Figure 3: We increase the model scale for 3T, LIT, and the baseline and report average retrieval, few- and zero-shot classification performance for both IN-21k and JFT pretraining. 3T and the baseline benefit more from increases in scale than LiT (their curves are steeper), with 3T performing better than the baseline. The baseline does not use a pretrained model and is displayed twice (at scale B and L...
scaling is more beneficial for 3T than for LiT: as we increase the scale, 3T's performance increases more than that of LiT. This means that 3T either extends its lead over, overtakes outright, or reduces increase scale. At L scale with IN-21k pretraining, 3T gives the best average
its gap to LiT performance all methods across tasks. For JFT pretraining, LiT maintains an edge for classification performance, although the scaling behavior suggests this gap may fully collapse larger scales. We observe similar trends when scaling the number of examples seen during training, cf. Fig. A.4.
4.4 Pretraining Robustness
Table 4: 3T is more robust to the pretraining setup than LiT. Reot classification accuracies (t). full details in main text.
Next, study what happens when 3T and LiT are used with pretrained models that do not conform to expectations, We con- sider two setups: one that we call "mismatched" and one that consid- ers pretraining on the Places365 [87] dataset. In Table 4, we dis- play zero-shot accuracies on Pets, IN-1K, and CIFAR- -100 tasks for which LiT usually performs best-as well as the average performance over the full set of tasks, see Table A.7 for individual results.
Setup
Mismatched Basel.
Places365 Basel.
Method
LiT
3Т
LiT
3T
IN-IK
69.5
69.5
71.5
45.6
24.5
47.4
CIFAR-100
73.5
78.6
75.6
48.3
27.4
52.4
Pets
84.2
84.7
87.4
61.5
30.3
60.2
...
...
Full Average
66.4
61.7
69.8
47.5
29.4
49.3
Mismatched Setup. So far, we have always matched the scale of the pretrained model to the scale of the models trained contrastively. For the mismatched setup, we now break this symmetry: we use an IN-2 B/32 scale image model (3T and LiT) with an L scale text tower (all approaches) and an L/16 unlocked image tower (3T and baseline). This setup is relevant when pretrained models are not available at the desired scale: for example, given ever larger contrastive learning datasets one may want to train larger image models than are available from supervised training, cf. $4.3. Of course, increasing the image tower scale also comes at increased compute costs for 3T and the baseline. We observe that LiT suffers from this mismatched setup much more than 3T, which is not restricted by the smaller pretrained model and now achieves higher accuracy than LiT on Pets and IN-
Places365. Zhai et al. [85] demonstrate that LiT performs badly when used with models pretrained on Places365. Table we reproduce this result and observe LiT performing much worse than the baseline. (We here train B scale models for 900M examples seen, and based on our discussion in $4.3, would expect LiT to perform even worse, in comparison to the baseline and 3T, when training longer or with larger scale models.) The embeddings obtained from Places365 pretraining do not allow LiT to perform well on our set of downstream tasks. 3T behaves much more robustly and does not suffer from any performance collapse because it incorporates both the pretrained model and contrastive data when training the image tower. Notably, 3T manages to improve average performance over the baseline even for Places365 pretraining. We further suspect the linear projection heads afford 3T some flexibility aligning to the pretrained model without restricting the generality of the embeddings learned in the main two towers.
7

---

Retrieval
Sew-Shot Classification
Zero-Shot Classification
Average Performance
48
76
45
....
Lit
72
ST
42
66
68
62
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Convex Combination a
Figure 4: Convex combination of the image models in 3T: or + ((7). By varying a, can generally interpolate between 3T and LiT performance. Interestingly, for a broad range of weights, the retrieval and few-shot classification performance of the combination outperforms 3T and LiT.
4.5 Benefits From Using 3T With All Three Towers at Test Time
We usually discard the pretrained model when applying 3T to downstream tasks, cf. Fig. (b). In this section, instead explore if we can find benefits from using the locked third tower at test time, similar to LiT. More specifically, we study the convex combination of the main image tower and locked pretrained model the third tower, (1-a())+(-a))-((()
if can interpolate between 3T- and LiT-like prediction at a = O and a = respectively. We train 3T without linear projection heads to make embeddings from all towers compatible. Additional details of this setup can be found in $A.10.
Fig. 4 shows we can generally interpolate between LiT- and 3T-like performance as we vary a from 0 to 1. Note that do not always recover LiT or 3T performance at E {0, 1} as explained in $A.10 10. Interestingly, for retrieval and few-shot classification tasks, the convex combination yields better performance than either of the underlying methods for relatively broad region around a a 0.5. We believe that further study the convex combination could be exciting future work: the method is entirely post-hoe and no additional training costs incurred, although inference costs do increase.
4.6 Ablations
Next, provide ablations for some of the design de- cisions of 3T as well as insights into LiT training. We perform the ablation study B scale with patch size 32, training for 900M examples seen, and use JFT- pretrained models. In Table 5, report the average difference in performance to our default runs across all tasks, together with standard errors computed the downstream tasks as an indication of statistical sig- nificance. We refer to $A.11 for full details and results.
Table 5: Ablation study, see text for details.
Difference to 3T
Rerun
-0.22 + 0.25
No Crtsg Loss
-6.66 10.61
No Ciatthy Loss
-1.19 + 0.75
No Cgstshy Loss
-2.77 0.91
Head Variants
0.09 + 0.35
MLP Embedding
-0.08 0.35
To study per-run variance, we perform a rerun of the base 3T model, obtaining aver- age performance difference of 0.22 %p across tasks. 'No L.,..:: When leaving out either of the three loss terms, average performance suffers significantly. 'Head Variants': We try a selection of different projection head variants, see $A.11 None give significantly better performance than our default setup. 'MLP Embedding':: Replacing the linear projection h in the third tower with an MLP does not improve performance. 'More Temper- atures': Using three learned temperatures, one per 3T loss term, instead of global temperature in Eq. (4), does not improve results. 'Loss weights': Replacing the loss with a weighted objective, 3 (Ceezg + w (Ci + Cgutthh))) )), does not improve per- formance significantly across a variety of choices for w. 'L2 Transfer": Using squared losses for the representation transfer objectives fi,,, and Lgg, ef. [60], results in significantly worse performance, even when optimizing the weight of the transfer terms. '3T Finetuning': Initializing the main tower in 3T with the same JFT-pretrained model as the third tower increases performance significantly; however, we find this effect becomes negligible for larger scale experiments, cf. $A.11.
3T Ablations. 'Rerun':
More Temperatures
-0.26
Loss Weights
0.17
L2 Transfer
3.80 + 1.13
3T Finetuning
1.85 - 1.27
Difference LiT
Rerun
-0.10 + 0.22
LiT Finetune FlexiLiTT FlexiLiT2
-14.99 + 6.09
-4.63 + 1.36
-5.04 + 1.54

---

LiT Ablations. 'Rerun': We observe similar between-run variance for LIT. 'LiT Finetune': We confirm the result of Zhai et al. [85] that finetuning from a pretrained model results in worse performance than locking. 'FlexiLiTT 1/2": We investigate simple ways of modifying LiT such that it can adjust the image tower during contrastive learning. see $A.11, but find these are not successful.
Additional Experiments. In $A.1,, we study the optimization behavior of 3T, finding evidence for beneficial knowledge transfer from the pretrained model. In $A.2, we study the calibration of all meth- ods for zero-shot classification, as well as their performance for out-of-distribution (OOD) detection: 3T is generally better calibrated than LiT, and for OOD tasks, we find trends similar $4.3, with all methods generally performing well. In $A.3, we confirm the results of Zhai et al. [85] that there are no benefits from using pretrained language models. In $A.4, detailed investigation of predictions suggests that 3T performs well because it combines knowledge from contrastive learning and the pre- trained model. Lastly, $A.5 shows 3T continues to perform well with other pretrained image models.
5 Related Work
CLIP [58] and ALIGN [34] are examples of vision-language models that have received significant at- tention, e.g. for their impressive ImageNet zero-shot results. Concurrently with LiT [85], BASIC [55] investigates locking and finetuning from JFT-pretrained models. Previously, [46, 57, 74, 67] have ex- plored representation learning from images with natural language descriptions before the deep learning era. Subsequently, [21. 36, 35, 19, 63, 8] explore image-text understanding with CNNs or Transform- ers. In this context, Li et al. [42] introduced the idea of zero-shot transfer to novel classification tasks. The loss objective. Eq. (I), was proposed by Sohn [66] for image representation learning, and also appears in [77, 51, 9]. Zhang et al. [86] then used the objective align images and captions, although their setting used medical data. Lots of work has built on CLIP and ALIGN. For example, [82, 79] have augmented the objective to optionally allow for labels, [89, 88] proposed methods for improving zero-shot prompts. [2, 43] applied CLIP to video, [54, 49, 62, 33] used CLIP embeddings to improve generative modelling, and [47] study different ways of incorporating image-only self-supervision objectives into CLIP-style contrastive learning. Relatedly, vast amounts of work have explored self- supervised or contrastive representation learning of images only, e.g. [16, 25, 24, 6]. Transfer learning [52] applies embeddings from large-scale (weakly) labelled datasets to downstream task [68, 44, 38].
6 Limitations, Impact, and Conclusions
Limitations. While 3T consistently improves over LiT for retrieval tasks, for classification, 3T out- performs LiT with ImageNet-21k-preee p-retrriiee models only at large scales, and may require even larger scales for JFT pretraining. Further, while inference costs are equal for all methods, 3T incurs addi- tional training costs compared to LiT. We have compared methods at matching inference cost for sim- plicity because there are many ways to account for the cost of pretraining and embedding computation.
Impact. We believe that locking is suboptimal way to incorporate pretrained image models, and we have demonstrated clear benefits from exposing the image tower to both the contrastive learning dataset and the pretrained model, particularly as scale increases. 3T is a simple and effective method to incorporate pretrained models into contrastive learning and should be considered by future research and applications whenever strong pretrained models are available. For future work that seeks to improve 3T, we consider it important to understand the differences between embeddings from 3T, LIT, and the baseline. If we can obtain insights into why they excel different tasks, we can perhaps (learn to) combine them for further performance improvements. Our convex combination experiments are a starting point; it would be interesting to continue this direction, possibly looking at combinations in parameter space [75, 76). Lastly, future work could explore 3T for distillation of large pretrained models into smaller models, extend 3T to multiple pretrained models, potentially from diverse modalities, or explore the benefits of 3i-like ideas for other approaches such as CoCa [81].
Conclusions. We have introduced the Three Tower (3T) method, a straight- and effective approach for incorporating pretrained image models into the contrastive learning of models. Unlike the previously proposed LiT, which directly uses a frozen pretrained model, 3T allows the image tower to benefit from both contrastive training and embeddings from the pretrained model. Empirically, 3T outperforms both LiT and the CLIP/ALIGN baseline for retrieval tasks. In contrast to LIT,, 3T consistently improves over the baseline across all tasks. Further, for le- preetrained I--rerrrnd models, 3T also outperforms LiT for few- and zero-shot classification. We believe that the robustness and simplicity of 3T makes it attractive to practitioners and an exciting object of further research.

---

References
[1] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z.. Citro, C., Corrado, S.S. Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, Harp, A., Irving, G., Isard. M., Jia, Y., Jozefowiez. R.. Kaiser, L., Kudlur, M.. Levenberg, J., Mané, D... Monga, R.,. Moore, S., Murray, D.. Olah, C., Schuster, M.. Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P. Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. TensorFlow: Large-scale machine learning on heterogencous systems, 2015. URL httt:://www.ttnsorooo..oom www.srewo.ooooooooooooooooo.ee.e org/. Software available from tensorflow.org.
[2] Bain, M., Nagrani, A., Varol, G., and Zisserman, A. Frozen in time: A joint video and image encoder for end-to-end retrieval. In International Conference on Computer Vision (ICCV), 2021. URL https: // //arxiv.. org/ aba/2104. 2104.00650.
[3] Barbu, A., Mayo, D.. Alverio, J., Luo, W., Wang. C.,, Gutfreund, D., Tenenbaum, J., and Katz, B. Objectnet: large-scale bias-controlled dataset for pushing the limits of object recognition models, In Advances Neural Information Processing Systems (NeurIPS), 2019, URL https: // https/////coenngeeeeeeeeeee wtw.ecoeddnnnnnnurs....... neurips.ce. cc/ paper /2per/2019/ 1tt///////////////////////////// 37 www.eefffaafffoo 301733092 Pwt.0000009----.....-.. pdf.
[4] Beyer. L., Zhai, X., and Kolesnikov, A. Big vision. atttps://gite com./google-ressarch//ba vision, 2022.
[5] Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman- S., and Zhang, Q. JAX: composable transformations Python+NumPP programs, 2018. URL http:// //gith/b..tttt/t com/ goole//ga//ja.
[6] Caron, M., Touvron, H., Misra, I,, Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in supe-supe vision transformers. In International conference computer vision (ICCV), 2021. URL attts:///arxte org/ abs/ /2104.21444 14294.
[7] Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Conceptual 12m: Pushing web-scale image-text pre- training recognize long-tail visual concepts. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. URL https:/ / /aaxxvvvvv///////////e/////aall/ org/aba aa/aaba/211000 12102.08981.
[8] Chen, J., Hu, H., Wu, H., Jiang, Y., and Wang, embedding. In Conference on computer vision and pattern recognition (CVPR), 2021. URL https: //arxiv. .ogg ha//bba/2000 200b/221.004355.
C. Learning the best pooling strategy for visual semantic
[9] Chen, T., Kormblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning (ICML), 2020. URL https: //arxiv. Marxiv.org/abs.. attpaab/////.//e///// 2002.05709.
[10] Chen, X., Fang. H. Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server, 2015. URL attps://aat /tapii//////iv/// org/ http:/pd//1100/ 00325 pdf.
[11] Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Gryener, A., Mustafa, B., Beyer, L., et al. Pali: A jointly-scaled multilingual language- image model, 2022. URL https: // /ttxxvv..vv/////////////////wtll org/ abs 22t///299.000 06794.
[12] Cheng, G.. Han, J.. and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 2017. URL https: //arxiv www.riwwww....voreeeee absg/abs//a 11701.00111....
[13] Cimpoi, M., Maji, S., Kokkinos, Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Conference on Computer Vision and Pattern Recognition (CVPR), 2014. URL https:// https://arxii.... orrg/ abs/1311. 3618.
[14] Deng, J., Dong, W., Socher, R., Li, I..JJ, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009. URL. https: //ipee////eeeeeoreeee leee. org/ htgg///ccment////e// 55206448.
[15] Djolonga, J., Hubis, F., Minderer, M., Nado, Z., Nixon, J., Romijnders, R., Tran, D., and Lucic, M. Robustness Metrics, 2020. URL https: / thub. com/ google wongggle-reesssee reseereh/robbeeeeeeee htw.eaarrcrrrobuuttsssseeeeeteee
[16] Doersch, C., Gupta, A., and Efros, A. A. Unsupervised visual representation learning by context prediction. In International conference on computer vision (ICCV), 2015. URL https: pectps://arxtas //arxxiv.../////////// org/ //aabb/1110 15055. 05192.
10

---

[17] Dosovitskiy, A., Beyer. L., Kolesnikov, A., Weissenborn, D.. Zhai, X., Unterthiner, T., Dehghani. M.. Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby. N. An image is worth 16x16 words: Transformers for image recognition at scale, International Conference on Learning Representations (ICLR), 2021. URL https: http:::///axx..oro//// abs/ 2000 11929.
[18] Esmaeilpour, S., Liu, B., Robertson, E., and Shu, L. sero out- distribution detection based on the pre-trained model clip. In AAAI conference on artificial intelligence (AAAI). 2022. URL https: // //arxiv. , org/ abaaabaa/aaa 22t./211...a 2109.02748.
[19] Faghri, F., Fleet, D.J., Kiros, J. R., and Fidler, S. Vsc++: Improving -isual-semantic embeddings with hard negatives. British Machine Vision Conference (BMVC), 2017. URL https:///axxxv/ wwtp:/iiv.or///s//.//// 1707. 05612.
[20] Fort, S., Ren, J., and Lakshninarayanan. B. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems, 2021. URL. httpa: //trri//aaiiv////aa orgg aaa///aaa2100 2106. 030044.
[21] Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, T. Devise: A deep
embedding model. Advances in neural information process- ing systems (NeurIPS), 2013. URL https://papers. httpppppppp.pp...pppppe.p www.miip......... httppcccppppeeeeeeee paalle/paper//21333 (ccG5cff967742277777777777
Abstract. .hmll.
[22] Gneiting, T. and Raftery, F. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 2007. URI.https: pdf.
wtt.:esstttttttttttttteee...e stat washington httpttton.dd//aftteeeeeeeee tery
Research, ww/Pp//PP//// /tDFF//eeitngg207aaasss
[23] Griffin, G., Holub, A., and Perona, P. Caltech object category dataset, 2007. URL wttth:t//ttthortt library. caltech. (eddu///9//// h/t////CCCSTTR-AR TR-2077-0000 .df.
[24] Grill, J.-B., Strub, F., Altché, F., Tallec, C.,, Richemond, P., Buchatskaya, E.,, Doersch, C., Avila Pires, B., Guo, 7., Gheshlaghi Azar, M., et Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems (NeurIPS), 2020. URL httpt::////attt org. /bbb//
2006. 07733.
[25] He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. In Conference computer vision and pattern recognition (CVPR), 2020. URL https: //arxiv.orz wwwp:///gg//aaaaeeeee. taabs/1911..e 05722.
[26] Heek, J., Levskaya, A., Oliver, A., Ritter, M., Rondepierre, B., Steiner, A., and van Zee, M. Flax: neural network library and ecosystem for JAX, 2023. URL http:/ /ttpttittttuu/..//// com/ google/flel
[27] Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Selected Topics in Applied Earth Observations and Remote Sensing, 2017. URL https: arxiv wwwpr/iwwwr///b/////aooa/// .00099.
[28] Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representutions (ICLR), 2019. URL https: //arxiv.org org/ p99//993. 12261v1 .pff.
[29] Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus), 2016. URL abs/1606 abs/1606.08415. 08415.
[30] Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and ont-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017. URL https: // //arxiv.orgg aaaaaaasa.11 1tt.//1111.0......11
[31] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang. F., Dorundo, E.,, Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of out-of-disttibutio generalization. In International Conference on Computer Vision (ICCV), 2021. URL https: //tpxv/.vvv/////////////////wtvl org/ abs 2206 16241.
[32] Hendrycks, D., Zhao, K.. Basart, S., Steinhardt, J., and Song. D. Natural adversarial examples. In Conference on Computer Visien and Pattern Recognition (CVPR), 2021. URL https://arziy 0bs//197.u. 177
[33] Jain, A., Tancik, M., and Abbeel, P. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In International Conference on Computer Vision (ICCV), 2021. URL https::/ //trx/aaavvvv//////////////aaall org/ aba/ab/// /204.. 00677.
11

---

[34] Jia, C., Yang. Y., Xia, Y., Chen, Y.-T.. Parekh, Z., Pham, H. Le, Q. Sung, Y.-H., Li, Z., and Duerig. T. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), 2021. URL https: //arxiv/..///////a http://wwwvv..r.aaaaa/e abs http//bb//20222 2102 05918.
[35] Joulin, A., Van Der Maaten, I.., Jabri, A., and Vasilache, N. Learning visual features from large weakly supervised data, In Faropean Conference on Computer Vision (ECCV), 2016. URL https: /arxiv. org/
[36] Karpathy, A. and Fei-Fei, L. Deep visual-semantic alignments for generating image descriptions. In Conference on computer vision and pattern recognition (CVPR), 2015. URL https:///arraee wwwp:/i..orr//aaa.....aa 1412. 2306.
[37] Kather, J. N., Weis, C.-A.. Bianconi, F. Melchers, S. M.. Schad, L. R.. Gaiser, T., Marx. A., and Zöllner, F. G. Multi-class texture analysis in colorectal cancer histology. Scientific Reports, 2016. URL https: //ww. nature. com/ arti cles/ erep27988.
[38] Kolesnikov, A., Beyer, I.., Zhai, X., Puigcerver, J., Yung. J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. European Conference on Computer Vision (ECCV), 2020. URL setps://arrivv org/ abs 191// 11370.
[39] Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In International Conference on Computer Vision (ICCV) Workshops, 2013. URL / /ttttp://ieeeeppooee ieee. org/ document/ wttporrcg//o///o/ooooo/o /2/5//7555555
[40] Krizhevsky, A., Hinton, G., ct al. Learning multiple layers of features from tiny images. Technical report, 2009. URL https: //www/ cs. toronto. aedu////ri/leeaaaae /eaii//eeaainnggg features----eeeeeee 2009- TR. påf.
[41] Kudo, T. and Richardson, J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Empirical Methods in Natural Language Processing (EMNLP), 2018. URL https: / arxiv. org/ abs/ 1808. 06226.
[42] Li, A., Jabri, A., Joulin, A., and Van Der Maaten, L. Learning visual n-grams from web data. In International Conference on Computer Vision (ICCV), 2017. URL https: // arxiv , org/ 09161.
[43] Luo, H., Ji. I.., Zhong, M., Chen, Y., Lei, W., Duan, N., and Li, T. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 2022. URL https:t////// h/appiv///////.////////a/aa org/bbs/ 2104. 088600
[44] Mahajan, D., Girshick, R., Ramanathan, V., Hc, K., Paluri, M., Li, Y., Bharambe, A., and Van Der Maaten, L. Exploring the limits of weakly supervised pretraining. In European conference on computer vision (ECCV), 2018. URL https: http://aa/vvv//a///aa//eae.awava wwg/abs/////eeaaoooe/..ooeeeeees abs/ 1803.0093.
[45] Minderer, M.. Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D.,, and Lucic, M. Revisiting the calibration modemn neural networks. Advances Neural Information Processing Systems (NeurlPS), 2021. htts:////rrxx arriv org/ ab/a//a////e abs/2106. 07998.
[46] Mori, Y., Takahashi, H., and Oka, R. Image-to- word transformation based on dividing and vector quantizing images with words. In First international workshop on multimedia intelligent storage and retrieval management, 1999. URL https: //d1. acm. org/doi//g/oooooooooooooog doi/ ptft//f//////////////////// / 100. 5555/2835865..... 2835895.
[47] Mu, N., Kirillov, A., Wagner, D., and Xie, S. Slip: Seff-supervision meets language-image pre-training. In European Conference Computer Vision (ECCV), 2022. URL https:://aaaaiii orgo htt//////////// 2211. 12750.
[48] Naeini, M. P., Cooper, G., and Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning. In AAAI conference on artificial intelligence (AAAI), 2015. URL sattps://oji aaai org/ index. php/ AAAI/ TAAAl/rrtice//ie wwwe/////eeeeeeeeeeeeeeeeeeeeeee /ttp//90////00 19411.
[49] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning (ICML), 2022. URL https: / /arxiv..ivvv org/ abs/211. 10741
[50] Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics and Image Processing, 2008
[51] Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding, 2018. URL https: //arxiv....v/////a org/ abs/ 1807 03748.
12

---

[52] Pan. S. J. and Yang, Q. A survey on transfer learning. IEEE Transactions knowledge and data engineering, 2010. URL
ieee. . org/ (eloccument/22885222
[53] Parkhi, O. M. Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012. URI. https:///eeexpllcr leee. org org/denumett/ 6248092.
[54] Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D. Styleclip: Text-driven manipulation of stylegan imagery. In International Conference on Computer Vision (ICCV), 2021. URL https: // //arxiv. org/ abw/ab/s/../ 17249.
[55] Pham, H., Dai, Z., Ghiasi, G., Liu, H., Yu, A. W. Luong, M.-T, Tan, M., and Le, Q V. Combined scaling for zero-shot transfer leamning 2021. URL https: //avivv/////iv////a//aa/e wwwg/bs//////eeoooo/ooo/o /tbb////11.. 10050.
[56] Plummer, B. A., Wang. I.., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentencce models. International Conference on Computer Vision (ICCV), 2015. URL https: //axiiv. org/ abs/1505. /1///1155000......
[57] Quattoni, A., Collins, M., and Darrell, T. Learning visual representations using images with captions. In Conference on Computer Vision and Pattern Recognition, 2007. URL https: /tte::///rraa aarxiv.vvvvv/ org/ www.oorg/////ooooeooeoe ab08 01392.
[58] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021. URL btttpa///arxivv org/ aaa//ba//2120 2103. http://21...000.... 00020.
[59] Recht, B., Roelofs, R., Schmidt, L., and Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML), 2019. URL https: //tp:i//aaaaa////aa org/ abs/1902. 10811.
[60] Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets, 2014, URL http::t/:///rre //aarv//////v/////a org/ /412. 6550.
[61] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang. Z., Karpathy, A., Khosla, A.. Bernstein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision (ICV)), 2015. URL https: // /arxxvv.vvv////////////////a org/ /1409. 0105905999.. 0576.
[62] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Syxtems, 2022. URL https: //arxiv. org/ abs 2200../22....... 11487.
[63] Sariyildiz, M. B., Perez. J., and Larlus, D. Learning visual representations with caption annotations. In European Conference on Computer Vision (ECCV), 2020. URL https://www. ecva. net. nettpppers/ accv_2020 pawpo/aapprrreeeee ECCV CCCC///pper/////ccccccccccsssses 123330154 pdf.
[64] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., al. Laion-5b: An open large-scale dataset for training next generation image- text models. Advances neural information processing systems. Truck on Datasets and Benchmarks, (NeurIPS Datasets and Benchmarks), 2022. URL https:// //tp///arvvvv//////aa//eae/aaava http//.iiiia//aaaaaa../.a/..
org/abs abs/ /2100. 08402.
[65] Shazeer, and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. International Conference on Machine Learning (ICMI., 2018. URL. htwps:ttt//eeeete http::///aawvreeeeaeaeeeee www./o/////.//////.//..o/../ee/a htt//bbb/880 1bb.1880..04255
[66] Sohn, K. Improved deep metric learning with multi-class n-pair loss objective. Advances neural information processing systems (NewrIPS), 2016. URL https: sttpps//pppprs waipa.cc/papee. /ffee// Pap.aaa Pww.:e..p.................... pdf.
[67] Srivastava. and Salakhutdinov, R. R. Multimodal learning with deep boltzmann machines. Advances in neural information processing systems (NeurIPS), 2012. URL https : //papers...p.///e////e//eeeeeeee nips. co/ co/paper_ f files/ http//pper///0/ 2atpper/211//halaeeea hash/ all.1l2.7.022222e2 wweeeee15522ceeeem -wt.aaddd-bbbbberee Abstract .htll.
[68] Sun, C., Shrivastava, A., Singh, S., and Gupta, A. Revisiting unreasonable effectiveness of data in deep lcarning era. In International conference on computer vision (ICCV), 2017. URL. httpp://aaxiv..ttt abs/1707 02968.
[69] Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J. Y0cm100m: The new data in multimedia research. Communications of the ACM, 2016. URL https: // /arxivv org/ wabb/b00. 01817.
13

---

[70] Tian. R., Wu, Z., Dai, Q., Hu, H. Qiao, Y., and Jiang. Y.-G. Resformer: Scaling vits with multi-resolution training, 2022. URL https:: bettpa://arii,,. .ogg abs waabb/2200 22122 00776.
[71] Tian, Y., Krishnan, D., and Isola, P. Contrastive representation distillation. In International Conference on Learning Representations (ICLR), 2020. URL https://arxxa org/ abs /1910. 10699.
[72] Tran, D.. Liu, J., Dusenberry, W. Phan, D.. Collier, M., Ren, J., Han, K.. Wang. Z., Mariet, Z.. Hu, H., Band, N., Rudner, I. G. Singhal, K., Nado, Z., van Amersfoort, J., Kirsch, A.. Jenatton, R.. Thain, N., Yuan, H., Buchanan, K., Murphy, K., Sculley. D., Gal, Y., Ghahramani. Z. Snoek, J., and Lakshninarayanan, B. Plex: Towards reliability using pretrained large model extensions, 2022. URL https: /trps://arxiv.... org/ abs/abb//77/// 2207 07411.
[73] Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, The caltech- uesd birds-200-200 dataset, 2011. URL https::////ttt h/wtpp::/www.../ vision. caltech. edu/ datasets/ cub_200.___00 _0r_2000220000 _011//
[74] Wang, J., Markert, K., Everingham, M., et al. Learning models for object recognition from natural language descriptions. In British Machine Vision Conference (BMVC), 2009. URL stttp://wwe bmva. org/ b 2009/Papers/PPP PwaaaaeeeeePeee 106 pdf.
[75] Wortsman, M., Ilharco. G., Gadre, S. Y., Roelofs, R., Gontijo- R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML.), 2022. URL http::ttttaaaaeeeeeeeee : http///wwi..o/aaaaaa/ee
org/ awgg//bb/220..... 05482.
[76] Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Komblith, S., Roclofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tuning of zero-shot models. In Conference Computer Vision and Pattern Recognition (CVPR), 2022. URL https: /tt./iiiiv/.///////// www.aawww....oaaaoeeeeeeeooeeeee wttg/b//////////o/a// 01903.
[77] Wu, Z.,, Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non-parametric instance discrimination, Conference on computer vision and pattern recognition, 2018. URL. petstp://aartie org/ /ba//baa/// 110...10....000 01978.
[78] Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., and Oliva, A. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision (ICV),, 2016. URL httts://link: springer com/ art.cle/ci........ 1007 htt.///11223001140 014-0748--7
[79] Yang, J., Li, C., Zhang, P., Xiao, B., Liu, C., Yuan, I.., and Gao, J. Unified contrastive learning in image-text-lahel space. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. URL. attps://arxing org/ abs/2204 http//224.03665 03610.
[80] Yang. Y. and Newsam, S. Bag-of-visual-wonde and spatial extensions for land-use classification. In SIGSPATIAL international conference advances in geographic information systems, 2010. URL bctpa://dl. acm. hetm.oorg/////li/ 1145/1869790 1869829.
[81] Yu, J., Wang, 7., Vasudevan, V., Yeung, I.., Scyedhosseini, M., and Wu, Y. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research (TMLR), 2022. URL
https: : //trx//avvvvv///////////ee/aawa/ org/ abs/2205
[82] Yuan, L., Chen, D., Chen, Y.L.,, Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. Florence: A new foundation model for computer vision, 2021. URL https: pettpps://ariivv .rrg/ abs/2111. 11432.
[83] Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, B., Riguelme, C., Lucic, M.. Djolonga, Pinto, S.. Neumann, M., Dosovitskiy, A., al. A large-scale study of representation learning with the visual task adaptation benchmark, 2019. URL https: htappss://arrrvv// org/ aba/ aba///11/ 1910. 04867.
[84] Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. URL htttps://arrae www.aaawwww..oooaooaaaaaerr oww.oos//////ooooooooooo.ooo.ss wwwpaa/b....//////////a//..a...a 01w.//26066.06..... pdf.
[85] Zhai, X., Wang, X., Mustafa, B., Steiner, A.. Keysers, D., Kolesnikov, A., and Beyer, L. Lit: Zero-shot transfer with locked-image text tuning. In Conference on Computer Vision and Puttern Recognition (CVPR), 2022. URL https:/ /ttps::///axvvv////aa org/ abbs/2111.0795
[86] Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., and Langlotz, C. P. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, 2022. URL https: arriiv org/ abs /2000. 00747.
14

---

[87] Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A. Places: A 10 million image database for scene recognition. Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2017. URL ieee. org/ cocuunt///// 779888877
https::/
[88] Zhou, K.. Yang, J., Loy, C. C.,, and Liu, Z. Conditional prompt learning for vision-language models. In Conference on Computer Vision and Puttern Recognition (CVPR), 2022. URL https::/arxii..om abs/ /
Zhou, K., Yang. J., Loy, C., and Liu, Z. Leaminng to prompt vision- language models. International 189]
Journal of Computer Vision (IJCV), 2022. URL https: http://axxvv.rreeee aataa/bs//220 01134.
15

---

Appendix
Three Towers: Flexible Contrastive Learning with Pretrained Image Models
A
Additional Experiments & Results
A.1
Training Dynamics
(b) x10-1
(c) x101
Basel.
Basel, ST fuu
Retrieval Coco img2txt
4
LIT-3T fo
3T Lyme
4
5
Dift
Loss
2
3
Reseline
Lil
0.1
0.0
0
Steps
x x100
Steps
x 10
Steps
x100
Figure A.1: Training dynamics: (a) The transfer losses, & fi, , and L ++h, improve the image-text loss, feeg. in 3T relative to the baseline. (b) Difference between matching loss terms for 3T, LiT, and the baseline. 3T obtains better image-to-text loss than the baseline and similar locked-image-to-t loss as LiT. (c) While the loss advantage of over the baseline shrinks during training, this does not happen for downstream applications; we display image-to-text retrieval on COCO as example. Moving averages applied to (a-b) for legibility.
In Fig. 1, compare the 3T training losses LiT and the from-scratch baseline, using the familiar I. scale setup with JFT pretraining. For 3T, we display all loss terms separately: the image-text loss fittgg the smage-to-third-towe and Fig. 2. For LiT and the baseline, there is only the image-to-text loss as per Eq. (1). As we train for less than one epoch, we do not observe any overfitting, in the sense that contrastive losses are identical on the training and validation set.
loss f and the text-to-third-tower loss Eggtth, cf. Eq. (4)
The image-to-tine to-thoor--ttoooo loss, Ci quickly reduces to zero, indicating successful knowledge transfer of the pretrained model into the image tower for 3T. Further, Ly++g behaves similar to the baseline loss; this makes sense because both objectives compute a loss between unlocked image tower and a text tower. Lastly, L.,, , closely follows LiT's loss; this also makes sense because both losses between locked pretrained image model and text tower trained from scratch.
In Fig. A.1 (b), we compute the difference of the baseline (LiT) loss and matching 3T Ci,eehy (Cgg, ,>,,,) loss, and observe that 3T generally achieves lower (similar) values for the same objective. This suggests a mutually beneficial effect for the individual loss terms of the 3T objective. By aligning the main image and text towers to the pretrained model, 3T obtains improved alignment between the main towers themselves. For the training loss, this effect is large early in training and then decreases. find that the gap between 3T and the baseline persists;
However, for downstream task application, we display retricval on COCO as an example in Fig. A.1 (c).
A.2 Robustness Metrics
In this section, we study 3T, LiT, and the from- from-scratch baseline from robustness perspective, evaluating on a subset of the tasks considered by Tran et al. [72]. Following $4.3, we evaluate all methods across multiple model scales and for both JFT and IN-21k pretraining. We use the full Unfiltered WebLI for all results here. We apply models in zero-shot fashion to these datasets, following the same protocol as for the main zero-shot classification experiments. We continue to use the global temperature т, ef. $3, learned during training to temper the probabilistic zero-shot predictions.
16

---

A.2.1 Probabilistic Prediction and Calibration on CIFAR and ImageNet Variants
In Fig. A.2, we report accuracy, negative log likelihood (NLL), Brier score [22], and expected calibration error (ECE) [48, 22] for 3T, LiT, and the baseline across scales for the following datasets: CIFAR-10. CIFAR-IO-C, ImageNet- 1k (IN- Ik), IN-A, IN-v2, IN-C. and IN-R.
Accuracy. Across all datasets, we find the familiar scaling behavior discussed in $4.2: 3T is consistently better than the baseline, 3T benefits more from increases in scale than LiT, LiT performs well with JFT pretraining but shows weaknesses when pretrained on IN-21k. Note that, for the ImageNet variants, have previously reported the accuracies (if only at L scale) in $4.2. (Note further, that there might be small discrepancies, because we actually recompute all numbers from a different codebase for the robustness evaluations (15].) For CIFAR-I0 [40] and CIFAR-IO-C [28], which we have not previously discussed, we also find the familiar scaling behavior. The absolute reduction of performance between CIFAR-10 and CIFAR-IO-C is similar across methods, indicating that no approach is significantly more robust to shifts. We observe the same comparing IN-IK to
IN-C.
Probabilistic Prediction and Calibration. NLL and Brier scores follow the general trend laid out by the accuracy results. Evidently, the probabilistic zero-shot predictions of the methods are all of similar high quality, cf., for example, Tran et al. [72], who investigate probabilistic few-shot predictions. This is confirmed by the ECE results: across tasks, ECE values do not exceed 0.1 L scale. For 3T, calibration results are regularly better than for LiT, particularly if pretrained on IN-21K and comparable those of the baseline: 3T and the baseline have lower calibration error than LiT on 6 out of 7 tasks at L scale with IN-21K pretraining.
We find the low magnitude of the calibration errors surprising. It is striking that the softmax temperature learned during contrastive training would work so well across the various downstream task applications. After all, finding matches across a batch from the contrastive learning dataset and assigning images to labels are, at least superficially, quite distinct tasks. We stress again that no task adaptation of either the models, prompt templates, softmax-tent temperrtttssssss was performed. We refer to Minderer et al. [45] for a general categorization of our calibration results and discussion in the context of deep learning models.
A.2.2 Out-Of-Distribution Detection
We evaluate the performance of 3T, LiT, and the baseline for out-of- (OOD) detection. We follow the common practice of thresholding the maximum softmax probabilities (MSP) of the models to obtain a binary classifier into in- and out-of-distributie metrics: area under the precision-recal curve (AUC(PR)), the area under the receiver operating curve (AUROC), as well as the false positive rate at 95 % true positives (FPR95). Following Tran et al. [72], we study CIFAR- 10 as in-distribution against CIFAR- 100, DTD, Places365, and SVHN as out-of- distribution. We also report numbers for IN-1k (in-distribution) vs. Places365 (out-of-distributtio))
[30, 20]. We report the following
Typically for OOD evaluations, the model is trained on the in-distribution data. Here, we apply methods in zero-shot manner: we only condition the text tower on the label set of the particular in-distribution dataset. Our image and text towers are trained on the contrastive learning data (image tower trained on JFT/IN-21K for LiT) and not adapted to the in in-distribution samples. Our contrastive learning methods 'learn' about the in-distribution data only through the label set, and they have to classify each incoming sample as 'in-distribution" or aligns with the given set of labels. If a given sample does not match any of the in-distribution labels well, prediction confidence is low, and the sample is classified as OOD. This setup diverges from typical assumptions about OOD experiments and should be interpreted with care. For example, if there were label overlap between the in- and out-of-diitribution data (c.g. would be the case for SVHN vs. MNIST), it would be impossible for the model to classify between in-distribution and OOD without further assumptions. OOD for CLIP/ALIGN-style models has been studied in similar settings by Fort et al. [20], Esmaeilpour et al. [18].
based solely on how well it
We display results in Fig. A.3. Generally, OOD detection works well with the contrastively learned models, despite conditioning only on the label set: for example, the AUROC for CIFARIÓ 10 exceeds 0.95 for both 3T and LiT at L scale for both IN-21k and JFT pretraining- The different metrics, AUC(PR), AUROC, and FPR95, are generally consistent in their ranking across scales and methods. We again find the familiar patterm 3T is consistently improving over the baseline, and 3T catches up
17

---

Accuracy 1
NLL
Brier 4
ECF.
0.75
0.10
CIFAR- 10
90
0.3
0.50
Baseline
0.2
0.05
80
L.T
ST
0.25
0.1
0.100
CIFAR 10cc
80
1.0
0.4
0.075
0.050
60
0.5
0.2
80
0.075
0.6
INet
0.050
0.4
0.025
75
1.0
INet-a
50
0.1
0.5
25
0.04
INet-v2
60
0.6
0.4
0.02
40
0.8
60
0.10
INet-c
0.6
40
0.05
0.4
20
0.08
80
2
0.50
INct-r
0.06
0.25
0.04
i21K-S
121k-B
121K-L
JFT-B
JFT-
JFT-g
121k-S
121K-B
121K-L
JFT-B
JFT-
1211-S
i21k-B
121K-L
JFT-B
JFTL
JFT-g
i21K-S
121K-B
121K-L
JFT-B
JET-L
JFT
Figure A.2: Robustness evaluation: Accuracy, negative log likelihood (NLL), Brier score, and
expected calibration error (ECE) for 3T, LiT, and the baseline for IN-21k and JFT pretraining across scales.
18

---

AUC(PR)
AUROC 1
FPRY5
0.6
0.95
C0 vs. Cifar
0.9
0.90
0.4
0.85
0.8
Raseline
LIT ST
0.80
0.2
Cifarl0 vs. DTD
0.8
0.95
0.3
0.90
0.6
0.2
0.85
0.1
0.4
0.95
Cifar10 Places365
0.7
0.995
0.90
0.6
0.85
0.990
0.5
0.80
0.985
0.75
0.20
Cffarl0 SVHN
0.99
0.98
0.15
0.98
0.96
0.10
0.96
0.94
0.05
INet vs. Places365
0.8
0.94
0.7
0.90
0.6
121K-S
121k-B
121k-L
JFT-B
JFT-L
JFT
121k-S
i21k-B
12Ik-L
JFT-B
JFT-L
JFT-g g
121k-5
121k-B
121k-L
JFT-B
JFT-L
JFT-
Figure A.3: Robustness evaluation: 3T, LIT, and the baseline for zero-shot duttof-distiibution
detection (OOD). Reported metrics are area under the precision-recall curve (AUC(PR)), the area under the receiver operating curve (AUROC), and the false positive rate 95 % true positives (FPR95).
19

---

Table A.I: Using pretrained language models instead of image encoders degrades performance drastically for LiT, confirming the results of Zhai al. [85]. While 3T does not suffer a drastic collapse in performance, we also do not observe gains from using the pretrained language model. B/32 ViT image tower, BERT models BASE scale for LiT and 3T, scale unlocked text tower for 3T and baseline, training for 900M examples seen at batch size 10 240.
Unfiltered WebLI
Pair-Filtered WebLI
Basel.
LiT
3T
Basel.
LiT
3T
Retrieval
Flickr" img2txt
55.2
4.4
50.6
65.7
10.6
64.4
Flickr" txt2img
36.2
2.4
32.4
45.2
5.2
43.7
Coco img2txt
34.5
2.2
29.9
42.8
5.0
38.4
Coco tx12img
20.7
1.4
18.2
26.8
3.2
24.6
Few-Shot Classification
Caltech
87.4
80.3
87.0
88.7
88.0
90.0
UC Merced
84.2
67.7
81.4
90.5
81.0
87.8
Cars
56.8
38.0
56.8
76.9
67.1
76.8
DTD
59.2
45.8
56.3
63.8
58.3
63.1
Col-Hist
72.0
59.3
69.0
75.8
63.8
73.2
Birds
32.1
16.9
29.5
48.5
35.1
46.7
Pets
57.8
31.9
53.5
72.7
70.2
78.1
ImageNet
37.8
24.2
35.2
46.5
41.6
47.9
CIFAR-100
50.2
33.8
45.2
57.5
43.7
53.3
Zero-Shot Classífication
Pets
58.7
6.1
56.5
79.2
36.8
79.4
ImageNet
45.8
11.3
41.8
58.0
26.9
56.9
CIFAR-100
52.2
19.1
44.0
57.6
37.6
54.7
LiT as scale is increased. For OOD detection, LiT generally does better than 3T and the baseline, perhaps owing to the fact that our choice of CIFAR/IN-II 1k in-distribution datasets is advantageous for LiT (similar to how LiT performs particularly well for these datasets for classífication).
We find differences between JFT and IN-21k pretraining to be much smaller for the OOD detection task. In fact, in some cases, IN-21k pretraining outperforms JFT pretraining, for example with LiT for the CIFAR- 10 vs. Places365 detection task. (This might again be due to the fact that IN-21k pretraining is sufficient for application to CIFAR- 10, and only struggles to perform well for other, more varied datasets.) Further, we can observe rare victory of 3T over JFT-LiT and the baseline and g scale in terms of FPR95 on CIFAR- 10 vs. Places365 and CIFAR- vs. SVHN. Lastly, we see LiT has almost fixed performance at x 0.98 for the CIFAR-10 vs. SVHN task across scales, perhaps due to early task saturation.
A.3 Pretrained Language Models
Table A.1, we explore if there are benefits to using pretrained language models instead of pretrained image encoders. For LiT, we confirm the results of Zhai et al. [85] that performance suffers drastically when using locked pretrained language encoder as the text tower (together with an unlocked image tower). In contrast, for 3T, we do not observe drastic decrease performance, once again showing that 3T is more robust to deficiencies in the pretrained model than LiT. However, we also do not observe gains from using pretrained language model with 3T, justifying pretrained image encoders in the main text. Nevertheless, we think it is plausible that future work may yet find benefits of incorporating knowledge from pretrained language models in other settings or downstream tasks, e.g. tasks that require more complex language reasoning abilities.
choice of focusing
Results here are for B/32 VIT image towers, we use BERT models at BASE scale pretrained language models for LiT and 3T, a scale unlocked text tower for 3T and the baseline, and we train for 900M examples seen at batch size 10 240.
20

---

Table A.2: Predictions between the bascline and LiT are different and 3T benefits from combining them. The first column gives the proportion of datapoints where the baseline but LiT predicts correctly. The second column gives the proportion of datapoints where LiT but not the baseline predicts correctly. Columns three five show the proportion of datapoints where 3T predicts differently than the baseline (3), LiT (4), or differently from both (5).
Base, Not LiT
LiT, Not Base
3T of Base
3T o LiT
3T A Base & 3T LiT
IN-Ik
6.9
13.3
22.8
26.5
14.5
CIFAR-100
7.5
18.0
28.6
32.8
20.4
Caltech
3.9
4.3
11.5
16.1
8.4
Pets
7.2
7.1
10.9
14.4
5.5
DTD
14.3
7.6
27.2
40.3
19.5
IN-A
19.0
11.7
38.3
52.8
28.4
IN-R
23.5
3.8
13.3
34.2
10.0
IN-v2
8.4
13.3
27.2
32.8
18.0
ObjectNet
20.7
6.4
34.1
52.9
26.7
EuroSat
19.2
14.2
62.3
75.1
49.4
Flowers
4.0
14.8
25.4
27.0
16.0
RESISC
33.2
4.1
31.9
62.6
25.7
Sun397
11.8
9.6
21.9
31.2
14.2
A.4 Investigating Prediction Differences
Table A.2 provides a detailed evaluation of how predictions differ between 3T, LiT, and the baseline on per datapoint (and per task) level. The results in Table A.2 for the zero-shot tasks of the IN-21k pretrained L scale model setup of Table 3. This evaluation gives meaningful insights into understanding how 3T improves predictions over LiT and the baseline. The contrastive learning baseline and the pretrained model have different strengths, and 3T can benefit from combining them.
The first columns of Table A.2 show that there is predictive diversity between the baseline and LiT. The first column gives the proportion of datapoints where the baseline but not LiT predicts correctly. The second column gives the proportion of datapoints where LiT but not the baseline predicts correctly. We can see that, for both the baseline and LiT, there is significant proportion of inputs, where only one of the models predicts correctly. Hence, the strengths and, equivalently. make different mistakes. A combination of the two approaches, such as 3T, can benefit from this if learns to combine their predictions in the right way.
models have different
In columns three, four, and five, we illustrate that 3T predicts differently from LiT and the baseline. The columns show the proportion of datapoints where 3T predicts differently than the baseline (3), LiT (4), or differently from both (5). Clearly, 3T learns a novel predictive mechanism that is meaningfully different from LiT and the baseline. Whenever 3T outperforms both LiT and the Baseline (Caltech, DTD, IN-A, IN-R, ObjectNet, EuroSat, and Sun397 in Table 3) it must have learned to combine knowledge from the pretrained model and contrastive learning mechanism in beneficial way.
A.5 Additional Image Encoders
In Table A.3, we study 3T and LiT for additional pretrained image encoders: a ResNet-based BiT model trained on IN-21k [38] and a ViT-based self-supervised encoder trained with DINO on IN-IK as in [85]. 3T consistently outperforms the baseline and LiT for DINO- and BiT-baser pretrained models for all retrieval scenarios. For few-shot classification, LiT performs best on average for BiT- based models but 3T performs best on average for our DINO experiments. For zero-shot classification, 3T performs best average for both DINO- and BiT-based experiments. In total, 3T performs best on average over all tasks for both experiment setups.
Here, we use B/16 scale image tower and B scale text encoder trained on Unfiltered WebLI. The pretrained image models are a BiT-R50x3 pretrained on IN-21K. and we follow Zhai et al. [85] for DINO pretraining IN-Ik. We also report a 'BiT-Baseline': for this we replace the ViT B/16 image tower of the standard baseline with a BiT-R50x3 trained from scratch.
21

---

Table A.3: Results for additional pretrained image encoders: a ResNet-based BiT trained on IN-21k and a self- self-supervised DINO encoder trained on IN-Ik. 3T outperforms LiT and the baseline on average for retrieval and classification task, with the exception of few-shot classification for BiT, where LiT performs best. B/16 scale image tower, B scale text encoder, BiT-R50x3 pretrained on IN-21K, DINO pretrained on IN-lk as Zhai et [85], Unfiltered WebLI.
BiT Experiments
DINO
Experiments
Basel.
BiT-Basee
LiT
3T
Basel.
LiT
3T
Flickr" img2txt
71.2
72.5
61.1
74.6
71.2
60.4
74.0
Retrieval
Flickr txr2img
49.3
50.6
39.1
52.5
49.3
35.0
52.1
Flickr" tx12img
51.3
53.7
42.9
54.5
51.3
38.3
54.0
Flickr imtt
68.5
68.1
59.2
70.1
68.5
57.5
69.2
COCO img2txt
44.3
45.0
40.9
46.8
44.3
38.9
45.7
COCO txt2img
29.0
29.8
24.7
32.0
29.0
21.2
31.4
Average
52.3
53.3
44.7
55.1
52.3
41.9
54.4
Caltech
89.8
90.0
90.4
90.9
89.8
91.0
92.1
Few- Shot Classification
UC Merced
89.3
84.0
91.6
91.2
89.3
94.2
92.1
Cars
75.0
73.4
42.4
77.1
75.0
49.4
78.7
DTD
66.5
64.0
66.8
69.4
66.5
63.5
70.1
Col-Hist
68.3
66.6
84.8
72.9
68.3
85.6
75.2
Birds
44.5
38.5
85.5
53.2
44.5
62.1
52.5
Pets
75.8
65.5
89.2
78.9
75.8
82.5
76.3
IN-1K
52.0
57.8
73.8
56.2
52.0
60.4
56.4
CIFAR-100
57.8
40.0
74.1
62.1
57.8
62.3
61.0
Average
68.8
64.4
77.6
72.4
68.8
72.3
72.7
Pets
75.4
79.0
80.9
80.2
75.4
83.5
78.7
IN-Ik
59.8
60.6
67.1
62.8
59.8
62.9
62.5
CIFAR-100
60.0
45.0
72.6
59.6
60.0
56.0
61.1
Zero-Shot Shot Classification
IN-A
32.8
31.4
26.2
34.9
32.8
21.7
32.5
IN-R
75.4
73.4
55.4
78.1
75.4
57.0
77.4
IN-v2
53.1
53.3
58.8
55.7
53.1
55.0
55.6
ObjectNet
44.7
42.7
34.4
47.3
44.7
25.3
45.0
Caltech
79.1
81.0
76.7
79.3
79.1
79.0
80.7
DTD
51.4
52.8
46.6
54.1
51.4
41.9
52.4
Eurosat
36.5
19.7
28.1
32.6
36.5
32.6
36.2
Flowers
51.9
54.3
64.1
57.1
51.9
48.2
55.6
RESISC
53.1
49.9
27.8
52.9
53.1
24.4
48.0
Sun397
62.6
63.4
60.9
65.3
62.6
53.7
63.2
Average
57.1
55.0
54.8
59.2
57.1
50.4
58.4
Average
59.6
57.4
59.5
62.2
59.6
55.1
61.8
22

---

A.6 IN-21k Pretraining Additional Results
In Table A.4, we report retrieval and few-/zero-shot classification performance for 3T, LiT, and the baseline across a selection of pretraining datasets for L scale models and IN-21k pretraining. In addition to the unfiltered WebLI split reported in the main text, we here report results on the pair-filtered and text-filtered splits of the WebI.I dataset, cf. $4.2. Further, we report results on the same dataset used by Zhai et [85] to train LiT. Across all datasets, 3T outperforms LiT and the baseline on average for retrieval and few-/zero-shot classification tasks, confirming our results for the unfiltered WebLI split in $4.1 and $4.2.
Table A.4: Results for the baseline, LIT, and 3T for L scale models and IN-21K pretraining. Across all datasets, 3T outperforms LiT and the baseline average for retrieval, few-shot image classification, and zero-shot image classification tasks.
Dataset
Pait-Filtered WebLI
Text-Filtered WebLI
LiT Dataset
Method
Basel.
LiT
3T
Basel.
LiT
3T
Basel.
LiT
Flickr img2txt
79.4
72.4
81.6
79.8
72.1
83.9
79.1
71.7
82.0
Retrieval
Flickr' img2txt
81.0
73.5
83.8
84.5
75.8
86.9
80.2
74.5
84.2
Flickr tx12img
59.6
48.3
51.3
64.1
Flickr' txt2img
61.8
51.5
64.0
67.9
54.2
70.6
61.5
50.8
66.1
COCO img2Ixt
56.4
47.8
58.4
58.2
50.3
62.5
53.0
47.2
57.6
COCO txt2img
39.1
29.5
40.9
43.0
31.8
45.6
38.3
29.6
41.3
Average
62.9
53.8
65.2
66.4
55.9
69.7
62.1
53.6
65.9
Few Shot Classification
IN-1K
67.8
79.0
71.9
64.9
79.0
69.6
63.3
79.0
68.6
CIFAR- 100
71.0
83.6
75.1
73.6
83.6
77.0
73.0
83.6
74.9
Caltech
88.8
88.4
90.8
89.8
88.4
90.5
90.7
88.4
92.0
Pets
91.3
89.2
91.7
85.7
89.2
87.7
84.1
89.2
86.9
DTD
73.2
69.2
74.8
71.5
69.2
75.7
71.3
69.2
73.1
UC Mercod
94.3
92.8
96.1
94.8
92.8
95.8
92.8
92.8
94.6
Cars
91.5
41.9
92.4
86.5
41.9
89.0
85.3
41.9
87.8
Col-Hist
77.5
86.4
81.3
76.1
86.4
80.4
72.2
86.4
78.6
Birds
70.9
83.4
79.0
56.7
83.4
68.2
62.1
83.4
70.4
Average
80.7
79.3
83.7
77.7
79.3
81.6
77.2
79.3
80.8
IN-1k
75.1
77.5
76.5
72.1
76.6
73.8
72.1
77.4
74.8
CIFAR-100
73.0
82.4
74.0
76.8
82.9
78.0
76.0
83.1
77.0
Zero Shot Classificatonn
Caltech
84.2
78.5
83.9
80.3
80.8
84.5
81.4
80.7
84.7
Pets
93.5
91.6
94.4
87.7
88.3
90.8
86.5
88.4
88.6
DTD
56.4
50.2
57.8
61.2
49.1
59.4
61.8
54.6
60.7
IN-A
51.8
46.7
54.8
57.6
46.7
60.4
56.2
47.3
58.9
IN-R
88.4
67.4
89.6
89.0
67.2
90.3
88.4
67.
90.0
IN-v2
67.8
68.9
69.8
65.4
68.0
67.5
65.3
68.6
68.3
ObjectNet
52.6
43.3
54.6
57.3
42.4
59.0
54.3
42.6
56.5
Eurosat
37.2
28.5
44.2
38.2
20.2
42.4
44.0
29.1
45.8
Flowers
79.7
81.1
80.3
63.8
77.8
67.7
67.3
79.0
73.4
Resisc
61.6
32.5
30.0
63.1
58.3
28.4
57.2
Sun397
67.1
63.9
67.6
65.6
69.6
65.8
69.5
Average
68.9
64.1
70.7
68.7
63.2
70.8
68.5
64.2
70.5
Average
71.1
66.4
73.3
70.8
66.3
73.6
69.7
66.4
72.5
23

---

A.7 JFT Pretraining Additional Results
In Table A.5, we report few- and zero-shot classification performance for 3T, LiT, and the baseline across our selection of datasets for L scale models and JFT pretraining. LiT outperforms 3T and the baseline on average for few- and zero-shot classification tasks.
In Table A.6, we report performance for scale models and JFT pretraining across all three splits of the WebL.I dataset described in $4. Retrieval performance is generally best for all methods for the Text-Filtered WebLI split, with 3T generally performing best across splits and tasks. For classification, for 3T and the baseline, performance on Text- and Pair-Filtered WebLI significantly better than on Unfiltered WebLI, with LiT generally performing best across splits. In line with our previous observations, the differences between the WebLI splits are smaller for LiT. As the image tower is kept fixed during contrastive training, LiT performance is influenced less by the contrastive learning setup.
Comparison to SOTA. While our retrieval performance is competitive, 3T does not set a new state-of-the art, see, for example, the CoCa paper [81] (Table 3) for a comparison of current methods. While SOTA results never the aim of this paper-we instead study pretrained models for contrastive learning- 3T would likely benefit, there are a few advantages the CoCa setup has, and from which Most notably, CoCa trains for about 6 times more examples seen than we do here (32B vs. 5B). Our scaling experiments, cf. Fig. A.4, suggest we would expect significant performance increase for longer training. There are further differences that likely benefit CoCa, such as the use of a larger batch size (65k for them vs 14k for us) or training on images with higher resolution for a portion of training (CoCa goes from 288x288 to 576x576, we stay at 288 x 2 available to us: while CoCa training takes 'about 5 days on 2,048 CloudTPUv4 chips' [81],, our g scale runs train for about the same duration on only 512 v4 TPU chips. It would be interesting to see if, in a fairer comparison, 3T matches outperforms CoCa for retrieval tasks. Alternatively, ideas from 3T could also be used to improve CoCa-like architectures.
Retrieval Results:
of these changes significantly increase computational costs beyond the budget
24

---

Table A.5: For JFT-pretraining, LiT outperforms 3T and the baseline on average on few- and zero-shot classification tasks. L scale models trained Unfiltered WebLI.
Method
Basel.
LiT
3T
IN-1k
62.8
81.3
67.7
Few- Shot Classification
CIFAR-100
70.4
83.2
74.3
Caltech
91.0
89.0
91.8
Pets
85.9
96.8
88.4
DTD
70.3
72.1
72.4
UC Merced
91.8
95.5
93.1
Cars
81.5
92.9
87.1
Col-Hist
71.7
81.3
77.0
Birds
53.4
85.6
62.4
IN-1k
69.5
80.1
72.0
CIFAR-I00
73.5
80.1
75.2
Caltech
81.9
79.5
82.5
Vero-Shot Classification
Pets
84.2
96.3
88.7
DTD
58.6
59.0
59.0
IN-C
49.6
68.1
52.8
IN-A
53.0
69.1
56.4
IN-R
85.8
91.7
88.4
IN-v2
62.2
74.0
65.4
ObjectNet
56.2
61.9
59.3
EuroSat
32.7
36.6
54.7
Flowers
62.0
76.7
66.6
RESISC
58.0
58.9
60.9
Sun397
67.6
69.7
68.1
Average
68.4
77.4
72.4
25

---

Table A.6: Results for the baseline, LiT, and 3T for g scale models and JFT pretraining for selection of different splits of the WebLI dataset. 3T outperforms LiT for retrieval tasks, while LiT performs better for image classification. The from-scratch CLIP/ALIGN-styl baseline not competitive.
Dataset Method
Unfiltered WebLI
Pair-Filtered WebLI
Text-Filterd WebLI
Basel.
LiT
3T
Basel.
LiT
3T
Basel.
LiT
3T
Flickr img2txt
75.2
83,0
81.5
81.4
83.2
84.0
85.0
83.9
87.3
Retrieval
Flickr" img21xt
80.0
84.8
84.2
80.7
83.9
85.6
86.7
85.2
88.3
Flickr tx12img
58.2
61.3
64.3
61.4
63.9
66.5
67.0
66.5
72.1
Flickr" tx12img
60.1
63.1
65.6
62.7
65.4
68.4
68.2
67.6
72.9
COCO img2txt
52.3
57.7
57.5
58.4
59.7
61.7
60.0
59.5
64.1
COCO txt2img
37.5
40.0
41.1
41.2
41.9
43.9
44.7
43.6
48.5
IN-1k
67.5
84.6
72.8
71.8
84.6
75.7
69.6
84.6
73.9
Few Shot Classification
CIFAR-
72.7
83.2
78.0
83.2
78.7
83.2
80.0
Caltech
91.8
90.0
93.3
89.7
90.0
90.9
90.8
90.0
92.4
Pets
88.4
97.8
91.5
93.0
97.8
94.3
88.8
97.8
91.4
DTD
70.7
74.6
74.7
74.2
74.6
76.1
73.6
74.6
76.0
UC Merced
92.9
96.9
94.7
95.2
96.9
95.6
95.2
96.9
96.5
Cars
84.1
93.3
88.6
92.6
93.3
93.5
89.0
93.3
91.6
Col-Hist
72.0
83.6
76.2
77.8
83.6
80.9
73.5
83.6
79.4
Birds
60.7
89.7
69.8
76.4
89.7
80.7
62.5
89.7
71.1
IN-1k
73.5
84.0
76.3
78.0
84.7
79.6
75.8
84.3
78.2
CIFAR-
77.5
81.3
76.2
81.3
79.5
81.8
82.3
Caltech
79.8
81.4
82.3
84.0
82.4
82.9
79.5
80.9
81.9
Zero Shot Classification
Pets
87.0
96.4
92.7
92.8
97.7
93.0
88.1
96.5
91.5
DTD
59.2
62.1
64.9
58.9
55.6
60.1
61.4
62.0
62.1
IN-C
54.9
72.9
58.2
57.7
73.3
60.3
57.6
73.3
61.3
IN-A
64.9
80.2
67.8
59.9
79.5
65.1
67.8
80.5
70.8
IN-R
89.8
94.4
91.8
90.5
94.2
92.8
91.8
94.6
93.3
IN-v2
66.4
78.1
69.5
70.8
79.2
73.0
69.1
78.5
71.4
Objectnet
62.7
70.3
56.9
68.3
59.5
70.0
65.9
Eurosat
55.7
33.6
48.9
32.9
30.7
42.8
47.9
36.1
52.1
Flowers
71.0
84.2
73.5
82.4
86.3
83.0
69.4
86.6
72.5
RESISC
61.5
58.4
60.5
59.8
56.5
64.8
65.4
57.8
61.7
Sun397
68.8
71.0
70.3
68.9
71.9
69.8
70.2
71.6
70.9
Average
70.2
77.0
73.7
72.4
77.0
75.3
73.1
77.7
75.9
26

---

A.8 Pretraining Robustness Additional Results
In Table A.7, we report results on additional tasks for 3T, LiT, and the baseline for both the 'mis- matched' setup and Places365 pretraining of $4.4. We find again that 3T is much more robust in both setups, significantly outperforming LiT. The difference particularly striking when using models pretrained on Places365, where LiT's performance degrades drastically while 3T is still able to improve over the baseline.
Table A.7: Testing robustness to the 'mismatched setup' and Places365 pretraining (instead IN-2IK/FFT) for 3T and LiT. In both cases, 3T performs significantly better than LiT. In particular when using models pretrained on Places365, LiT's performance degrades dramatically while 3T continues to improve over the baseline average. (Note that the baselines here are different not because they use the pretraining dataset, but because we compare to an L scale baseline for the mismatched setup and a B scale bascline (trained for only 900M examples seen) for Places365 pretraining.) We refer to the main text for full details.
Experiment Method
Mismatched Setup
Places365 Pretraining
Basel.
LiT
3T
Basel.
LiT
3T
Retrieval
Flickr" img2txt
75.6
66.5
80.2
56.0
35.5
58.1
Flickr" txt2img
57.1
45.1
62.1
36.2
19.5
38.4
COCO img2txt
51.0
44.1
54.5
34.1
19.3
36.5
COCO tx12img
34.2
26.4
37.8
21.0
10.9
22.1
IN-1K
62.8
70.3
67.6
37.8
16.6
41.5
Few Shot Classification
CIFAR-100
70.4
80.3
73.8
47.1
33.9
52.7
Caltech
91.0
88.1
91.7
87.9
66.5
88.5
Pets
85.9
86.0
86.8
56.8
20.3
59.9
DTD
70.3
66.3
73.4
58.4
39.7
63.1
UC Merced
91.8
91.5
93.8
85.8
80.8
89.4
Cars
81.5
36.7
85.3
57.0
10.1
58.6
Col-Hist
71.7
84.4
74.3
72.9
70.7
78.7
Birds
53.4
76.8
65.2
33.2
15.7
38.1
IN-1k
69.5
69.5
71.5
45.6
24.5
47.4
CIFAR-100
73.5
78.6
75.6
48.3
27.4
52.4
Caltech
81.9
82.0
81.2
76.6
62.7
77.0
Zero Shot Classification
Pets
84.2
84.7
87.4
61.5
30.3
60.2
DTD
58.6
49.4
60.6
39.8
23.6
39.7
IN-C
49.6
55.5
51.8
25.3
14.4
27.3
IN-A
53.0
29.1
54.1
12.0
4.7
12.5
IN-R
85.8
60.7
87.9
56.1
20.3
58.2
IN-v2
62.2
61.1
65.0
39.4
20.7
40.5
Objectnet
56.2
34.9
57.8
28.4
7.3
29.6
Eurosat
32.7
33.1
52.5
33.7
15.6
27.3
Flowers
62.0
74.1
66.2
37.6
17.4
37.3
RESISC
58.0
29.0
57.4
37.9
24.0
38.3
Sun397
67.6
62.0
68.4
55.1
60.6
57.3
Average
66.4
61.7
69.8
47.5
29.4
49.3
27

---

A.9 Scaling Model Sizes and Training Duration Additional Results
Complementing the results of $4.3, in Fig. A.4 report the performance when scaling only the number of examples seen during training, keeping the model sizes fixed at B scale. We observe a similar trend to $4.3 / Fig. 3, where 3T benefits more from increases in scale than LiT. Compared to the baseline. 3T consistently improves performance, even as the number of examples grows very large. We do not observe any evidence that 3T only improves performance for particular compute budgets. Note that, because the dataset size is 10B samples, all of our runs equate to less than a full epoch.
Retrieval
Few-Shot Classification
Zero-Shot Classification
Average Performancs
45
75
Baseline
50
70
40
65
35
60
40
30
450M
900M
L8B
3.6B
5B
450M
900M
LSB
3.6B
5B
450M
900M
L.B
3.6B
5B
No. Examples Seen
Figure A.4: Increasing training duration of 3T, LiT, and the baseline; average retrieval, few- and zero-shot classification performance. The model scale is B (B/32 for ViTs) for all approaches and towers. 3T and the baseline benefit more from increases in scale than LiT. with 3T maintaining a consistent increase in performance over the baseline. Note that the few-shot performance for LiT is fixed, as only the locked pretrained image tower is used for fewshot applications.
A.10 Benefits From Using 3T With All Three Towers at Test Time - Extended Version
We usually discard the pretrained model when applying 3T to downstream tasks, cf. Fig. 2 (b). Instead, this section, we explore whether we can find benefits from using the locked third tower at test time, similar to LiT. More specifically, we are interested in interpolating between the main image tower and locked pretrained model the third tower. Can we interpolate between 3T- and LiT-like prediction by combining the image embeddings?
This idea does not work directly with the default 3T due to our use of linear projection heads, ef. Fig. 2 (a), since there is no unified embedding space that all towers embed to. Therefore, we introduce a 'headless 3T" variant, for which we do not use the linear projection heads, h hg. fh. and Gr. (Alternatively, one may think of all linear projection heads fixed to identity mappings.) Thus, all losses directly use the same embeddings, f(T). p(T) and g(T), making the embedding spaces directly comparable. Here, we train B scale models for 3.6B examples seen and use an IN-21 model. Further note that the average zero-shot classification performance we report here over only subset of the list of tasks used in $4.2 we consider IN- CIFAR- 100, and Pets. The selection of few-shot classification and retrieval tasks remains the same, although we do not use the Karpathy split for Flickr here."
In Fig. A.5, we display the average retrieval, few-shot classification, and zero-shot classification performance for the convex combination, alongside a comparable LiT run and a 3T run with default projection head setup. Across all tasks, we observe similar behavior: for third tower), we obtain performance close to, but ultimately below, LiT; performance then increases with a, peaking for a E 3/44, before decreasing again. At a = 1 (full weight on main image tower), we recover the performance of the headless 3T setup. Interestingly, for retrieval and few-shot classification tasks, the convex combination yields better performance than either of the towers separately across relatively broad band of a values.
(full weight on the
28

---

Retrieval
Few-Shot Classification
Zero-Shot Classification
Average Porformanse
48
74
76
45
LTT
70
ST Hendiess
72
42
66
Conres
39
68
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Convex Combination a
Figure A.5: Convex combination of the image models in 3T: can generally interpolate between 3T and LiT performance. Interestingly, for a broad range of weights, the retrieval and few-shot classification performance of the combination outperforms 3T and LiT.
By varying a, we
Perhaps counterintuitively, for a = O, we do not recover the performance of LiT exactly, The reasons for this differ between tasks: For retrieval and zero-shot applications, while the image tower is identical to that of LiT, the text tower different as it has been trained with the 3T objective. For few-shot application, the default evaluation procedure of Zhai et al. [85] uses the prelogits of the ViTs underlying ff and as inputs to the few-shot classifier, i.c. not the final embeddings. As the prelogit spaces of f and h are not aligned, here, need to instead construct the convex combination in embedding space, which does however mean that a = 0 does not give performance equivalent to LiT. Lastly, although the 3T run with the default projection heads does not seem to perform better than '3T headless' in this instance, we have seen 'headless' setups underperform in preliminary experiments and would suggest additional experiments before opting for headless design, see also $A.11.
We believe that further study of this approach is exciting future work: the method is entirely post-hoc and no additional training costs are incurred, although inference costs do increase.
A.11 Ablation
In this section, give additional results and details for the ablation study presented in $4.6. Table A.8 gives additional results, extending Table in the main paper. In addition to the mean and two standard errors, we also report standard deviations over tasks here. Note that, for zero classification performance, we only have access to subset of the full list of tasks used in Section 4.2: we consider IN-IK, CIFAR-100, and Pets. The selection of few-shot classification and retrieval tasks remains the same, although we do not use the Karpathy split for Flickr here." *
No C.ey. Details. For this ablation we consider leaving out either of the three loss terms. 'No Gyes"" We replace the 3T loss by _ (Cf,, + Lyn ++,,). 'No Lin+thy": We replace the 3T loss by (Cferg + L.y ).h,)) 'No Lg, ": We replace the 3T loss by (Eferg + fo,,)))) When leaving out either of the three loss terms, average performance suffers significantly. Leaving out the loss between the main two towers (obviously) has the biggest negative effect, as the main embeddings, f(I) and g(T',, are not aligned during training.
Head Variants - Details and Additional Results. In the main part of the paper, we have only given results for the best alternative variant for the projection head setup. Here, describe all variants and report results individually. We refer to Fig. 2 (a) for the projection head notation. 'Heads only on Third Tower': The main tower projection heads fr. and 9r. are fixed to identity mappings. 'Heads Only on Main Towers": The third tower projection heads h and hy are fixed to identity mappings. 'No Heads/Heedless This is the setup described in $A.10: all linear projections hy, Shg ʃú, Ih. fixed to identity mappings. 'Heads Fully Independent': This setup adds linear projection heads before the computation of Cy+tg i.e we compute (I) = [i)(f(()) and (T') Lin(g(T)) and then compute the loss L (instead of C)+zg). In Table A.8, we give results for all variants that we try; none outperform the base variant significantly, while some underperform.
MLP Embedding- Details. When replacing the linear projection h in the third tower with an MLP, use the following architecture: MLP(x) Liny(GELU(Ling (z)), where we use GELU non-linearities [29], Lin, expands the embedding dimensionality of the input by a factor of 4, and Linz maps the shared embedding dimension D.
3T with Loss Weights Details and Additional Results. We replace the standard 3T loss with weighted objective } (Creeg + w (Cfatthy Eggeth,)) )). For the weights w, sweep over
29

---

Table A.8: Extended results for the 3T ablation study. Difference to the 3T reference run for various architecture ablations. We report mean, standard deviation, and two standard errors of the differences over the downstream task selection.
Difference
Mean
Standard Deviation
Two Standard Errors
Rerun
-0.22
0.50
0.25
No Litg
-26.63
21.22
10.61
No titt
-1.19
1.51
0.75
No Cgg
-2.77
1.83
0.91
(Head Variants (best))
0.09
0.70
0.35
Heads Only on Third Tower
0.09
0.70
0.35
Heads Only on Main Towers
-0.67
0.66
0.33
Heads Fully Independent
-0.60
0.63
0.32
No Heads/Headless
-0.47
1.04
0.52
ML.P Embedding
-0.08
0.69
0.35
More Temperatures
-0.26
0.95
0.48
(Loss weight 2 (best))
0.17
1.06
0.53
Loss weight 0.1
-2.11
1.33
0.67
Loss weight 0.5
-0.90
0.81
0.41
Loss weight
0.17
1.06
0.53
Loss weight 10
-0.56
1.74
0.87
(L2 Transfer (best))
-3.80
2.27
1.13
L2 Transfer w=0.0001
-4.40
1.89
0.94
1.2 Transfer w=0.001
-3.80
2.27
1.13
I.2 Transfer w=0.05
-4.41
2.24
1.12
1.2 Transfer w=0.01
-4.17
1.97
0.99
L2 Transfer w=.1
-3,97
2.06
1.03
L2 Transfer w=.5
-7.12
2.95
1.48
L2 Transfer w=1
-11.38
4.39
2.19
L2 Transfer w=2
-16.09
5.14
2.57
L2 Transfer w=10
-46.80
14.32
7.16
3T Finetuning
1.85
2.53
1.27
w E 10.1, 0.5, 2, 10). All weights except w: lead to an average performance decrease. However, the size of the effect for w is small relative to twice the standard error.
L2 Representation Transfer Details and Additional Results. investigate the use of squared losses for the representation transfer between the main towers and the third tower instead of relying on the contrastive loss. Concretely, we replace the loss, Eq. (4), with
L Cs+g +
Σ (1. hy(4)1 (T.) - ho (4))))) 1
(5)
3
N
For the weight hyperparameters w, we sweep over a large set of values, (0.0001, 0.001, 0.05. 0.01, 0.1, 0.5, 2, 10). L.2 representation transfer gives worse results than the contrastive loss for all values of w we try, corroborating the results of Tian et al. [71].
Finetuning Details and Additional Results. Initializing the main tower in 3T with the same JFT- pretrained model as the third tower boosts performance significantly, increasing average performance from 56.76 to 58.61. A rerun confirmed these results; obtained increase from 56.46 58.82. Excited by this, we explored the 3T finetuning setup at other scales, and report performance in Table A.9. Note that here, we increase the numbers of examples seen during training from 450M (S scale) to 900M (B scale) to 5B (L scale). We observe that, as we increase the scale of the experiments, gains from finetuning the main image tower decrease until they are negligible (compared to rerun variance). We therefore have opted to not make finetuning the main tower part of the standard 3T setup, as it (a) complicates the setup and (b) restricts the main tower be the same model architecture and scale the third tower.
30

---

Table A.9: Finetuning for 3T: Initializing the main tower in 3T with the same pretrained model as the third tower improves performance significantly at smaller but not larger experiment scales.
Pretraining
Scale
Avg. Performance 3T
Avg. Performance 3T Finetuned
JFT
56.76
58.61
L
73.97
74.22
IN-21k
44.39
47.61
B
56.30
58.83
73.63
73.83
'FlexiLiT 1/2" Details. With the FlexiLiT variants, cf. Table 5 in the main body of the pa- per, we investigate if there are other, simple ways to improve LiT. For both variants, we create new 'half-locked" image tower by adding learnable components the frozen pretrained image model. For FlexiLiT 1, we add a lightweight learnable 4-layer MLP on top of the frozen back- bone: FlexiLiT-1(7)
MLP(LiT(()). The MLP has layers, uses GIEL.U-nolineearities, and an expansion factor of 4. For FlexiLiT' 2, we add an additional learnable ViT next to the locked backbone (adding sigmificant cost) and merge representations with an MLP: FlexiLiT-2(() MLP((cnnca) (concat(LLi(()) ViT(()). The additional VIT is B/32, following the main locked image tower. The MLP merging the two representations an MLP with the same configuration for FlexiLiT 1.
B Implementation Details
We follow Zhai et al. [85] for optimization and implementation details. We use the open-source vision transformer implementation available from Beyer et al. [4].
Unless otherwise mentioned, we use Transformers of scale L, with a 16x 16 patch size for the ViT image towers, i.c. L/16. We train for 5B examples seen batch size of 14 1024, i.c. for about 350 000 steps. We resize input images to 224 x 224 resolution, and normalize pixel values to the [-1,1] range. Note that for experiments with scale models, we resize images to 288 288 instead. We use learning rate of 0.001, warming up linearly for 10 000 steps, before following cosine decay schedule. We use the Adafactor optimizer [65] with default B, 0.9 and (3 0.99, and we clip gradients if their norm exceeds 1.0. For scale runs, we set A3 = 0.95 by default, which we found to important ensure training stability. We use weight decay of 0.001.
We aggregate embeddings across tokens using multihead attention pooling, i.c. an attention block where the query single learned latent vector, and the keys and values are the outputs of the vision transformer (cf. vit.p" py in the code base [4]).
For details how the different model scales and patch sizes relate to transformer width, depth, MLP dimension, the number of heads, or parameter count, we refer to Table [17] and Table 2 in [84].
Compute Cost. We train our models on v3 and v4 TPUs. For our main experiments at L scale, use 256 TPU chips per experiment. Our 3T runs converge in about three days, for example, the 3T run with JFT pretraining took 63 hours of training time to converge over 348772 training steps. The baseline converges in 54 hours, and LiT in 35. For our five main experiments at I. scale-3T, LiT for JFT and IN-21k pretraining, and a baseline run-the total runtime about 280 hours, about 8 TPU-Chip years worth of compute for the scale experiments of this project. At g scale, we use 512 TPU chips per run, and our 3T runs converge in about 5 days.
Below we mention additional details pertaining only some of the experiments.
Details on Few-Shot Classification. Following Zhai et al. [85], we use the prelogits of the ViTs instead of the final embeddings input the linear few-shot classifier.
Details on Places Experiment. Following Zhai al. [85], for the Places365 experiment, we use B/16 ResFormer [70] as the pretrained model.
31

---

C
Societal Impact
With 3T, we introduce a novel machine learning method for learning joint embeddings of images and text. We train on large datasets of noisy and potentially biased data crawled from the internet. The same general caveats that apply to CLIP/ALIGN and LiT may also apply to 3T. We refer $7 in Radford et al. [58] for general discussion of the societal impact these methods may have.
Additionally, wish highlight the importance of carefully evaluating these models, testing for specific undesired behavior, before applying them in production. While the zero- and few-shot classification capabilities of these models generally impressive, it also important consider their limitations and not succumb to wishful thinking when it comes to the real-world performance of these models on arbitrary tasks. For example, all of the approaches we study here do not perform well for zero-shot prediction on the structured and specialized tasks contained in VTAB, which include, for example, medical applications. performance of these methods when applied to real-world applications. Lastly, because 3T and LiT rely on two datasets for training, classification and a contrastive learning dataset, this can complicate investigations into undesired biases in the final model.
is therefore particularly important to carefully evaluate the
D Libraries & Dataset
We rely on the Jax [5], Flax [26]. and TensorFlow [1] Python libraries for our implementation. Additionally, make use of the Big Vision [4] and Robustness Metries [15] code bases.
For retrieval performance, we evaluate on Microsoft COCO [10] and Flickr30k [56]. For image
classification, we evaluate on IN-1 [40, 61], CIFAR-100 [40], Caltech-256 [23]. Oxford-IIIT Pet
[53], Describable Textures (DTD) [13], UC Merced Land Use [80]. Stanford Cars [39], Cos-
[37]. Birds [73], ImageNet variants -C [28], -A [32], -R [31], -v2 [59], ObjectNet [3], EuroSat [27].
Oxford Flowers- 102 [50], NWPU-RESISC45 [12], and Sun397 [78].
We take the EuroSat, Flowers, RESISC, and Sun397 datasets from the Visual Task Adaptation Benchmark (VTAB) [83]. They are the only VTAB datasets for which least method achieved better than trivial performance.
32


Reviews:
[{"review_id": "5l5vXaur4M", "reviewer": "Reviewer_GeUR", "summary": "This paper focuses on improving LiT. A concern with LiT is that it may be overly reliant on the pretrained model, completely missing out on any potential beneﬁts the image tower might get from contrastive training. The proposed 3T method is formalized for ﬂexible and effective transfer of pretrained classiﬁers into contrastive vision-language models.", "questions": "Please refer to the “Weakness”.", "limitations": "yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed 3T method is formalized for ﬂexible and effective transfer of pretrained classiﬁers into contrastive vision-language models. 3T consistently improves over LiT and a from-scratch baseline for retrieval tasks. 3T also supports a simple post-hoc method to further improve performance by combining 3T- and LiT-like prediction.", "weaknesses": "1. Text-Encoder.\nLiT showed that locked text tower performs badly, and finetuning gives small to negligible gains over training from scratch. But The 3T approach includes pretrained knowledge without suffering from the inﬂexibility of directly using the pretrained model as the main image tower. Maybe applying 3T-like architecture on the text tower can consistently solves the problem on text tower of LiT.\n\n2. Dataset\nIn LiT, CC, YFCC datasets are used to train the whole model, while 3T uses WebLi. Why not uses exactly the same dataset as LiT? In that way I think the comparison would be more convincing.\n\n3. Although 3T removes the reliance on the pretrained image tower, it brings more reliance on the constrastive-learning dataset. If the contrastive-learning is not diverse enough, the chances of encountering “blindspots” in downstream applications wound be higher.\n\n4. Experiments with more pretrained models\nLiT designed a control experiments with different pretrained models, i.e., MoCo, \tDINO, AugReg. I believe this experiment is also necessary for 3T. \n\n5. Computational efficiency\nOne of the biggest adavantages of LiT is the computational efficiency, as it only optimizes the text encoder. In 3T, both image and text towers are trained during training, as well the additional cost of the pretrained image tower.\n\n6. Generalization ability.\nLit has discussed the reason why locked is better than unlocked. The image and text towers will be aligned better if they are tuned together, but it will also make them specialized the dataset used for alignment. This is somewhat opposed to analysis in 3T.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer GeUR,\n\n\nThank you for your hard work and helpful feedback. We will gladly incorporate some of your suggestions and look forward to engaging with you during the discussion period to clarify any remaining points. Please read our comment above, addressed to all reviewers, first.\n\n\n> 1 [...]\n\n\nThanks for this great idea! We have now performed experiments with pre-trained language models. You can find the results in our reply to all reviewers. In short, unlike LiT, 3T does not suffer catastrophically when using pre-trained language models – however, it also does not seem to benefit from them.\n\n\n> 2 [...]\n\n\nFirstly, we would like to highlight that we always compare models that we have trained on the _same_ contrastive learning dataset. This ensures a fair comparison that avoids differences in contrastive learning datasets as a confounder.\n\n\nSecondly, in the LiT paper, Zhai et al. explore two different setups: once the small-scale CC/YFCC setup you mention and once a large-scale dataset of 4B image-text pairs, for which LiT performs better.\nThere are no fundamental differences between the WebLI and large scale LiT data: both are noisy datasets of image-caption pairs. \nWe simply prefer WebLI because it is newer and about 2.5 times larger for its largest split.\n\n\nFurther, we presents results across three subsets of WebLI: the standard dataset, a pair-filtered split, and a text-filtered split (cf. S.4 and Table A.2).\nThe standard WebLI in particular applies very similar (minimal) data filtering as that used in the LiT paper, and therefore, our results on this split should be comparable to those of Zhai et al.\nOur experiments across the three WebLI splits should therefore be fair, representative, and not underestimate LiT performance.\n\n\nHowever, to make sure, we are currently running additional experiments on the _exact same_ data that Zhai et al. use to train LiT. \nIn Figure R.1 in the rebuttal pdf, we provide training curves of the ongoing runs.\nSo far, we observe no significant difference to our WebLI runs.\nWe will provide final numbers across all tasks as soon as possible.\n\n\n> 3 [...]\n\n\nThanks for this suggestion. We will happily extend our discussion on 'Risk in Locking’ to include benefits of locking, too, as soon as we can update the paper.\n\n\nWe would generally expect contrastive learning datasets to have significantly fewer blindspots than classification datasets, as they contain a wide variety of web-scraped data, are much larger, and do not rely on manual annotation.\nIt is true that, compared to LiT (but not the baseline), 3T increases dependence on the contrastive learning data.\nWe believe this is usually a good thing and not a weakness: unlike LiT and the baseline, 3T makes use of two sources of information for learning the image tower.\nTo some extent, this allows 3T to guard against ‘blindspots’ of the pre-trained model _and_ the contrastive learning data.\nE.g., Stanford Cars is a blindspot in IN-21k models, but 3T does not suffer from the same failure as LiT and even manages to improve over the baseline (cf. Table 1 here and in the following).\nConversely, it seems that the Birds task could be considered a blindspot of the contrastive learning data: it requires fine-grained knowledge of bird species, which is heavily present in IN-21k and presumably more so than in our contrastive learning data.\nOn Birds, the baseline performs worse than LiT by 30 %p., but 3T significantly closes the gap to about 18 %p.\n\n\nTherefore, in terms of robustness to blindspots – or just overall robustness, cf. S. 4 – 3T is clearly advantageous to both LiT and the CLIP-style baseline. Unlike 3T, both LiT and the baseline sometimes perform significantly worse than all other approaches, but this never happens for 3T.\n\n\n> 4 [...]\n\n\nThanks for this great suggestion! We have performed experiments with self-supervised DINO models as well as supervised, but ResNet-based BiT models. \nPlease see our reply to all reviewers (‘Additional Image Encoders’) for results and discussion.\nIn short, we find that 3T continues to outperform the baseline and LiT for retrieval tasks, and we are working towards providing results for additional models and tasks for the final submission.\n\n\n> 5 [...]\n\n\nWe agree increased training cost is a weakness of the 3T architecture over LiT.\nHowever:\n(1) We acknowledge increased training costs clearly as a limitation in S. 6;\n(2) As benefits for the increased training costs, 3T obtains better retrieval performance and increased robustness;\n(3) Inference costs are the same for all methods. These can practically be much more important than training costs if a model is trained once and then deployed many times (see e.g. Patterson et al., 2022, https://arxiv.org/abs/2204.05149).\nTherefore, we feel that increased training costs should not hold back acceptance of this paper.\n\n\n> 6 [...]\n\n\nApologies, but we are not quite sure what exactly you are referring to here. It would be great if you could clarify where exactly you think our analysis opposes conclusions of Zhai et al. (2022).\n\n\nAre you saying that Zhai et al. claim that, in CLIP, the image and text towers are better aligned on contrastive data than for LiT, but this does not transfer to better downstream task performance?\nWe do not oppose, but rather fully agree with this: the contrastive training loss is often not indicative of downstream task performance. We also observe this in Appendix A.\nWe agree with Zhai et al. that there are benefits to using pre-trained models in contrastive learning but prefer a more flexible approach, which achieves better retrieval performance (where LiT’s fixed embeddings are perhaps not fine-grained enough) and higher robustness (where LiT’s embeddings may have blindspots).\n\n\nAgain, thank you very much for your review. We hope that our reply has addressed your concerns. Please let us know if there are any further changes you would like to see or if there is anything else that we can clarify.\n"}, {"review_id": "6GD8hkYrFr", "reviewer": "Reviewer_ni71", "summary": "In this paper, a technique called Three Towers (3T) is presented as a means to enhance the contrastive learning of vision-language models. The approach involves integrating pretrained image classifiers to achieve flexible and effective transfer in contrastive vision-language models. The authors provide a formalization of the 3T method, which serves as an extension to the LIT model by introducing an additional trainable Image tower.", "questions": "- Regarding Line 214, the sentence raises a question about the caveat on LiT's few-shot evaluation. It is important to understand why there might be a caveat in this aspect of LiT's performance. According to my understanding, if the authors used the same data split as LIT for a few-shot evaluation, there should not be any caveat. Therefore, I request further elaboration on this matter.\n\n- Grammar correction: Line 91 should be modified as follows: \"then keep them frozen during training\" (remove the second \"them\").", "limitations": "There are not any limitations. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Based on the results shown in Tables 1, 2, and Table A.1.2, Three Towers (3T) demonstrates superior performance compared to both LIT and Baselines in retrieval tasks. Specifically, 3T outperforms LIT and Baselines in 2 out of 3 different experiments related to retrieval, indicating its effectiveness in this task.\n- The paper can be considered as an experimental paper, as it presents a reasonable number of experiments and provides a thorough analysis of the results.\n- While the contribution of the paper may not be groundbreaking, it explores a novel direction in contrastive learning of vision and language, which adds value to the existing research in the field.", "weaknesses": "One weakness of the Three Towers (3T) method is its performance in zero-shot and few-shot evaluations. While 3T outperforms LIT and baselines in the specific scenario of IN-21k pertaining to L scale models, Unfiltered WebLI, it underperforms LIT in other settings as reported in Table A.1 and A.2. This discrepancy is concerning, as it indicates that 3T excels only in 1 out of 4 different zero-shot and few-shot evaluations. Consequently, it is recommended that the authors modify the paper's title to accurately reflect this limitation, emphasizing the superior performance of 3T in retrieval tasks rather than zero- and few-shot tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer ni71,\n\n\nThank you for your hard work and helpful feedback. We look forward to actively engaging with you during the discussion period to clarify any remaining points. Please read our comment above, addressed to all reviewers, first.\n\n\n> One weakness of the Three Towers (3T) method is its performance in zero-shot and few-shot evaluations. \n\nWe agree that 3T’s most positive results are achieved on retrieval tasks.\nHowever, we would like to push back on the claim that 3T is _only_ valuable for retrieval.\n\n\nFirstly, LiT’s few-shot classification performance is identical to the few-shot performance of the pretrained model and is not in any way improved during contrastive training – only methods like 3T can improve the few-shot classification performance of pre-trained image models using contrastive learning.\n\n\nSecondly, 3T outperforms LiT and the CLIP-style baseline when using IN-21k pre-trained models. JFT-pretrained models are not public, and 3T’s superior classification performance using public IN-21k pre-trained models will be relevant to a broad audience.\n\n\nThirdly, even for JFT-pretrained models, LiT performance is sometimes worse than the baseline/3T for specialized tasks such as Eurosat and RESISC (Table A.2).\n(For IN-21k-based models there are more examples: DTD, IN-A, IN-R, ObjectNet, EuroSat, and RESISC, see Table 1.)\nWe believe this is indicative of 3T being more robust than LiT, as its image tower can flexibly benefit from the diverse contrastive learning data _and_ knowledge in the pre-trained embeddings.\nBoth the baseline and LiT sometimes perform much worse than all other methods – this is never the case for 3T.\nOur results in S. 4.4. with Places365 and the ‘mismatched’ setup further show 3T is more robust than LIT and the baseline.\n\n\nFurther, S. 4.5 shows how 3T allows us to interpolate between LiT- and CLIP-style prediction, which boosts zero- and few-shot classification performance, and essentially allows one to to trade off their respective strengths and weaknesses at test time for a particular downstream task.\n\n\nIn summary, we believe that, for classification tasks, 3T has three major selling points: (1) increased performance for IN-21k-pretrained models, (2) increased robustness across all setups, (3) flexibility in interpolating between LiT and 3T predictions.\nWhile we are happy to discuss changes to the paper title, we believe that any new title should account for these benefits of the 3T architecture.\n\n\nLastly, we believe our detailed evaluation and open reporting of the classification performance of 3T – both its strengths and weaknesses – should not count against this submission.\nThe performance of 3T on image classification does not take away from its performance on retrieval tasks: we do not believe it is necessary for a new method to always beat prior work (e.g. LiT) across all possible downstream tasks.\nNote, for example, that LiT itself usually does not improve over CLIP/ALIGN on retrieval tasks.\n\n\n> [...] This discrepancy is concerning, as it indicates that 3T excels only in 1 out of 4 different zero-shot and few-shot evaluations. \n\n\nWe disagree with this summary of 3T’s classification performance.\nOf those 4 evaluations, only one pertains to IN-21k-pretraining; three use JFT-pretraining across different WebLI splits.\nWe do not find large differences between the three WebLI splits in terms of classification performance in Table A.2.\nWe therefore think it is best, and fairer, to summarize 3T classification performance as: (1) We do not find significant differences between models for different WebLi splits; (2) 3T outperforms LiT for IN-21k-pretrained models but not for JFT-pretrained models.\n\n\nTo make sure of this, we are currently running experiments with IN-21k-pretrained models on the pair- and text-filtered WebLi splits, giving us a full set of WebLI splits for both IN-21k and JFT-pretrained models. \nPlease see our reply to all reviewers (‘Additional WebLI Splits for IN-21k Pre-Training (Table R.3)) for these intermediate results. We will provide a full set of classification results as soon these runs are completed.\n\n\nHowever, as discussed above, even if 3T would only improve retrieval performance – noting it actually is significantly helpful in many classification scenarios – we do not believe it is necessary for a new method to always beat prior work across all possible downstream tasks (e.g. LiT does not improve retrieval compared to CLIP).\n\n\n> Regarding Line 214, the sentence raises a question about the caveat on LiT's few-shot evaluation. It is important to understand why there might be a caveat in this aspect of LiT's performance. According to my understanding, if the authors used the same data split as LIT for a few-shot evaluation, there should not be any caveat. \n\nApologies, this is a misunderstanding and we will update the text in the revised version to clarify this point.\n\n\nOf course, we are using the same data split for all models for all downstream evaluations: the results are directly comparable.\n\n\nBy ‘caveat’ we are referring to our discussion of LiT’s few-shot classification performance in l. 159.\nFor few-shot classification evaluations, we train a linear classifier atop the features of the image tower (cf. l. 85).\nFor LiT, these are the fixed embeddings from the pretrained model.\nTherefore, LiT’s few-shot performance does not change during contrastive training and is exactly the same as that of the pre-trained image model (see also Figure A.4).\nThis is what we mean by 'caveat’.\nWe apologize for the lack of clarity and will be explicit about this in the updated version of the draft.\n\n\n> Grammar correction …\n\n\nThanks for spotting this! We will happily update the paper accordingly.\n\n\nAgain, thank you very much for your review. We hope that our reply has addressed your concerns. Please let us know if there are any further changes you would like to see or if there is anything else that we can clarify.\n"}, {"review_id": "1sqIjWUbBB", "reviewer": "Reviewer_Wqsw", "summary": "This paper presents Three Towers (3T), a method to enhance the contrastive learning of vision-language models by incorporating pretrained image classifiers. Unlike the previously proposed Locked-Image Text Tuning (LiT) method, which directly uses frozen pretrained classifier embeddings, 3T introduces a third tower containing the frozen pretrained embeddings and encourages alignment between the third tower and the main image-text towers. The paper shows that 3T consistently outperforms LiT and the CLIP/ALIGN baseline for retrieval tasks, and for ImageNet-21k-pretrained models, 3T also outperforms LiT for few- and zero-shot classification.", "questions": "Have you ever attempted to calculate the loss function using the third tower at different layers? For instance, in a 12-layer model, you could compute the loss with the third tower in the hidden layer of layer 8 while calculating the regular loss in layer 12. This approach might offer valuable insights into the model's performance at various depths.", "limitations": "Addressed", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a straightforward and effective approach to incorporating pretrained image models into contrastive learning of vision-language models.\n2. The paper is well-written and easy to follow, with a clear explanation of the motivation, methodology, and experimental results.\n3. The results demonstrate the effectiveness of the 3T method in improving the performance of vision-language models, particularly for retrieval tasks and few- and zero-shot classification with ImageNet-21k pretraining.\n4. The additional training cost is limited, as the paper said \"as frozen embeddings from the third tower can be pre-computed and then stored with the dataset\"\n5. The experiments in this paper are very solid and present extensive experiments on various tasks, pretraining datasets, and model sizes.", "weaknesses": "1 The reviewers perceive a resemblance between this work and SLIP[1], as both approaches combine CLIP with existing visual pretraining methods. While 3T employs a direct strategy using pretrained encoders, SLIP opts for a joint training approach. The reviewers recommend including a discussion and comparison between this paper and SLIP to provide a more comprehensive understanding. \n\n2. This paper has less exploration of pretrained classifier embeddings and does not explore different pre-trained models as the third tower.\nExperiments similar to SLIP[1] can be taken to explore the adaptation of visual coders with different pre-training methods to the CLIP framework.\n\n3. The overall engineering style of the paper is heavy, lacking some insightful analysis and limited innovation.\n\n[1] Mu, Norman, et al. \"Slip: Self-supervision meets language-image pre-training.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Wqsw,\n\n\nThank you for your hard work and helpful feedback. We will very gladly incorporate some of your excellent suggestions and look forward to engaging with you during the discussion period to clarify any remaining points. Please read our comment above, addressed to all reviewers, first.\n\n\n>The reviewers perceive a resemblance between this work and SLIP[1], as both approaches combine CLIP with existing visual pretraining methods. While 3T employs a direct strategy using pretrained encoders, SLIP opts for a joint training approach. The reviewers recommend including a discussion and comparison between this paper and SLIP to provide a more comprehensive understanding.\n\n\nThanks for suggesting this! We were not aware of SLIP and agree it is related work. We have included a citation and discussion of SLIP in our updated version of the draft – note, we cannot currently update the submission on OpenReview. Below we give a detailed discussion of the relation between SLIP and 3T, which we believe is mostly _complementary_.\n\n\nWith 3T, we study the benefits of incorporating _pre-trained image models_ into CLIP/ALIGN-style contrastive learning.\nThe SLIP paper instead studies different ways of incorporating different _image-only self-supervision objectives_ into CLIP/ALIGN-style contrastive learning.\nWith 3T, we assume there is a pre-trained model available.\nSLIP cannot include pre-trained models (except perhaps for initializing their encoders, i.e. the ‘fi’/’ft’ in Algorithm 1 of the SLIP paper).\nTherefore, SLIP is orthogonal to 3T: future work could _combine_ SLIP with 3T by replacing the loss between the main image and text tower in 3T with a SLIP loss, such that the main image tower benefits from the pre-trained model and SLIP objective.\nWe note that, in addition to SLIP, there are many other approaches that improve CLIP (see related work) and that could benefit from pretrained models by combining them with a 3T-style approach.\n\n\n> This paper has less exploration of pretrained classifier embeddings and does not explore different pre-trained models as the third tower. Experiments similar to SLIP[1] can be taken to explore the adaptation of visual coders with different pre-training methods to the CLIP framework.\n\n\nThanks for this great suggestion! We have performed experiments with self-supervised DINO models as well as supervised, but ResNet-based BiT models. \nPlease see our reply to all reviewers (‘Additional Image Encoders’) for the results and discussion.\nIn short, we find that 3T continues to outperform the baseline and LiT for retrieval tasks, and we are working towards providing results for additional models and tasks for the final submission.\n\n\n> The overall engineering style of the paper is heavy [...] and limited innovation.\n\n\nWe agree that our proposal of the 3T architecture is largely straight-forward, although we do not think this is a bad thing at all: 3T is easy to implement and provides solid and consistent performance improvements.\nSee, e.g., reviewer Y95u, who highlights this as a strength of our approach.\nNote the reviewer guidelines explicitly mention that even a ‘novel combination of well-known techniques’ can be ‘valuable’, although we believe our paper provides more than that.\n\n\n> lacking some insightful analysis\n\n\nCould you please clarify what you mean by ‘lacking some insightful analysis’? \nWe believe our paper provides extensive empirical analysis of the 3T architecture, evaluating performance across many different downstream tasks, pre-trained models, and contrastive learning datasets (S. 4.1, S. 4.2, S. A.2.3).\nFurther, we explore performance at multiple model scales (S. 4.3), the robustness of our method (S. 4.4), propose a novel interpolation approach between CLIP- and LiT-style prediction (S. 4.5, S. A.4), provide a thorough ablation (S. 4.6, A. 5), study performance at different training budgets (S. A.3), evaluate and interpret training dynamics (S. A.1), and study probabilistic robustness and out-of-distribution detection (S. A.2).\nAll of these empirical results and ablations add valuable insight about the inner workings of the 3T method.\n\n\n> Have you ever attempted to calculate the loss function using the third tower at different layers? For instance, in a 12-layer model, you could compute the loss with the third tower in the hidden layer of layer 8 while calculating the regular loss in layer 12. This approach might offer valuable insights into the model's performance at various depths.\n\n\nThanks for the suggestion! We have not tried this but agree that future work could consider this. It seems you might be suggesting that deeper layers implement more fine-grained features and that aligning to those might perhaps not be desirable. Are you aware of work that studies ViT losses/representations at different layer depths?\n\n\nMore generally, our ablation study in S. 4.6 (and A.5) already investigates many different modifications to the 3T architecture, such as introducing weight hyperparameters or additional softmax temperatures.\nWe find these do not give significant gains, and therefore favor the simplest version of 3T that works well.\nComputing losses between different layers would likely require that both image towers in 3T are comparable architectures, which reduces the flexibility of the method.\n\n\nAgain, thank you very much for your review. We hope that our reply has addressed your concerns. Please let us know if there are any further changes you would like to see or if there is anything else that we can clarify.\n"}, {"review_id": "nDudy2DWn4", "reviewer": "Reviewer_Y95u", "summary": "CLIP is trained using contrastive loss. Previous works have shown that freezing the image tower, and only training the text encoder on pretraining datasets yields better performance compared to training both the encoders. This paper proposes that completely freezing the image encoder is drastic and will underperform when the downstream data is not covered in pretraining data span for the image tower. \n\nHence they propose training both the image and text encoder from scratch (as in standard training), albeit with an additional contrastive loss with the pretrained representations for the image encoder. They show that this approach outperforms the baseline standard CLIP training as well as LiT.\n\n However, it is not clear if the gains of the proposed approach will be seen as the pretraining image-caption corpus data size is increased, especially to the point at which the baseline has a similar zeroshot accuracy as JFT/Im21k pretrained representations.\n", "questions": "See the cons section.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Pros:\n- The empirical gains are good, especially in retrieval tasks in the scales and settings considered.\n- The idea is simple to implement and well described. \n- Proper ablations have been done. \n", "weaknesses": "Cons:\n- The novelty is limited. It is a well-established fact that distilling knowledge from pretrained models improves performance. \n- LiT had the additional advantage of being cost-effective, whereas this will not be true for the proposed approach 3T. However, it is not a major concern or the goal of this work and will not be part of my evaluation. Probably, it would be a good idea to tone down the connection to LiT in the paper and not make it feel like a build-up over LiT or else make it clear early on that we do not want to be cost effective. \n- Section 4.3 Scaling pretraining data: It would be much better to consider showing gains as the pretraining image-caption corpus is scaled i.e. LAION400M to 5B to 12B or so based on your compute budgets. More importantly, as the image-caption corpus is scaled, intuitively it seems that there will be diminishing returns from using the 3rd tower of pretrained image representations from say Im21k or JFT. Can authors maybe do some mini experiments where training samples are scaled from say 2M to 64M exponentially (5-6 points) or whatever scale is feasible for them (I believe this is quite small a scale given other experiments in the paper). \n- As the pretraining image-caption scales are increased to 5B and 12B(https://www.datacomp.ai/leaderboard.html), the zeroshot accuracies have approached near 80% similar to that of JFT (81%, which is a proprietary dataset moreover) or higher than that of Im21k (79%). Would using 3T based training with the JFT/Im21k pretrained image (3rd tower) tower help here? If not, then does it suggest that gains when training on WebLI was just because of additional data seen by JFT or Im21k, the advantage which doesn’t hold as these image-caption corpuses are being scaled. \n- However, at the same, it would be interesting if you can show that 3T can help improving the zeroshot accuracies of a smaller CLIP model if the representations from the bigger CLIP model are used. This is because such an observation might hold true even at large image-caption corpus scales.\n- Beyond the fact that distillation, in general, helps improve the quality of learned representations, do authors have any other intuition behind why 3T is better than the baseline (not LiT, that is clear).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Y95u,\n\n\nThank you for your hard work and helpful feedback. We look forward to actively engaging with you during the discussion period to clarify any remaining points. Please read our comment above, addressed to all reviewers, first.\n\n\n> Section 4.3 Scaling pretraining data [...]\n\n\nThanks for raising this point! Our experiments in Appendix 3 may address your concerns already. In A.3, we increase the training set size from 450M to 5B samples.\nOur results indicate that gains from 3T over the CLIP/ALIGN baseline are constant across dataset sizes, and that 3T benefits more from larger datasets than LiT.\n\n\nWe agree that it _could_ be possible, that at very large contrastive learning dataset sizes, the benefits from 3T over the baseline _could_ grow smaller as contrastive learning outweighs benefits from the pretrained model.\nHowever, we have not observed this, despite training on _very large_ datasets already. \nWe also observe benefits for 3T when the pre-trained model is weak (see both experiments in S. 4.4), which is another setting in which one would perhaps expect 3T to not gain much over the CLIP baseline.\n\n\nAgain, thanks for bringing this up and we will include this discussion in the paper, as soon as we are able to update it.\n\n\n> As the pretraining image-caption scales are increased [...]\n\n\nOur results indicate that 3T will still improve performance here.\nIn Table A.2, we show performance for g scale models trained on the Pair-Filtered WebLi split.\nFor the baseline, we obtain 78.0% zero-shot on IN-1k (our best baseline result and close to the JFT performance you cite).\nDespite this, 3T still improves IN-1k performance over the baseline with 79.6% accuracy.\n\n\nAnother example are tasks such as Cars, DTD, IN-A, IN-R, ObjectNet, or EuroSat in Table 3: despite _the baseline outperforming LiT_, 3T still benefits from the pre-trained model, and _3T outperforms both the baseline and LiT_.\nSee also our robustness results (S. 4.4 and discussion below), where we also observe 3T performing best, despite the baseline outperforming LiT.\n\n\nWe are therefore confident that, even when the baseline is as good or better than the pre-trained model, 3T can improve performance over both.\n\n\n> However, at the same, it would be interesting if you can show [...]\n\n\nThanks for this suggestion! We agree that distilling larger CLIP models into smaller CLIP models is interesting future work that deserves careful attention on its own.\n\n\nOur study of the ‘mismatched setup’ in S. 4.4 might get closest to answering the question of whether 3T will continue to be beneficial when contrastive learning dataset size increases further.\nInstead of increasing the contrastive learning dataset scale, we here _decrease_ the capabilities of the pre-trained model instead, by using a smaller ViT for the pre-trained model than for the main image tower.\nPredictably, we now observe that the baseline often outperforms LiT (cf. Table A.3).\nHowever, despite this, 3T continues to provide benefits over the baseline (and LiT).\n\n\nTogether with our dataset scaling results (S. A.3, see discussion above), we are therefore confident that 3T would continue to provide benefits over both the baseline and LiT at larger dataset sizes.\n\n\n> The novelty is limited [...]\n\n\nFirstly, we agree the idea behind 3T is intuitive.\nHowever, our ablation (S. 4.6, A. 5) shows that actually obtaining benefits from distillation is not as simple: naive approaches, e.g. standard L2 losses, actually do not perform well, and we require _contrastive_ distillation objectives to reap benefits.\n\n\nSecondly, we further think it is an advantage that the final 3T architecture is straight-forward, not a disadvantage: 3T is easy to implement and provides consistent and convincing performance improvements.\nThe NeurIPS reviewer guidelines explicitly mention that a ‘novel combination of well-known techniques’ can be ‘valuable’.\nThis is at least what we provide by combining contrastive learning jointly with distillation, providing value by improving the performance of CLIP-style contrastive learning.\nOur results are not found in the existing literature: the primary competing prior work, LiT, uses pretrained models in a totally different fashion.\n\n\n> Beyond the fact that distillation [...]\n\n\nWe suspect that distillation is the main mechanism behind why 3T improves performance over the baseline, and would be curious to hear if you have any reason to think otherwise.\nUnlike 3T, the baseline does not include knowledge from a pre-trained model.\nIf that prior knowledge is incorporated appropriately, 3T should perform at least as good as the baseline – which is exactly what we usually observe.\nWe find the following example from Table 3 in S. 4.2 illustrative:\nIN-21k contains a large variety of labels of bird species, and therefore, the IN-21k-pretrained model is much more knowledgeable about birds than the baseline.\nWith 3T, we can incorporate this knowledge from the pretrained model about birds, and significantly outperform the baseline model on the birds task.\n\n\nAs discussed, sometimes 3T improves over the baseline despite LiT performing worse than the baseline.\nOn one hand, this can still be explained by distillation: regardless of performing worse, the pretrained model may contain information about the task that the baseline does not.\nOn the other hand, there could also be more indirect benefits from aligning the image embedding space to the pre-trained classifier: there might be a general mismatch between the contrastive learning objective and downstream tasks that is ‘corrected’ by also aligning the towers to the classification embeddings.\n\n\nIf you think this discussion is helpful, we would be more than happy to add it to the revised version of the paper.\n\n\nAgain, thank you very much for your review. We hope that our reply has addressed your concerns. Please let us know if there are any further changes you would like to see or if there is anything else that we can clarify.\n"}]

Author rebuttal:
We thank all reviewers for their careful consideration, insightful comments, and helpful suggestions. We are glad to see the paper was generally well received and hope that our responses alleviate the concerns that were raised. We believe your input has already helped improve the paper and look forward to engaging with you further during the discussion period.


We were glad to see all reviewers recognize the empirical improvements of 3T, highlighting that ‘3T consistently improves over LiT and a from-scratch baseline for retrieval tasks’ (GeUR), ‘demonstrates superior performance compared to both LIT and Baselines in retrieval tasks’ (ni71), ‘the results demonstrate the effectiveness of the 3T method in improving the performance of vision-language models, particularly for retrieval tasks and few- and zero-shot classification with ImageNet-21k pretraining' (Wqsw), and 'empirical gains are good, especially in retrieval tasks’ (Y95u).


Further, we were glad to see you felt that 3T 'explores a novel direction [...] which adds value to the existing research in the field’ (ni71), that our presentation is 'excellent ‘ (ni71, Y95u), that the paper is 'well-written and easy to follow, with a clear explanation of the motivation, methodology, and experimental results’ (Wqsq), and that our idea is 'well described’ (Y95u). Lastly, we were glad to see you recognize our empirical evaluation as ‘thorough’ (ni71), 'very solid’/'extensive’ (Wqsw), highlighting that we perform 'proper ablations’ (Y95u).


Following your excellent feedback, we have added a number of new experiments, which are attached to this message as a PDF. Note that, as per NeurIPS guidelines, we cannot update the paper directly during the rebuttal. We will include these results in the final submission. However, many of our experiments require a large number of accelerators and multiple days of compute which makes it difficult to finish them within the time constraints of the rebuttal period.
We therefore sometimes need to present intermediate results for ongoing experiments.
Note, this early comparison generally benefits LiT as the benefits of a locked image tower are strongest early on in training, cf. Figures A.1 (c) and A.4.
For the baseline, the gap to 3T is usually consistent throughout training.
Therefore, our intermediate results here should be representative for comparing 3T and the baseline and conservative for comparing 3T and LiT.
We will provide a final and full set of results as soon as possible.


Concretely, we study the following new experiments:


* **Additional Image Encoders** (Table R.1): We study additional pre-trained image encoders: a ResNet-based BiT model trained on IN-21k (Kolesnikov et al., ECCV 2020, arXiv: 1912.11370) and a self-supervised encoder based on DINO as in Zhai et al. (2022). We find that 3T continues to outperform the baseline and LiT for retrieval tasks for both encoders. We are currently working on results for additional image encoders, such as MoCo-v3 and AugReg as in Zhai et al. (2022), as well as a full set of classification evaluations.

* **Pre-Trained Language Models** (Table R.2): We explore if there are benefits to using pre-trained _language models_ instead of image encoders with 3T. For LiT, we confirm the results of Zhai et al. (2022) that performance suffers drastically when using a locked pre-trained language encoder as the text tower. In contrast, for 3T, we do not observe a drastic decrease in performance, once more showing that 3T is more robust to deficiencies in the pre-trained model than LiT. However, we also do not observe gains from using a pre-trained language model with 3T, justifying our choice of focusing on pre-trained image encoders in the original submission. Nevertheless, we think it is plausible that future work may yet find benefits of incorporating knowledge from pretrained language models in other settings or downstream tasks, e.g. tasks that require more complex language reasoning abilities.

* **Additional WebLI Splits for IN-21k Pre-Training** (Table R.3): We study IN-21k pre-trained models at L scale for the text-filtered and pair-filtered splits of the WebLI dataset (see S. 4). We find that 3T continues to outperform the baseline and LiT for retrieval tasks on these additional WebLi subsets.


* **Results on the Original LiT Dataset** (Figure R.1): We have started experiments for the baseline, LiT, and 3T on the exact same dataset Zhai et al. (2022) use to train LiT. While runs are still ongoing, so far, we observe no significant differences in behavior to our previous experiments on splits of the WebLI dataset.


We address your specific comments in the individual replies below.
We look forward to engaging with you during the discussion period to clear up any remaining concerns and allow us to improve the paper further.


Meta-review: This paper introduces an innovative technique known as Three Towers (3T) aimed at enhancing the contrastive learning of vision-language models by integrating pretrained image classifiers. In contrast to the LiT method, the 3T approach acts as an extension to the LIT model, introducing an additional trainable Image tower to promote alignment between the third tower and the primary image-text towers. The results demonstrate that 3T has led to notable improvements in retrieval tasks and few-/zero-shot classification.

Reviewers have praised the simplicity and clarity of the proposed 3T method, which is well-documented and supported by robust experimental evidence. The authors' responses to reviewers' concerns, including the exploration of additional image encoders, elucidating how 3T enhances predictions over LiT and the baseline, its applicability to Pre-Trained Language Models, essential comparisons with SLIP, considerations of pretraining scale, and detailed clarifications regarding fair comparisons with LiT and performance in zero-/few-shot evaluation, have all been meticulously addressed.

Given these substantial enhancements, all reviewers have expressed scores surpassing the acceptance threshold, and AC concurs with their recommendation to accept this paper. AC strongly encourages the authors to integrate these valuable insights, extensive experiments and constructive feedback from the reviewers into the final version of the manuscript.
Decision: Accept (poster)