SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The above results support our hypothesis that image embeddings trained on the highly diverse contrastive learning dataset will be more broadly applicable. Strikingly, even when using the same IN-21k model, 3T almost always improves over the baseline and never suffers the same failures as LiT. However, it seems that JFT-pretrained models can fix many of LiT's shortcomings for image classification. JFT-LiT performs remarkably well and almost never underperforms significantly compared to the baseline. A deviation from this pattern are the results for Eurosat and RESISC at g scale for JFT pretraining in Table A.6, where, e.g. on the Text-Filtered WebLI split, LiT lacks behind 3T by  $12\%$  p and  $8\%$  p.
Table 3: For IN-21k pretraining, 3T has the best average classification accuracy  $(\uparrow)$  (L scale models, Unfiltered WebLI).
Method	Basel.	LiT	3T
Few-Shot Classification	IN-1k	62.8	79.0	68.0
CIFAR-100	70.4	83.6	72.5
Caltech	91.0	88.4	92.3
Pets	85.9	89.2	86.5
DTD	70.3	69.2	73.3
UC Merced	91.8	92.8	94.0
Cars	81.5	41.9	84.9
Col-Hist	71.7	86.4	76.6
Birds	53.4	83.4	65.0
Zero-Shot Classification	IN-1k	69.5	76.0	71.7
CIFAR-100	73.5	82.9	73.4
Caltech	81.9	82.4	84.1
Pets	84.2	87.1	87.0
DTD	58.6	51.8	60.3
IN-C	49.6	62.0	51.8
IN-A	53.0	45.6	54.3
IN-R	85.8	66.1	88.1
IN-v2	62.2	67.2	64.9
ObjectNet	56.2	41.9	58.3
EuroSat	32.7	27.6	42.8
Flowers	62.0	72.6	65.7
RESISC	58.0	29.0	57.9
Sun397	67.6	65.4	68.7
Average	68.4	68.3	71.4
4.3 Scaling Model Sizes and Training Duration
Since the publication of LiT, the scale of contrastive learning datasets, both public and proprietary, has increased, for example with the release of LAION-5B [64] or WebLI-10B [11]; we use the latter here. Given their cheap collection costs, it seems likely that this growth will continue to outpace that of more expensive classification datasets. However, locking the image tower ignores any potential benefits from the increased contrastive learning data for the image tower. Additionally, larger datasets often lead to increases in model scales to fully make use of the additional information [84]; unlike LiT, 3T can increase the scale of the main image tower independently of the pretrained model, cf. §4.4.
Here, we separately study the effects that model scale and dataset size have on 3T, LiT, and the baseline. First, we vary the scale of all involved towers, including the pretrained model. We study both IN-21k and JFT pretraining, always train contrastively for 5B examples seen, and the S, B, L, and g scales correspond to S/32, B/32, L/16, and g/14 for the ViT models. We compute averages separately across retrieval, few-shot, and zero-shot classification tasks, where the tasks are those from Tables 2 and 3.
In Fig. 3, we observe that 3T's lead over LiT and the baseline in retrieval performance holds across scales and pretraining datasets. Further, across all tasks and scales, 3T maintains a consistent performance gain over the baseline. Also across tasks and pretraining datasets, we observe that
Figure 3: We increase the model scale for 3T, LiT, and the baseline and report average retrieval, few- and zero-shot classification performance for both IN-21k and JFT pretraining. 3T and the baseline benefit more from increases in scale than LiT (their curves are steeper), with 3T performing better than the baseline. The baseline does not use a pretrained model and is displayed twice (at scale B and L).
Model Scale
scaling is more beneficial for 3T than for LiT: as we increase the scale, 3T's performance increases more than that of LiT. This means that 3T either extends its lead over, overtakes outright, or reduces its gap to LiT as we increase scale. At L scale with IN-21k pretraining, 3T gives the best average performance of all methods across tasks. For JFT pretraining, LiT maintains an edge for classification performance, although the scaling behavior suggests this gap may fully collapse at larger scales. We observe similar trends when scaling the number of examples seen during training, cf. Fig. A.4.
4.4 Pretraining Robustness
Next, we study what happens when 3T and LiT are used with pretrained models that do not conform to expectations. We consider two setups: one that we call 'mismatched' and one that considers pretraining on the Places365 [87] dataset. In Table 4, we display zero-shot accuracies on Pets, IN-1k, and CIFAR-100—tasks for which LiT usually performs
Table 4: 3T is more robust to the pretraining setup than LiT. Zero-shot classification accuracies  $(\uparrow)$ , full details in main text.
Setup Method	Mismatched	Places365
Basel.	LiT	3T	Basel.	LiT	3T
IN-1k	69.5	69.5	71.5	45.6	24.5	47.4
CIFAR-100	73.5	78.6	75.6	48.3	27.4	52.4
Pets	84.2	84.7	87.4	61.5	30.3	60.2
...	...	...
Full Average	66.4	61.7	69.8	47.5	29.4	49.3
best—as well as the average performance over the full set of tasks, see Table A.7 for individual results.
Mismatched Setup. So far, we have always matched the scale of the pretrained model to the scale of the models trained contrastively. For the mismatched setup, we now break this symmetry: we use an IN-21k-pretrained B/32 scale image model (3T and LiT) with an L scale text tower (all approaches) and an L/16 unlocked image tower (3T and baseline). This setup is relevant when pretrained models are not available at the desired scale: for example, given ever larger contrastive learning datasets one may want to train larger image models than are available from supervised training, cf. §4.3. Of course, increasing the image tower scale also comes at increased compute costs for 3T and the baseline. We observe that LiT suffers from this mismatched setup much more than 3T, which is not restricted by the smaller pretrained model and now achieves higher accuracy than LiT on Pets and IN-1k.
Places365. Zhai et al. [85] demonstrate that LiT performs badly when used with models pretrained on Places365. In Table 4, we reproduce this result and observe LiT performing much worse than the baseline. (We here train B scale models for 900M examples seen, and based on our discussion in §4.3, would expect LiT to perform even worse, in comparison to the baseline and 3T, when training longer or with larger scale models.) The embeddings obtained from Places365 pretraining do not allow LiT to perform well on our set of downstream tasks. 3T behaves much more robustly and does not suffer from any performance collapse because it incorporates both the pretrained model and contrastive data when training the image tower. Notably, 3T manages to improve average performance over the baseline even for Places365 pretraining. We further suspect the linear projection heads afford 3T some flexibility in aligning to the pretrained model without restricting the generality of the embeddings learned in the main two towers.
Figure 4: Convex combination of the image models in 3T:  $\alpha \cdot h(I) + (1 - \alpha) \cdot f(I)$ . By varying  $\alpha$ , we can generally interpolate between 3T and LiT performance. Interestingly, for a broad range of weights, the retrieval and few-shot classification performance of the combination outperforms 3T and LiT.
4.5 Benefits From Using 3T With All Three Towers at Test Time
We usually discard the pretrained model when applying 3T to downstream tasks, cf. Fig. 2 (b). In this section, we instead explore if we can find benefits from using the locked third tower at test time, similar to LiT. More specifically, we study the convex combination of the main image tower and locked pretrained model in the third tower,  $\alpha \cdot h(I) + (1 - \alpha)\cdot f(I)$ , to see if we can interpolate between 3T- and LiT-like prediction at  $\alpha = 0$  and  $\alpha = 1$  respectively. We train 3T without linear projection heads to make embeddings from all towers compatible. Additional details of this setup can be found in §A.10.
Fig. 4 shows we can generally interpolate between LiT- and 3T-like performance as we vary  $\alpha$  from 0 to 1. Note that we do not always recover LiT or 3T performance at  $\alpha \in \{0,1\}$  as explained in §A.10. Interestingly, for retrieval and few-shot classification tasks, the convex combination yields better performance than either of the underlying methods for a relatively broad region around  $\alpha \approx 0.5$ . We believe that further study of the convex combination could be exciting future work: the method is entirely post-hoc and no additional training costs are incurred, although inference costs do increase.
Next, we provide ablations for some of the design decisions of 3T as well as insights into LiT training. We perform the ablation study at B scale with patch size 32, training for 900M examples seen, and use JFT-pretrained models. In Table 5, we report the average difference in performance to our default runs across all tasks, together with two standard errors computed over the downstream tasks as an indication of statistical significance. We refer to §A.11 for full details and results.
3T Ablations. 'Rerun': To study per-run variance, we perform a rerun of the base 3T model, obtaining an average performance difference of  $-0.22\%$  across tasks. 'No  $\mathcal{L}_{\leftrightarrow}$ ': When leaving out either of the three loss terms, average performance suffers significantly. 'Head Variants': We try a selection of different projection head variants, see §A.11. None give significantly better performance than our default setup. 'MLP Embedding': Replacing the linear projection  $h$  in the third tower with an MLP does not improve performance. 'More Temperatures': Using three learned temperatures, one per 3T loss term, instead of a global temperature as in Eq. (4), does not improve results. 'Loss weights': Replacing
the loss with a weighted objective,  $\frac{1}{3} \cdot (\mathcal{L}_{f \leftrightarrow g} + w \cdot (\mathcal{L}_{f_h \leftrightarrow h_f} + \mathcal{L}_{g_h \leftrightarrow h_g}))$ , does not improve performance significantly across a variety of choices for  $w$ . 'L2 Transfer': Using squared losses for the representation transfer objectives  $\mathcal{L}_{f_h \leftrightarrow h_f}$  and  $\mathcal{L}_{g_h \leftrightarrow h_g}$ , cf. [60], results in significantly worse performance, even when optimizing the weight of the transfer terms. '3T Finetuning': Initializing the main tower in 3T with the same JFT-pretrained model as the third tower increases performance significantly; however, we find this effect becomes negligible for larger scale experiments, cf. §A.11.
Table 5: Ablation study, see text for details.
Difference to 3T
Rerun	-0.22 ± 0.25
No Lf←gLoss	-26.63 ± 10.61
No Lfh←hfLoss	-1.19 ± 0.75
No Lgh←hgLoss	-2.77 ± 0.91
Head Variants	0.09 ± 0.35
MLP Embedding	-0.08 ± 0.35
More Temperatures	-0.26 ± 0.48
Loss Weights	0.17 ± 0.53
L2 Transfer	-3.80 ± 1.13
3T Finetuning	1.85 ± 1.27
Difference to LiT
Rerun	-0.10 ± 0.22
LiT Finetune	-14.99 ± 6.09
FlexiLiT1	-4.63 ± 1.36
FlexiLiT2	-5.04 ± 1.54
LiT Ablations. 'Rerun': We observe similar between-run variance for LiT. 'LiT Finetune': We confirm the result of Zhai et al. [85] that finetuning from a pretrained model results in worse performance than locking. 'FlexiLiT 1/2': We investigate simple ways of modifying LiT such that it can adjust the image tower during contrastive learning, see §A.11, but find these are not successful.
Additional Experiments. In §A.1, we study the optimization behavior of 3T, finding evidence for beneficial knowledge transfer from the pretrained model. In §A.2, we study the calibration of all methods for zero-shot classification, as well as their performance for out-of-distribution (OOD) detection: 3T is generally better calibrated than LiT, and for OOD tasks, we find trends similar to §4.3, with all methods generally performing well. In §A.3, we confirm the results of Zhai et al. [85] that there are no benefits from using pretrained language models. In §A.4, a detailed investigation of predictions suggests that 3T performs well because it combines knowledge from contrastive learning and the pretrained model. Lastly, §A.5 shows 3T continues to perform well with other pretrained image models.
CLIP [58] and ALIGN [34] are examples of vision-language models that have received significant attention, e.g. for their impressive ImageNet zero-shot results. Concurrently with LiT [85], BASIC [55] investigates locking and finetuning from JFT-pretrained models. Previously, [46, 57, 74, 67] have explored representation learning from images with natural language descriptions before the deep learning era. Subsequently, [21, 36, 35, 19, 63, 8] explore image-text understanding with CNNs or Transformers. In this context, Li et al. [42] introduced the idea of zero-shot transfer to novel classification tasks. The loss objective, Eq. (1), was proposed by Sohn [66] for image representation learning, and also appears in [77, 51, 9]. Zhang et al. [86] then used the objective to align images and captions, although their setting used medical data. Lots of work has built on CLIP and ALIGN. For example, [82, 79] have augmented the objective to optionally allow for labels, [89, 88] proposed methods for improving zero-shot prompts, [2, 43] applied CLIP to video, [54, 49, 62, 33] used CLIP embeddings to improve generative modelling, and [47] study different ways of incorporating image-only self-supervision objectives into CLIP-style contrastive learning. Relatedly, vast amounts of work have explored self-supervised or contrastive representation learning of images only, e.g. [16, 25, 24, 6]. Transfer learning [52] applies embeddings from large-scale (weakly) labelled datasets to downstream task [68, 44, 38].
6 Limitations, Impact, and Conclusions
Limitations. While 3T consistently improves over LiT for retrieval tasks, for classification, 3T outperforms LiT with ImageNet-21k-pretrained models only at large scales, and may require even larger scales for JFT pretraining. Further, while inference costs are equal for all methods, 3T incurs additional training costs compared to LiT. We have compared methods at matching inference cost for simplicity because there are many ways to account for the cost of pretraining and embedding computation.
Impact. We believe that locking is a suboptimal way to incorporate pretrained image models, and we have demonstrated clear benefits from exposing the image tower to both the contrastive learning dataset and the pretrained model, particularly as scale increases. 3T is a simple and effective method to incorporate pretrained models into contrastive learning and should be considered by future research and applications whenever strong pretrained models are available. For future work that seeks to improve 3T, we consider it important to understand the differences between embeddings from 3T, LiT, and the baseline. If we can obtain insights into why they excel at different tasks, we can perhaps (learn to) combine them for further performance improvements. Our convex combination experiments are a starting point; it would be interesting to continue this direction, possibly looking at combinations in parameter space [75, 76]. Lastly, future work could explore 3T for distillation of large pretrained models into smaller models, extend 3T to multiple pretrained models, potentially from diverse modalities, or explore the benefits of 3T-like ideas for other approaches such as CoCa [81].
Conclusions. We have introduced the Three Tower (3T) method, a straight-forward and effective approach for incorporating pretrained image models into the contrastive learning of vision-language models. Unlike the previously proposed LiT, which directly uses a frozen pretrained model, 3T allows the image tower to benefit from both contrastive training and embeddings from the pretrained model. Empirically, 3T outperforms both LiT and the CLIP/ALIGN baseline for retrieval tasks. In contrast to LiT, 3T consistently improves over the baseline across all tasks. Further, for ImageNet-21k-pretrained models, 3T also outperforms LiT for few- and zero-shot classification. We believe that the robustness and simplicity of 3T makes it attractive to practitioners and an exciting object of further research.
[87] Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A. Places: A 10 million image database for scene recognition. Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2017. URL https://ieeexplore.ieee.org/document/7968387.
[88] Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional prompt learning for vision-language models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. URL https://arxiv.org/abs/2203.05557.
[89] Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV), 2022. URL https://arxiv.org/abs/2109.01134.
A Additional Experiments & Results
A.1 Training Dynamics
Figure A.1: Training dynamics: (a) The transfer losses,  $\mathcal{L}_{f_h\leftrightarrow h_f}$  and  $\mathcal{L}_{g_h\leftrightarrow h_g}$ , improve the image-text loss,  $\mathcal{L}_{f\leftrightarrow g}$ , in 3T relative to the baseline. (b) Difference between matching loss terms for 3T, LiT, and the baseline. 3T obtains better image-to-text loss than the baseline and similar locked-image-to-text loss as LiT. (c) While the loss advantage of 3T over the baseline shrinks during training, this does not happen for downstream applications; we display image-to-text retrieval on COCO as an example. Moving averages applied to (a-b) for legibility.
In Fig. A.1, we compare the 3T training losses to LiT and the from-scratch baseline, using the familiar L scale setup with JFT pretraining. For 3T, we display all loss terms separately: the image-text loss  $\mathcal{L}_{f\leftrightarrow g}$ , the image-to-third-tower loss  $\mathcal{L}_{f_h\leftrightarrow h_f}$ , and the text-to-third-tower loss  $\mathcal{L}_{g_h\leftrightarrow h_g}$ , cf. Eq. (4) and Fig. 2. For LiT and the baseline, there is only the image-to-text loss as per Eq. (1). As we train for less than one epoch, we do not observe any overfitting, in the sense that contrastive losses are identical on the training and validation set.
The image-to-third-tower loss,  $\mathcal{L}_{f_h\leftrightarrow h_f}$ , quickly reduces to near zero, indicating successful knowledge transfer of the pretrained model into the image tower for 3T. Further,  $\mathcal{L}_{f\leftrightarrow g}$  behaves similar to the baseline loss; this makes sense because both objectives compute a loss between an unlocked image tower and a text tower. Lastly,  $\mathcal{L}_{gh\leftrightarrow h_g}$  closely follows LiT's loss; this also makes sense because both are losses between a locked pretrained image model and a text tower trained from scratch.
In Fig. A.1 (b), we compute the difference of the baseline (LiT) loss and matching 3T  $\mathcal{L}_{f_h\leftrightarrow h_f}$ $(\mathcal{L}_{g_h\leftrightarrow h_g})$  loss, and observe that 3T generally achieves lower (similar) values for the same objective. This suggests a mutually beneficial effect for the individual loss terms of the 3T objective. By aligning the main image and text towers to the pretrained model, 3T obtains improved alignment between the main towers themselves. For the training loss, this effect is large early in training and then decreases. However, for downstream task application, we find that the gap between 3T and the baseline persists; we display retrieval on COCO as an example in Fig. A.1 (c).
A.2 Robustness Metrics
In this section, we study 3T, LiT, and the from-scratch baseline from a robustness perspective, evaluating on a subset of the tasks considered by Tran et al. [72]. Following §4.3, we evaluate all methods across multiple model scales and for both JFT and IN-21k pretraining. We use the full Unfiltered WebLI for all results here. We apply models in zero-shot fashion to these datasets, following the same protocol as for the main zero-shot classification experiments. We continue to use the global temperature  $\tau$ , cf. §3, learned during training to temper the probabilistic zero-shot predictions.
A.2.1 Probabilistic Prediction and Calibration on CIFAR and ImageNet Variants
In Fig. A.2, we report accuracy, negative log likelihood (NLL), Brier score [22], and expected calibration error (ECE) [48, 22] for 3T, LiT, and the baseline across scales for the following datasets: CIFAR-10, CIFAR-10-C, ImageNet-1k (IN-1k), IN-A, IN-v2, IN-C, and IN-R.
Accuracy. Across all datasets, we find the familiar scaling behavior discussed in §4.2: 3T is consistently better than the baseline, 3T benefits more from increases in scale than LiT, LiT performs well with JFT pretraining but shows weaknesses when pretrained on IN-21k. Note that, for the ImageNet variants, we have previously reported the accuracies (if only at L scale) in §4.2. (Note further, that there might be small discrepancies, because we actually recompute all numbers from a different codebase for the robustness evaluations [15].) For CIFAR-10 [40] and CIFAR-10-C [28], which we have not previously discussed, we also find the familiar scaling behavior. The absolute reduction of performance between CIFAR-10 and CIFAR-10-C is similar across methods, indicating that no approach is significantly more robust to shifts. We observe the same comparing IN-1k to IN-C.
Probabilistic Prediction and Calibration. NLL and Brier scores follow the general trend laid out by the accuracy results. Evidently, the probabilistic zero-shot predictions of the methods are all of similar high quality, cf., for example, Tran et al. [72], who investigate probabilistic few-shot predictions. This is confirmed by the ECE results: across tasks, ECE values do not exceed 0.1 at L scale. For 3T, calibration results are regularly better than for LiT, particularly if pretrained on IN-21k, and comparable to those of the baseline: 3T and the baseline have lower calibration error than LiT on 6 out of 7 tasks at L scale with IN-21k pretraining.
We find the low magnitude of the calibration errors surprising. It is striking that the softmax temperature learned during contrastive training would work so well across the various downstream task applications. After all, finding matches across a batch from the contrastive learning dataset and assigning images to labels are, at least superficially, quite distinct tasks. We stress again that no task adaptation of either the models, prompt templates, or softmax-temperatures was performed. We refer to Minderer et al. [45] for a general categorization of our calibration results and discussion in the context of deep learning models.
A.2.2 Out-Of-Distribution Detection
We evaluate the performance of 3T, LiT, and the baseline for out-of-distribution (OOD) detection. We follow the common practice of thresholding the maximum softmax probabilities (MSP) of the models to obtain a binary classifier into in- and out-of-distribution [30, 20]. We report the following metrics: area under the precision-recall curve (AUC(PR)), the area under the receiver operating curve (AUROC), as well as the false positive rate at  $95\%$  true positives (FPR95). Following Tran et al. [72], we study CIFAR-10 as in-distribution against CIFAR-100, DTD, Places365, and SVHN as out-of-distribution. We also report numbers for IN-1k (in-distribution) vs. Places365 (out-of-distribution).