SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Typically for OOD evaluations, the model is trained on the in-distribution data. Here, we apply methods in a zero-shot manner: we only condition the text tower on the label set of the particular in-distribution dataset. Our image and text towers are trained on the contrastive learning data (image tower trained on JFT/IN-21k for LiT) and not adapted to the in-distribution samples. Our contrastive learning methods 'learn' about the in-distribution data only through the label set, and they have to classify each incoming sample as 'in-distribution' or 'out-of-distribution' based solely on how well it aligns with the given set of labels. If a given sample does not match any of the in-distribution labels well, prediction confidence is low, and the sample is classified as OOD. This setup diverges from typical assumptions about OOD experiments and should be interpreted with care. For example, if there were label overlap between the in- and out-of-distribution data (e.g. as would be the case for SVHN vs. MNIST), it would be impossible for the model to classify between in-distribution and OOD without further assumptions. OOD for CLIP/ALIGN-style models has been studied in similar settings by Fort et al. [20], Esmaeilpour et al. [18].
We display results in Fig. A.3. Generally, OOD detection works well with the contrastively learned models, despite conditioning only on the label set: for example, the AUROC for CIFAR10 exceeds 0.95 for both 3T and LiT at L scale for both IN-21k and JFT pretraining. The different metrics, AUC(PR), AUROC, and FPR95, are generally consistent in their ranking across scales and methods. We again find the familiar pattern: 3T is consistently improving over the baseline, and 3T catches up
Figure A.2: Robustness evaluation: Accuracy, negative log likelihood (NLL), Brier score, and expected calibration error (ECE) for 3T, LiT, and the baseline for IN-21k and JFT pretraining across scales.
Figure A.3: Robustness evaluation: 3T, LiT, and the baseline for zero-shot out-of-distribution detection (OOD). Reported metrics are area under the precision-recall curve (AUC(PR)), the area under the receiver operating curve (AUROC), and the false positive rate at  $95\%$  true positives (FPR95).
Table A.1: Using pretrained language models instead of image encoders degrades performance drastically for LiT, confirming the results of Zhai et al. [85]. While 3T does not suffer a drastic collapse in performance, we also do not observe gains from using the pretrained language model. B/32 ViT image tower, BERT models at BASE scale for LiT and 3T, B scale unlocked text tower for 3T and baseline, training for 900M examples seen at batch size 10240.
Unfiltered WebLI	Pair-Filtered WebLI
Basel.	LiT	3T	Basel.	LiT	3T
Retrieval	Flickr* img2txt	55.2	4.4	50.6	65.7	10.6	64.4
Flickr* txt2img	36.2	2.4	32.4	45.2	5.2	43.7
Coco img2txt	34.5	2.2	29.9	42.8	5.0	38.4
Coco txt2img	20.7	1.4	18.2	26.8	3.2	24.6
Few-Shot Classification	Caltech	87.4	80.3	87.0	88.7	88.0	90.0
UC Merced	84.2	67.7	81.4	90.5	81.0	87.8
Cars	56.8	38.0	56.8	76.9	67.1	76.8
DTD	59.2	45.8	56.3	63.8	58.3	63.1
Col-Hist	72.0	59.3	69.0	75.8	63.8	73.2
Birds	32.1	16.9	29.5	48.5	35.1	46.7
Pets	57.8	31.9	53.5	72.7	70.2	78.1
ImageNet	37.8	24.2	35.2	46.5	41.6	47.9
CIFAR-100	50.2	33.8	45.2	57.5	43.7	53.3
Zero-Shot Classification	Pets	58.7	6.1	56.5	79.2	36.8	79.4
ImageNet	45.8	11.3	41.8	58.0	26.9	56.9
CIFAR-100	52.2	19.1	44.0	57.6	37.6	54.7
to LiT as scale is increased. For OOD detection, LiT generally does better than 3T and the baseline, perhaps owing to the fact that our choice of CIFAR/IN-1k as in-distribution datasets is advantageous for LiT (similar to how LiT performs particularly well for these datasets for classification).
We find differences between JFT and IN-21k pretraining to be much smaller for the OOD detection task. In fact, in some cases, IN-21k pretraining outperforms JFT pretraining, for example with LiT for the CIFAR-10 vs. Places365 detection task. (This might again be due to the fact that IN-21k pretraining is sufficient for application to CIFAR-10, and only struggles to perform well for other, more varied datasets.) Further, we can observe a rare victory of 3T over JFT-LiT and the baseline at L and g scale in terms of FPR95 on CIFAR-10 vs. Places365 and CIFAR-10 vs. SVHN. Lastly, we see LiT has almost fixed performance at  $\approx 0.98$  for the CIFAR-10 vs. SVHN task across scales, perhaps due to early task saturation.
A.3 Pretrained Language Models
In Table A.1, we explore if there are benefits to using pretrained language models instead of pretrained image encoders. For LiT, we confirm the results of Zhai et al. [85] that performance suffers drastically when using a locked pretrained language encoder as the text tower (together with an unlocked image tower). In contrast, for 3T, we do not observe a drastic decrease in performance, once again showing that 3T is more robust to deficiencies in the pretrained model than LiT. However, we also do not observe gains from using a pretrained language model with 3T, justifying our choice of focusing on pretrained image encoders in the main text. Nevertheless, we think it is plausible that future work may yet find benefits of incorporating knowledge from pretrained language models in other settings or downstream tasks, e.g. tasks that require more complex language reasoning abilities.
Results here are for B/32 ViT image towers, we use BERT models at BASE scale as pretrained language models for LiT and 3T, a B scale unlocked text tower for 3T and the baseline, and we train for 900M examples seen at batch size 10240.
Table A.2: Predictions between the baseline and LiT are different and 3T benefits from combining them. The first column gives the proportion of datapoints where the baseline but not LiT predicts correctly. The second column gives the proportion of datapoints where LiT but not the baseline predicts correctly. Columns three to five show the proportion of datapoints where 3T predicts differently than the baseline (3), LiT (4), or differently from both (5).
Base, Not LiT	LiT, Not Base	3T ≠ Base	3T ≠ LiT	3T ≠ Base & 3T ≠ LiT
IN-1k	6.9	13.3	22.8	26.5	14.5
CIFAR-100	7.5	18.0	28.6	32.8	20.4
Caltech	3.9	4.3	11.5	16.1	8.4
Pets	7.2	7.1	10.9	14.4	5.5
DTD	14.3	7.6	27.2	40.3	19.5
IN-A	19.0	11.7	38.3	52.8	28.4
IN-R	23.5	3.8	13.3	34.2	10.0
IN-v2	8.4	13.3	27.2	32.8	18.0
ObjectNet	20.7	6.4	34.1	52.9	26.7
EuroSat	19.2	14.2	62.3	75.1	49.4
Flowers	4.0	14.8	25.4	27.0	16.0
RESISC	33.2	4.1	31.9	62.6	25.7
Sun397	11.8	9.6	21.9	31.2	14.2
A.4 Investigating Prediction Differences
Table A.2 provides a detailed evaluation of how predictions differ between 3T, LiT, and the baseline on a per datapoint (and per task) level. The results in Table A.2 are for the zero-shot tasks of the IN-21k pretrained L scale model setup of Table 3. This evaluation gives meaningful insights into understanding how 3T improves predictions over LiT and the baseline. The contrastive learning baseline and the pretrained model have different strengths, and 3T can benefit from combining them.
The first two columns of Table A.2 show that there is predictive diversity between the baseline and LiT. The first column gives the proportion of datapoints where the baseline but not LiT predicts correctly. The second column gives the proportion of datapoints where LiT but not the baseline predicts correctly. We can see that, for both the baseline and LiT, there is a significant proportion of inputs, where only one of the models predicts correctly. Hence, the two models have different strengths and, equivalently, make different mistakes. A combination of the two approaches, such as 3T, can benefit from this if it learns to combine their predictions in the right way.
In columns three, four, and five, we illustrate that 3T predicts differently from LiT and the baseline. The columns show the proportion of datapoints where 3T predicts differently than the baseline (3), LiT (4), or differently from both (5). Clearly, 3T learns a novel predictive mechanism that is meaningfully different from LiT and the baseline. Whenever 3T outperforms both LiT and the Baseline (Caltech, DTD, IN-A, IN-R, ObjectNet, EuroSat, and Sun397 in Table 3) it must have learned to combine knowledge from the pretrained model and contrastive learning mechanism in a beneficial way.
A.5 Additional Image Encoders
In Table A.3, we study 3T and LiT for additional pretrained image encoders: a ResNet-based BiT model trained on IN-21k [38] and a ViT-based self-supervised encoder trained with DINO on IN-1k as in [85]. 3T consistently outperforms the baseline and LiT for DINO- and BiT-based pretrained models for all retrieval scenarios. For few-shot classification, LiT performs best on average for BiT-based models but 3T performs best on average for our DINO experiments. For zero-shot classification, 3T performs best on average for both our DINO- and BiT-based experiments. In total, 3T performs best on average over all tasks for both experiment setups.
Here, we use a B/16 scale image tower and a B scale text encoder trained on Unfiltered WebLI. The pretrained image models are a BiT-R50x3 pretrained on IN-21k, and we follow Zhai et al. [85] for DINO pretraining on IN-1k. We also report a 'BiT-Baseline': for this we replace the ViT B/16 image tower of the standard baseline with a BiT-R50x3 trained from scratch.
Table A.3: Results for additional pretrained image encoders: a ResNet-based BiT trained on IN-21k and a self-supervised DINO encoder trained on IN-1k. 3T outperforms LiT and the baseline on average for retrieval and classification task, with the exception of few-shot classification for BiT, where LiT performs best. B/16 scale image tower, B scale text encoder, BiT-R50x3 pretrained on IN-21k, DINO pretrained on IN-1k as in Zhai et al. [85], Unfiltered WebLI.
BiT Experiments	DINO Experiments
Basel.	BiT-Basel.	LiT	3T	Basel.	LiT	3T
Retrieval	Flickr* img2txt	71.2	72.5	61.1	74.6	71.2	60.4	74.0
Flickr txt2img	49.3	50.6	39.1	52.5	49.3	35.0	52.1
Flickr* txt2img	51.3	53.7	42.9	54.5	51.3	38.3	54.0
Flickr img2txt	68.5	68.1	59.2	70.1	68.5	57.5	69.2
COCO img2txt	44.3	45.0	40.9	46.8	44.3	38.9	45.7
COCO txt2img	29.0	29.8	24.7	32.0	29.0	21.2	31.4
Average	52.3	53.3	44.7	55.1	52.3	41.9	54.4
Few-Shot Classification	Caltech	89.8	90.0	90.4	90.9	89.8	91.0	92.1
UC Merced	89.3	84.0	91.6	91.2	89.3	94.2	92.1
Cars	75.0	73.4	42.4	77.1	75.0	49.4	78.7
DTD	66.5	64.0	66.8	69.4	66.5	63.5	70.1
Col-Hist	68.3	66.6	84.8	72.9	68.3	85.6	75.2
Birds	44.5	38.5	85.5	53.2	44.5	62.1	52.5
Pets	75.8	65.5	89.2	78.9	75.8	82.5	76.3
IN-1k	52.0	57.8	73.8	56.2	52.0	60.4	56.4
CIFAR-100	57.8	40.0	74.1	62.1	57.8	62.3	61.0
Average	68.8	64.4	77.6	72.4	68.8	72.3	72.7
Zero-Shot Classification	Pets	75.4	79.0	80.9	80.2	75.4	83.5	78.7
IN-1k	59.8	60.6	67.1	62.8	59.8	62.9	62.5
CIFAR-100	60.0	45.0	72.6	59.6	60.0	56.0	61.1
IN-A	32.8	31.4	26.2	34.9	32.8	21.7	32.5
IN-R	75.4	73.4	55.4	78.1	75.4	57.0	77.4
IN-v2	53.1	53.3	58.8	55.7	53.1	55.0	55.6
ObjectNet	44.7	42.7	34.4	47.3	44.7	25.3	45.0
Caltech	79.1	81.0	76.7	79.3	79.1	79.0	80.7
DTD	51.4	52.8	46.6	54.1	51.4	41.9	52.4
Eurosat	36.5	19.7	28.1	32.6	36.5	32.6	36.2
Flowers	51.9	54.3	64.1	57.1	51.9	48.2	55.6
RESISC	53.1	49.9	27.8	52.9	53.1	24.4	48.0
Sun397	62.6	63.4	60.9	65.3	62.6	53.7	63.2
Average	57.1	55.0	54.8	59.2	57.1	50.4	58.4
Average	59.6	57.4	59.5	62.2	59.6	55.1	61.8
A.6 IN-21k Pretraining - Additional Results
In Table A.4, we report retrieval and few-/zero-shot classification performance for 3T, LiT, and the baseline across a selection of pretraining datasets for L scale models and IN-21k pretraining. In addition to the unfiltered WebLI split reported in the main text, we here report results on the pair-filtered and text-filtered splits of the WebLI dataset, cf. §4.2. Further, we report results on the same dataset used by Zhai et al. [85] to train LiT. Across all datasets, 3T outperforms LiT and the baseline on average for retrieval and few-/zero-shot classification tasks, confirming our results for the unfiltered WebLI split in §4.1 and §4.2.
Table A.4: Results for the baseline, LiT, and 3T for L scale models and IN-21k pretraining. Across all datasets, 3T outperforms LiT and the baseline on average for retrieval, few-shot image classification, and zero-shot image classification tasks.
Dataset Method	Pair-Filtered WebLI	Text-Filtered WebLI	LiT Dataset
Basel.	LiT	3T	Basel.	LiT	3T	Basel.	LiT	3T
Retrieval	Flickr img2txt	79.4	72.4	81.6	79.8	72.1	83.9	79.1	71.7	82.0
Flickr* img2txt	81.0	73.5	83.8	84.5	75.8	86.9	80.2	74.5	84.2
Flickr txt2img	59.6	48.3	62.5	65.1	51.3	68.8	60.6	48.1	64.1
Flickr* txt2img	61.8	51.5	64.0	67.9	54.2	70.6	61.5	50.8	66.1
COCO img2txt	56.4	47.8	58.4	58.2	50.3	62.5	53.0	47.2	57.6
COCO txt2img	39.1	29.5	40.9	43.0	31.8	45.6	38.3	29.6	41.3
Average	62.9	53.8	65.2	66.4	55.9	69.7	62.1	53.6	65.9
Few-Shot Classification	IN-1k	67.8	79.0	71.9	64.9	79.0	69.6	63.3	79.0	68.6
CIFAR-100	71.0	83.6	75.1	73.6	83.6	77.0	73.0	83.6	74.9
Caltech	88.8	88.4	90.8	89.8	88.4	90.5	90.7	88.4	92.0
Pets	91.3	89.2	91.7	85.7	89.2	87.7	84.1	89.2	86.9
DTD	73.2	69.2	74.8	71.5	69.2	75.7	71.3	69.2	73.1
UC Merced	94.3	92.8	96.1	94.8	92.8	95.8	92.8	92.8	94.6
Cars	91.5	41.9	92.4	86.5	41.9	89.0	85.3	41.9	87.8
Col-Hist	77.5	86.4	81.3	76.1	86.4	80.4	72.2	86.4	78.6
Birds	70.9	83.4	79.0	56.7	83.4	68.2	62.1	83.4	70.4
Average	80.7	79.3	83.7	77.7	79.3	81.6	77.2	79.3	80.8
Zero-Shot Classification	IN-1k	75.1	77.5	76.5	72.1	76.6	73.8	72.1	77.4	74.8
CIFAR-100	73.0	82.4	74.0	76.8	82.9	78.0	76.0	83.1	77.0
Caltech	84.2	78.5	83.9	80.3	80.8	84.5	81.4	80.7	84.7
Pets	93.5	91.6	94.4	87.7	88.3	90.8	86.5	88.4	88.6
DTD	56.4	50.2	57.8	61.2	49.1	59.4	61.8	54.6	60.7
IN-A	51.8	46.7	54.8	57.6	46.7	60.4	56.2	47.3	58.9
IN-R	88.4	67.4	89.6	89.0	67.2	90.3	88.4	67.1	90.0
IN-v2	67.8	68.9	69.8	65.4	68.0	67.5	65.3	68.6	68.3
ObjectNet	52.6	43.3	54.6	57.3	42.4	59.0	54.3	42.6	56.5
Eurosat	37.2	28.5	44.2	38.2	20.2	42.4	44.0	29.1	45.8
Flowers	79.7	81.1	80.3	63.8	77.8	67.7	67.3	79.0	73.4
Resisc	61.6	32.5	63.1	61.8	30.0	63.1	58.3	28.4	57.2
Sun397	67.1	63.9	67.6	69.3	65.6	69.6	69.3	65.8	69.5
Average	68.9	64.1	70.7	68.7	63.2	70.8	68.5	64.2	70.5
Average	71.1	66.4	73.3	70.8	66.3	73.6	69.7	66.4	72.5
A.7 JFT Pretraining - Additional Results
In Table A.5, we report few- and zero-shot classification performance for 3T, LiT, and the baseline across our selection of datasets for L scale models and JFT pretraining. LiT outperforms 3T and the baseline on average for few- and zero-shot classification tasks.
In Table A.6, we report performance for g scale models and JFT pretraining across all three splits of the WebLI dataset described in §4. Retrieval performance is generally best for all methods for the Text-Filtered WebLI split, with 3T generally performing best across splits and tasks. For classification, for 3T and the baseline, performance on Text- and Pair-Filtered WebLI is significantly better than on Unfiltered WebLI, with LiT generally performing best across splits. In line with our previous observations, the differences between the WebLI splits are smaller for LiT. As the image tower is kept fixed during contrastive training, LiT performance is influenced less by the contrastive learning setup.
Retrieval Results: Comparison to SOTA. While our retrieval performance is competitive, 3T does not set a new state-of-the-art, see, for example, the CoCa paper [81] (Table 3) for a comparison of current methods. While SOTA results were never the aim of this paper—we instead study pretrained models for contrastive learning—there are a few advantages the CoCa setup has, and from which 3T would likely benefit, too. Most notably, CoCa trains for about 6 times more examples seen than we do here (32B vs. 5B). Our scaling experiments, cf. Fig. A.4, suggest we would expect a significant performance increase for longer training. There are further differences that likely benefit CoCa, such as the use of a larger batch size (65k for them vs 14k for us) or training on images with higher resolution for a portion of training (CoCa goes from  $288 \times 288$  to  $576 \times 576$ , we stay at  $288 \times 288$ )—both of these changes significantly increase computational costs beyond the budget available to us: while CoCa training takes 'about 5 days on 2,048 CloudTPUv4 chips'[81], our g scale runs train for about the same duration on only 512 v4 TPU chips. It would be interesting to see if, in a fairer comparison, 3T matches or outperforms CoCa for retrieval tasks. Alternatively, ideas from 3T could also be used to improve CoCa-like architectures.
Table A.5: For JFT-pretraining, LiT outperforms 3T and the baseline on average on few- and zero-shot classification tasks. L scale models trained on Unfiltered WebLI.
Method	Basel.	LiT	3T
Few-Shot Classification	IN-1k	62.8	81.3	67.7
CIFAR-100	70.4	83.2	74.3
Caltech	91.0	89.0	91.8
Pets	85.9	96.8	88.4
DTD	70.3	72.1	72.4
UC Merced	91.8	95.5	93.1
Cars	81.5	92.9	87.1
Col-Hist	71.7	81.3	77.0
Birds	53.4	85.6	62.4
Zero-Shot Classification	IN-1k	69.5	80.1	72.0
CIFAR-100	73.5	80.1	75.2
Caltech	81.9	79.5	82.5
Pets	84.2	96.3	88.7
DTD	58.6	59.0	59.0
IN-C	49.6	68.1	52.8
IN-A	53.0	69.1	56.4
IN-R	85.8	91.7	88.4
IN-v2	62.2	74.0	65.4
ObjectNet	56.2	61.9	59.3
EuroSat	32.7	36.6	54.7
Flowers	62.0	76.7	66.6
RESISC	58.0	58.9	60.9
Sun397	67.6	69.7	68.1
Average	68.4	77.4	72.4
Table A.6: Results for the baseline, LiT, and 3T for g scale models and JFT pretraining for a selection of different splits of the WebLI dataset. 3T outperforms LiT for retrieval tasks, while LiT performs better for image classification. The from-scratch CLIP/ALIGN-style baseline is not competitive.
Dataset Method	Unfiltered WebLI	Pair-Filtered WebLI	Text-Filtered WebLI
Basel.	LiT	3T	Basel.	LiT	3T	Basel.	LiT	3T
Retrieval	Flickr img2txt	75.2	83.0	81.5	81.4	83.2	84.0	85.0	83.9	87.3
Flickr* img2txt	80.0	84.8	84.2	80.7	83.9	85.6	86.7	85.2	88.3
Flickr txt2img	58.2	61.3	64.3	61.4	63.9	66.5	67.0	66.5	72.1
Flickr* txt2img	60.1	63.1	65.6	62.7	65.4	68.4	68.2	67.6	72.9
COCO img2txt	52.3	57.7	57.5	58.4	59.7	61.7	60.0	59.5	64.1
COCO txt2img	37.5	40.0	41.1	41.2	41.9	43.9	44.7	43.6	48.5
Few-Shot Classification	IN-1k	67.5	84.6	72.8	71.8	84.6	75.7	69.6	84.6	73.9
CIFAR-100	72.7	83.2	78.0	73.1	83.2	78.7	76.4	83.2	80.0
Caltech	91.8	90.0	93.3	89.7	90.0	90.9	90.8	90.0	92.4
Pets	88.4	97.8	91.5	93.0	97.8	94.3	88.8	97.8	91.4
DTD	70.7	74.6	74.7	74.2	74.6	76.1	73.6	74.6	76.0
UC Merced	92.9	96.9	94.7	95.2	96.9	95.6	95.2	96.9	96.5
Cars	84.1	93.3	88.6	92.6	93.3	93.5	89.0	93.3	91.6
Col-Hist	72.0	83.6	76.2	77.8	83.6	80.9	73.5	83.6	79.4
Birds	60.7	89.7	69.8	76.4	89.7	80.7	62.5	89.7	71.1
Zero-Shot Classification	IN-1k	73.5	84.0	76.3	78.0	84.7	79.6	75.8	84.3	78.2
CIFAR-100	77.5	81.3	80.3	76.2	81.3	79.5	80.6	81.8	82.3
Caltech	79.8	81.4	82.3	84.0	82.4	82.9	79.5	80.9	81.9
Pets	87.0	96.4	92.7	92.8	97.7	93.0	88.1	96.5	91.5
DTD	59.2	62.1	64.9	58.9	55.6	60.1	61.4	62.0	62.1
IN-C	54.9	72.9	58.2	57.7	73.3	60.3	57.6	73.3	61.3
IN-A	64.9	80.2	67.8	59.9	79.5	65.1	67.8	80.5	70.8
IN-R	89.8	94.4	91.8	90.5	94.2	92.8	91.8	94.6	93.3
IN-v2	66.4	78.1	69.5	70.8	79.2	73.0	69.1	78.5	71.4
Objectnet	62.7	70.3	65.3	56.9	68.3	59.5	63.3	70.0	65.9
Eurosat	55.7	33.6	48.9	32.9	30.7	42.8	47.9	36.1	52.1
Flowers	71.0	84.2	73.5	82.4	86.3	83.0	69.4	86.6	72.5
RESISC	61.5	58.4	60.5	59.8	56.5	64.8	65.4	57.8	61.7
Sun397	68.8	71.0	70.3	68.9	71.9	69.8	70.2	71.6	70.9
Average	70.2	77.0	73.7	72.4	77.0	75.3	73.1	77.7	75.9
A.8 Pretraining Robustness - Additional Results
In Table A.7, we report results on additional tasks for 3T, LiT, and the baseline for both the 'mismatched' setup and Places365 pretraining of §4.4. We find again that 3T is much more robust in both setups, significantly outperforming LiT. The difference is particularly striking when using models pretrained on Places365, where LiT's performance degrades drastically while 3T is still able to improve over the baseline.
Table A.7: Testing robustness to the 'mismatched setup' and Places365 pretraining (instead of IN-21k/JFT) for 3T and LiT. In both cases, 3T performs significantly better than LiT. In particular when using models pretrained on Places365, LiT's performance degrades dramatically while 3T continues to improve over the baseline on average. (Note that the baselines here are different not because they use the pretraining dataset, but because we compare to an L scale baseline for the mismatched setup and a B scale baseline (trained for only 900M examples seen) for Places365 pretraining.) We refer to the main text for full details.
Experiment Method	Mismatched Setup	Places365 Pretraining
Basel.	LiT	3T	Basel.	LiT	3T
Retrieval	Flickr* img2txt	75.6	66.5	80.2	56.0	35.5	58.1
Flickr* txt2img	57.1	45.1	62.1	36.2	19.5	38.4
COCO img2txt	51.0	44.1	54.5	34.1	19.3	36.5
COCO txt2img	34.2	26.4	37.8	21.0	10.9	22.1
Few-Shot Classification	IN-1k	62.8	70.3	67.6	37.8	16.6	41.5
CIFAR-100	70.4	80.3	73.8	47.1	33.9	52.7
Caltech	91.0	88.1	91.7	87.9	66.5	88.5
Pets	85.9	86.0	86.8	56.8	20.3	59.9
DTD	70.3	66.3	73.4	58.4	39.7	63.1
UC Merced	91.8	91.5	93.8	85.8	80.8	89.4
Cars	81.5	36.7	85.3	57.0	10.1	58.6
Col-Hist	71.7	84.4	74.3	72.9	70.7	78.7
Birds	53.4	76.8	65.2	33.2	15.7	38.1
Zero-Shot Classification	IN-1k	69.5	69.5	71.5	45.6	24.5	47.4
CIFAR-100	73.5	78.6	75.6	48.3	27.4	52.4
Caltech	81.9	82.0	81.2	76.6	62.7	77.0
Pets	84.2	84.7	87.4	61.5	30.3	60.2
DTD	58.6	49.4	60.6	39.8	23.6	39.7
IN-C	49.6	55.5	51.8	25.3	14.4	27.3
IN-A	53.0	29.1	54.1	12.0	4.7	12.5
IN-R	85.8	60.7	87.9	56.1	20.3	58.2
IN-v2	62.2	61.1	65.0	39.4	20.7	40.5
Objectnet	56.2	34.9	57.8	28.4	7.3	29.6
Eurosat	32.7	33.1	52.5	33.7	15.6	27.3
Flowers	62.0	74.1	66.2	37.6	17.4	37.3
RESISC	58.0	29.0	57.4	37.9	24.0	38.3
Sun397	67.6	62.0	68.4	55.1	60.6	57.3
Average	66.4	61.7	69.8	47.5	29.4	49.3
A.9 Scaling Model Sizes and Training Duration - Additional Results
Complementing the results of §4.3, in Fig. A.4 we report the performance when scaling only the number of examples seen during training, keeping the model sizes fixed at B scale. We observe a similar trend to §4.3 / Fig. 3, where 3T benefits more from increases in scale than LiT. Compared to the baseline, 3T consistently improves performance, even as the number of examples grows very large. We do not observe any evidence that 3T only improves performance for particular compute budgets. Note that, because the dataset size is 10B samples, all of our runs equate to less than a full epoch.
Figure A.4: Increasing training duration of 3T, LiT, and the baseline; average retrieval, few- and zero-shot classification performance. The model scale is B (B/32 for ViTs) for all approaches and towers. 3T and the baseline benefit more from increases in scale than LiT, with 3T maintaining a consistent increase in performance over the baseline. Note that the few-shot performance for LiT is fixed, as only the locked pretrained image tower is used for fewshot applications.
No. Examples Seen
A.10 Benefits From Using 3T With All Three Towers at Test Time - Extended Version
We usually discard the pretrained model when applying 3T to downstream tasks, cf. Fig. 2 (b). Instead, in this section, we explore whether we can find benefits from using the locked third tower at test time, similar to LiT. More specifically, we are interested in interpolating between the main image tower and locked pretrained model in the third tower. Can we interpolate between 3T- and LiT-like prediction by combining the image embeddings?
This idea does not work directly with the default 3T due to our use of linear projection heads, cf. Fig. 2 (a), since there is no unified embedding space that all towers embed to. Therefore, we introduce a 'headless 3T' variant, for which we do not use the linear projection heads,  $h_f$ ,  $h_g$ ,  $f_h$ , and  $g_h$ . (Alternatively, one may think of all linear projection heads fixed to identity mappings.) Thus, all losses directly use the same embeddings,  $f(I)$ ,  $p(I)$ , and  $g(T)$ , making the embedding spaces directly comparable. Here, we train B scale models for 3.6B examples seen and use an IN-21k-pretrained model. Further note that the average zero-shot classification performance we report here is over only a subset of the list of tasks used in §4.2: we consider IN-1k, CIFAR-100, and Pets. The selection of few-shot classification and retrieval tasks remains the same, although we do not use the Karpathy split for Flickr here.*