SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Fig. A.5, we display the average retrieval, few-shot classification, and zero-shot classification performance for the convex combination, alongside a comparable LiT run and a 3T run with default projection head setup. Across all tasks, we observe similar behavior: for  $\alpha = 0$  (full weight on the third tower), we obtain performance close to, but ultimately below, LiT; performance then increases with  $\alpha$ , peaking for  $\alpha \in [1/4, 3/4]$ , before decreasing again. At  $\alpha = 1$  (full weight on main image tower), we recover the performance of the headless 3T setup. Interestingly, for retrieval and few-shot classification tasks, the convex combination yields better performance than either of the towers separately across a relatively broad band of  $\alpha$  values.
Figure A.5: Convex combination of the image models in 3T:  $\alpha \cdot h(I) + (1 - \alpha)\cdot f(I)$ . By varying  $\alpha$ , we can generally interpolate between 3T and LiT performance. Interestingly, for a broad range of weights, the retrieval and few-shot classification performance of the combination outperforms 3T and LiT.
Perhaps counterintuitively, for  $\alpha = 0$ , we do not recover the performance of LiT exactly. The reasons for this differ between tasks: For retrieval and zero-shot applications, while the image tower is identical to that of LiT, the text tower is different as it has been trained with the 3T objective. For few-shot application, the default evaluation procedure of Zhai et al. [85] uses the prelogits of the ViTs underlying  $f$  and  $h$  as inputs to the few-shot classifier, i.e. not the final embeddings. As the prelogit spaces of  $f$  and  $h$  are not aligned, here, we need to instead construct the convex combination in embedding space, which does however mean that  $\alpha = 0$  does not give performance equivalent to LiT. Lastly, although the 3T run with the default projection heads does not seem to perform better than '3T headless' in this instance, we have seen 'headless' setups underperform in preliminary experiments and would suggest additional experiments before opting for a headless design, see also §A.11.
We believe that further study of this approach is exciting future work: the method is entirely post-hoc and no additional training costs are incurred, although inference costs do increase.
A.11 Ablation
In this section, we give additional results and details for the ablation study presented in §4.6. Table A.8 gives additional results, extending Table 5 in the main paper. In addition to the mean and two standard errors, we also report standard deviations over tasks here. Note that, for zero-shot classification performance, we only have access to a subset of the full list of tasks used in Section 4.2: we consider IN-1k, CIFAR-100, and Pets. The selection of few-shot classification and retrieval tasks remains the same, although we do not use the Karpathy split for Flickr here.*
No  $\mathcal{L}_{\leftrightarrow}$ . - Details. For this ablation we consider leaving out either of the three loss terms. 'No  $\mathcal{L}_{f\leftrightarrow g}$ ': We replace the 3T loss by  $\frac{1}{2} \cdot (\mathcal{L}_{f_h \leftrightarrow h_f} + \mathcal{L}_{g_h \leftrightarrow h_g})$ . 'No  $\mathcal{L}_{f_h \leftrightarrow h_f}$ ': We replace the 3T loss by  $\frac{1}{2} \cdot (\mathcal{L}_{f\leftrightarrow g} + \mathcal{L}_{g_h \leftrightarrow h_g})$ . 'No  $\mathcal{L}_{g_h \leftrightarrow h_g}$ ': We replace the 3T loss by  $\frac{1}{2} \cdot (\mathcal{L}_{f\leftrightarrow g} + \mathcal{L}_{f_h \leftrightarrow h_f})$ . When leaving out either of the three loss terms, average performance suffers significantly. Leaving out the loss between the main two towers (obviously) has the biggest negative effect, as the main embeddings,  $f(I)$  and  $g(T)$ , are not aligned during training.
Head Variants - Details and Additional Results. In the main part of the paper, we have only given results for the best alternative variant for the projection head setup. Here, we describe all variants and report results individually. We refer to Fig. 2 (a) for the projection head notation. 'Heads only on Third Tower': The main tower projection heads  $f_{h}$  and  $g_{h}$  are fixed to identity mappings. 'Heads Only on Main Towers': The third tower projection heads  $h_{f}$  and  $h_{g}$  are fixed to identity mappings. 'No Heads/Headless': This is the setup described in §A.10: all linear projections  $h_{f}, h_{g}, f_{h}, g_{h}$  are fixed to identity mappings. 'Heads Fully Independent': This setup adds linear projection heads before the computation of  $\mathcal{L}_{f \leftrightarrow g}$ , i.e. we compute  $f_{g}(I) = \operatorname{Lin}(f(I))$  and  $g_{f}(T) = \operatorname{Lin}(g(T))$ , and then compute the loss  $\mathcal{L}_{f_{g} \leftrightarrow g_{f}}$  (instead of  $\mathcal{L}_{f \leftrightarrow g}$ ). In Table A.8, we give results for all variants that we try; none outperform the base variant significantly, while some underperform.
MLP Embedding - Details. When replacing the linear projection  $h$  in the third tower with an MLP, we use the following architecture:  $\mathrm{MLP}(x) = \mathrm{Lin}_2(\mathrm{GELU}(\mathrm{Lin}_1(x))$ , where we use GELU non-linearities [29],  $\mathrm{Lin}_1$  expands the embedding dimensionality of the input by a factor of 4, and  $\mathrm{Lin}_2$  maps to the shared embedding dimension  $D$ .
3T with Loss Weights - Details and Additional Results. We replace the standard 3T loss with a weighted objective  $\frac{1}{3} \cdot (\mathcal{L}_{f \leftrightarrow g} + w \cdot (\mathcal{L}_{f_h \leftrightarrow h_f} + \mathcal{L}_{g_h \leftrightarrow h_g}))$ . For the weights  $w$ , we sweep over
Table A.8: Extended results for the 3T ablation study. Difference to the 3T reference run for various architecture ablations. We report mean, standard deviation, and two standard errors of the differences over the downstream task selection.
Difference	Mean	Standard Deviation	Two Standard Errors
Rerun	-0.22	0.50	0.25
No Lf→g	-26.63	21.22	10.61
No Lfh←hf	-1.19	1.51	0.75
No Lgh←hg	-2.77	1.83	0.91
(Head Variants (best))	0.09	0.70	0.35
Heads Only on Third Tower	0.09	0.70	0.35
Heads Only on Main Towers	-0.67	0.66	0.33
Heads Fully Independent	-0.60	0.63	0.32
No Heads/Headless	-0.47	1.04	0.52
MLP Embedding	-0.08	0.69	0.35
More Temperatures	-0.26	0.95	0.48
(Loss weight = 2 (best))	0.17	1.06	0.53
Loss weight 0.1	-2.31	1.33	0.67
Loss weight 0.5	-0.90	0.81	0.41
Loss weight 2	0.17	1.06	0.53
Loss weight 10	-0.56	1.74	0.87
(L2 Transfer (best))	-3.80	2.27	1.13
L2 Transfer w=0.0001	-4.40	1.89	0.94
L2 Transfer w=0.001	-3.80	2.27	1.13
L2 Transfer w=0.05	-4.41	2.24	1.12
L2 Transfer w=0.01	-4.17	1.97	0.99
L2 Transfer w=.1	-3.97	2.06	1.03
L2 Transfer w=.5	-7.12	2.95	1.48
L2 Transfer w=1	-11.38	4.39	2.19
L2 Transfer w=2	-16.09	5.14	2.57
L2 Transfer w=10	-46.80	14.32	7.16
3T Finetuning	1.85	2.53	1.27
$w \in \{0.1, 0.5, 2, 10\}$ . All weights except  $w = 2$  lead to an average performance decrease. However, the size of the effect for  $w = 2$  is small relative to twice the standard error.
L2 Representation Transfer - Details and Additional Results. We investigate the use of squared losses for the representation transfer between the main towers and the third tower instead of relying on the contrastive loss. Concretely, we replace the 3T loss, Eq. (4), with
$$
\frac {1}{3} \left\{\mathcal {L} _ {f \leftrightarrow g} + w \frac {1}{N} \sum_ {i = i} ^ {N} \left[ \| f _ {h} (I _ {i}) - h _ {f} (I _ {i}) \| ^ {2} + \| g _ {h} (T _ {i}) - h _ {g} (I _ {i}) \| ^ {2} \right] \right\}. \tag {5}
$$
For the weight hyperparameters  $w$ , we sweep over a large set of values,  $w \in \{0.0001, 0.001, 0.05, 0.01, 0.1, 0.5, 1, 2, 10\}$ . L2 representation transfer gives worse results than the contrastive loss for all values of  $w$  we try, corroborating the results of Tian et al. [71].
Finetuning - Details and Additional Results. Initializing the main tower in 3T with the same JFT-pretrained model as the third tower boosts performance significantly, increasing average performance from 56.76 to 58.61. A rerun confirmed these results; we obtained an increase from 56.46 to 58.82. Excited by this, we explored the 3T finetuning setup at other scales, and report performance in Table A.9. Note that here, we increase the numbers of examples seen during training from 450M (S scale) to 900M (B scale) to 5B (L scale). We observe that, as we increase the scale of the experiments, the gains from finetuning the main image tower decrease until they are negligible (compared to rerun variance). We therefore have opted to not make finetuning the main tower part of the standard 3T setup, as it (a) complicates the setup and (b) restricts the main tower to be the same model architecture and scale as the third tower.
Table A.9: Finetuning for 3T: Initializing the main tower in 3T with the same pretrained model as the third tower improves performance significantly at smaller but not larger experiment scales.
Pretraining	Scale	Avg. Performance 3T	Avg. Performance 3T Finetuned
JFT	B	56.76	58.61
L	73.97	74.22
IN-21k	S	44.39	47.61
B	56.30	58.83
L	73.63	73.83
'FlexiLiT 1/2' - Details. With the FlexiLiT variants, cf. Table 5 in the main body of the paper, we investigate if there are other, simple ways to improve LiT. For both variants, we create a new 'half-locked' image tower by adding learnable components to the frozen pretrained image model. For FlexiLiT 1, we add a lightweight learnable 4-layer MLP on top of the frozen backbone: FlexiLiT-1(I) = MLP(LiT(I)). The MLP has 4 layers, uses GELU-nonlinearities, and an expansion factor of 4. For FlexiLiT 2, we add an additional learnable ViT next to the locked backbone (adding significant cost) and merge representations with an MLP: FlexiLiT-2(I) = MLP(concat(LiT(I)), ViT(I)). The additional ViT is B/32, following the main locked image tower. The MLP merging the two representations is an MLP with the same configuration as for FlexiLiT 1.
We follow Zhai et al. [85] for optimization and implementation details. We use the open-source vision transformer implementation available from Beyer et al. [4].
Unless otherwise mentioned, we use Transformers of scale L, with a  $16 \times 16$  patch size for the ViT image towers, i.e. L/16. We train for 5B examples seen at a batch size of  $14 \cdot 1024$ , i.e. for about 350 000 steps. We resize input images to  $224 \times 224$  resolution, and normalize pixel values to the  $[-1,1]$  range. Note that for experiments with g scale models, we resize images to  $288 \times 288$  instead. We use a learning rate of 0.001, warming up linearly for 10 000 steps, before following a cosine decay schedule. We use the Adafactor optimizer [65] with default  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.99$ , and we clip gradients if their norm exceeds 1.0. For g scale runs, we set  $\beta_{2} = 0.95$  by default, which we found to be important to ensure training stability. We use weight decay of 0.001.
We aggregate embeddings across tokens using multihead attention pooling, i.e. an attention block where the query is a single learned latent vector, and the keys and values are the outputs of the vision transformer (cf. vit.py in the code base [4]).
For details on how the different model scales and patch sizes relate to transformer width, depth, MLP dimension, the number of heads, or parameter count, we refer to Table 1 in [17] and Table 2 in [84].
Compute Cost. We train our models on v3 and v4 TPUs. For our main experiments at L scale, we use 256 TPU chips per experiment. Our 3T runs converge in about three days, for example, the 3T run with JFT pretraining took 63 hours of training time to converge over 348772 training steps. The baseline converges in 54 hours, and LiT in 35. For our five main experiments at L scale—3T, LiT for JFT and IN-21k pretraining, and a baseline run—the total runtime was about 280 hours, or about 8 TPU-Chip years worth of compute for the L scale experiments of this project. At g scale, we use 512 TPU chips per run, and our 3T runs converge in about 5 days.
Below we mention additional details pertaining to only some of the experiments.
Details on Few-Shot Classification. Following Zhai et al. [85], we use the prelogits of the ViTs instead of the final embeddings as input to the linear few-shot classifier.
Details on Places Experiment. Following Zhai et al. [85], for the Places365 experiment, we use a B/16 ResFormer [70] as the pretrained model.
With 3T, we introduce a novel machine learning method for learning joint embeddings of images and text. We train on large datasets of noisy and potentially biased data crawled from the internet. The same general caveats that apply to CLIP/ALIGN and LiT may also apply to 3T. We refer to §7 in Radford et al. [58] for a general discussion of the societal impact these methods may have.
Additionally, we wish to highlight the importance of carefully evaluating these models, testing for specific undesired behavior, before applying them in production. While the zero- and few-shot classification capabilities of these models are generally impressive, it is also important to consider their limitations and not succumb to wishful thinking when it comes to the real-world performance of these models on arbitrary tasks. For example, all of the approaches we study here do not perform well for zero-shot prediction on the structured and specialized tasks contained in VTAB, which include, for example, medical applications. It is therefore particularly important to carefully evaluate the performance of these methods when applied to real-world applications. Lastly, because 3T and LiT rely on two datasets for training, a classification and a contrastive learning dataset, this can complicate investigations into undesired biases in the final model.
D Libraries & Dataset
We rely on the Jax [5], Flax [26], and TensorFlow [1] Python libraries for our implementation. Additionally, we make use of the Big Vision [4] and Robustness Metrics [15] code bases.
For retrieval performance, we evaluate on Microsoft COCO [10] and Flickr30k [56]. For image classification, we evaluate on IN-1k [40, 61], CIFAR-100 [40], Caltech-256 [23], Oxford-IIIT Pet [53], Describable Textures (DTD) [13], UC Merced Land Use [80], Stanford Cars [39], Col-Hist [37], Birds [73], ImageNet variants -C [28], -A [32], -R [31], -v2 [59], ObjectNet [3], EuroSat [27], Oxford Flowers-102 [50], NWPU-RESISC45 [12], and Sun397 [78].
We take the EuroSat, Flowers, RESISC, and Sun397 datasets from the Visual Task Adaptation Benchmark (VTAB) [83]. They are the only VTAB datasets for which at least one method achieved better than trivial performance.