SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The results for both datasets are presented in Table 1 for the zero-shot setting. For all three base models, adding DN almost always improves recall for both zero-shot image-to-text and text-to-image retrieval tasks, with an average  $1.1\%$  increase in top-1 recall. This provides substantial evidence that DN is effective for both tasks even in datasets with diverse images like MSCOCO. The improvement can be as large as a  $2.2\%$  increase in top-1 image-to-text recall as we see for CLIP on Flickr30K. We also observe that, in general, adding DN brings a larger improvement for top-1 recall than for top-5/10 recall, suggesting that DN is particularly suitable for distinguishing similar samples.
While adding test-time augmentation techniques to CLIP are effective in themselves as shown in Table 1, we also show that adding DN or  $\mathrm{DN^{*}}$  on top of TTA techniques further improves retrieval accuracy. Specifically,  $\mathrm{CLIP + TTA + DN}$  achieves another average  $1.3\%$  improvement over  $\mathrm{CLIP + TTA}$ . A similar improvement is achieved by  $\mathrm{CLIP + TTA + DN^{*}}$  as well. However, it is noteworthy that although  $\mathrm{CLIP + TTA + DN}$  and  $\mathrm{CLIP + TTA + DN^{*}}$  achieves similar average improvement over  $\mathrm{CLIP + TTA}$  ( $1.3\%$  and  $1.2\%$  respectively), the improvement of  $\mathrm{CLIP + TTA + DN^{*}}$  has a smaller standard deviation ( $0.8\%$  compared to  $1.3\%$ ), suggesting that averaging DN with direct dot product indeed improves the stability.
4.4 Zero-shot Classification Results
Table 2 gives the zero-shot classification results for all three base models and their variants with DN on ImageNet1K [9], Cifar100 [28], SUN397 [67], Stanford Cars [27], Caltech 101 [31], and Flowers 102 [44] (see Appendix C.2 for datasets details). In these experiments, we perform DN before the final layer normalization of image/text encoders.
DN consistently enhanced the performance of all three base models across six datasets with an average  $4.4\%$  increase in top-1 accuracy. It demonstrates effectiveness not just in general-purpose vision datasets like ImageNet, but also in specialized datasets such as Stanford Cars, where average accuracy increases by  $1.5\%$ . Specifically,  $\mathrm{CLIP + TTA + DN / DN^{*}}$  is the best variant for a CLIP base model, signifying simultaneous benefits from DN and TTA. While  $\mathrm{CLIP + TTA + DN}$  and  $\mathrm{CLIP + TTA + DN^{*}}$
MSCOCO (5K test set)	Flickr30K (1K test set)
Image → Text	Text → Image	Image → Text	Text → Image
R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
CLIP [50]	52.4	76.0	84.5	30.2	55.1	66.4	81.3	95.0	98.5	62.7	86.0	92.0
CLIP + TTA [55]	53.9	77.5	85.5	32.1	57.5	68.3	83.2	96.8	98.4	65.2	87.9	92.9
CLIP + TTA + DN	53.6 ± 0.1	76.9 ± 0.1	84.8 ± 0.1	34.8 ± 0.0	60.4 ± 0.0	70.8 ± 0.1	85.8 ± 0.2	97.5 ± 0.1	99.1 ± 0.0	68.1 ± 0.1	89.4 ± 0.1	94.1 ± 0.0
CLIP + TTA + DN*	54.7 ± 0.1	77.8 ± 0.1	85.6 ± 0.1	33.8 ± 0.0	59.4 ± 0.0	70.1 ± 0.0	85.8 ± 0.1	97.5 ± 0.1	98.8 ± 0.1	67.6 ± 0.0	89.1 ± 0.0	93.9 ± 0.1
CLIP + DN	51.7 ± 0.1	75.8 ± 0.1	84.0 ± 0.1	33.4 ± 0.0	58.6 ± 0.1	69.4 ± 0.1	83.3 ± 0.2	96.4 ± 0.1	98.6 ± 0.1	66.2 ± 0.1	88.2 ± 0.1	93.3 ± 0.1
CLIP + DN*	52.9 ± 0.1	76.4 ± 0.1	84.9 ± 0.1	32.1 ± 0.1	57.4 ± 0.0	68.3 ± 0.1	83.5 ± 0.1	96.2 ± 0.0	98.5 ± 0.1	64.8 ± 0.2	87.5 ± 0.1	93.1 ± 0.0
TCL [68]	57.6	84.3	91.8	41.8	70.6	80.6	73.8	93.3	96.9	59.1	84.6	91.1
TCL + DN	60.6 ± 0.1	85.8 ± 0.1	92.4 ± 0.1	43.2 ± 0.0	71.8 ± 0.1	81.6 ± 0.0	77.5 ± 0.5	94.1 ± 0.2	96.9 ± 0.2	59.8 ± 0.2	84.9 ± 0.1	91.1 ± 0.1
TCL + DN*	59.5 ± 0.1	85.2 ± 0.0	92.2 ± 0.1	42.7 ± 0.0	71.5 ± 0.0	81.3 ± 0.0	75.5 ± 0.0	94.4 ± 0.1	96.9 ± 0.1	60.0 ± 0.1	85.1 ± 0.0	91.1 ± 0.0
ALBEF [33]	62.5	85.9	92.2	40.2	68.4	78.9	78.2	95.5	97.9	59.9	84.8	90.6
ALBEF + DN	63.0 ± 0.2	85.8 ± 0.1	92.4 ± 0.1	44.8 ± 0.1	72.5 ± 0.0	82.0 ± 0.0	80.6 ± 0.1	96.2 ± 0.1	98.3 ± 0.1	64.1 ± 0.0	87.1 ± 0.1	92.3 ± 0.1
ALBEF +DN*	63.0 ± 0.1	86.0 ± 0.0	92.5 ± 0.1	42.8 ± 0.1	70.8 ± 0.0	80.7 ± 0.0	79.2 ± 0.1	96.2 ± 0.0	98.0 ± 0.0	62.4 ± 0.1	86.1 ± 0.1	91.9 ± 0.1
Table 1: Cross-modal retrieval performance on MSCOCO and Flickr30K in the zero-shot setting. Means for DN are estimated using 100 random unlabeled validation samples. Average recalls and standard deviations are calculated with 5 random seeds.
display similar improvements over  $\mathrm{CLIP + TTA}$  (0.82% and 0.9% respectively),  $\mathrm{CLIP + TTA + DN^{*}}$  again exhibits a smaller standard deviation (0.22% vs. 0.76%), indicating improved stability. Furthermore, DN consistently surpasses CALIP in all model-dataset combinations, despite CALIP's slight higher computational cost from incorporating an attention module. Comparing TPT with our best variant  $\mathrm{CLIP + TTA + DN^{*}}$ , we find that they achieve comparable top-1 accuracy across all datasets while TPT suffers from a significant drop in top-5 accuracy, potentially due to poor calibrations in performing gradient steps on each single test image. It is also important to note that  $\mathrm{DN^{*}}$  is much more computationally efficient with only an overhead of subtracting the mean while TPT requires performing gradient steps on 64 different augmented view of each test image.
Notably, TCL saw the largest gain of  $7\%$  on average in top-1 accuracy. On ImageNet1K particularly, TCL's top-1 accuracy is improved from  $20.0\%$  to  $30.5\%$  by adding DN. We believe the fact that TCL and ALBEF are pre-trained on a smaller dataset (4M & 14M images) than CLIP (400M images) resulted in TCL and ALBEF getting a larger improvement than CLIP, as DN may have helped ease the instability when the model is not adequately trained. All these improvements are achieved with a single layer that incurs minimal computational overhead. Lastly, comparing DN and  $\mathrm{DN^{*}}$  for TCL and ALBEF, we found that DN consistently gives a greater improvement than  $\mathrm{DN^{*}}$ , potentially because averaging with vanilla dot product halves the benefits from DN when the base models are instable due to inadequate training.
ImageNet1K	Cifar100	SUN397	Stanford Cars	Caltech 101	Flowers 102
Acc@1	Acc@5	Acc@1	Acc@5	Acc@1	Acc@5	Acc@1	Acc@5	Acc@1	Acc@5	Acc@1	Acc@5
CLIP [50]	61.0	87.4	63.9	88.7	56.1	89.4	58.6	90.9	82.3	95.0	62.1	83.8
CALIP [14]	61.2 ± 0.2	87.5 ± 0.1	64.2 ± 0.1	88.9 ± 0.0	56.1 ± 0.1	89.3 ± 0.1	58.7 ± 0.0	90.1 ± 0.0	82.5 ± 0.1	95.1 ± 0.0	62.2 ± 0.1	83.4 ± 0.0
TPT [56]	63.5	87.1	65.2	88.1	59.4	88.8	61.5	90.2	83.2	96.0	64.5	81.3
CLIP + TTA	62.4	88.5	66.0	90.5	56.9	90.0	60.8	92.3	82.5	95.4	62.5	84.0
CLIP + TTA + DN	63.0 ± 0.0	88.8 ± 0.0	67.5 ± 0.1	90.7 ± 0.1	58.8 ± 0.2	91.0 ± 0.1	60.3 ± 0.2	91.5 ± 0.1	83.3 ± 0.1	95.4 ± 0.0	63.1 ± 0.1	84.3 ± 0.1
CLIP + TTA + DN *	63.2 ± 0.0	88.9 ± 0.0	67.1 ± 0.1	90.7 ± 0.0	58.1 ± 0.1	90.7 ± 0.0	61.5 ± 0.1	92.2 ± 0.0	83.1 ± 0.0	95.5 ± 0.0	63.5 ± 0.1	84.5 ± 0.0
CLIP + DN	61.6 ± 0.1	87.7 ± 0.1	65.5 ± 0.2	89.2 ± 0.2	57.9 ± 0.1	90.5 ± 0.1	57.5 ± 0.1	90.5 ± 0.1	82.9 ± 0.1	94.9 ± 0.1	62.7 ± 0.1	84.1 ± 0.1
CLIP + DN*	61.7 ± 0.1	87.8 ± 0.0	65.1 ± 0.1	89.4 ± 0.0	57.3 ± 0.0	90.2 ± 0.1	58.6 ± 0.1	90.7 ± 0.0	82.8 ± 0.0	95.1 ± 0.0	62.9 ± 0.1	84.3 ± 0.1
TCL [68]	20.0	42.9	39.1	68.2	28.6	63.4	2.0	8.7	58.8	80.1	24.4	42.6
TCL + DN	30.5 ± 0.2	54.3 ± 0.2	45.4 ± 0.1	73.1 ± 0.1	36.2 ± 0.2	70.8 ± 0.3	2.6 ± 0.1	10.7 ± 0.1	66.7 ± 0.1	81.8 ± 0.1	28.5 ± 0.1	48.4 ± 0.2
TCL + DN*	25.1 ± 0.1	48.9 ± 0.2	42.2 ± 0.0	71.1 ± 0.1	31.8 ± 0.1	66.8 ± 0.2	2.4 ± 0.0	9.5 ± 0.1	63.9 ± 0.1	81.1 ± 0.0	27.2 ± 0.1	45.7 ± 0.1
ALBEF [33]	37.3	65.8	38.5	65.7	45.6	81.5	25.0	61.0	66.0	86.3	26.9	49.1
ALBEF + DN	39.9 ± 0.2	68.5 ± 0.2	50.2 ± 0.1	77.0 ± 0.2	46.7 ± 0.1	82.4 ± 0.1	26.0 ± 0.2	62.0 ± 0.3	71.5 ± 0.3	88.9 ± 0.1	33.1 ± 0.2	54.4 ± 0.1
ALBEF + DN*	38.7 ± 0.1	67.3 ± 0.0	44.8 ± 0.2	72.0 ± 0.2	46.3 ± 0.1	82.0 ± 0.0	26.0 ± 0.1	61.7 ± 0.2	68.9 ± 0.4	87.6 ± 0.2	29.9 ± 0.1	51.9 ± 0.1
Table 2: Zero-shot classification performance on ImageNet1K, Cifar100, SUN397, Stanford Cars, Caltech 101, and Flowers 102. Means for DN are estimated using 100 random unlabeled validation samples. Average accuracies and standard deviations are calculated with 5 random seeds.
4.5 Image Captioning Metric Results
Inspired by the success of CLIPScore [17] as a metric for image captioning, we conduct experiments to test whether DN and  $\mathrm{DN^{*}}$  can help various cross-modal representation models align better with human judgments. To compare the performance of different methods, we report the Kendall's  $\tau_{b}$  or  $\tau_{c}$  coefficients against human judgments on Flickr8k-expert, Flickr8k-cf [18] and THumb [24] in Table 3, following the standard practice in [17]. We also report the resulting accuracy for caption-caption preferences on Pascal-50S [51] in Table 7. Details about these datasets are given in appendix C.3. In the reference-based setting, for CLIP-ref, we follow [17] in using an averaged score given by:
$$
\operatorname {C L I P - r e f} \left(x _ {0}, y _ {0}\right) = \mathrm {H - m e a n} \left(\phi \left(x _ {0}\right) ^ {\intercal} \psi \left(y _ {0}\right), \max  _ {c \in R} \psi (c) ^ {\intercal} \psi \left(y _ {0}\right)\right),
$$
where H-mean is the harmonic mean and  $R$  is the set of provided references. On the other hand, for a distribution normalized  $\mathrm{CLIP} + \mathrm{DN}$  -ref, we use an algebraic mean for simplicity:
$$
\operatorname {C L I P} + \operatorname {D N - r e f} (x _ {0}, y _ {0}) = \operatorname {A - m e a n} (S _ {(1)} (x _ {0}, y _ {0}), \max  _ {c \in R} (\psi (c) - \mu_ {y}) ^ {\intercal} (\psi (y _ {0}) - \mu_ {y})),
$$
where A-mean is the algorithmic mean,  $R$  is the set of provided references,  $\mu_y$  is the mean of text representations, and  $S$  is defined in Eqn. 9.
In Table 3, we report the results on Flickr8K-Expert, Flicker8K-CF, and THumb for all three base models. Some methods, such as the BLEU score, are not applicable in the reference-free setting and are thus only compared with in the reference-based setting. The results strongly support that adding DN significantly improves the  $\tau$  correlation for all three base models on all dataset in the reference-free setting. In particular, CLIP  $^+$  DN improves CLIP by  $2.9\%$  on Flickr8k-expert,  $1.1\%$  on Flickr8k-cf and  $3.6\%$  on THumb, and CLIP  $^+$  DN* performs comparably. In the reference-based setting, to give a better context, we also provide the performance of other reference-based SOTA metrics that are not contrastively trained for comparison. We find that CLIP and CLIP-ref outperform all of them. However, by simply adding a DN layer, CLIP-ref  $^+$  DN again yields an average  $1.3\%$  gain across the board over previous leaders. Fur
Flickr8k-expert
τc	Flickr8k-cf
τb	THumb
τc
Ref-free	CLIP [17]	51.4	34.3	19.9
CLIP + TTA	51.9	34.7	20.7
CLIP + TTA + DN	53.6	35.7	23.7
CLIP + TTA + DN*	53.5	35.5	22.7
CLIP + DN	54.3	35.4	23.5
CLIP + DN*	53.2	35.1	22.2
TCL [68]	31.0	20.6	8.1
TCL + DN	42.0	26.4	14.4
TCL + DN*	36.0	23.3	11.1
ALBEF [33]	24.9	15.4	0.9
ALBEF + DN	34.8	21.8	5.5
ALBEF + DN*	29.2	18.1	2.5
Ref-based	BLEU-1	32.3	17.9	11.1
BLEU-4	30.8	16.9	6.9
CIDEr [63]	43.9	24.6	13.8
NUBIA* [23]	49.5	-	-
ViLBERTScore-F [69]	50.1	-	-
CLIP-ref [17]	53.0	36.4	24.7
CLIP + DN-ref	55.3	37.0	25.8
CLIP + DN*-ref	54.3	36.9	26.2
Table 3: Image captioning metric results on Flickr8k-Expert, Flickr8k-CF, and THumb.
thermore, analogous to the results in cross-modal retrieval and zeroshot classification, we found that TTA is effective in improving on results of image captioning metric (CLIP + TTA Has on average  $0.6\%$  improvement over CLIP) but TTA combined with DN gives another  $1.9\%$  boost on average. A similar pattern is also observed in our experiment results on Pascal-50S in Appendix 7. Our main results on cross-modal retrieval, zero-shot classification, and image caption metrics provide extensive evidence that our proposed DN is a widely applicable modification that can improve the performance of cross-modal representation models across a variety of downstream applications.
Figure 2: Comparison of the effects of fine-tuning between CLIP and  $\mathrm{CLIP} + \mathrm{DN}^*$  for Acc@1. Results on MSCOCO's 5k test set are shown to the left, and Flickr30K's 1k test set are shown to the right. For each set, image-to-text retrieval (left) and image-to-text retrieval (right) are reported. For each of the 5 checkpoints we trained, we find its average accuracy and standard deviation with another 5 iterations random sampling for mean estimation, and plot the mean of these 5 accuracies and standard deviations from 5 independently fine-tuned checkpoints.
4.6.1 Results on Different Variants of CLIP
In addition to the CLIP-B32 model used in most experiments, in Table 4 and Table 5, we investigate into the performance of  $\mathrm{DN^{*}}$  on other popular variants of CLIP models, including CLIPB16, CLIPL14
[50] that come with more parameters and smaller patch size, and CLIPB32-Laion [54] that is trained on a larger image-caption pair dataset. The results show that DN* brings a similar performance gain for CLIP with more parameters or trained on a larger dataset, implying the universal applicability of our proposed method irrespective to the size of the CLIP model.
MSCOCO (5K test set)	Flickr30K (1K test set)
Image → Text	Text → Image	Image → Text	Text → Image
R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
CLIP(B16) + TTA	53.6	77.5	85.1	33.8	58.7	69.1	85.4	97.9	99.1	66.6	89.0	93.7
CLIP(B16) + TTA + DN*	54.6	78.5	86.1	35.7	60.7	70.8	87.3	98.0	99.6	69.3	90.2	94.6
CLIP(L14) + TTA	57.7	80.1	87.8	36.8	61.3	71.2	88.4	98.9	99.9	69.9	90.6	94.8
CLIP(L14) + TTA + DN*	58.8	81.3	88.4	38.6	63.1	72.9	89.3	98.8	99.8	72.1	91.7	95.5
CLIP(B32-Laion) + TTA	58.5	80.9	88.1	40.0	65.8	76.0	85.8	96.7	98.9	71.1	91.4	94.8
CLIP(B32-Laion) + TTA + DN*	60.7	82.3	88.9	40.8	66.5	76.4	86.6	97.1	98.9	71.7	91.6	94.7
Table 4: Cross-modal retrieval performance on MSCOCO and Flickr30K in the zero-shot setting for more CLIP variants. Means for DN* are estimated using 100 random unlabeled validation samples. Average recalls are calculated with 5 random seeds.
ImageNet1K	Cifar100	SUN397	Stanford Cars	Caltech 101	Flowers 102
Acc@1	Acc@5	Acc@1	Acc@5	Acc@1	Acc@5	Acc@1	Acc@5	Acc@1	Acc@5	Acc@1	Acc@5
CLIP(B16) + TTA	67.1	91.5	67.7	90.1	60.0	91.4	63.3	93.1	84.5	96.7	69.8	84.8
CLIP(B16) + TTA + DN *	67.8	92.0	71.1	92.2	61.9	92.4	64.6	93.9	84.9	96.6	70.0	85.0
CLIP(L14) + TTA	73.1	93.4	77.6	94.0	62.1	92.5	76.3	97.7	86.0	97.3	72.8	89.1
CLIP(L14) + TTA + DN *	74.2	94.1	80.4	95.4	63.8	93.3	77.2	97.8	86.2	97.1	74.0	89.1
CLIP(B32-Laion) + TTA	66.9	89.4	75.7	94.0	63.9	93.5	87.1	99.2	87.3	97.7	70.3	86.4
CLIP(B32-Laion) + TTA + DN *	67.2	90.3	76.2	94.3	64.3	93.7	87.1	99.1	86.8	97.2	71.1	85.8
Table 5: Zero-shot classification performance on ImageNet1K, Cifar100, SUN397, Stanford Cars, Caltech 101, and Flowers 102. Means for DN* are estimated using 100 random unlabeled validation samples. Average accuracies are calculated with 5 random seeds.
4.6.2 Effect of Fine-Tuning
We also perform ablations on the effects of  $\mathrm{DN^{*}}$  on fine-tuned models. We fine-tune CLIP on the MSCOCO training set, for a total of 10 epochs on 4 Nvidia 2080Ti. We use the Adam optimizer with a learning rate of 1e-5 and a weight decay of 0.1. We then evaluated each checkpoint on the same MSCOCO test split used in Table 1, and estimate the mean for  $\mathrm{CLIP} + \mathrm{DN^{*}}$  using 100 randomly sampled validation points. A total of 5 checkpoints were trained, and we evaluated each checkpoint for 5 iterations. For each checkpoint, we calculate the average accuracy and standard deviation, then plot the mean of the accuracies and standard deviations over all 5 checkpoints. We repeat this process for Flickr30K, using the same number of epochs, checkpoints, and model of GPUs. We again use the Adam optimizer with a learning rate of 1e-5, but with a weight decay of 0.02. We record accuracy scores at 1, 5, and 10 for both image-to-text retrieval and text-to-image retrieval on the two datasets. We present Acc@1 here, and leave Acc@5 and Acc@10 in Appendix D.1.
The following analysis focuses specifically on the case of Acc@1 for MSCOCO, but the general trend is also present in Flickr30K. As seen in Figure 2, fine-tuning allows for an improvement in accuracy over the zero-shot setting initially by  $8.06\%$  and  $9.47\%$  for  $\mathrm{CLIP} + \mathrm{DN}^*$  and  $\mathrm{CLIP}$  respectively in the text-to-image case, and  $6.02\%$  and  $4.88\%$  in the image-to-text case in the first epochs. But the fine-tuning performance gradually decays after that due to overfitting to the training set. At their highest peak,  $\mathrm{CLIP} + \mathrm{DN}^*$  achieves  $43.06\%$  accuracy compared to  $\mathrm{CLIP}^{\prime}s$ $42.39\%$  for text-to-image retrieval, and  $59.82\%$  compared to  $59.57\%$  for image-to-text retrieval. We observe for both image-to-text retrieval and text-to-image retrieval that  $\mathrm{CLIP} + \mathrm{DN}^*$  performs better than CLIP across all fine-tuning epochs by an average of  $0.80\%$  for text-to-image, and  $0.60\%$  for image-to-text case. A similar pattern is also observed in Flickr30K. Our experiments on the fine-tuning setting provide an affirmative answer to the last question we seek to answer, that DN* provides a non-trivial performance boost to both pre-trained and fine-tuned cross-modal representation models.
4.6.3 Number of Samples for Mean Estimation
We conduct experiments to examine the performance of DN* when different numbers of samples are used to estimate the mean. Using the ImageNet1K, Cifar100, SUN397, and Stanford Cars datasets,
we first test the retrieval accuracy of  $\mathrm{DN}^*$  by estimating the mean with the entire test set. We use the performance achieved here as an upper bound, because the mean is most reliable if it is directly calculated on all of the unlabeled testing samples. We compare it with the estimated means using 1, 10, and 100 random samples from the training set, and present our results in Table 6.
As shown in the table, decreasing the number of samples to just 100 from the training set results in an accuracy drop of no more than  $0.2\%$  compared to the upper bound, with an average of only  $0.09\%$ . We find that even decreasing the number of samples to as few as 10 does not significantly hurt the result. For 10 samples, there is an average drop in accuracy of  $0.14\%$ , with the greatest difference of no more than  $0.4\%$ . Moreover, for CLIP, TCL, and ALBEF, we find that estimating the mean with as few as 10 samples still improved accuracy over the base model by an average of  $1.96\%$  for Acc@1, and  $1.92\%$  for Acc@5. Furthermore, our results are generally stable despite the low number of sample points used to estimate the mean. Across 5 random samplings, the standard deviation of our accuracy is on average  $0.08\%$  for 100 training set samples, and  $0.22\%$  for 10 training set samples. Even in the extreme case of just 1 sample, we still observe a slight improvement of  $1.15\%$  over the base model averaged across Acc@1 and Acc@5, but with a slightly larger standard deviation of  $0.77\%$ . These ablation results show that DN* can generalize very well even in the setting of extremely scarce data, possibly because DN* only needs to know the global mean of the distribution and does not require fine-grained local details.
ImageNet1K	Cifar100	SUN397	Stanford Cars
Acc@1	Acc@5	Acc@1	Acc@5	Acc@1	Acc@5	Acc@1	Acc@5
CLIP [50]	61.0	87.4	63.9	88.7	56.1	89.4	58.6	90.9
CLIP + DN* (test)	61.8	87.9	65.1	89.4	57.3	90.1	58.7	90.7
CLIP + DN* (100)	61.7 ± 0.0	87.8 ± 0.0	65.1 ± 0.1	89.4 ± 0.0	57.3 ± 0.0	90.2 ± 0.1	58.6 ± 0.1	90.7 ± 0.0
CLIP + DN* (10)	61.6 ± 0.1	87.8 ± 0.1	65.1 ± 0.2	89.2 ± 0.1	57.1 ± 0.1	90.1 ± 0.1	58.3 ± 0.1	90.7 ± 0.1
CLIP + DN* (1)	60.6 ± 0.4	87.2 ± 0.2	64.1 ± 0.6	88.4 ± 0.6	56.9 ± 0.5	89.7 ± 0.2	56.5 ± 0.4	89.7 ± 0.5
TCL [68]	20.0	42.9	39.1	68.2	28.6	63.4	2.0	8.7
TCL + DN* (test)	25.4	49.4	42.2	71.1	31.9	66.9	2.4	9.5
TCL + DN* (100)	25.1 ± 0.1	48.9 ± 0.2	42.2 ± 0.0	71.1 ± 0.1	31.8 ± 0.1	66.8 ± 0.2	2.4 ± 0.0	9.5 ± 0.1
TCL + DN* (10)	25.3 ± 0.1	49.1 ± 0.2	42.2 ± 0.2	70.9 ± 0.1	31.9 ± 0.4	66.8 ± 0.5	2.3 ± 0.1	9.5 ± 0.0
TCL + DN* (1)	25.5 ± 0.3	49.2 ± 0.3	40.8 ± 0.7	69.4 ± 0.9	31.4 ± 0.5	65.8 ± 0.9	2.3 ± 0.1	9.5 ± 0.3
ALBEF [33]	37.3	65.8	38.5	65.7	45.6	81.5	25.0	61.0
ALBEF + DN* (test)	38.7	67.4	45.0	72.1	46.2	81.9	26.0	61.8
ALBEF + DN* (100)	38.7 ± 0.1	67.3 ± 0.0	44.8 ± 0.2	72.0 ± 0.2	46.3 ± 0.1	82.0 ± 0.0	26.0 ± 0.1	61.7 ± 0.2
ALBEF + DN* (10)	38.7 ± 0.1	67.3 ± 0.1	45.1 ± 0.6	72.2 ± 0.5	45.9 ± 0.3	81.7 ± 0.2	25.8 ± 0.3	61.3 ± 0.6
ALBEF + DN* (1)	37.3 ± 1.2	65.6 ± 1.4	44.3 ± 2.8	71.1 ± 3.1	45.4 ± 0.3	81.4 ± 0.6	25.0 ± 0.6	60.1 ± 1.2
Table 6: Ablation study on the effect of the number of samples to estimate the mean for ImageNet1K, Cifar100, SUN397 and Stanford Cars. Parenthesized is the number of unlabeled samples from the training set. (test) means that DN is conducted on the entire unlabeled test set. Mean and standard deviation are calculated based on 5 random samplings.
5 Limitations and Social Impact
The success of distribution normalization builds on access to a small amount (10-100) of unlabeled data from the test distribution. Although much cheaper and more easily accessible than the setting of fine-tuning or few-shot learning where more labeled data is needed, the use of unlabeled data can still be a potential limitation in cases when the test distribution is entirely unknown. This work builds on top of a recent line of work that consolidates vast amounts of data from public resources on the Internet to learn a joint representation space of images and text. Therefore, it is possible that DN might inherit or strengthen the social biases in those large visual-language models.
In this paper, we identify a mismatch between the pre-training objective of cross-modal representation models like CLIP and its downstream use with a direct dot product. To address this problem, we propose Distribution Normalization (DN) that not only displays a significant advantage over direct dot product in a wide variety of settings but is also straightforward to implement. However, though being extremely sample efficient, DN still requires estimating a mean separately for each distribution and it can be of interest if a universal mean can be found so that DN is more easily applied to a wider range of downstream applications. Future research can also explore the implications of DN to the contrastive training process of cross-modal representation models.
[69] Y. Yi, H. Deng, and J. Hu. Improving image captioning evaluation by considering inter references variance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 985–994, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.93. URL https://aclanthology.org/2020.acl-main.93.
[70] C. You, R. Zhao, L. H. Staib, and J. S. Duncan. Momentum contrastive voxel-wise representation learning for semi-supervised volumetric medical image segmentation. CoRR, abs/2105.07059, 2021. URL https://arxiv.org/abs/2105.07059.
[71] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models, 2022. URL https://arxiv.org/abs/2205.01917.
[72] X. Yuan, Z. Lin, J. Kuen, J. Zhang, Y. Wang, M. Maire, A. Kale, and B. Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6995-7004, June 2021.
[73] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer. Lit: Zero-shot transfer with locked-image text tuning. CoRR, abs/2111.07991, 2021. URL https://arxiv.org/abs/2111.07991.
[74] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. CoRR, abs/2010.00747, 2020. URL https://arxiv.org/abs/2010.00747.
[75] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
[76] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.