Test-Time Distribution Normalization for Contrastively Learned Visual-language Models
Authors: Yifei Zhou
Juntao Ren
Fengyu Li
Ramin Zabih
Ser-Nam Lim
Venue: NeurIPS 2023 poster (2023)
Keywords: contrastive learning
pre-trained visual-language models
zero-shot learning
test-time augmentation
TL;DR: A test-time plug-in module for pre-trained visual-language models that enhances zero-shot performance in a wide range of tasks.

Abstract: Advances in the field of visual-language contrastive learning have made it possible for many downstream applications to be carried out efficiently and accurately by simply taking the dot product between image and text representations.  One of the most representative approaches proposed recently known as CLIP has quickly garnered widespread adoption due to its effectiveness. CLIP is trained with an InfoNCE loss that takes into account both positive and negative samples to help learn a much more robust representation space. This paper however reveals that the common downstream practice of taking a dot product is only a zeroth-order approximation of the optimization goal, resulting in a loss of information during test-time. Intuitively, since the model has been optimized based on the InfoNCE loss, test-time procedures should ideally also be in alignment. The question lies in how one can retrieve any semblance of negative samples information during inference in a computationally efficient way. We propose Distribution Normalization (DN), where we approximate the mean representation of a batch of test samples and use such a mean to represent what would be analogous to negative samples in the InfoNCE loss. DN requires no retraining or fine-tuning and can be effortlessly applied during inference. Extensive experiments on a wide variety of downstream tasks exhibit a clear advantage of DN over the dot product on top of other existing test-time augmentation methods.

Test-Time Distribution Normalization for Contrastively Learned Vision-language Models
Yifei Zhou'
Juntao Ren" Corelll University j1r429ccormel1.om edu
Fengyu Li* Cornell University f13348corne11.edu
University of California, Berkeley yifei zhou@berkeley. edu
Ramin Zabih Cornell University rdz0cs cornell edu
Ser-Nam Lim University of Central Florida sernam@ucf edu
Abstract
Advances in the field of vision-language contrastive learning have made it possible for many downstream applications to be carried out efficiently and accurately by simply taking the dot product between image and text representations. One of the most representative approaches proposed recently known CL.IP (50) has garnered widespread adoption due its effectiveness. CLIP is trained with an InfoNCE loss that takes into account both positive and negative samples to help learn much more robust representation space. This paper reveals that the common downstream prac- tice of taking a dot product is only veroth-order approximation of the optimization goal, resulting in loss of information during test-time. Intuitively, since the model has been optimized based on the InfoNCE loss, test best-time procedures should also be in alignment. The question lies in how one can retrieve any semblance of negative samples information during inference in a computationally efficient way. To this end, we propose Distribution Normalization (DN), where we approximate the mean representation of a batch of test samples and use such mean to represent what would be analogous to negative samples in the InfoNCE loss. DN requires no retraining or fine-tuning and can be effortlessly applied during inference. Extensive experiments on wide variety of downstream tasks exhibit a clear advantage of DN over the dot product on top of other existing test-time augmentation methods. Our code is available at http:///hhhhtaoooo////////////lm
Introduction
has become an important area of research in recent years due its widespread success self self-supervised representation learning. The key idea behind contrastive learning is to group positive samples that share similar semantics (e.g., images resulting from applying different augmentations
Contrastive Learning 20
60 6 52
the same image), while separating semantically different negative samples (e.g., different images with different data augmentations). On the heels of this progress, contrastive learning has also been shown be very effective for creating a joint representation space between The CLIP approach proposed in [50] is at the forefront of this research effort and has been shown be the state-of-the-ar the-art multiple downstream tasks. The model is generally pre-trained on a large amount of image-text pairs that form the positive samples, while the negative samples are composed from random pairing of images and text. After the model is trained, similarities between text and images can be measured by simply taking a dot product between their corresponding representations. Such a paradigm has opened the doors to state-of-the-an performance
images and text [50 46 75
10
"Equal Contribution
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

in many cross-modal alignment tasks such as vero-shot classification [73 21 cross-modal retrieval
40 35 32].
and evaluation of machine-generater captions [17 22 30
marina with boets.
a) Contrastive training
b) Test-time distribution normalization
against a
Encoder
8/1 M/
Us
...
Mext
InfoNCE
Image Enoader
24
1
Prng
Figure I: An illustration of how Distribution Normalization better aligns with infoNCE pre-training objective. Pimg and plat are mean encoded representations for images and text respectively.
Despite CLIP's successes, we observe "mismatch" between the pre-training objective of such cross-modal representation models and their downstream usage. Most if not all of these models are trained taking into account of both positive and negative samples, of which one of the most popular optimization objectives the InfoNCE loss [58] 62 (4). The InfoNCE loss considers negative samples from the data distribution per positive sample and has been shown to be very effective for cross-modal representation learning. contrast, during test-time the similarity is often measured using the dot product between the image and the text representations, which does not utilize any information with regard to the data distribution.
This paper presents our findings that rectifying such misalignment can boost performance consistently across variety of downstream tasks. However, naively applying InfoNCE loss for downstream tasks requires iterating over all negative samples for every test sample, which is not a tractable operation. Instead, found that first-order approximation of the InfoNCE loss, consisting of simply subtracting the distribution mean in the representation space before taking the dot product, is able to achieve similar effect. We call this proposed approach Distribution Normalization (DN) DN is very easy to implement and does not require any retraining, fine-tuning or labeled data.
To investigate how this simple modification affects the performance of CLIP and its state-of-the-ar variants practice, we conduct extensive experiments on a wide variety of cross-modal alignment tasks, including zero-shot classification, cross moos- retrieval, and evaluation of image captions. In all the settings that we have studied, DN displays a clear benefit over direct dot product and can be combined with other existing test-time augmentation methods [55], which we will show in our experiments. Our contributions are: (1) We present an insightful analysis of why the dot product as a similarity measure is only zeroth-order approximation; (2) We propose DN to overcome this shortcoming by utilizing an approximation of the test-time distribution, and also provide guidance on how DN can be conveniently implemented in practice; (3) We provide extensive supporting empirical evidence and ablation studies that demonstrate the fidelity of DN for practical use.
2 Background & Related Works
Learning via Natural Language Supervision Several recent studies 7 12
34 35
leverage large-
scale datasets of image-text pairs to supervise vision-language pre-training (VLP) with a contrastive objective. CLIP [50] is one such example that is contrastively trained to create a joint vision-lan vision-language representation space, which has produced empirical success in downstream tasks such zero-shot
classification and cross cross-moda modal retrieval over previous state-of-the models [40 65 68 33). Since
CLIP's introduction, several other works have utilized similar contrastive objective create good dision-language representations and achieve state-of- tittee-ffthhe-e the-art performance on many downstream tasks involving datasets that the model was not tre-trained [33 11 26.. For example, ALIGN scales joint pre- training to minimally-filee filtered datasets (21). while ALBEF (3) and TCL [68] focus on learning more semantically meaningful multi-modal interactions. It is noteworthy that these SOTA

---

models all leverage the InfoNCE loss as their key training objective, and apply a simple dot product in downstream tasks.
for Downstream Applications Researchers have explored nu- merous methods to adapt CLIP for downstream tasks, capitalizing its remarkable transferability. CoOp [7] [6] employs learnable textual prompts, inspired by prompt learning [36], for few-shot training data, while VT-CLIP [49 incorporates visual-guides texts for enhanced vision-language alignment. However, these adaptations demand extra training data and fine-tuning resources, under- mining CLIP's core strength of efficient zero-shot recognition. To address this issue, CALIP [1] presents a training-free attention module to boost zero-shot image classification, and SuS-X 6 offers training-free name-only transfer by adapting CLIP with a curated support set. However, both CALIP and Sus-X are tailored for zeroshot classification only, while this paper, we propose DN, a training-fie frrengg-feeeeeee adaptation that is conceptually simpler, easier to implement, and applicable to wide range of downstream tasks.
CLIP Augmentation//Aaalatiee
Test-time adaptation and bist-time& augmentation strategies share a common goal adapt distrribuiiiiio scenarios test-time adaptation techniques make adjustments to the batch normalization layers to normalize
Test-time
53
43 during test time. Many
feature statistics [37 411 while test-time augmentation techniques 664 25
29 15 19 57
aim to ameliorate performance by taking some aggregation (55) of the model's prediction over batch of augmented images. Though DN is also applied during test-time, DN's task is to align the mismatch between training and inference objectives and not to deal with out of distribution test samples. This renders DN orthogonal TTA. In experiments, we use representative TTA model from Shanmugam et al. (55) and show that while TTA improves model performance, adding DN to TTA gives an additional increase.
3 Methodology
This section is organized as follows: in Section 3.2 we show that dot product is only zeroth-order approximation of the pre-training objective and how better approximation arises naturally from the InfoNCE objective, and in Section 3.3 we explain how this idea can be efficiently implemented in practice.
3.1 Notations
In our analysis, Ds is the training distribution and Dr is the test distribution, To. are images and text belonging to the same pair, I1, ...,,,, Iy and y1, Mn are random images and text in the same batch, d, ti are image and text encoders respectively, and 7 is temperature hyperparameter used in InfoNCE loss while training. Since in most applications, only relative comparisons of the distances matter (e.g., retrieving the best candidate caption describing the image), we consider two similarity measures S and S2 be equivalent if one monotone function of the other, namely
Vz, V1, x2, /2, S1 (I1, v1) 2 Si (32, U2) + S2(11, 91) 2 S2(22,12) U2).
(1)
The goal here is to find a test-time similarity measure that is similar to the train-time optimization goal through careful simplification of InfoNCE loss.
3.2 Analysis of InfoNCE Loss
We start from the original form of InfoNCE loss
58
62
74
(NC)(Dg) Dg Pyp Dg log
Ei Alo 90.1
Pe log
14 El ( 17
Since Mn (oee))
etro) T is usually quite small (< 0.1) due to the exponentiation, we can taylor expand the above term around 0 and discard second and higher order terms, resulting in
En e ((2))) B Br DD (
Σn
(Ds) Ro ER ( No ~Dz
(3)
3

---

which can be simplified to
(NC)(Ds) D D e (oo voool)
E (lee ((20) (4)
successful generalization the test distribution means that the infoNCE loss (Drr(Dr) is small when the expectation over training distribution for To. Yo, #1, y1 is substituted with the expectation over test distribution, where
LNCE(DT) Ren Ex..Bro.g0-- ~Ds BB (wo))
e oto (5)
3.2.1 Reduction to Zero-Order Dot Product
We argue that Eqn. 3 can be reduced to straightforwardly comparing the dot products. When we do not have of the test distribution Pr, the only thing that can do is to perform zeroth-orde
approximation of Eqn. [ The most intuitive way of doing so is to reduce the distribution of o(x1) and (111 to be 0 deterministically,
that:
(0) NNCE (Dr) 2n. Do +(uo))/
(6)
This shows that we can essentially just output e- - o(vo) as the distance between zo and yo- So we get the zeroth-order similarity:
Sto) (zo, yo) = ((x)) (g(0).
3.2.2 Efficient First-Order Approximation
The common practice of taking naive zeroth-order approximation can result in loss of some important information of the distribution of negative samples. On the other hand, computing Eqn for each pair Fp, No involves iterating over all samples in distribution Dr, and is computationally inefficient. We therefore turn to more efficient approximations using statistics of higher-order moments of the distribution. In our experiments, we found that this intuitive modification preserves most of the useful information as shown Appendix[D..
we can estimate the first moment of the distribution, i.e., the mean of the unlabeled distribution Mr and My for o(x,) and (v1),, we can perform a first-order approximation of the distribution with P()(z))) = [((xx)) = {r and P(v(w)) = [v(y)) =
pyy. so that P()(z) )), P(v()))
matches the true distribution in terms of the first-order moment. This simplifies Eqn. 5 to be:
(L) NCC = ED [e(ss)) (1y (1y-b((0))/-> + e 1ola-600 ((zo)) (h00))
(8)
Therefore, evaluate the distance of any Fp. yo. we can simply output e*(o) (M, v(Ho))/ ((()=)(2=))" w/y6o/77 without having calculate expectation by iterating through all samples in the distribution.
3.3 Distribution Normalization
Considering that (M (V--oo)))) (oxx--())))) is equivalent to taking an algorithmic convert this mean of (d (Ho))(()))/ and e(M= ((xl)) (owo)/ to simplify this further, which achieves algorithmic mean geometric mean (4y (((p)))//e(=oooo))))
similar effect averaging the distance measure of ((z))) (Mo (No-v(go))/r and ( (((()))))((())/// An additional empirical verification of the correlation between algorithmic mean and geometric mean in elts.- e(t=.-(=0)) v(o)/T is monotone function of (d(()) M (V(w)) (2)) after removing the square root and exponentiation, we can simply use (o( - ( (yy) to calculate the distance, resulting in a simplified
this step is provided in Appendix D.2 Since Veelao)"(uu-Wlin)
first-order similarity measure:
1 (a1 vo) = (6(zv) - (W(()) [k))
99)
Note that the new similarity measure is equivalent to subtracting the means immediately after the original representations are calculated, method we call Distribution Normalization (DN), after

---

which the similarity is just their dot product. Clearly, our proposal is extremely easy to implement when access the mean of the distribution of interest available.
Fortunately, we observed that the mean of the distribution can usually be robustly estimated from very few unlabeled samples as shown in Section 4.6.3 This is easily accessible in all the downstream tasks that we have considered in this paper and is much cheaper than fine-tuning where not only more data is required but also the data needs to be annotated. For example, in the task of cross-modal retrieval, the mean can be estimated from the gallery data and past queries (without knowing the correct answer to the queries); in the task of zero-shot classification, the mean can be estimated from small and unlabeled validation set; and in the task of image caption metrics, the mean can be estimated from the images to be captioned and the references provided.
Experiments
Our experiments are designed to answer the following questions: 1) whether our proposed DN can uniformly improve wide range of cross-modal alignment tasks for different kinds of cross-modal representation models and whether this gain is larger than that achieved by other zeroshot CLIIP augmentations, 2) whether DN can be used in parallel with other common test-time adaptation methods compatible with CLIP, 3) how robust DN is when only scarce, unlabeled data is available to estimate the mean from, and 4) whether DN can improve the performance of fine-tuned models in addition to pre-trained models.
4.1 Downstream Tasks
Cross-modal Retrieval includes two subtasks where images are used to query corresponding text and vice versa. Following [68] 33 for each subtask, we consider both the zero-shot setting and the fine-tuning setting on COCO 39 and Flickr30K [48). In the zero-shot setting, we directly apply the pre-trained model on the test set of each dataset while in the fine-tuned setting first fine-tune each pre-trained model on the training set of each dataset with the InfoNCE loss and apply the fine-tuned model to the test set. For each setting, we report Recall @k (k = 1,5) whereby the retrieval is considered a success if its corresponding image/text is in the top-k.
Zero-shot Classification is the task where we directly apply the pre-trained models to predict the category of the images in the test set. Each class is represented with a fixed textual template of "a photo of {class name).", and the corresponding representation is obtained by encoding those sentences with the text encoder of the pre-trained model. For each image, we encode it with the image encoder and output the top-k most similar class representations as predictions. We benchmark all
methods with top-1 and top-5 accuracy on ImageNetIK [9] C0far100 28 SUN397 67], Stanford Cars 27 Caltech 101 [31, and Flowers102 [4 datasets.
Image Captioning Metric is the task that measures the correlation between human ratings and similarity metrics on machine- machine-generated captions. Following [17], we consider both the reference- setting and the eeference-based setting. In the reference-free setting, the metric does not have access to numan-written example captions (references) and has to give a score based on the source image and the generated caption alone while in the sitrence-bbsed setting the metric can additionally take ad advantage of the information from the provided references.
4.2 Baselines
This paper compares the augmentations of Distribution Normalization and other baselines on CLIP [50], ALBEF (33). and TCL 168). More details of the base representation models can be found in
Appendix B In particular, the following are the main basclines being examined in the experiments.
TTA [55] aims to better adapt vision encoders to out-of-distribution test samples by augmenting test images and taking weighted reduction of the various augmentations. However, since DN is meant for the zero-shot setting, training these test-time augmenting layers is not applicable in our setting. Instead, we apply the standard test-time augmentation policy from Shanmugam et al. (55) which consists of a combination of flips, crops and scales, and take a mean over the model outputs on these augmented images.
CALIP 14 replaces CLIP's dot-product similarity with a parameter- free attention module. improves performance on zero-shot image classification by enabling visual and textual representations
5

---

to better explore cross- moosl features. This method results in improved adaptive zero-shot alignment by blending images with textual-aware signals. We implement CALIP using their official code and report our results on the "ViT-B/322--bas based CLIP architecture, consistent with our approach in other experiments. Since the original paper focuses solely on zero-shot image classification. CALIP serves as our baseline for this task.
TPT [56] is another CLIP zeroshot method that performs prompt tuning each single test image to promote the model's prediction consistency on different augmented views of the single test image. TPT is mainly proposed for classification tasks and it is unclear how it can be used for general cross-modal alignment tasks. We implement TPT using their official code and report our results on the VVTT-/32"-bbase CLIP architecture, consistent with our approach in other experiments. Importantly, TPT is much more computationally demanding than vanilla CLIP dot product because requires taking gradient steps for each single test image over its 64 different augmentations, while other methods do not suffer from such computation issues.
DN is our proposed plug-in augmentation using Eqn Estimated image mean and text mean are respectively subtracted from image and text embeddings immediately after the embeddings are calculated from base encoder models. Dot products are then taken between text and image representations as the similarity. Different ways estimate the image mean and text mean from few validation samples or references in different downstream tasks are detailed in the following sections.
DN Considering the motivation for DN is to better align the downstream similarity with the pre-training goal, it might introduce instability when there is a significant mismatch between the pre-raining and testing distribution. We therefore examine this variant called DN* where the only difference from DN that DN* averages the similarity output by DN and a vanilla zero-order dot product for improved robustness.
4.3 Cross-modal Retrieval Results
We conducted experiments on both image-to-text and text-to-image retrieval on Microsoft COCO 39 and Flicker30K [48] (see Appendix [.T for datasets details). For all the retrieval tasks, we estimated the mean with 100 random unlabeled samples from the validation set and calculated standard deviations and average recalls with 5 random seeds.
The results for both datasets are presented in Table for the zero-shot setting. For all three base models, adding DN almost always improves recall for both zero-shot image-to-text and text-to-image retrieval tasks, with an average 1.1 % increase in top-1 recall. This provides substantial evidence that DN is effective for both tasks even in datasets with diverse images like MSCOCO. The improvement can be as large as a 2.2% increase in top-1 image-to-text recall as we see for CLIP on Flickr30K. We also observe that, in general, adding DN brings larger improvement for top-1 recall than for top-5/10 recall, suggesting that DN is particularly suitable for distinguishing similar samples.
While adding test-time augmentation techniques to CLIP are effective in themselves as shown in Table we also show that adding DN or DN* on top of TTA techniques further improves retrieval accuracy. Specifically, CLIP+TTA+DN achieves another average 1.3% improvement over CLIP+TTA. similar improvement is achieved by CLIP+TTA+DN* as well. However, noteworthy that although CLiP+TTA+DN and CLIP+TTA+DN* achieves similar average improvement over CLIP + TTA (1.3% and 1.2% respectively), the improvement of CLIP+TTA+DN* has smaller standard deviation (0.8% compared to 1.3%), suggesting that averaging DN with direct dot product indeed improves the stability.
4.4 Zero-shot Classification Results
Table 2 gives the zero-shot classification results for all three base models and their variants with DN on ImageNetIK [9]. Cifar 100 28). SUN397 [67). Stanford Cars [27]. Caltech 101 [3I], and Flowers 102 44 (see Appendix C.2 for datasets details). In these experiments, we perform DN before the
final layer normalization of image/text encoders.
DN consistently enhanced the performance of all three base models across six datasets with average 4.4% increase in top-1 accuracy. It demonstrates effectiveness not just in general-pune purpose vision datasets like ImageNet, but also in specialized datasets such Stanford Cars, where average accuracy increases by 1.5%. Specifically, CLIP+TT++DN/DN= is the best variant for a CLIP base model, signifying simultancous benefits from DN and TTA. While CLIP+TTA+DN and CLIP+TTA+DNE

---

MSDOCOO
FlidefOKK
Imsge
Test
Test -Ingge
ROF1O
Itees
RerTo
R0F1
1OF1O
CLIP 50
32-4
76.0
84.5
30.2
56.1
06.4
X1.3
95.0
95.6
62.7
59.0
92.0
55.9
77-5
65.5
32.1
68-3
83.3
96-5
Q8.4
65.2
87.9
42:0
39.420..1
CLI TTA DN
2.0
23.5 =000
0.0
70.12:00
0.
07.000.0
$2.9
CLIP DN
517-001
75.830.1
$4.300.1
35, 35.0:5.0
$84-0011
00.450.
83.510.2
96.620.1
98.626.1
66.2:011
68.240.1
91.S40.1
CLIP DN
320:0011
76,420.1
S4.960.1
18.1:01
$7.4.00
08.2240
80540.1
95,2600
18.0e0.1
642-02
57.540.3
W1,150.0
61.3
01.5
50.6
71.8
M3.3
96-9
58.1
SUGER1
85
924200.
433400
718-011
S14310
17.540.5
969202
292-022
01.12011
42
94,456.1
88.0:00
ALBER D3
62.5
$6,9
92.2
40.2
68.4
78.9
78.2
05.5
97.9
30.0
54.5
90.6
AIREF= -DN
60F-2
85.840.1
92:100.1
46
72.
82.000.0
H06-00.1
96.2-0.1
9K316611
601-00
87.0-0.1
92.310.1
ALBEF IDN
630ER1
8600:00
92.5
42 82001
70:3:00
50.7240.0
79.266.11
96.36000
98,0:0000
62 22:
56. 56.130011
91, 91.960.1
Table 1: Cross-modal retrieval performance on MSCOCO and Flickr30K in the zero-shot setting. Means for DN are estimated using 100 random unlabeled validation samples. Average recalls and standard deviations are calculated with 5 random seeds.
display similar improvements over CLIP+TTA (0.82% and 0.9% respectively), CLIP+ITA+DN* again exhibits smaller standard deviation (0.22% vs. 0.76%). indicating improved stability. Further- more, DN consistently surpasses CALIP in all model-dataset combinations, despite CALIP's slight higher computational cost from incorporating variant CLI++TTA+DN* we find that they achieve comparable top-1 accuracy across all datasets while TPT suffers from significant drop in top-5 accuracy, potentially due to poor calibrations in performing gradient steps on each single test image. It is also important to note that DN is much more computationally efficient with only an overhead of subtracting the mean while TPT requires performing gradient steps on 64 different augmented view of each test image.
attention module. Comparing TPT with our best
Notably, TCL saw the largest gain of 7% on average in top-1 accuracy. On ImageNetl K particularly, TCL's top-1 accuracy is improved from 20.0% 30.5% 5% by adding DN. We believe the fact that TCL. and ALBEF are pre-trained on a smaller dataset (4M& & 14M images) than CLIP (400M images) resulted in TCL and ALBEF getting larger improvement than CLIP, DN may have helped ease the instability when the model is not adequately trained. All these improvements are achieved with single layer that incurs minimal computational overhead. Lastly, comparing DN and DN for TCL. and ALBEF, we found that DN consistently gives a greater improvement than DN potentially because averaging with vanilla dot product halves the benefits from DN when the base models instable due to inadequate training.
InsersNaik
Clieii0)
SUNNY
Sunfoel Can.
Calieds
102
Acco1
Acc
Accdil
Acci5
Accil 1
Acc
AucBi
Acce5
Acct
Acc05
Accil
Accil
CLIP 550 CALLI [4]
67.4
43.0
85.7
36.1
59.1
16.0
141.
87.3
15.0
01.1
83.8
61.3
68.
0.0
56.1 0.1
80.3
58.7
96-1 0.0
88.5 8:0.1
95,1
83 1
67.1
05.2
59.4
51.2
96.0
64.5
CLIP TTA
CLIP TTA
67.5 011
90.7 0.1
023=0.2
01.5=2001
833 0:1
$0.42:00
+:1
CLIP TTA IN
65.22:0.
58.9 1:001:000 0.0
62.1::0011
90.7 0.0
5..1
9022200 0.0
92.2:260 6.1
0.0
55.11:00 0.0
0.0
$4.5 1:00
CLIP DN
61.02:011
5 0.1
66.5 0.02
50.2
0.2
67.9::0.1
00.5
6..
00.5-1...J 0.1
92.0
9L.11.0.1
627
56.1 2.11
CLIP- DN
61.7+:01
67.8 4: 01
05.1
0.0
00.11-00
57.3 0.0
10. 0.1
56.6 6.3
90T 0.0
42.8
95.1
5:00
029
5431-011
42.0
65.1
58.5
12.6
ML-DD DN
0.
54.3
9.2
011
0.1
56.
70.8
0.3
266 0.1
0.1
66.7001
$1.8
28.5
TCL - DN
1:111:0
1..9...0 0.2
12.2 8
71.1 0.1
$1..200.1
06.8 0.2
2.6 0.0
0.5.0.11
05.9.000 0.1
81.1
1:00
272
1...100
ALBEF 53
87.3
62.8
08.5
65.7
45.6
81.5
26.0
61.0
00.0
80.3
25,9
49.1
AIBER-4
0.2
68.5
50.2
0.2
46. 0.5
82.4
26.0
11.5 0.5
$8.
1 02
ALBEF
673
02
020
0.3
8200
260 0.
07.7#00
07
200
51.9411
Table 2: Zero-shot classification performance
ImageNetlK, Cifar SUN397, Stanford Cars, Caltech 101, and Flowers 102. Means DN are estimated using 100 random unlabeled validation samples. Average accuracies and standard deviations are calculated with random seeds.
4.5 Image Captioning Metric Results
Inspired by the success of CLIPScore 17 as a metric for image captioning, we conduct experiments to test whether DN and DN can help various cross-modal representation models align better with human judgments. To compare the performance of different methods, we report the Kendall's 75 or To
coefficients against human judgments on Flickr8k-expert, Flickr8k-cf [18] and THumb [4]] in Table
3. following the standard practice in [17. We also report the resulting accuracy for caption-caption
preferences on Pascal-50S [1 in Table 7 Details about these datasets are given in appendix C.3 In
the based setting, for cef,-ref, we follow
I7 in using an averaged score given by:
CLIP-ref(Zo: Vo) (mea((((z)) Tyyo),, max (x)6))
cER
7

---

where H-mean is the harmonic mean and R is the set of provided references. On the other hand, for distribution normalized CLIP + DN-ref, we use an algebraic mean for simplicity:
CLIP + DN-ref(zo, yo) (-mean(S(1) y0), max(((()) M4y) (1b(0))))))) Hy)),
where A-mean is the algorithmic mean, the set of provided references, My is the mean of text representationss.add S is defined in Eqn. [
In Table 3 we report the results on Flickr8K- Expert, Flicker8K-CF, and THumb for all three base models. Some methods, such as the BLEU score, are not applicable the reference- setting and are thus only compared with in the reference- based setting. The results strongly support that adding DN significantly improves the correlation for all three base models on all dataset in the reference- ticular, CLIP + DN improves CLIP by 2.9% on Flickr8k-expert, 1.1% % on Flickr8k-cf and 3.6 % on THumb, and CLIP DN performs compa- rably. In the reference- based setting, to give better context, we also provide the performance of other reference-based SOTA metrics that are not contrastively trained for comparison. We find that CLIP and CLIP-ref outperform all of them. However, by simply adding a DN layer, CLIP-ref + DN again yields an average 1.3% gain across the board over previous leaders. Fur- thermore, analogous to the results in cross-modal retrieval and zeroshot classification, we found that TTA is effective improving on results of image captioning metric (CLIP TTA Has on average 0.6% improvement over CLIP) but TTA combined with DN gives another 1.9% boost on average. A similar pattern also observed in our experiment results Pascal-50S in Appendix 7 Our main results on cross-modal retrieval, zero-shot classification, and image caption metrics provide extensive evidence that our proposed DN is widely applicable modification that improve the performance of cross-modal representation models across a variety of downstream applications.
Flickrll-expeet
THumb
TE
CLIP [7]
51.4
34,3
19.9
Ref-
CLIP TTA
51,9
34,7
20.7
TTA
CLIP DN
53.5
35.5
22.7
CLIP DN
54.3
35.4
23.5
CLIP DN
53.2
35.1
22.2
TCL (8
31.0
DNN
42,0
14,4
setting. In par-
TCL. DN
36.0
23.3
11.1
ALBEF 33
24.9
15,4
0.9
ALBEF DN
34,8
21.8
5.5
ALBEF
BLEU-1
32,3
17.9
11.1
BLEU-4
30,8
16.9
6.9
CIDE: [3
43,9
24.6
13.8
NUBIA 23
49.1
bused
VILBERTScOre
CLIP-re
17
50.1
$3.00
36.4
24.7
CLIP DN-ref
55.3
37.0
25,8
DN
54,3
36,9
26.2
Table 3: Image captioning metric results on Flickr8k- Expert, Flickr8k-CI: and THumb.
MSCOCO 12I: Accurncy @ Top1
M MSCOCO 12T: Accuracy Top1
PLICKR T21: Accuracy @ Top1
FLICKR 12T: Accunacy @ Tep1
60
68
72
40
50
BG
Accuracy
38
82
36
55
60
CLIP Tup1 CLIPIYY Dupl
CLIP Eept CLIP BN Top1
64 CLIP Topt CLIP -DN Tup1
CLIP Ppp CLIP DNN Top!
30
10
Epochs Fineluned
Epocha Fineluned
Epocha Finelused
Epocha Fineluned
Figure Comparison the effects of fine-tuning between CLIP and CLIP + DN* for Acc@ 1. Results on MSCOCO's 5k test set are shown to the left, and Flickr30K's s 1K test set are shown to the right. For each set, image-to-text retrieval (left) and image-to-text retrieval (right) are reported. trained, we find its average accuracy and standard deviation with
For each of the 5 checkpoints another 5 iterations random sampling for mean estimation, and plot the mean of these accuracies and standard deviations from independently fine-tuned checkpoints.
4.6
Ablation Study
4.6.1
Results on Different Variants of CLIP
In addition to the CLIP-B32 model used in most experiments, in Table 4 and Table 5 we investigate into the performance of DN* on other popular variants of CLIP models, including CLIPB16. CLIPL14

---

[50] that come with more parameters and smaller patch size, and CLIPB32- [54] that is trained on a larger image-cattion pair dataset. The results show that DN* brings a similar performance gain for CLIP with more parameters or trained on a larger dataset, implying the universal applicability of our proposed method irrespective the size of the CLIP model.
MSCOCO (5K test set)
Flickr30K (IK test set)
Image
Text
Text
Image
Image
Text
Image
R@1
R@5
R 10
R@1
R@5
R@:0
R@1
R65
RB 10
R@1
R@5
R@
CLIP(B16) + TIA
53.6
77.5
85.1
33.8
58.7
69.1
85.4
97.9
99.1
66.6
89.0
93.77
CLIP(B16) + TIA + DN*
54.6
78.5
86.1
35,7
60.7
70.8
87.3
98.0
99.6
69.3
90.2
94,6
CLIP(1.14) + TTA
57.7
80.1
87.8
36.8
61.3
71.2
88.4
98.9
99.9
69.9
90.6
94.8
CLIP(L14) TTA + DN
58.8
81.3
88.4
38.6
63.1
72.9
89.3
98.8
99.8
72.1
91.7
95.5
CLi + TIA
58.5
80.9
88.1
10.01
63.8
76.0
85.8
96.7
98.9
71.1
91.4
94.8
CLIP(B32-Laion)- TTA+ DN
82.3
88,9
40.8
66.5
76.4
86.6
97.1
71.7
91.6
91.7
Table 4: Cross-modal retrieval performance on MSCOCO and Flickr30K in the zero-shot setting for more CLIP variants. Mcans for DN* are estimated using 100 random unlabeled validation samples. Average recalls are calculated with 5 random seeds.
ImageNett
Cifar100
SUN397
Stanford Cuns
Caltech 101
Flowers
Acceh1
Accel
Acceh1
Axcd1
Acc-t1 A0c
Acc
CLIPBI6)
07.7
63.3
98.1
CLIP(B16) à TTA DN
92.0
71.1
92.2
61.9
92.4
64.6
93.9
96.6
70.0
85.0
C
F3.1
83.4
77.6
91.0
62.1
92.5
76.3
97.7
86.0
97.3
728
89.
CLIPLL14+ TIA DN
74.2
94.1
80.4
95.4
63.8
93.3
77.2
97.8
86.3
97.1
74.0
89.1
LLP-IB3-Laiio)- + TTA
66.9
69.4
75.7
$1.0
63.9
93.5
87.1
99.2
87.3
97.7
70.3
86.4
CLIP(E33-Laio TIA DN
67.2
90.3
76.2
94.3
64.3
93.7
87.1
99.1
86.8
97.2
7L.
85.8
Table 5: Zero-shot classification performance on ImageNet1K, Cifar100, SUN397, Stanford Cars, Caltech 101, and Flowers 102. Means for DN are estimated using 100 random unlabeled validation samples. Average accuracies are calculated with random seeds.
4.6.2 Effect of Fine- Tunnng
We also perform ablations on the effects of DN* on fine-tuned models. We fine-tune CLIP on the MSCOCO training set, for total of 10 epochs on 4 Nvidia 2080Ti. We use the Adam optimizer with a learning rate of le-5 and a weight decay of 1. We then evaluated each checkpoint on the same MSCOCO test split used in Table D and estimate the mean for CLIP + DN* using 100 randomly sampled validation points. A total of 5 checkpoints were trained, and we evaluated each checkpoint for 5 iterations. For each checkpoint, we calculate the average accuracy and standard deviation, then plot the mean of the accuracies and standard deviations over all 5 checkpoints. We repeat this process for Flickr30K, using the same number of epochs, checkpoints, and model of GPUs. We again use the Adam optimizer with a learning rate of le-5, but with a weight decay of 0.02. We record accuracy scores at 1, 5, and 10 for both image-to-text retrieval and text-to-image retrieval on the two datasets. We present Acc@) here, and leave @ 5 and Ace@ 10 Apendix [D.]
The following analysis focuses specifically on the case of Acc@I for MSCOCO, but the general trend is also present in Flickr30K. As seen in Figure 2 fine-tuning allows for improvement in accuracy over the zero-shot setting initially by 8.06% and 9,47% for CLIP + DN* and CLIP respectively in the text-to-image case, and 6.02% and 4.88% in the image-to-text case in the first epochs. But the fine-tuning performance gradually decays after that due to overfitting to the training set. At their highest peak, CLIP DN achieves 43.06% accuracy compared to CLIP's 42.39% for text-to-image retrieval, and 59.82% compared to 59.57% for image-to-ter to-text retrieval. We observe for both image-to- -mage-to-exxt retrieval and text-to- to-image retrieval that CLIP + DN* performs better than CLIP across all fine-tuning epochs by an average of 0.80% for text-to-image and 0.60% for image-to-text case. A similar pattern is also observed in Flickr30K. Our experiments on the fine-tuning setting provide an affirmative answer to the last question we seek to answer, that DN provides a non non-trivial performance boost to both pre-trained and fine-tuned cross modal representation models.
4.6.3 Number of Samples for Mean Estimation
We conduct experiments to examine the performance of DN* when different numbers of samples are used to estimate the mean. Using the ImageNetIK, 100, SUN397, and Stanford Cars datasets,

---

we first test the retrieval accuracy of DN by estimating the mean with the entire test set. We use the performance achieved here as an upper bound, because the mean is most reliable if it is directly calculated on all of the unlabeled testing samples, We compare it with the estimated means using 1, 10, and 100 random samples from the training set, and present our results in Table 6
As shown in the table, decreasing the number of samples to just 100 from the training set results in an accuracy drop of no more than 0.2% compared to the upper bound, with average of only 0.09%. We find that even decreasing the number of samples to as few as 10 does not significantly hurt the result. For 10 samples, there is an average drop in accuracy of O. 14%, with the greatest difference of no more than 0.4%. Moreover, for CLIP, TCL, and ALBEF, we find that estimating the mean with as few 10 samples still improved accuracy over the base model by average of 1.96% for Ace@1 I, and 1.92% for Aco@5. Furthermore, our results are generally stable despite the low number of sample points used to estimate the mean. Across 5 random samplings, the standard deviation of our aceuracy on average 0.08% for 100 training set samples, and 0.22% for 10 training set samples. Even in the extreme case of just sample, we still observe a slight improvement of 1.15% over the base model averaged across Ace@1 and Ac, but with slightly larger standard deviation of 0.77%. These ablation results show that DN can generalize very well even in the setting of extremely scarce data, possibly because DN* only needs to know the global mean of the distribution and does not require fine-grat fine-grained local details.
ImageNetIK
Cifar100
SUN397
Stanford Cars
Acc@1
Acc@5
Acc@1 I
Acc@
Acc@1 I
Acc@5
Acc@1 I
Acc@
CLIP 150
61.0
87.4
63.9
88.7
56.1
89.4
58.6
90.9
CLIP + DN (test)
61.8
87.9
65.1
89.4
$7.3
90.1
58.7
90.7
CLIP+ (100)
61.7
87.8 0.0
65.1
+
89.4
$7.3
90.2
58.6 1
90.7
+
CLIP + DN* (10)
61.6+0.11
87.8 +-0,1
65.1
0.2
0.1 89.2
57.1
0.1
90.1
+.1
58.3
+ 0.1
90.7
+.11
CLIP + DN (1)
6..6
8.2
64.1
0.6
88.4 + 0.6
56.9
+ 0.5
89.7
0.2
56.5
0.4
89.7
0.5
TCL [68]
20.0
42,9
39.1
68.2
28.6
63.4
2.0
8.7
TCL + DN (test)
25.4
49.4
42,2
71.1
31,9
66.9
2.4
9.5
TCL DN (100)
0.1
48,9
0.2
42.2 0.0
71.1
31.8 0.1
66.8
0.2
2.4
TCL DN (10)
0,1
49.1
0.2
0.2
70,9
0.1
66.8
0.5
0.1
9.5 0.0
TCL + DN (1)
0.3
49.2: 0.3
40.8 0.7
69.4 0.9
31.4 0.5
65.8 0.9
2.3 2.1
9.5 0.3
ALBEF
37.3
65.8
38.5
65.7
45.6
81.5
25.0
61.0
ALBEF DN (test)
38.7
67.4
45.0
72.1
46.2
81.9
26.0
61.8
ALBEF DN (100)
0.1
67.3
0.0
44,8
72.0
46.3
0.1
82.0
26.0
6.22
ALBEF DN* (10)
38.7
0.1
67.3
0.1
45.1
+ 0.6
0.5
45,9 1 0.3
81.7
0.2
25.8 - 0.3
61.3
0.6
ALBEF DN (1)
37.3 4: 1.2
65.6 4: 1.4
44.3
1: 28
71.1
3.1
45.4
0.3
81.4
0.6
25.0
0.6
60.1
1.2
Table 6: Ablation study the effect of the number of samples estimate the mean for ImageNet1 K, C0f SUN397 and Stanford Cars. Parenthesized the number of unlabeled samples from the training set. (lest) means that DN is conducted on the entire unlabeled test set. Mean and standard deviation are calculated based on random samplings.
5 Limitations and Social Impact
The success of distribution normalization builds on access to small amount (10-100) of unlabeled data from the test distribution. Although much cheaper and more easily accessible than the setting of fine-tuning or few-shot learning where more labeled data is needed, the use of unlabeled data can still be a potential limitation in cases when the test distribution is entirely unknown. This work builds on top of a recent line of work that consolidates vast amounts of data from public resources on the Internet to learn a joint representation space of images and text. Therefore, it is possible that DN might inherit or strengthen the social biases in those large visual-language models.
6 Conclusion
In this paper, we identify a mismatch between the pre-training objective of cross-modal representation models like CLIP and its downstream with a direct dot product. To address this problem, propose Distribution Normalization (DN) that not only displays a significant advantage over direct dot product in a wide variety of settings but is also straightforward to implement. However, though being extremely sample efficient, DN still requires estimating a mean separately for each distribution and it can be of interest if universal mean can be found so that DN is more casily applied to wider range of downstream applications. Future research can also explore the implications of DN to the contrastive training process of cross-mode representation models.
10

---

References
[1] Anderson, Fernando, M. Johnson, and Gould. SPICE: semantic propositional image caption evaluation. CoRR, abs/1607.08822 2016. URI. httt:://axxiv.org/aaae
08822
[2] S. Banerjee and A. Lavie. METEOR:
An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and
Extrinsic Evaluation Measures for Machine Translation and/or Summarization. pages 65-72.
Ann Arbor, Michigan, June 2005, Association for Computational Linguistics. URL https
/aaclanthology org/ wrgg//0-0000000 -000
[3] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020. URL https://///xx org/ abs/2002.0577
[4] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big self-supervised models are strong semi-supervised learners. CoRR, abs/2006.10029, 2020. URL https://arxiv org/abs/2005 10029
[5] X. Chen, H. Fan, R. B. Girshick, and K. He. Improved baselines with momentum contrastive learning. CoRR, abs/2003.042)) 2020. URL
hbtpr///ab///00...////// 04297
[6] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. CoRR, abs/2003.04297, 2020. URL https::/arxiv..ortt http://aaxiiv..rrg/////0000. 2bs/
[7] Y. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, UNITER: learning universal image-text representations. CoRR, abs/1909. 17740, 2019. URL http:
//arxiv. farg/abe abs/ 1909. 11740
[8] Y. Cui, G. Yang, A. Veit, X. Huang, and J. Belongie. Leariing to evaluate image captioning. CoRR, abs/ dtbs/1806.0mm 18111880602222 2018. URL http://arxiv orgg/bbs/8000 http::///rrii..orrrab//1886666 06422
[9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Puttern Recognition, pages 248-255, 2009, doi:
(19/CCVP.20099522))
[10] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and W. Vaughan, editors, Ad- vances in Neural Information Processing Systems, volume 34, pages 8780-8794. Curran Associates, Inc., 2021. URL htttps://proceeddiie wewpiiinnnnnnnuuiiiee nwwrinuuri.... http./eerr.r.....ppp/////000101 cc/papper/202//file/
[11] Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler. VSE++: improved visual-semantic embeddings. CoRR, abs/1707.05612 2017. URL http://rxi:: wwt.:////////.......///. 00662
[12] Z. Gan, Y. Chen, L. Li, C. Zhu, Y. Cheng, and J. Liu. Large-scale adversarial training for vision-and- lammauuge representation learning. CoRR, abx/2006.06195, 2020. URL https
//arxiv. www.//gg////////o/oo//o.oo..e/// absa 20060 /00000000000005. 06195
[13] Grill, Strub, F. Altché, C. Tallec, H. Richemond, E. Buchatskaya, Doersch, B. Á. Pires, Z. D. Guo. M. G. Azar. B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent: A new approach self-supervised learning. CoRR, abs/2006.07733, 2020. htttttt///axxii www.aiwww....oraeeeeeo hwwrg/abs/2006.000
[14] Z. Guo, R. Zhang, L. Qiu, X. Ma. X. Miao, X. He, and B. Cui. Calip: Zero-shot enhancement of clip with parameter-free attention, 2022.
[15] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[16] K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick. Momentum contrast for unsupervised visual representation learning. CoRR, abs/1911.05722. 2019. URL http://///a/ //appvv.//////////a///a www.wwwwgwwwa////..............p org/abs/1911 05722

---

[17] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi. Clipscore: A reference-free evaluation metric for image captioning. CoRR, abs/2104.08713 2021. URL https://arxiv org/abs/2104.000
[18] M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as ranking task: Data, models and evaluation metries. Journal of Artificial Intelligence Research, 47:853-899, 2013.
[19] A. G. Howard. Some improvements on deep convolutional neural network based image classification. arXiv preprint arXiv: 2013.
[20] A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F. Makedon. survey on contrastive self-supervised learning. CoRR, abs/2011.00362, 2020. URL 2011 20362
[21] C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. V. Le, Sung. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. CoRR, abs/2102.05918, 2021. CtLthttpss://// //arxivv org/gogg/o abs/2102 05918
[22] M. Jiang, Q. Huang, I.. Zhang, X. Wang, P. Zhang, Z. Gan, J. Diesner, and J. Gao. Tiger: Text-to-image grounding for image caption evaluation. CoRR, abs/1909.02255, 2019. URL
http://arxiv... org. wwtrg/abs119900 wtw.b///99...0055555 02050
[23] H. Kane. M. Y. Kocyigit, A. Abdalla, P. Ajanoh, and M. Coulibali. NUBIA: NeUral based interchangeability assessor for text generation. In Proceedings of the Ist Workshop on Eval- uating NLG Evaluation, pages 28-37, Online (Dublin, Ireland). Dec. 2020. Association for Computational Linguistics. URL https: |tttstps::/a httpss///aaaaathlooogo.o oro.org////0 22000/2020..am evainlgeval-1..4
[24] Kasai, K. Sakaguchi, L. Dunagan, Morrison, R. L. Bras, Y. Choi, and N. A. Smith. Transparent human evaluation for image captioning. arXiv preprint wrXiv:2/11.08944 2021.
[25] I. Kim, Y. Kim, and S. Kim. Learning loss for test-time augmentation. Advances in Neural Information Processing Systems, 13:4163-4174 2020.
[26] W. Kim, B. Son, and I. Kim. Vilt: Vision-and-language transformer without convolution or region supervision, 2021.
[27] Krause, M. Stark, Deng, and L., Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D) Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
[28] A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[29] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[30] H. Lee, S. Yoon, Dernoncourt, D. S. Kim, T. Bui, and K. Jung. ViLBERTScore: Eval- uating image caption using vision-and-langee languuge BERT. In Proceedings of the First Work- shop on Evaluation and Comparison of NLP Systems, pages 34-39, Online, Nov. 2020. Association for Computational Linguistics. doi: 10. 00.186633/112222eevvla./. https://aclaartoooogyy org/ 2020. ev.14n1p11.4
URL
[31] F-F. Li, M. Andrecto, M. Ranzato, and P. Perona. Caltech 101, Apr 2022.
[32] G. Li, N. Duan, Y. Fang, D. Jiang, and M. Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. CoRR, hiss1908.06066 2019. http 7/arxiv. wwrrxii.org//a. abs///////// abs/1908. 09060006886666...
[33] J. Li, R. R. Selvaraju, A. D. Gotmare, S. R. Joty, C. Xiong, and S. C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. CoRR, abs/2107.07651. 2021. UtLptttps: http:///aaxxvvvv/e org/
U http://.orggaab/////e/eeeee aa///1/bbb///// 07651
[34] W. Li, C. Gao, G. Niu, X. Xiao, H. Liu, J. Liu, H. Wu. and H. Wang. UNIMO: towards unified unified-modal modal understanding and generation via cross- modal contrastive learning. CoRR, 154099, 2020. URL https: h org/ abs/ 2bss2201..em 15409
12

---

[35] X. Li, X. Yin. C. Li. P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, Y. Choi, and J. Gao. Oscar: Object-semanties aligned pre-traming for vision-languee tasks, CoRR, abs/2004.06165, 2020. URL httLtttss///xiiii http://.rrr/abb//00.00000 06165
[36] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.
[37] Y. Li, N. Wang, J. Shi, J. Liu, and X. Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04777 2016.
[38] C.nY. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004, Association for Computational Linguisties. URL URL htttps::/aallaatttol..gg www.:oyyyooo/00000000000000
[39] T. Lin, M. Maire, J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, Dollár, and C. L. Zitmick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312 2014. URLhttpp: /wwpvv..vvvvvv org/ abs
awtpaa/s.....//. 1405 0312
[40] X. Lu, T. Zhao, and K. Lee. Visualsparta: Sparse transformer fragment-level matching for large-scale text-to-image search. CoRR, abs/2101.00265, 2021. URL.tttss://arxi org/ abs/2101.00265
[41] F. Maria Carlucci, L. Porzi, B. Caputo, E. Ricci, and S. Rota Bulo. Autodial: Automatic domain alignment layers. In Proceedings of the IEEE international conference on computer vision, pages 5067-5075.
[42] N. Moshkov, B. Mathe, A. Kertesz-Farkas, R. Hollandi, and P. Horvath. Test-time augmentation for deep learning-base cell segmentation on microscopy images. Scientific reports, 10(1):1-7, 2020.
[43] Z. Nado, S. Padhy, D. Sculley, A. D' Ammour, B. Lakshminarayanan, and J. Snoek. Evaluat- ing prediction batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
[44] M.-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. Indian Conference on Computer Vision, Graphics and Image Processing. Dec 2008.
[45] K. Papineni. S. Roukos, T. Ward, and W.-J. Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[46] O. Patashnik, Z. Wu, E. Shechtman, D. Cohen- Or, and D. Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2095-2094, October 2021.
[47] X. Peng, K. Wang, Z. Zhu, and Y. You. Crafting better contrastive views for siamese representa- tion learning. CoRR, abs/2202.03277, 2022. URL
03278
[48] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. CoRR, /e/p/1550088877 2015. http: http:pp://aarii..oo/aab////////e 0
[49] L. Qiu, R. Zhang, Z. Guo, Z. Zeng. Y. Li, and G. Zhang. Vt-elip: Enhancing vision-language models with visual-guided texts, 2022.
[50] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. URL/https://are org/ abs/
[51] C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier. Collecting image annotations using amazon's 's mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's 's Mechanical Turk, pages 139-147, 2010.
[52] J. Robinson, C. Chuang, S. Sra, and S. Jegelka, Contrastive learning with hard negative samples. CoRR, abs/2010.04592. 2020. URL https: //trxiv//vv////////a//eaa/aaaav org/ aa/aa/as//000 20000 04592
13

---

[53] S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33:1 33:11539-11551, 2020.
[54] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
[55] D. Shanmugam, D. Blalock, G. Balakrishnan, and J. Guttag. Better aggregation in test-time augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1214-1223, 2021.
[56] M. Shu, W. Nie, D.-A Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models, 2022.
[57] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 15X.v:1409.1556, 2014.
[58] K. Sohn. Improved deep metric learning with multi-class n-pair loss objective. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL httplttttt//rrrceedtiiiiii pdf
[59] Y. Tian, D. Krishnan, and Isola. Contrastive multiview coding. CoRR, abs/1906.05849, 2019. URL http://arxiv...... org/ werggaa///ie a90/////b//1.///////// 19009996669999999
[60] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and Isola. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6827-6839. Curran Associates, Inc., 2020. URL http:ss//pprocceeinng.... neurips....pp...... http:///a...rrrppppcc////////// httll//c2222aeeee222200999666666
pdf
[61] V. Udandarao, A. Gupta, and S. Albanie. Sus-x: Training-free name-only transfer of vision- language models, 2023.
[62] A. van den Oord, Li, and O. Vinyals. Representation learmin with contrastive predictive coding. CoRR, abs/1807/ 1807003777 2018. URL UKL.htt:///arxvv..rr/aa///////11
[63] R. Vedantam, C. L. Zitmick, and D. Parikh. Cider: Consensus-based image description evaluation. CoRR. abbs/111..772 2014. URL/tttp:/ httt:://arriv.oorgabb///1111
[64] G. Wang, W. Li. M. Aertsen, J. Deprest, S. Ourselin, and T. Vercauteren. Alcatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing, 338:34-45, 2019.
[65] H. Wang, Y. Zhang, Z. Ji, Y. Pang, and L. Ma. Consensus-aware visual-semantic embedding for image-text matching. CoRR, abs/2007.0ss 008803, 2020. URL htt.atpss//axx.i..o//aba//////// 08883
[66] Wang and Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. CoRR, abs/2005.10242, 2020. URL htttttts///axii org/ abs/2005. 10242
[67] Xiao, Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485-3492, 2010. doi: 10. 109/CVPR.2010.55399
[68] J. Yang, J. Duan, S. Tran, Y. Xu. S. Chanda, L. Chen, B. Zeng. T. Chilimbi, and J. Huang. Vision-language pre-training with triple contrastive learning. 2022. URL lttth:tt///// :/apsii///xxvv org 2t/p//ab2/2222.110/
14

---

[69] Y. Yi, H. Deng, and J. Hu. Improving image captioning evaluation by considering inter refer- ences variance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 985-994, Online, July 2020. Association for Computational Linguistics. doi: 10.18653w112000.acccmm
URL https:://acaaaccooooggooooo/oo//
acl-mainn 93
[70] C. You, R. Zhao, L. H. Staib, and J. S. Duncan. Momentum contrastive voxel-wise representation learning for semi-supervised volumetric medical image segmentation. CoRR, abs/2105.07059, 2021. URL https: http:/aaxxvv....//////
07059
[71] Yu, Z. Wang, Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models, 2022. URL httph:ttp///aatt http:///axvvv.oo wwwxxv..rgggaaaee htrg/bbs/2200 01917
[72] X. Yuan, Z. Lin, Kuen, J. Zhang, Wang, M. Maire, A. Kale, and B. Faieta. Multimodal con- trastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6995-7004, June 2021.
[73] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer. Lit: Zero-shot transfer with locked sicked-image text tuning. CoRR, abs/2111.0799) 2021. URL https / /aariiv.org httg/ab//21111 07991
[74] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. CoRR, abs/2010.00747, 2020. URL https:/ arw.vv.vvvv. org/ abs/2010. 00747
[75] K. Zhou, Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2344 2022.
[76] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Conditional prompt learning for vision-language models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
15

---

A More Related Works
Contrastive Learning Contrastive learning has become popular method for self-supervised rep- resentation learning, where the goal is to learn embedding functions such that semantically similar samples are closer in embeddings space while semantically dissimilar samples are embedded further apart [59 (66] 47 70 72 [3]. Among various contrastive learning methods, this paper specifically focuses on the InfoNCE loss, which optimizes the probability of correctly identifying the positíve sample (62] as it has been one of the most frequently used objectives in recent contrastive learning tasks [6 4
Image Caption Evaluation Image caption evaluation is the task of automatically scoring the quality of a caption given the corresponding image and optionally reference captions. The quality of image caption metries evaluated with respect its correlation with human ratings. While traditional
image caption metrics such as BLEU-4 455 ROUGE-L (38), METEOR (2). CIDEr [63]. and
SPICE [I] evaluate the candidate caption by measuring the n-gram overlap of the candidate with
the references, more contemporary neural metrics [22 69 23 30, g judge the candidate caption by comparing the neural embeddings of the caption with that of the image and references, and much more flexible. Among them, the one most related to our work is CLiPScore [17]. By simply passing test-time image and caption pairs through the pre-trained CLIP image and text encoders and taking the dot product, CL.IPScore achieves state-of-the-ar performance using only the candidate caption and the corresponding image without the need for references.
B Base Representation Model Details
This paper examines the effect of a Distribution Normalization layer on the following state-of-the-art open-sourced cross-modal representation models:
one of the earliest pioneers that successfully created joint representation space of images and text through large-scale contrastive learning. It is trained on over 400M image-caption pairs collected by the authors. They have open-sourced multiple versions of pretrained models and this paper examines the commonly used version "ViT-B/32".
CLIP [50]
ALBEF [33] is one of the later variants of CLIP that adds molti-modal encoder to capture the interplay between images and text by predicting if they are paired samples or hard negatives. Addi- tional momentum distillation loss and masked language modeling loss are also added to improve the model performance. Two versions of pre- pre-trained models trained on dataset consisting of 4M and 14M unique images released and this work uses the latter. To focus the effect of DN cross-modal representations, we only keep the image and text encoder component of ALBEF and do not use the multi-modal encoder component.
TCL 68 is another state-of of-t--of-the-rr aht CLIP variant. It inherits the same model architecture as ALBEF, but adds triplet contrastive loss including Cross-modie Alignment, Intra-modal Contrastive, and Local MI Maximization. Their model is trained on 4M unique images. As with ALBEF, we only keep the image and the text encoder component.
C Datasets Details
There is total of 12 datasets mentioned this paper, as described below.
C.1 Cross-modal Retrieval Datasets
multi-purpose dataset known for its rich compatibility with object detection, segmentation, and image captioning tasks. It contains 1 118K images in its training split and more than 5K images in its test split.
MSCOCO BV
Flicker30K (48 is a popular benchmark for sentence-based picture portrayal. contains 31K images with each image having five reference sentences generated by human annotators. We took train-test split following [68) and (33), which involves a selection of 30K images for fine-tuning and IK images for testing.
16

---

C.2 Zeroshot Classification Datasets
ImageNetIK [9] is the most well-known dataset for image classification that contains more than 1200K images its training set and 100K images in its and covers 1000 categories of objects. Accuracy on ImageNetIK is especially widely used for evaluating state-of-the-art image classification models.
C0 28 a dataset containing 100 classes of objects, with each class having 500 training images and 100 test images.
SUN3977 67], the Scene UNderstanding datahase, contains 899 categories and about 130K images. 397 well-sampled categories are available for benchmarking state-of-the-art algorithms for scene recognition tasks.
Stanford Cars [27] an image classification dataset dedicated for cars. collects about 16K images for 196 types of cars and adopts 50-50 train-test split. Results on Stanford Cars test our method's effectiveness in improving special-purpose classification models.
dataset for object recognition tasks. It contains 101 object categories, with varying numbers of images in each category (between 40 800 images per category). The dataset has over 9,000 images total collected from the internet. is a popular benchmark for evaluating the performance of algorithms on general-purpont single-object recognition tasks.
Caltech101 31
Flowers 44 is an image classification dataset focused on flowers, comprising 102 distinet categories of flowers common to the UK. It includes diverse set of images, with each class containing between 40 258 images. The dataset comprises over 8,000 images.
C.3 Image Captions Evaluation Datasets
Blickr8K-Exper 18 is curated subset of the Flickr&K dataset that contains 17K human ratings of image-caption pairs. Each human score corresponds one pair and from to 4 (1 means the caption is irrelevant, 4 means the caption describes the image fully correctly.)
Flicker8K-CP [8] is similar dataset gathered from CloudFlower that contains 48K image-caption pairs and 145K binary ratings of these pairs.
THumb
2 is a dataset containing some machine- and human-generated captions from the MSCOCO dataset. It has 500 images, each with 5 candidates captions that are evaluated by human annotators.
Pascal-50S [51 contains 4K caption-caption pairs with each pair describing the same image. For all pairs, annotators give a preference on which caption in the pair provides better description of the image. Caption-captin pairs categorized based the origins of the captions (see Table for details about categorics.)
D Additional Results
D.1 Fine-tuning Epochs on MSCOCO and Flickr30K
we present the remaining two plots of fine-tuning ablations for MSCOCO, CLIP + DN* does better than CLIP by an average of 0.52% for Acc @5, and 0.37% for Acc@ 10. Both graphs observe a peak in accuracy around 4-5 epochs. For Acc@5, we observe an initial improvement of 9.10% for CLIP DN after just epoch, and 11.10% for CLIP on text-to-image retrieval. These respective numbers are 5. 117 and 4.39% for image-to-text retrieval. For Ace@ 10, we observe an initial improvement of 8.53% for CLIP DN and 10.03% for CLIP on text-to-image retrieval, and 3.14% and 14% respectively for image-to-ter retrieval.
In Figure 3
In Figure 4 present the effects of finetuning Acc @5 and Acc @ 10 for Flickr30K. Again, CLIP+DN & does better than CLIP by an average of 0.22% for Ace@ and 0. 14% for Ace @ 10. All results yield similar trend in support of the conclusion we have made in Section 4.6.2
17

---

HC
HM
MM
Mean
CLIP 17
55.0
99.2
96.9
71.8
80.7
CLIP TTA
55.2
99.2
96.8
71.8
80.8
CLIP+ + +TTA DN
55.1
99.2
97.4
73.7
81.4
CLIP + TTA DN
55.2
99.3
97.4
72.9
81.2
CLIP DN
56.1
99.1
97.3
73.9
81.6
CLIP + DN*
55.6
99.3
97.3
73.6
81.4
TCL 68
52.4
91.8
54.7
63.4
65.6
TCL DN
52.4
96.7
65.7
66.4
70.3
TCL DN
52.7
93.8
59.0
64.3
67.4
ALBEF [3]
56.9
98.1
82.5
67.2
76.2
ALBEF DN
56.1
97.9
81.2
67.0
75.6
ALBEF DN
56.9
98.1
81.7
67.8
76.1
BLEU-4
60.4
90.6
84,9
54.7
72.6
CIDEr 63
65.1
98.1
90.5
64.8
79.6
ViLBERTScore- 69
49.9
93.1
75.8
79.6
CLIP ref 17
62.4
99.7
96.7
73.0
83.0
CLIP+ DN -re CLIP DN -ref
60.8
99.6
97.8
75.1
83.3
61.1
99.7
97.3
74.8
83.2
Table 7: Accuracy results on Pascal-50S given different categories of caption-caption pairs. HC means two correct hhuman-generateee captions. HI means two human-generated captions with one incorrect. HM means human-generated and machine-generaann caption. MM means secchine-gnnrated captions.
Accuracy Top5
Ancuracy TepS
Accuracy Top10
Accuracy Top10
70
80
68
70
76
Accursty
44
Accuracy
24
8B
92
87
22
86
98
66
CLIP TepS CLIP CUI Tupo
CLIP Tep5 CLIP 1DX" Top3
CLIPI UN Top10
CLIP Tople
85
CLIP Top
CLIPI DM Tap
Epochs Pinetuned
Epochs Pinetuned
Epochs Finetuned
Epochs Finetuned
Figure 3: Comparison of the effects of fine-tuning between CLIP and CLIP + DN* on MSCOCO's 5k test set. We report text-to-image retrieval (left) and image-to-text retrieval (right) for Acc@5 and Acc@ 10. The average accuracy over 5 checkpoints trained with 5 random seeds is plotted. For each of the checkpoints we trained, we find its average accuracy and standard deviation with another 5 iterations random sampling for mean estimation, and plot the mean of these 5 accuracies and standard deviations from 5 independently fine-tuned checkpoints.
T21: Accurncy Top5
12T: Ancurncy @ TopS
121: Accuracy Top10
12T: Accuracy & Top10
92
85.5
$5.0
98.6
Accuracy
30
Accuracy
24.0
98.2
95.5
93.0
87.8
87
$2.5
86
$2.0
87.4
2 Epochs Finetuned
10
2
6
10
2
10
4
B
Epochs Finetuned
Epochs Finetuned
Epochs Finetuned
Figure 4: Comparison of the effects of fine-tuning between CLIP and CLIP + DN Flickr30K's 1k test set. We report text-to-image retrieval (left) and image-to-text retrieval (right) for all Acc@5 and Acc@ 10. The average accuracy over 5 checkpoints trained with 5 random seeds is plotted. For each of the checkpoints we trained, we find its average accuracy and standard deviation with another iterations random sampling for mean estimation, and plot the mean of these 5 accuracies and standard deviations from 5 independently tine-tuned checkpoints.
18

---

D.2 Information Lass in Approximation
Without considering computational efficiency, for each test sample, we can iterate over the entire unlabeled reference set to calculate a similarity measure given in Eqn
Sour( (o yo) = BDr e "(m)
(Won)) -
+ el ((xo)] "
(10)
To investigate how much information might be lost from the first-order approximation in Section 3.2.2 and algebraic mean to geometric mean conversion in Section 3.33 that simplifies Eqn 10 distribution normalization in Eqn() we carry out experiments to compare them back to back in the task of image captioning metric and present our results in Table B terms of their correlation with human judgments on image captioning datasets. Surprisingly, in all the base models and datasets that we have studied, we did not notice any significant difference between DN and Eqn. I0 (difference < 0.1%) This shows that higher-order information contributes negligibly to the downstream applications compared only taking the mean of the distribution, and taking algebraic mean achieves similar effect as taking a geometric mean.As a conclusion, we found that DN provides equivalent performance to the original Eqn 5 without incurring expensive computational costs.
D.2.1 Pixel-Level Normalization
A potential alternative to distribution normal- ization which normalizes the data the rep- resentation level is a commonly used trick that normalizes the data (mostly images) the in- put level (pixel level), which we refer as pixel- level normalization. In Table 9 we present the comparison results between CLIP + pixel- norm and our proposed methods. However, al- though using the same data for normalization on the representation level as DN and DN* yields a large gain over vanilla CLIP, changing the normalization to the pixel level wipes out all improvements. We hypothesize that this is because the difference of a constant vector (representation mean) vanishes while the input
FlicktSk-cxpeit
Flickr8k-ef
THumb
To
17
CLIP DN
54.3
35.4
23.5
CLIP DN
54.3
35.4
23.4
TCL [8
31.0
20.6
8.1
TCL +N
42.0
26.4
14.4
TCL (Eqo 10
41.8
26.4
14.3
ALBEF [3]
24.9
15.4
0.9
21.8
ALBEF DN (Engn 10
34,8
21.8
5,5
Table 8: Ahaltion study comparison with distribu- normalization and full Equi5 on Flickr8k-Expert Flickr8k-CE: and THumb.
goes through a large neural network like CLIP.
Flickr8k-cxpet
Flickr8k-ef
THumb
CI.IP 17
$1.4
34.3
19.9
Ref- free
CLIP + pixcl-norm
51.3
34.3
19.4
CLIP DN
54.3
35.4
23,5
CLIP+
Table Ablation study on pixel-level normalization on FlickrSk-Expert, Flickr8k-CF, and THumb.
19


Reviews:
[{"review_id": "x6DzO6xANd", "reviewer": "Reviewer_jN4m", "summary": "This paper focus on the misalignment of training and testing of CLIP model. Specifically, CLIP is trained with an InfoNCE\tloss containing both positive and negative samples, while tested lack negative samples information. They reveal that the common downstream practice of taking a dot product is only a zeroth-order approximation of the optimization goal, and propose distribution normalization to narrow the gap at test time considering both effectiveness and efficiency. Experimental results show some improvements on different datasets.", "questions": "see weakness.", "limitations": "The limitations are adequately addressed in the paper and seem reasonable.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-This paper propose an interesting perspective for the difference of the optimization goal between train- and test-time.\n-The analysis of InfoNCE Loss is easy to understand.\n-The paper is well written, and easy to follow.\n", "weaknesses": "-There is no relevant introduction about test-time tasks in the first paragraph, and the necessity of performing this task at test-time is questionable.\n-The proposed distribution normalization requires prior knowledge of the entire test set, which is often not met in practice, making it unable to handle sustained data.\n-The performance gain is limited. Is it the limitation of the first-order approximation?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1, The necessity of performing this task at test time is questionable.\nWe are not sure about what “this task” refers to, and would be happy to give more explanations if the reviewer can please clarify this point.  However, just to clarify as best as we could understand the comment, DN is conducted to improve performance by aligning training and test time objectives, which we proved with strong test-time results. Reviewer’s comment on why this is necessary and how our approach is questionable is puzzling.\n\n### 2, DN requires knowledge of the entire test, unable to handle sustained distribution\nWe have provided a general response on this issue and kindly ask the reviewer to refer to the “Need for Extra Test Samples” general response for this concern.\n\n### 3, The performance gain is limited\nWe have provided a general response on this issue and kindly ask the reviewer to refer to the “Performance Gain” general response for this concern.\n"}, {"review_id": "m9O1AhyEjH", "reviewer": "Reviewer_HYdZ", "summary": "This paper addresses the problem of retrieval and classification accuracy using vision-language representation.  The authors claimed that there is a mismatch between training objectives and inference-time operation. Specifically, InfoNCE used negative information but test time only use positive similarity scores. The authors presented findings that rectifying such misalignment can boost performance consistently across a variety of downstream tasks. They argued that it is sufficient to use the first-order approximation of InfoNCE loss, i.e. just subtracting the mean in a test batch before doing dot product at test time, and called it Distribution Normalization (DN). They applied their test-time augmentation to a number of existing approaches such as CLIP, ALBEF, and TCL, and show certain improvements over the baselines on common classification and retrieval benchmarks such as MSCOCO and Flickr30K. ", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and the proposed approach and statement are simple and easy to understand. The experiments show consistent improvements over baselines that don't use test-time augmentation. \nThe authors applied their approach to some of the top performers and tested them on the various common benchmarks. They also provided a decent set of ablation studies on how test-time batch size affects performance and how this simple normalization help in the fine-tuning setting. ", "weaknesses": "On line 110, the claim that 0.1 is small is not very convincing, since in practice, the value of the referred expressions is typically much smaller, especially after L2 normalization. Such oversimplification as this makes the final outcome, I.e. DN method, looks simplistic.\n\nThe approach in this paper required test-time augmentation, and when compared to other test-time augmentation approaches such as TTA of [54], the result was even inferior in some cases. Only when in combination with TTA, there was marginal improvements (over TTA[54])", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1, The claim that 0.1 is small is not convincing\nIn our paper, we claim that the value of the referred expression is smaller than 0.1 (or even much smaller than 0.1 according to the reviewer) and therefore second and higher order terms are negligible. The result of this simplification is to make DN as simple as possible so that it is easy and efficient to implement. As provided in Appendix D.2, we have also shown that adding the second and higher order terms won’t make a big difference compared to our first order approximation:\nPart of Appendix D.2\n|  |Flickr8k-expert|    Flickr8k-cfI       |THumb|  \n|-----------|---------------|------|--------|\n| CLIP+ DN |    54.3     | 35.4 |  23.5  |    \n| CLIP+ DN (full) |    54.3     | 35.4|  23.4  |    \n| TCL+ DN |    42.0     | 26.4 |  14.4  |    \n| TCL+ DN (full) |    41.8     | 26.4 |  14.3  |    \n| ALBEF+ DN |    34.8     | 21.8|  5.5  |    \n| ALBEF+ DN (full) |     34.8     | 21.8|  5.5  |    \nWe would be happy to share more details if the reviewer can please explain more why this is a weakness. \n\n### 2, The method is inferior to TTA. TTA + DN only has marginal improvement over TTA\nWe have provided a general response on this issue and kindly ask the reviewer to refer to the “Performance Gain” general response for this concern.\n\n"}, {"review_id": "2MbqrrkbI9", "reviewer": "Reviewer_UGnD", "summary": "CLIP is trained using InfoNCE loss where positive and negative pair alignment is done during training. However, at inference, we simply take the dot product with text embeddings which is not the optimal/similar to the way pretraining was done. Authors propose to rectify this inference and training objective misalignment, based on a first order approximation of InfoNCE loss. Specifically, they subtract the mean representation (calculated over a validation set) from the test image and label/caption representation before applying the dot product. The empirical evaluations of the proposed approach however do not give strong enough gains, especially on CLIP.", "questions": "Please see the weakness section.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "- Approach is easy to implement.\n- Paper writing is generally smooth and clear. However, section around Eqn 8 (L130 to L133) should be improved and is not clear. \n", "weaknesses": "- Any reason why the proposed approach does not work well on MSCOCO retrieval image to text. \n- Empirical gains are pretty nominal, especially for the CLIP models. The authors average their gains over CLIP and other not widely used models (TCL and ALBEF). However, the gains seems to be much lower on CLIP itself. \n- Moreover, as mentioned in L252, it seems that there are larger gains with models pretrained on smaller datasets. This suggests that as the pretraining corpus size is increased, say we use CLIP pretrained on LAION5B, there may not be any gains. Can the reviewers test their approach on CLIP pretrained on larger datasets like DataComp XL ( https://huggingface.co/laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K ) or LAION5B.\n- Authors should test their approach on larger and more standard CLIP architecture like ViT-B-16 and ViT-L, which have higher zeroshot accuracies. \n- Table 2, authors report 61.0 as the zeroshot accuracy of CLIP ViT-B-32, which is however 63.2 (https://arxiv.org/pdf/2103.00020.pdf). Can the authors please clarify this.\n- Can the authors provide finetuning experiments on image classification datasets as well like ImageNet or StanfordCars/Flowers/etc. considered in Table 2. \n- (typo) Eq 5, D_s should be D_t in expectation.\n- The supplementary material is supposed to be submitted as a separate PDF and not part of the main paper, to the best of my knowledge. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n### 1, proposed approach does not work well on MSCOCO retrieval image to text.\nIn the setting of mscoco image to text, while DN does not perform well with CLIP-B32, it has shown effectiveness with other CLIP variants such as CLIP-B16 and CLIP-L14 as follows. A method's performance may depend on the specific characteristics of the dataset and the pre-trained model, and it’s important to highlight that it has shown improvement in the vast majority of settings. This indicates that DN is generally a reliable enhancement for multimodal models.\n\n\n### 2, Improvements not strong, especially on CLIP\nWe have provided a general response on this issue and kindly ask the reviewer to refer to the “Performance Gain” general response for this concern.\n\n### 3, Improvements on CLIP are smaller compared to ALBEF and TCL. What about CLIP-B16, CLIP-L, and CLIP pre-trained on DataComp XL?\nWe have conducted experiments on CLIP-B16, CLIP-L14 and found a similar if not larger improvement of DN for those larger models compared to CLIP-B32 used in the paper. Although the improvement is smaller for CLIP pre-trained on laion for the classification task, its improvement on cross-modal retrieval is still significant and comparable to the improvement for the original CLIP. This shows that DN is most effective when the downstream task resembles the contrastive pre-training procedure as cross-modal retrieval datasets contain diverse image-caption pairs similar to the pre-training dataset, as the purpose of DN is to align the downstream use with the pre-training objective. \n\nFurthermore, we believe the large improvement of DN on ALBEF and TCL is due to the large distribution shift from the small pre-training dataset to other datasets. This shows that DN is particularly successful in adapting to out-of-distribution scenarios. This out-of-distribution adaptation ability can be important even for models pre-trained on large datasets, especially when applied to real-world sustained distributions different from the training set.\n\nCross-modal Retrieval (Top1/Top5/Top10)\n|  | MSCOCO I2T|    MSCOCO T2I       | Flickr30k  I2T|      Flickr30k T2I   | avg |\n|-----------|---------------|------|--------|---------------|---|\n| CLIP-B16 + TTA |    53.6/77.5/85.1     | 33.8/58.7/69.1 |  85.4/97.9/99.1  |    66.6/89.0/93.7      | 59.9/80.8/86.8 |\n| CLIP-B16 + TTA + DN* |    54.6/78.5/86.1      |35.7/60.7/70.8  | 87.3/98.0/99.6  |     69.3/90.2/94.6      | 61.7/81.9/87.8 |\n| CLIP-L14 + TTA |    57.7/80.1/87.8      | 36.8/61.3/71.2 | 88.4/98.9/99.9   |   69.9/90.6/94.8       | 63.2/82.7/88.4 |\n| CLIP-L14 + TTA + DN* |   58.8/81.3/88.4       | 38.6/63.1/72.9 | 89.3/98.8/99.8  |   72.1/91.7/95.5        | 64.7/83.7/89.2 |\n| CLIP-B32-Laion + TTA |     58.5/80.9/88.1     | 40.0/65.8/76.0 | 85.8/96.7/98.9   |   71.1/91.4/94.8       | 63.9/83.7/89.5 |\n| CLIP-B32-Laion + TTA + DN* |    60.7/82.3/88.9      | 40.8/66.5/76.4 | 86.6/97.1/98.9  |   71.7/91.6/94.7        | 65.0/84.4/89.7 |\n\n\nClassification (Top1/Top5)\n|  |imagenet1k |   Cifar100   |   SUN397     | Stanford Cars |  Caltech101   |   Flowers102     | avg |\n|-----------|---------------|------|--------|---------------|------|--------|---|\n| CLIP-B16 + TTA     | 67.1/91.5    | 67.7/90.1 | 60.0/91.4  |  63.3/93.1      | 84.5/96.7 | 69.8/84.8   | 68.7/91.3 |\n| CLIP-B16 + TTA + DN*    |  67.8/92.0   | 71.1/92.2 | 61.9/92.4  |  64.6/93.9      | 84.9/96.6  |  70.0/85.0  | 70.1/92.0 |\n| CLIP-L14 + TTA     |  73.1/93.4   | 77.6/94.0 | 62.1/92.5  |   76.3/97.7     | 86.0/97.3 |  72.8/89.1  | 74.6/94.0 |\n| CLIP-L14 + TTA  + DN*   |  74.2/94.1   | 80.4/95.4 | 63.8/93.3  |  77.2/97.8      | 86.2/97.1 | 74.0/89.1   | 76.0/94.5  |\n| CLIP-B32-Laion + TTA     | 66.9/89.4    | 75.7/94.0 | 63.9/93.5  |   87.1/99.2     | 87.3/97.7  |  70.3/86.4  | 75.2/93.4 |\n| CLIP-B32-Laion + TTA  + DN*  |  67.2/90.3   | 76.2/94.3 | 64.3/93.7  |  87.1/99.1    | 86.8/97.2  |  71.1/85.8  | 75.5/93.4 |\n\n\n\n### 4, 61.0 is different from 63.2. \nThe original CLIP paper maintains a list of potential prompts and validates the best prompt from the list using a validation dataset. In our paper, to avoid dependence on a validation dataset, we use a fixed prompt template “a photo of a {}” for all datasets, and this prompt template may not be the optimal one from the template list from the original CLIP paper.\n\n### 5, Finetuning experiments on Imagenet. \nFinetuning CLIP on the image classification dataset will effectively turn the pre-trained CLIP into an image-classification model. This is different from the pre-training contrastive objective of CLIP. Since the goal of DN is to better align the contrastive objective with downstream uses, this extension is outside the scope of DN, and we believe it is more appropriate to be left for a future direction. We believe that our current extensive results over the tasks of cross-modal retrieval, zero-shot classification, image-caption metrics, and contrastive fine-tuning on image-caption datasets are sufficient in validating the role of DN to better align downstream use of vision-language models with their pre-training objectives.\n"}, {"review_id": "QWgTRlx1Y9", "reviewer": "Reviewer_5JmX", "summary": "This paper proposes distribution normalization (DN) for contrastively trained vision-language models. The idea is motivated by an analysis of the InfoNCE loss. The authors identify that the common practice of taking dot product for zero-shot inference is only a zero-order approximation of the InfoNCE loss. The proposed method improves the dot product inference by using unlabeled test data to make a first-order approximation.", "questions": "1. Regarding performance, I find DN's improvement margins on CLIP to be smaller than on other models. Any idea what would cause this discrepancy between different contrastively-trained models? ", "limitations": "1. As mentioned by the author, as a limitation of this work, DN requires access to multiple test samples and assumes they are all from the same distribution. Such an assumption may not hold in practical settings.\n2. Many existing works have explored The idea of using test data for normalization at test time [2, 3, 4]. The authors should consider citing them and discussing their connections and differences. In addition, there is another work also analyzes the loss functions for test-time adaptation [5]. \n\n[3]. Schneider, Steffen, et al. \"Improving robustness against common corruptions by covariate shift adaptation.\" NeurIPS 2020.    \n[4]. Wang, Dequan, et al. \"Tent: Fully test-time adaptation by entropy minimization.\", ICLR 2021.    \n[5]. Goyal, Sachin, et al. \"Test-time adaptation via conjugate pseudo-labels.\", NeurIPS 2022.    ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The analysis in Section 3.2 is intuitive and motivates the proposed method.\n2. The method is efficient: it only needs to subtract the mean of the text/image features of the test data. \n3. The experiment section evaluates the method on multiple tasks and various contrastively pre-trained models.\n", "weaknesses": "1. There are some missing baselines for CLIP zero-shot classification. For example, [1] proposes test-time prompt tuning for CLIP, which is less efficient than DN, but it works on a single test sample. [2] uses multiple test samples from the same distribution to do logit normalization, which is similar to the proposed method. \n\n\n[1]. Shu, Manli, et al. \"Test-time prompt tuning for zero-shot generalization in vision-language models.\", NeurIPS 2022.       \n[2]. Allingham, James Urquhart, et al. \"A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models.\", ICML 2023.    ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1, Two more baselines missing\nThanks for providing the related work, and we will include the comparisons in a revised version. Here are the results of TPT in our setting and a comparison to our results. (top1/top5). We found that our CLIP+TTA+DN* achieves a comparable top-1 accuracy and a higher top-5 accuracy compared to TPT, while at the same time being much more efficient as TPT needs to do gradient updates for each test sample.\n|  |imagenet1k |   Cifar100   |   SUN397     | Stanford Cars |  Caltech101   |   Flowers102     |avg |\n|-----------|---------------|------|--------|---------------|------|--------|---|\n| TPT     |  63.5/87.1       | 65.2/88.1 | 59.4/88.8   | 61.5/90.2        | 83.2/96.0 | 64.5/81.3   | 66.2/88.6 |\n| CLIP+TTA+DN*    |   63.2/88.9     | 67.1/90.7  |  58.1/90.7  |   61.5/92.2     | 83.1/95.5 |  63.5/84.4  | 66.1/90.4 |\n\nFor Zero-shot Prompt Weighting (ZPW), we agree with the reviewer that this would be an interesting baseline to have, but we would also like to emphasize that our proposed method is, in principle, meant to be auxiliary to ZPW. Specifically, since ZPW only weighs the importance of the dot-product similarity of each prompt template from a prompt pool, DN can still be applied by subtracting the mean of these embeddings from the logits. This also does not interfere with the weighted average of logits cross-product with prompt scores. Further, results from the ZPW paper are actually tested on a slightly different dataset split and we have previously looked into the released code for this paper but found that it is currently still in development with several unresolved dependencies. Should the stable version of the code be released prior to submission, we would be happy to include the performance of ZPW + DN as an additional baseline. \n\n### 2, Are Improvements on CLIP smaller than other methods?\nYes, your observation is correct. We have provided a general response on this issue and kindly ask the reviewer to refer to the “Performance Gain” general response for this concern.\n\n\n### 3, Differences with other related works.\nThanks for providing these related works. [3] estimates statistics for batch normalization within the large models. This method is not applicable to CLIP because CLIP does not have batch normalization to avoid information about the batch leaked in the pre-training. [2] and [3] involves minimizing a loss over all test samples. They require more unlabeled samples, are less efficient than DN, and cannot handle general multimodal alignment tasks such as cross-modal retrieval and image caption metrics. We will add detailed comparisons with those works in a revised version.\n\n[3]. Schneider, Steffen, et al. \"Improving robustness against common corruptions by covariate shift adaptation.\" NeurIPS 2020.\n\n[4]. Wang, Dequan, et al. \"Tent: Fully test-time adaptation by entropy minimization.\", ICLR 2021.\n\n[5]. Goyal, Sachin, et al. \"Test-time adaptation via conjugate pseudo-labels.\", NeurIPS 2022.\n"}, {"review_id": "PFgfw8xTvB", "reviewer": "Reviewer_XtRE", "summary": "This paper reveals a mismatch between the pre-training objective of contrastively trained vision-language models and their downstream usage. The authors propose  Distribution Normalization (DN) to solve this problem. The results on a wide variety of downstream tasks show the effectiveness of the proposed method.\n", "questions": "Please see the weaknesses.", "limitations": "The authors have adequately discussed the limitations of this paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The finding about the misalignment between pretraining and downstream tasks for contrastively trained vision-language models is important.\n\n2. The proposed method, DN, is simple but effective, and also conveniently implemented in practice.\n\n3. The experiments on various tasks show the effectiveness of the proposed method, and also there are sufficient ablation studies.\n", "weaknesses": "Overall, I think this paper is well-presented and reveal an important problem in contrastively trained vision-language models. However, I still have the below concerns:\n\n1. A recent work, TPT [1], is missing, which aims to learn prompts for vision-language models at the test-time. I recommend that the authors will compare DN with TPT and integrate DN into the TPT’s framework.\n\n2. I suggest that the authors will modify “visual-language” to “vision-language” in the next version.\n\n3. Another concern is that because DN  needs a small amount of unlabeled data, its setting actually is not the zero-shot one, thus I suggest the authors will refine the related descriptions, e.g. Line 175.\n\n\n[1] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models. NeurIPS 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": " ### 1, Add TPT baseline\nThanks for providing the related work, and we will include the comparisons in a revised version. Here are the results of TPT in our setting and a comparison of our results. (top1/top5). We found that our CLIP+TTA+DN* achieves a comparable top-1 accuracy and a higher top-5 accuracy compared to TPT, while at the same time being much more efficient as TPT needs to do gradient updates for each test sample.\n|  |imagenet1k |   Cifar100   |   SUN397     | Stanford Cars |  Caltech101   |   Flowers102     |avg |\n|-----------|---------------|------|--------|---------------|------|--------|---|\n| TPT     |  63.5/87.1       | 65.2/88.1 | 59.4/88.8   | 61.5/90.2        | 83.2/96.0 | 64.5/81.3   | 66.2/88.6 |\n| CLIP+TTA+DN*    |   63.2/88.9     | 67.1/90.7  |  58.1/90.7  |   61.5/92.2     | 83.1/95.5 |  63.5/84.4  | 66.1/90.4 |\n\n### 2, Change to vision-language models and refine related descriptions\nThanks for the reminder. We will change to “vision-language models” and refine the related descriptions to include the need for extra unlabeled samples.\n"}]

Author rebuttal:
We appreciate the reviewers’ agreement on the efficiency and wide applicability of our proposed DN, and reviewer XtRE for recognizing the effectiveness of DN. There were several issues pointed out by multiple reviewers which we will address here in the general response. 

### Performance Gain: 
First of all, while reviewer HYdZ is right in saying that DN is comparable to TTA on average, we want to emphasize that DN is not mutually exclusive to TTA methods as they serve different purposes. DN aligns objectives between training and testing while TTA resolves issues caused by the instability of a single view of the image.  As such, they can work hand in hand together to produce strong results. For example, as shown in Section 4, DN can be combined with TTA techniques such that CLIP+TTA+DN consistently performs better than CLIP+TTA across all tasks. 

More importantly, our absolute performance gain over baseline methods is comparable to the recently published [1, 2] works that reviewers XtRE and 5Jmx cited in their reviews. For cross-modal retrieval task top-1 retrieval accuracy, our CLIP+TTA+DN* achieves another 1.6% average performance boost over CLIP+TTA. The equivalent average performance boosts for classification and image captioning metrics are 0.9% and 1.6%. A similar improvement of CLIP+DN compared to CLIP is 2.2% for retrieval, 0.8% for classification and 1.3% for image caption metrics. In fact, when using the same CLIP architecture CLIP-B16 as in [1] and [2], the improvements are even as large as 1.4% top-1 accuracy for classification (see full results in the response to UGnD). In comparison, the average improvements of top-1 accuracy of [1] and [2] for zero-shot cross-dataset classification are 1.52% and 2.12% respectively compared to their baselines, but their method does not generalize to tasks beyond classification. In contrast, DN is widely applicable to a wide variety of important multi-modal alignment tasks including cross-modal retrieval, classification, and image captioning metric and at the same time achieve decent improvements across the board.

In addition, we would like to direct the reviewers’ attention to the more neglected experiments with TCL and ALBEF that achieve a much larger gain compared to CLIP (as large as 7% average top-1 improvement on classification). We surmise that this is because TCL and ALBEF are pre-trained on a smaller dataset compared to CLIP, which makes them more susceptible to larger distribution shifts in downstream applications. The relatively larger improvement from adding DN here should not be overlooked, as it shows that it can be particularly helpful when added to models with significant real-world distribution shifts such as specialized expert models pre-trained on a smaller dataset. This does not mean that DN will be ineffective for CLIP with a larger model architecture and pre-trained on a larger dataset. As presented in response to reviewer UGnD, DN brings a comparable if not larger performance boost to CLIP-B16, CLIP-L14, and CLIP pre-trained on the laion2B dataset. 

### Need for Extra Test Samples: 
There were also several comments about the need for test samples. We want to ensure that it is clear that we only need a very small number of test samples as shown in Table 4, not the entire test set as reviewer jN4m claimed. However, we do admit that these few test samples need to be of the same distribution as reviewer 5Jmx pointed out. But we don’t agree that this is impractical in many real-world applications as long as we collect these test samples close in time (e.g., trending images on social networks). This requirement is strictly the weakest among the test-time adaptation literature mentioned by reviewer 5Jmx. We also agree that since we need a small number of test samples, we should be clear that DN is few-shots rather than zero-shots (L175), and will refine related descriptions in the paper.


### Practicality: 
Finally, DN is particularly valuable in light of the practical tradeoff between efficiency and accuracy. [1] requires doing test-time adaptation gradient updates for each single test sample and [2] requires iterating over hundreds of prompt templates in the prompt pool. Both methods create a considerable computational overhead. In contrast, DN is particularly simple to implement — we simply subtract the mean.


[1] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models.", NeurIPS 2022.

[2] Allingham, James Urquhart, et al. "A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models.", ICML 2023.


Meta-review: This paper proposes a simple method to improve zero-shot accuracy of CLIP style models. The method is well motivated by the info-nce loss used in contrastive learning. They have quite strong results and somewhat thorough evaluation (the inclusion of other test time training methods was particularly useful). Reviewers also agreed to increase the scores after the rebuttal. Thus
Decision: Accept (poster)