Title: Test-Time Distribution Normalization for Contrastively Learned Visual-language Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: contrastive learning

pre-trained visual-language models

zero-shot learning

test-time augmentation
Abstract: Advances in the field of visual-language contrastive learning have made it possible for many downstream applications to be carried out efficiently and accurately by simply taking the dot product between image and text representations.  One of the most representative approaches proposed recently known as CLIP has quickly garnered widespread adoption due to its effectiveness. CLIP is trained with an InfoNCE loss that takes into account both positive and negative samples to help learn a much more robust representation space. This paper however reveals that the common downstream practice of taking a dot product is only a zeroth-order approximation of the optimization goal, resulting in a loss of information during test-time. Intuitively, since the model has been optimized based on the InfoNCE loss, test-time procedures should ideally also be in alignment. The question lies in how one can retrieve any semblance of negative samples information during inference in a computationally efficient way. We propose Distribution Normalization (DN), where we approximate the mean representation of a batch of test samples and use such a mean to represent what would be analogous to negative samples in the InfoNCE loss. DN requires no retraining or fine-tuning and can be effortlessly applied during inference. Extensive experiments on a wide variety of downstream tasks exhibit a clear advantage of DN over the dot product on top of other existing test-time augmentation methods.

Test-Time Distribution Normalization for Contrastively Learned Vision-language Models

Abstract

Advances in the field of vision-language contrastive learning have made it possible for many downstream applications to be carried out efficiently and accurately by simply taking the dot product between image and text representations. One of the most representative approaches proposed recently known as CLIP [50] has garnered widespread adoption due to its effectiveness. CLIP is trained with an InfoNCE loss that takes into account both positive and negative samples to help learn a much more robust representation space. This paper reveals that the common downstream practice of taking a dot product is only a zeroth-order approximation of the optimization goal, resulting in a loss of information during test-time. Intuitively, since the model has been optimized based on the InfoNCE loss, test-time procedures should also be in alignment. The question lies in how one can retrieve any semblance of negative samples information during inference in a computationally efficient way. To this end, we propose Distribution Normalization (DN), where we approximate the mean representation of a batch of test samples and use such a mean to represent what would be analogous to negative samples in the InfoNCE loss. DN requires no retraining or fine-tuning and can be effortlessly applied during inference. Extensive experiments on a wide variety of downstream tasks exhibit a clear advantage of DN over the dot product on top of other existing test-time augmentation methods. Our code is available at https://github.com/fengyuli2002/distribution-normalization.

1 Introduction

Contrastive Learning [20, 3, 60, 6, 52] has become an important area of research in recent years due to its widespread success in self-supervised representation learning. The key idea behind contrastive learning is to group positive samples that share similar semantics (e.g., images resulting from applying different augmentations to the same image), while separating semantically different negative samples (e.g., different images with different data augmentations). On the heels of this progress, contrastive learning has also been shown to be very effective for creating a joint representation space between images and text [50, 46, 75, 10]. The CLIP approach proposed in [50] is at the forefront of this research effort and has been shown to be the state-of-the-art in multiple downstream tasks. The model is generally pre-trained on a large amount of image-text pairs that form the positive samples, while the negative samples are composed from random pairing of images and text. After the model is trained, similarities between text and images can be measured by simply taking a dot product between their corresponding representations. Such a paradigm has opened the doors to state-of-the-art performance

in many cross-modal alignment tasks such as zero-shot classification [73, 21], cross-modal retrieval [71, 40, 35, 32], and evaluation of machine-generated captions [17, 22, 30].

A marinawith

A

leans

a)Contrastive training

boats.

b) Test-time distribution

against a net.

normalization

Text

Text

Encoder

Encoder

y1y2y3..yn

μtxt

21

C2

Image

Image

Encoder

Encoder

.··

Cn

(21:n -μimg)(y1：n-μtxt)

Pimg

Despite CLIP's successes, we observe a 'mismatch' between the pre-training objective of such cross-modal representation models and their downstream usage. Most if not all of these models are trained taking into account of both positive and negative samples, of which one of the most popular optimization objectives is the InfoNCE loss [58, 62, 74]. The InfoNCE loss considers negative samples from the data distribution per positive sample and has been shown to be very effective for cross-modal representation learning. In contrast, during test-time, the similarity is often measured using the dot product between the image and the text representations, which does not utilize any information with regard to the data distribution.

This paper presents our findings that rectifying such misalignment can boost performance consistently across a variety of downstream tasks. However, naively applying InfoNCE loss for downstream tasks requires iterating over all negative samples for every test sample, which is not a tractable operation. Instead, we found that a first-order approximation of the InfoNCE loss, consisting of simply subtracting the distribution mean in the representation space before taking the dot product, is able to achieve a similar effect. We call this proposed approach Distribution Normalization (DN) DN is very easy to implement and does not require any retraining, fine-tuning or labeled data.

To investigate how this simple modification affects the performance of CLIP and its state-of-the-art variants in practice, we conduct extensive experiments on a wide variety of cross-modal alignment tasks, including zero-shot classification, cross-modal retrieval, and evaluation of image captions. In all the settings that we have studied, DN displays a clear benefit over direct dot product and can be combined with other existing test-time augmentation methods [55], which we will show in our experiments. Our contributions are: (1) We present an insightful analysis of why the dot product as a similarity measure is only a zeroth-order approximation; (2) We propose DN to overcome this shortcoming by utilizing an approximation of the test-time distribution, and also provide guidance on how DN can be conveniently implemented in practice; (3) We provide extensive supporting empirical evidence and ablation studies that demonstrate the fidelity of DN for practical use.

2 Background & Related Works

Learning via Natural Language Supervision Several recent studies [7, 12, 34, 35] leverage largescale datasets of image-text pairs to supervise vision-language pre-training (VLP) with a contrastive objective. CLIP [50] is one such example that is contrastively trained to create a joint vision-language representation space, which has produced empirical success in downstream tasks such as zero-shot classification and cross-modal retrieval over previous state-of-the-art models [40, 65, 68, 33]. Since CLIP's introduction, several other works have utilized a similar contrastive objective to create good vision-language representations and achieve state-of-the-art performance on many downstream tasks involving datasets that the model was not pre-trained on [33, 21, 7, 11, 26]. For example, ALIGN scales joint pre-training to minimally-filtered datasets [21], while ALBEF [33] and TCL [68] focus on learning more semantically meaningful multi-modal interactions. It is noteworthy that these SOTA

models all leverage the InfoNCE loss as their key training objective, and apply a simple dot product in downstream tasks.

CLIP Augmentation/Adaptation for Downstream Applications Researchers have explored numerous methods to adapt CLIP for downstream tasks, capitalizing on its remarkable transferability. CoOp [75, 76] employs learnable textual prompts, inspired by prompt learning [36], for few-shot training data, while VT-CLIP [49] incorporates visual-guided texts for enhanced vision-language alignment. However, these adaptations demand extra training data and fine-tuning resources, undermining CLIP's core strength of efficient zero-shot recognition. To address this issue, CALIP [14] presents a training-free attention module to boost zero-shot image classification, and SuS-X [61] offers training-free name-only transfer by adapting CLIP with a curated support set. However, both CALIP and Sus-X are tailored for zeroshot classification only, while in this paper, we propose DN, a training-free adaptation that is conceptually simpler, easier to implement, and applicable to a wide range of downstream tasks.

Test-time Augmentation/Adaptation Test-time adaptation and test-time augmentation strategies share a common goal to adapt to out-of-distribution scenarios [53, 43] during test time. Many test-time adaptation techniques make adjustments to the batch normalization layers to normalize feature statistics [37, 41], while test-time augmentation techniques [64, 25, 42, 29, 15, 19, 57] aim to ameliorate performance by taking some aggregation [55] of the model's prediction over a batch of augmented images. Though DN is also applied during test-time, DN's task is to align the mismatch between training and inference objectives and not to deal with out of distribution test samples. This renders DN orthogonal to TTA. In our experiments, we use a representative TTA model from Shanmugam et al. [55] and show that while TTA improves model performance, adding DN to TTA gives an additional increase.

3 Methodology

This section is organized as follows: in Section 3.2, we show that a dot product is only a zeroth-order approximation of the pre-training objective and how a better approximation arises naturally from the InfoNCE objective, and in Section 3.3, we explain how this idea can be efficiently implemented in practice.

3.1 Notations

In our analysis, D S is the training distribution and D T is the test distribution, x 0 , y 0 are images and text belonging to the same pair, x 1 , ..., x n and y 1 , ..., y n are random images and text in the same batch, ϕ, ψ are image and text encoders respectively, and τ is a temperature hyperparameter used in InfoNCE loss while training. Since in most applications, only relative comparisons of the distances matter (e.g., retrieving the best candidate caption describing the image), we consider two similarity measures S 1 and S 2 to be equivalent if one is a monotone function of the other, namely

The goal here is to find a test-time similarity measure that is similar to the train-time optimization goal through careful simplification of InfoNCE loss.

3.2 Analysis of InfoNCE Loss

We start from the original form of InfoNCE loss [58, 62, 74]:

Since ∑ n i =1 e ϕ ( x i ) ⊺ ψ ( y 0 ) /τ e ϕ ( x 0 ) ⊺ ψ ( y 0 ) /τ is usually quite small ( < 0 . 1 ) due to the exponentiation, we can taylor expand the above term around 0 and discard second and higher order terms, resulting in

which can be simplified to

A successful generalization to the test distribution means that the infoNCE loss L NCE ( D T ) is small when the expectation over training distribution for x 0 , y 0 , x 1 , y 1 is substituted with the expectation over test distribution, where

3.2.1 Reduction to Zero-Order Dot Product

We argue that Eqn. 5 can be reduced to straightforwardly comparing the dot products. When we do not have of the test distribution D T , the only thing that we can do is to perform a zeroth-order approximation of Eqn. 5. The most intuitive way of doing so is to reduce the distribution of ϕ ( x 1 ) and ψ ( y 1 ) to be 0 deterministically, so that:

This shows that we can essentially just output e -ϕ ( x 0 ) ⊺ ψ ( y 0 ) as the distance between x 0 and y 0 . So we get the zeroth-order similarity:

3.2.2 Efficient First-Order Approximation

The common practice of taking a naive zeroth-order approximation can result in a loss of some important information of the distribution of negative samples. On the other hand, computing Eqn.5 for each pair x 0 , y 0 involves iterating over all samples in a distribution D T , and is computationally inefficient. We therefore turn to more efficient approximations using statistics of higher-order moments of the distribution. In our experiments, we found that this intuitive modification preserves most of the useful information as shown in Appendix D.2.

If we can estimate the first moment of the distribution, i.e., the mean of the unlabeled distribution µ x and µ y for ϕ ( x 1 ) and ψ ( y 1 ) , we can perform a first-order approximation of the distribution with ̂ P ( ϕ ( x 1 )) = I { ϕ ( x 1 ) = µ x } and ̂ P ( ψ ( y 1 )) = I { ψ ( y 1 ) = µ y } , so that ̂ P ( ϕ ( x 1 )) , ̂ P ( ψ ( y 1 )) matches the true distribution in terms of the first-order moment. This simplifies Eqn. 5 to be:

Therefore, to evaluate the distance of any x 0 , y 0 , we can simply output e ϕ ( x 0 ) ⊺ ( µ y -ψ ( y 0 )) /τ + e ( µ x -ϕ ( x 0 )) ⊺ ψ ( y 0 ) /τ without having to calculate an expectation by iterating through all samples in the distribution.

3.3 Distribution Normalization

Considering that e ϕ ( x 0 ) ⊺ ( µ y -ψ ( y 0 )) /τ + e ( µ x -ϕ ( x 0 )) ⊺ ψ ( y 0 ) /τ is equivalent to taking an algorithmic mean of e ϕ ( x 0 ) ⊺ ( µ y -ψ ( y 0 )) /τ and e ( µ x -ϕ ( x 0 )) ⊺ ψ ( y 0 ) /τ , to simplify this further, we can convert this algorithmic mean to a geometric mean √ e ϕ ( x 0 ) ⊺ ( µ y -ψ ( y 0 )) /τ e ( µ x -ϕ ( x 0 )) ⊺ ψ ( y 0 ) /τ which achieves a similar effect as averaging the distance measure of e ϕ ( x 0 ) ⊺ ( µ y -ψ ( y 0 )) /τ and e ( µ x -ϕ ( x 0 )) ⊺ ψ ( y 0 ) /τ . An additional empirical verification of the correlation between algorithmic mean and geometric mean in this step is provided in Appendix D.2. Since √ e ϕ ( x 0 ) ⊺ ( µ y -ψ ( y 0 )) /τ e ( µ x -ϕ ( x 0 )) ⊺ ψ ( y 0 ) /τ is a monotone function of -( ϕ ( x 0 ) -1 2 µ x ) ⊺ ( ψ ( y 0 ) -1 2 µ y ) after removing the square root and exponentiation, we can simply use -( ϕ ( x 0 ) -1 2 µ x ) ⊺ ( ψ ( y 0 ) -1 2 µ y ) to calculate the distance, resulting in a simplified first-order similarity measure:

Note that the new similarity measure is equivalent to subtracting the means immediately after the original representations are calculated, a method we call Distribution Normalization (DN), after

which the similarity is just their dot product. Clearly, our proposal is extremely easy to implement when access to the mean of the distribution of interest is available.

Fortunately, we observed that the mean of the distribution can usually be robustly estimated from very few unlabeled samples as shown in Section 4.6.3. This is easily accessible in all the downstream tasks that we have considered in this paper and is much cheaper than fine-tuning where not only more data is required but also the data needs to be annotated. For example, in the task of cross-modal retrieval, the mean can be estimated from the gallery data and past queries (without knowing the correct answer to the queries); in the task of zero-shot classification, the mean can be estimated from a small and unlabeled validation set; and in the task of image caption metrics, the mean can be estimated from the images to be captioned and the references provided.

4 Experiments

Our experiments are designed to answer the following questions: 1) whether our proposed DN can uniformly improve a wide range of cross-modal alignment tasks for different kinds of cross-modal representation models and whether this gain is larger than that achieved by other zeroshot CLIP augmentations, 2) whether DN can be used in parallel with other common test-time adaptation methods compatible with CLIP, 3) how robust DN is when only scarce, unlabeled data is available to estimate the mean from, and 4) whether DN can improve the performance of fine-tuned models in addition to pre-trained models.

4.1 Downstream Tasks

Cross-modal Retrieval includes two subtasks where images are used to query corresponding text and vice versa. Following [68, 33], for each subtask, we consider both the zero-shot setting and the fine-tuning setting on COCO [39] and Flickr30K [48]. In the zero-shot setting, we directly apply the pre-trained model on the test set of each dataset while in the fine-tuned setting we first fine-tune each pre-trained model on the training set of each dataset with the InfoNCE loss and apply the fine-tuned model to the test set. For each setting, we report Recall@k (k = 1,5) whereby the retrieval is considered a success if its corresponding image/text is in the top-k.

Zero-shot Classification is the task where we directly apply the pre-trained models to predict the category of the images in the test set. Each class is represented with a fixed textual template of 'a photo of { class name } .', and the corresponding representation is obtained by encoding those sentences with the text encoder of the pre-trained model. For each image, we encode it with the image encoder and output the top-k most similar class representations as predictions. We benchmark all methods with top-1 and top-5 accuracy on ImageNet1K [9], Cifar100 [28], SUN397 [67], Stanford Cars [27], Caltech 101 [31], and Flowers102 [44] datasets.

Image Captioning Metric is the task that measures the correlation between human ratings and similarity metrics on machine-generated captions. Following [17], we consider both the reference-free setting and the reference-based setting. In the reference-free setting, the metric does not have access to human-written example captions (references) and has to give a score based on the source image and the generated caption alone while in the reference-based setting the metric can additionally take advantage of the information from the provided references.

4.2 Baselines

This paper compares the augmentations of Distribution Normalization and other baselines on CLIP [50], ALBEF [33], and TCL [68]. More details of the base representation models can be found in Appendix B. In particular, the following are the main baselines being examined in the experiments.

TTA [55] aims to better adapt vision encoders to out-of-distribution test samples by augmenting test images and taking a weighted reduction of the various augmentations. However, since DN is meant for the zero-shot setting, training these test-time augmenting layers is not applicable in our setting. Instead, we apply the standard test-time augmentation policy from Shanmugam et al. [55] which consists of a combination of flips, crops and scales, and take a mean over the model outputs on these augmented images.

CALIP [14] replaces CLIP's dot-product similarity with a parameter-free attention module. It improves performance on zero-shot image classification by enabling visual and textual representations

to better explore cross-modal features. This method results in improved adaptive zero-shot alignment by blending images with textual-aware signals. We implement CALIP using their official code and report our results on the 'ViT-B/32'-based CLIP architecture, consistent with our approach in other experiments. Since the original paper focuses solely on zero-shot image classification, CALIP serves as our baseline for this task.

TPT [56] is another CLIP zeroshot method that performs prompt tuning on each single test image to promote the model's prediction consistency on different augmented views of the single test image. TPT is mainly proposed for classification tasks and it is unclear how it can be used for general cross-modal alignment tasks. We implement TPT using their official code and report our results on the 'ViT-B/32'-based CLIP architecture, consistent with our approach in other experiments. Importantly, TPT is much more computationally demanding than vanilla CLIP dot product because it requires taking gradient steps for each single test image over its 64 different augmentations, while other methods do not suffer from such computation issues.

DN is our proposed plug-in augmentation using Eqn.9. Estimated image mean and text mean are respectively subtracted from image and text embeddings immediately after the embeddings are calculated from base encoder models. Dot products are then taken between text and image representations as the similarity. Different ways to estimate the image mean and text mean from few validation samples or references in different downstream tasks are detailed in the following sections.

DN* Considering the motivation for DN is to better align the downstream similarity with the pre-training goal, it might introduce instability when there is a significant mismatch between the pre-raining and testing distribution. We therefore examine this variant called DN* where the only difference from DN is that DN* averages the similarity output by DN and a vanilla zero-order dot product for improved robustness.

4.3 Cross-modal Retrieval Results

We conducted experiments on both image-to-text and text-to-image retrieval on Microsoft COCO [39] and Flicker30K [48] (see Appendix C.1 for datasets details). For all the retrieval tasks, we estimated the mean with 100 random unlabeled samples from the validation set and calculated standard deviations and average recalls with 5 random seeds.

The results for both datasets are presented in Table 1 for the zero-shot setting. For all three base models, adding DN almost always improves recall for both zero-shot image-to-text and text-to-image retrieval tasks, with an average 1.1% increase in top-1 recall. This provides substantial evidence that DN is effective for both tasks even in datasets with diverse images like MSCOCO. The improvement can be as large as a 2.2% increase in top-1 image-to-text recall as we see for CLIP on Flickr30K. We also observe that, in general, adding DN brings a larger improvement for top-1 recall than for top-5/10 recall, suggesting that DN is particularly suitable for distinguishing similar samples.

While adding test-time augmentation techniques to CLIP are effective in themselves as shown in Table 1, we also show that adding DN or DN* on top of TTA techniques further improves retrieval accuracy. Specifically, CLIP+TTA+DN achieves another average 1.3% improvement over CLIP+TTA. A similar improvement is achieved by CLIP+TTA+DN* as well. However, it is noteworthy that although CLIP+TTA+DN and CLIP+TTA+DN* achieves similar average improvement over CLIP + TTA (1.3% and 1.2% respectively), the improvement of CLIP+TTA+DN* has a smaller standard deviation (0.8% compared to 1.3%), suggesting that averaging DN with direct dot product indeed improves the stability.

4.4 Zero-shot Classification Results

Table 2 gives the zero-shot classification results for all three base models and their variants with DN on ImageNet1K [9], Cifar100 [28], SUN397 [67], Stanford Cars [27], Caltech 101 [31], and Flowers 102 [44] (see Appendix C.2 for datasets details). In these experiments, we perform DN before the final layer normalization of image/text encoders.

DNconsistently enhanced the performance of all three base models across six datasets with an average 4 . 4% increase in top-1 accuracy. It demonstrates effectiveness not just in general-purpose vision datasets like ImageNet, but also in specialized datasets such as Stanford Cars, where average accuracy increases by 1.5%. Specifically, CLIP+TTA+DN/DN* is the best variant for a CLIP base model, signifying simultaneous benefits from DN and TTA. While CLIP+TTA+DN and CLIP+TTA+DN*

display similar improvements over CLIP+TTA (0.82% and 0.9% respectively), CLIP+TTA+DN* again exhibits a smaller standard deviation (0.22% vs. 0.76%), indicating improved stability. Furthermore, DN consistently surpasses CALIP in all model-dataset combinations, despite CALIP's slight higher computational cost from incorporating an attention module. Comparing TPT with our best variant CLIP+TTA+DN*, we find that they achieve comparable top-1 accuracy across all datasets while TPT suffers from a significant drop in top-5 accuracy, potentially due to poor calibrations in performing gradient steps on each single test image. It is also important to note that DN* is much more computationally efficient with only an overhead of subtracting the mean while TPT requires performing gradient steps on 64 different augmented view of each test image.

Notably, TCL saw the largest gain of 7% on average in top-1 accuracy. On ImageNet1K particularly, TCL's top-1 accuracy is improved from 20.0% to 30.5% by adding DN. We believe the fact that TCL and ALBEF are pre-trained on a smaller dataset (4M & 14M images) than CLIP (400M images) resulted in TCL and ALBEF getting a larger improvement than CLIP, as DN may have helped ease the instability when the model is not adequately trained. All these improvements are achieved with a single layer that incurs minimal computational overhead. Lastly, comparing DN and DN* for TCL and ALBEF, we found that DN consistently gives a greater improvement than DN*, potentially because averaging with vanilla dot product halves the benefits from DN when the base models are instable due to inadequate training.

4.5 Image Captioning Metric Results

Inspired by the success of CLIPScore [17] as a metric for image captioning, we conduct experiments to test whether DN and DN* can help various cross-modal representation models align better with human judgments. To compare the performance of different methods, we report the Kendall's τ b or τ c coefficients against human judgments on Flickr8k-expert, Flickr8k-cf [18] and THumb [24] in Table 3, following the standard practice in [17]. We also report the resulting accuracy for caption-caption preferences on Pascal-50S [51] in Table 7. Details about these datasets are given in appendix C.3. In the reference-based setting, for CLIP-ref, we follow [17] in using an averaged score given by:

where H-mean is the harmonic mean and R is the set of provided references. On the other hand, for a distribution normalized CLIP + DN-ref, we use an algebraic mean for simplicity:

where A-mean is the algorithmic mean, R is the set of provided references, µ y is the mean of text representations,and S is defined in Eqn. 9.

In Table 3, we report the results on Flickr8KExpert, Flicker8K-CF, and THumb for all three base models. Some methods, such as the BLEU score, are not applicable in the reference-free setting and are thus only compared with in the reference-based setting. The results strongly support that adding DN significantly improves the τ correlation for all three base models on all dataset in the reference-free setting. In particular, CLIP + DN improves CLIP by 2.9% on Flickr8k-expert, 1.1% on Flickr8k-cf and 3.6 % on THumb, and CLIP + DN* performs comparably. In the reference-based setting, to give a better context, we also provide the performance of other reference-based SOTA metrics that are not contrastively trained for comparison. We find that CLIP and CLIP-ref outperform all of them. However, by simply adding a DN layer, CLIP-ref + DN again yields an average 1.3% gain across the board over previous leaders. Fur-

thermore, analogous to the results in cross-modal retrieval and zeroshot classification, we found that TTA is effective in improving on results of image captioning metric (CLIP + TTA Has on average 0.6% improvement over CLIP) but TTA combined with DN gives another 1.9% boost on average. A similar pattern is also observed in our experiment results on Pascal-50S in Appendix 7. Our main results on cross-modal retrieval, zero-shot classification, and image caption metrics provide extensive evidence that our proposed DN is a widely applicable modification that can improve the performance of cross-modal representation models across a variety of downstream applications.

0

2

4

6

8

10

30

32

34

36

38

40

42

CLIP Top1

CLIP+DN* Top1

0

2

4

6

8

10

53

54

55

56

57

58

59

60

MSCOCO I2T: Accuracy @ Top1

CLIP Top1

CLIP+DN* Top1

0

2

4

6

8

10

64

66

68

70

72

CLIP Top1

CLIP+DN* Top1

0

2

4

6

8

10

81

82

83

84

85

86

87

88

FLICKR I2T: Accuracy @ Top1

CLIP Top1

CLIP+DN* Top1

4.6 Ablation Study

4.6.1 Results on Different Variants of CLIP

In addition to the CLIP-B32 model used in most experiments, in Table 4 and Table 5, we investigate into the performance of DN* on other popular variants of CLIP models, including CLIPB16, CLIPL14

[50] that come with more parameters and smaller patch size, and CLIPB32-Laion [54] that is trained on a larger image-caption pair dataset. The results show that DN* brings a similar performance gain for CLIP with more parameters or trained on a larger dataset, implying the universal applicability of our proposed method irrespective to the size of the CLIP model.

Table 4: Cross-modal retrieval performance on MSCOCO and Flickr30K in the zero-shot setting for more CLIP variants. Means for DN* are estimated using 100 random unlabeled validation samples. Average recalls are calculated with 5 random seeds.

Table 5: Zero-shot classification performance on ImageNet1K, Cifar100, SUN397, Stanford Cars, Caltech 101, and Flowers 102. Means for DN* are estimated using 100 random unlabeled validation samples. Average accuracies are calculated with 5 random seeds.

4.6.2 Effect of Fine-Tuning

We also perform ablations on the effects of DN* on fine-tuned models. We fine-tune CLIP on the MSCOCO training set, for a total of 10 epochs on 4 Nvidia 2080Ti. We use the Adam optimizer with a learning rate of 1e-5 and a weight decay of 0.1. We then evaluated each checkpoint on the same MSCOCO test split used in Table 1, and estimate the mean for CLIP + DN* using 100 randomly sampled validation points. A total of 5 checkpoints were trained, and we evaluated each checkpoint for 5 iterations. For each checkpoint, we calculate the average accuracy and standard deviation, then plot the mean of the accuracies and standard deviations over all 5 checkpoints. We repeat this process for Flickr30K, using the same number of epochs, checkpoints, and model of GPUs. We again use the Adam optimizer with a learning rate of 1e-5, but with a weight decay of 0.02. We record accuracy scores at 1, 5, and 10 for both image-to-text retrieval and text-to-image retrieval on the two datasets. We present Acc@1 here, and leave Acc@5 and Acc@10 in Apendix D.1.

The following analysis focuses specifically on the case of Acc@1 for MSCOCO, but the general trend is also present in Flickr30K. As seen in Figure 2, fine-tuning allows for an improvement in accuracy over the zero-shot setting initially by 8.06% and 9.47% for CLIP + DN* and CLIP respectively in the text-to-image case, and 6.02% and 4.88% in the image-to-text case in the first epochs. But the fine-tuning performance gradually decays after that due to overfitting to the training set. At their highest peak, CLIP + DN* achieves 43.06% accuracy compared to CLIP's 42.39% for text-to-image retrieval, and 59.82% compared to 59.57% for image-to-text retrieval. We observe for both image-to-text retrieval and text-to-image retrieval that CLIP + DN* performs better than CLIP across all fine-tuning epochs by an average of 0.80% for text-to-image, and 0.60% for image-to-text case. A similar pattern is also observed in Flickr30K. Our experiments on the fine-tuning setting provide an affirmative answer to the last question we seek to answer, that DN* provides a non-trivial performance boost to both pre-trained and fine-tuned cross-modal representation models.

4.6.3 Number of Samples for Mean Estimation

We conduct experiments to examine the performance of DN* when different numbers of samples are used to estimate the mean. Using the ImageNet1K, Cifar100, SUN397, and Stanford Cars datasets,

we first test the retrieval accuracy of DN* by estimating the mean with the entire test set. We use the performance achieved here as an upper bound, because the mean is most reliable if it is directly calculated on all of the unlabeled testing samples. We compare it with the estimated means using 1, 10, and 100 random samples from the training set, and present our results in Table 6.

As shown in the table, decreasing the number of samples to just 100 from the training set results in an accuracy drop of no more than 0.2% compared to the upper bound, with an average of only 0.09%. We find that even decreasing the number of samples to as few as 10 does not significantly hurt the result. For 10 samples, there is an average drop in accuracy of 0.14%, with the greatest difference of no more than 0.4%. Moreover, for CLIP, TCL, and ALBEF, we find that estimating the mean with as few as 10 samples still improved accuracy over the base model by an average of 1.96% for Acc@1, and 1.92% for Acc@5. Furthermore, our results are generally stable despite the low number of sample points used to estimate the mean. Across 5 random samplings, the standard deviation of our accuracy is on average 0.08% for 100 training set samples, and 0.22% for 10 training set samples. Even in the extreme case of just 1 sample, we still observe a slight improvement of 1.15% over the base model averaged across Acc@1 and Acc@5, but with a slightly larger standard deviation of 0.77%. These ablation results show that DN* can generalize very well even in the setting of extremely scarce data, possibly because DN* only needs to know the global mean of the distribution and does not require fine-grained local details.

5 Limitations and Social Impact

The success of distribution normalization builds on access to a small amount (10-100) of unlabeled data from the test distribution. Although much cheaper and more easily accessible than the setting of fine-tuning or few-shot learning where more labeled data is needed, the use of unlabeled data can still be a potential limitation in cases when the test distribution is entirely unknown. This work builds on top of a recent line of work that consolidates vast amounts of data from public resources on the Internet to learn a joint representation space of images and text. Therefore, it is possible that DN might inherit or strengthen the social biases in those large visual-language models.

6 Conclusion

In this paper, we identify a mismatch between the pre-training objective of cross-modal representation models like CLIP and its downstream use with a direct dot product. To address this problem, we propose Distribution Normalization (DN) that not only displays a significant advantage over direct dot product in a wide variety of settings but is also straightforward to implement. However, though being extremely sample efficient, DN still requires estimating a mean separately for each distribution and it can be of interest if a universal mean can be found so that DN is more easily applied to a wider range of downstream applications. Future research can also explore the implications of DN to the contrastive training process of cross-modal representation models.

References

P. Anderson, B. Fernando, M. Johnson, and S. Gould. SPICE: semantic propositional image caption evaluation. CoRR , abs/1607.08822, 2016. URL http://arxiv.org/abs/1607. 08822 .

S. Banerjee and A. Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL https: //aclanthology.org/W05-0909 .

T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. CoRR , abs/2002.05709, 2020. URL https://arxiv.org/ abs/2002.05709 .

T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big self-supervised models are strong semi-supervised learners. CoRR , abs/2006.10029, 2020. URL https://arxiv. org/abs/2006.10029 .

X. Chen, H. Fan, R. B. Girshick, and K. He. Improved baselines with momentum contrastive learning. CoRR , abs/2003.04297, 2020. URL https://arxiv.org/abs/2003.04297 .

X. Chen, H. Fan, R. B. Girshick, and K. He. Improved baselines with momentum contrastive learning. CoRR , abs/2003.04297, 2020. URL https://arxiv.org/abs/2003.04297 .

Y. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. UNITER: learning universal image-text representations. CoRR , abs/1909.11740, 2019. URL http: //arxiv.org/abs/1909.11740 .

Y. Cui, G. Yang, A. Veit, X. Huang, and S. J. Belongie. Learning to evaluate image captioning. CoRR , abs/1806.06422, 2018. URL http://arxiv.org/abs/1806.06422 .

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009. doi: 10.1109/CVPR.2009.5206848.

P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 8780-8794. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/ 49ad23d1ec9fa4bd8d77d02681df5cfa-Paper.pdf .

F. Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler. VSE++: improved visual-semantic embeddings. CoRR , abs/1707.05612, 2017. URL http://arxiv.org/abs/1707.05612 .

Z. Gan, Y. Chen, L. Li, C. Zhu, Y. Cheng, and J. Liu. Large-scale adversarial training for vision-and-language representation learning. CoRR , abs/2006.06195, 2020. URL https: //arxiv.org/abs/2006.06195 .

J. Grill, F. Strub, F. Altch´ e, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. ´ A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent: A new approach to self-supervised learning. CoRR , abs/2006.07733, 2020. URL https://arxiv.org/abs/2006.07733 .

Z. Guo, R. Zhang, L. Qiu, X. Ma, X. Miao, X. He, and B. Cui. Calip: Zero-shot enhancement of clip with parameter-free attention, 2022.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick. Momentum contrast for unsupervised visual representation learning. CoRR , abs/1911.05722, 2019. URL http://arxiv.org/abs/1911. 05722 .

J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi. Clipscore: A reference-free evaluation metric for image captioning. CoRR , abs/2104.08718, 2021. URL https://arxiv. org/abs/2104.08718 .

M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research , 47:853-899, 2013.

A. G. Howard. Some improvements on deep convolutional neural network based image classification. arXiv preprint arXiv:1312.5402 , 2013.

A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F. Makedon. A survey on contrastive self-supervised learning. CoRR , abs/2011.00362, 2020. URL https://arxiv.org/abs/ 2011.00362 .

C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. CoRR , abs/2102.05918, 2021. URL https://arxiv.org/abs/2102.05918 .

M. Jiang, Q. Huang, L. Zhang, X. Wang, P. Zhang, Z. Gan, J. Diesner, and J. Gao. Tiger: Text-to-image grounding for image caption evaluation. CoRR , abs/1909.02050, 2019. URL http://arxiv.org/abs/1909.02050 .

H. Kane, M. Y. Kocyigit, A. Abdalla, P. Ajanoh, and M. Coulibali. NUBIA: NeUral based interchangeability assessor for text generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation , pages 28-37, Online (Dublin, Ireland), Dec. 2020. Association for Computational Linguistics. URL https://aclanthology.org/2020.evalnlgeval-1.4 .

J. Kasai, K. Sakaguchi, L. Dunagan, J. Morrison, R. L. Bras, Y. Choi, and N. A. Smith. Transparent human evaluation for image captioning. arXiv preprint arXiv:2111.08940 , 2021.

I. Kim, Y. Kim, and S. Kim. Learning loss for test-time augmentation. Advances in Neural Information Processing Systems , 33:4163-4174, 2020.

W. Kim, B. Son, and I. Kim. Vilt: Vision-and-language transformer without convolution or region supervision, 2021. URL https://arxiv.org/abs/2102.03334 .

J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) , Sydney, Australia, 2013.

A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

H. Lee, S. Yoon, F. Dernoncourt, D. S. Kim, T. Bui, and K. Jung. ViLBERTScore: Evaluating image caption using vision-and-language BERT. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems , pages 34-39, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.eval4nlp-1.4. URL https://aclanthology.org/2020.eval4nlp-1.4 .

F.-F. Li, M. Andreeto, M. Ranzato, and P. Perona. Caltech 101, Apr 2022.

G. Li, N. Duan, Y. Fang, D. Jiang, and M. Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. CoRR , abs/1908.06066, 2019. URL http: //arxiv.org/abs/1908.06066 .

J. Li, R. R. Selvaraju, A. D. Gotmare, S. R. Joty, C. Xiong, and S. C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. CoRR , abs/2107.07651, 2021. URL https://arxiv.org/abs/2107.07651 .

W. Li, C. Gao, G. Niu, X. Xiao, H. Liu, J. Liu, H. Wu, and H. Wang. UNIMO: towards unified-modal understanding and generation via cross-modal contrastive learning. CoRR , abs/2012.15409, 2020. URL https://arxiv.org/abs/2012.15409 .

X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, Y. Choi, and J. Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. CoRR , abs/2004.06165, 2020. URL https://arxiv.org/abs/2004.06165 .

X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.

Y. Li, N. Wang, J. Shi, J. Liu, and X. Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04779 , 2016.

C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013 .

T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick. Microsoft COCO: common objects in context. CoRR , abs/1405.0312, 2014. URL http://arxiv.org/abs/1405.0312 .

X. Lu, T. Zhao, and K. Lee. Visualsparta: Sparse transformer fragment-level matching for large-scale text-to-image search. CoRR , abs/2101.00265, 2021. URL https://arxiv.org/ abs/2101.00265 .

F. Maria Carlucci, L. Porzi, B. Caputo, E. Ricci, and S. Rota Bulo. Autodial: Automatic domain alignment layers. In Proceedings of the IEEE international conference on computer vision , pages 5067-5075, 2017.

N. Moshkov, B. Mathe, A. Kertesz-Farkas, R. Hollandi, and P. Horvath. Test-time augmentation for deep learning-based cell segmentation on microscopy images. Scientific reports , 10(1):1-7, 2020.

Z. Nado, S. Padhy, D. Sculley, A. D'Amour, B. Lakshminarayanan, and J. Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.10963 , 2020.

M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing , Dec 2008.

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311-318, 2002.

O. Patashnik, Z. Wu, E. Shechtman, D. Cohen-Or, and D. Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2085-2094, October 2021.

X. Peng, K. Wang, Z. Zhu, and Y. You. Crafting better contrastive views for siamese representation learning. CoRR , abs/2202.03278, 2022. URL https://arxiv.org/abs/2202.03278 .

B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. CoRR , abs/1505.04870, 2015. URL http://arxiv.org/abs/1505.04870 .

L. Qiu, R. Zhang, Z. Guo, Z. Zeng, Y. Li, and G. Zhang. Vt-clip: Enhancing vision-language models with visual-guided texts, 2022.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. CoRR , abs/2103.00020, 2021. URL https://arxiv.org/abs/ 2103.00020 .

C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier. Collecting image annotations using amazon's mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's Mechanical Turk , pages 139-147, 2010.

J. Robinson, C. Chuang, S. Sra, and S. Jegelka. Contrastive learning with hard negative samples. CoRR , abs/2010.04592, 2020. URL https://arxiv.org/abs/2010.04592 .

S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems , 33:11539-11551, 2020.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.

D. Shanmugam, D. Blalock, G. Balakrishnan, and J. Guttag. Better aggregation in test-time augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1214-1223, 2021.

M. Shu, W. Nie, D.-A. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models, 2022.

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

K. Sohn. Improved deep metric learning with multi-class n-pair loss objective. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016. URL https://proceedings. neurips.cc/paper/2016/file/6b180037abbebea991d8b1232f8a8ca9-Paper.pdf .

Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. CoRR , abs/1906.05849, 2019. URL http://arxiv.org/abs/1906.05849 .

Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6827-6839. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/ file/4c2e5eaae9152079b9e95845750bb9ab-Paper.pdf .

V. Udandarao, A. Gupta, and S. Albanie. Sus-x: Training-free name-only transfer of visionlanguage models, 2023.

A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. CoRR , abs/1807.03748, 2018. URL http://arxiv.org/abs/1807.03748 .

R. Vedantam, C. L. Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. CoRR , abs/1411.5726, 2014. URL http://arxiv.org/abs/1411.5726 .

G. Wang, W. Li, M. Aertsen, J. Deprest, S. Ourselin, and T. Vercauteren. Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing , 338:34-45, 2019.

H. Wang, Y. Zhang, Z. Ji, Y. Pang, and L. Ma. Consensus-aware visual-semantic embedding for image-text matching. CoRR , abs/2007.08883, 2020. URL https://arxiv.org/abs/2007. 08883 .

T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. CoRR , abs/2005.10242, 2020. URL https://arxiv.org/ abs/2005.10242 .

J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 3485-3492, 2010. doi: 10.1109/CVPR.2010.5539970.

J. Yang, J. Duan, S. Tran, Y. Xu, S. Chanda, L. Chen, B. Zeng, T. Chilimbi, and J. Huang. Vision-language pre-training with triple contrastive learning, 2022. URL https://arxiv. org/abs/2202.10401 .

Y. Yi, H. Deng, and J. Hu. Improving image captioning evaluation by considering inter references variance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 985-994, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.93. URL https://aclanthology.org/2020.acl-main.93 .

C. You, R. Zhao, L. H. Staib, and J. S. Duncan. Momentum contrastive voxel-wise representation learning for semi-supervised volumetric medical image segmentation. CoRR , abs/2105.07059, 2021. URL https://arxiv.org/abs/2105.07059 .

J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models, 2022. URL https://arxiv.org/abs/2205. 01917 .

X. Yuan, Z. Lin, J. Kuen, J. Zhang, Y. Wang, M. Maire, A. Kale, and B. Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6995-7004, June 2021.

X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer. Lit: Zero-shot transfer with locked-image text tuning. CoRR , abs/2111.07991, 2021. URL https: //arxiv.org/abs/2111.07991 .

Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. CoRR , abs/2010.00747, 2020. URL https://arxiv.org/abs/2010.00747 .

K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

A More Related Works

Contrastive Learning Contrastive learning has become a popular method for self-supervised representation learning, where the goal is to learn embedding functions such that semantically similar samples are closer in embeddings space while semantically dissimilar samples are embedded further apart [59, 66, 47, 70, 72, 13]. Among various contrastive learning methods, this paper specifically focuses on the InfoNCE loss, which optimizes the probability of correctly identifying the positive sample [62], as it has been one of the most frequently used objectives in recent contrastive learning tasks [16, 5, 3, 4].

Image Caption Evaluation Image caption evaluation is the task of automatically scoring the quality of a caption given the corresponding image and optionally reference captions. The quality of image caption metrics are evaluated with respect to its correlation with human ratings. While traditional image caption metrics such as BLEU-4 [45], ROUGE-L [38], METEOR [2], CIDEr [63], and SPICE [1] evaluate the candidate caption by measuring the n-gram overlap of the candidate with the references, more contemporary neural metrics [22, 69, 23, 30, 8] judge the candidate caption by comparing the neural embeddings of the caption with that of the image and references, and are much more flexible. Among them, the one most related to our work is CLIPScore [17]. By simply passing test-time image and caption pairs through the pre-trained CLIP image and text encoders and taking the dot product, CLIPScore achieves state-of-the-art performance using only the candidate caption and the corresponding image without the need for references.

B Base Representation Model Details

This paper examines the effect of a Distribution Normalization layer on the following state-of-the-art open-sourced cross-modal representation models:

CLIP [50] is one of the earliest pioneers that successfully created a joint representation space of images and text through large-scale contrastive learning. It is trained on over 400M image-caption pairs collected by the authors. They have open-sourced multiple versions of pretrained models and this paper examines the commonly used version 'ViT-B/32'.

ALBEF [33] is one of the later variants of CLIP that adds a multi-modal encoder to capture the interplay between images and text by predicting if they are paired samples or hard negatives. Additional momentum distillation loss and masked language modeling loss are also added to improve the model performance. Two versions of pre-trained models trained on dataset consisting of 4M and 14M unique images are released and this work uses the latter. To focus on the effect of DN on cross-modal representations, we only keep the image and text encoder component of ALBEF and do not use the multi-modal encoder component.

TCL [68] is another state-of-the-art CLIP variant. It inherits the same model architecture as ALBEF, but adds triplet contrastive loss including Cross-modal Alignment, Intra-modal Contrastive, and Local MI Maximization. Their model is trained on 4M unique images. As with ALBEF, we only keep the image and the text encoder component.

C Datasets Details

There is a total of 12 datasets mentioned in this paper, as described below.

C.1 Cross-modal Retrieval Datasets

MSCOCO [39] is a multi-purpose dataset known for its rich compatibility with object detection, segmentation, and image captioning tasks. It contains 118K images in its training split and more than 5K images in its test split.

Flicker30K [48] is a popular benchmark for sentence-based picture portrayal. It contains 31K images with each image having five reference sentences generated by human annotators. We took a train-test split following [68] and [33], which involves a selection of 30K images for fine-tuning and 1K images for testing.

C.2 Zeroshot Classification Datasets

ImageNet1K [9] is the most well-known dataset for image classification that contains more than 1200K images in its training set and 100K images in its test set and covers 1000 categories of objects. Accuracy on ImageNet1K is especially widely used for evaluating state-of-the-art image classification models.

Cifar100 [28] is a dataset containing 100 classes of objects, with each class having 500 training images and 100 test images.

SUN397 [67], the Scene UNderstanding database, contains 899 categories and about 130K images. 397 well-sampled categories are available for benchmarking state-of-the-art algorithms for scene recognition tasks.

Stanford Cars [27] is an image classification dataset dedicated for cars. It collects about 16K images for 196 types of cars and adopts a 50-50 train-test split. Results on Stanford Cars test our method's effectiveness in improving special-purpose classification models.

Caltech101 [31] is a dataset for object recognition tasks. It contains 101 object categories, with varying numbers of images in each category (between 40 to 800 images per category). The dataset has over 9,000 images in total collected from the internet. It is a popular benchmark for evaluating the performance of algorithms on general-purpose single-object recognition tasks.

Flowers102 [44] is an image classification dataset focused on flowers, comprising 102 distinct categories of flowers common to the UK. It includes a diverse set of images, with each class containing between 40 to 258 images. The dataset comprises over 8,000 images.

C.3 Image Captions Evaluation Datasets

Flickr8K-Expert [18] is a curated subset of the Flickr8K dataset that contains 17K human ratings of image-caption pairs. Each human score corresponds to one pair and is from 1 to 4 (1 means the caption is irrelevant, 4 means the caption describes the image fully correctly.)

Flicker8K-CF [18] is a similar dataset gathered from CloudFlower that contains 48K image-caption pairs and 145K binary ratings of these pairs.

THumb [24] is a dataset containing some machine- and human-generated captions from the MSCOCO dataset. It has 500 images, each with 5 candidates captions that are evaluated by human annotators.

Pascal-50S [51] contains 4K caption-caption pairs with each pair describing the same image. For all pairs, annotators give a preference on which caption in the pair provides better description of the image. Caption-caption pairs are categorized based on the origins of the captions (see Table 7 for details about categories.)

D Additional Results

D.1 Fine-tuning Epochs on MSCOCO and Flickr30K

In Figure 3, we present the remaining two plots of fine-tuning ablations for MSCOCO. CLIP + DN* does better than CLIP by an average of 0.52% for Acc@5, and 0.37% for Acc@10. Both graphs observe a peak in accuracy around 4-5 epochs. For Acc@5, we observe an initial improvement of 9.10% for CLIP + DN* after just 1 epoch, and 11.10% for CLIP on text-to-image retrieval. These respective numbers are 5.17% and 4.39% for image-to-text retrieval. For Acc@10, we observe an initial improvement of 8.53% for CLIP + DN* and 10.03% for CLIP on text-to-image retrieval, and 3.14% and 3.14% respectively for image-to-text retrieval.

In Figure 4, we present the effects of finetuning on Acc@5 and Acc@10 for Flickr30K. Again, CLIP+DN* does better than CLIP by an average of 0.22% for Acc@5 and 0.14% for Acc@10. All results yield a similar trend in support of the conclusion we have made in Section 4.6.2.

0

2

4

6

8

10

56

58

60

62

64

66

68

70

T2I: Accuracy @ Top5

CLIP Top5

CLIP+DN* Top5

0

2

4

6

8

10

76

77

78

79

80

81

82

83

I2T: Accuracy @ Top5

CLIP Top5

CLIP+DN* Top5

0

2

4

6

8

10

66

68

70

72

74

76

78

80

CLIP Top10

CLIP+DN* Top10

0

2

4

6

8

10

85

86

87

88

89

90

I2T: Accuracy @ Top10

CLIP Top10

CLIP+DN* Top10

0

2

4

6

8

10

86

87

88

89

90

91

92

T2I: Accuracy @ Top5

CLIP Top5

CLIP+DN* Top5

0

2

4

6

8

10

95.0

95.5

96.0

96.5

97.0

97.5

I2T: Accuracy @ Top5

CLIP Top5

CLIP+DN* Top5

0

2

4

6

8

10

92.0

92.5

93.0

93.5

94.0

94.5

95.0

95.5

CLIP Top10

CLIP+DN* Top10

0

2

4

6

8

10

97.4

97.6

97.8

98.0

98.2

98.4

98.6

98.8

I2T: Accuracy @ Top10

CLIP Top10

CLIP+DN* Top10

D.2 Information Loss in Approximation

Without considering computational efficiency, for each test sample, we can iterate over the entire unlabeled reference set to calculate a similarity measure given in Eqn.5:

To investigate how much information might be lost from the first-order approximation in Section 3.2.2 and algebraic mean to geometric mean conversion in Section 3.3 that simplifies Eqn.10 to distribution normalization in Eqn.9, we carry out experiments to compare them back to back in the task of image captioning metric and present our results in Table 8 in terms of their correlation with human judgments on image captioning datasets. Surprisingly, in all the base models and datasets that we have studied, we did not notice any significant difference between DN and Eqn. 10 (difference < 0 . 1% ). This shows that higher-order information contributes negligibly to the downstream applications compared to only taking the mean of the distribution, and taking an algebraic mean achieves a similar effect as taking a geometric mean.As a conclusion, we found that DN provides equivalent performance to the original Eqn.5 without incurring expensive computational costs.

D.2.1 Pixel-Level Normalization

A potential alternative to distribution normalization which normalizes the data on the representation level is a commonly used trick that normalizes the data (mostly images) on the input level (pixel level), which we refer as pixellevel normalization. In Table 9, we present our the comparison results between CLIP + pixelnorm and our proposed methods. However, although using the same data for normalization on the representation level as DN and DN* yields a large gain over vanilla CLIP, changing the normalization to the pixel level wipes out all the improvements. We hypothesize that this is because the difference of a constant vector

(representation mean) vanishes while the input goes through a large neural network like CLIP.

Peer reviews:
comments: SUMMARY:
This paper reveals a mismatch between the pre-training objective of contrastively trained vision-language models and their downstream usage. The authors propose  Distribution Normalization (DN) to solve this problem. The results on a wide variety of downstream tasks show the effectiveness of the proposed method.

STRENGTHS:
1. The finding about the misalignment between pretraining and downstream tasks for contrastively trained vision-language models is important.

2. The proposed method, DN, is simple but effective, and also conveniently implemented in practice.

3. The experiments on various tasks show the effectiveness of the proposed method, and also there are sufficient ablation studies.

WEAKNESSES:
Overall, I think this paper is well-presented and reveal an important problem in contrastively trained vision-language models. However, I still have the below concerns:

1. A recent work, TPT [1], is missing, which aims to learn prompts for vision-language models at the test-time. I recommend that the authors will compare DN with TPT and integrate DN into the TPT’s framework.

2. I suggest that the authors will modify “visual-language” to “vision-language” in the next version.

3. Another concern is that because DN  needs a small amount of unlabeled data, its setting actually is not the zero-shot one, thus I suggest the authors will refine the related descriptions, e.g. Line 175.


[1] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models. NeurIPS 2023.

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
The authors have adequately discussed the limitations of this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes distribution normalization (DN) for contrastively trained vision-language models. The idea is motivated by an analysis of the InfoNCE loss. The authors identify that the common practice of taking dot product for zero-shot inference is only a zero-order approximation of the InfoNCE loss. The proposed method improves the dot product inference by using unlabeled test data to make a first-order approximation.\n\nSTRENGTHS:\n1. The analysis in Section 3.2 is intuitive and motivates the proposed method.\n2. The method is efficient: it only needs to subtract the mean of the text/image features of the test data. \n3. The experiment section evaluates the method on multiple tasks and various contrastively pre-trained models.\n\nWEAKNESSES:\n1. There are some missing baselines for CLIP zero-shot classification. For example, [1] proposes test-time prompt tuning for CLIP, which is less efficient than DN, but it works on a single test sample. [2] uses multiple test samples from the same distribution to do logit normalization, which is similar to the proposed method. \n\n\n[1]. Shu, Manli, et al. \"Test-time prompt tuning for zero-shot generalization in vision-language models.\", NeurIPS 2022.       \n[2]. Allingham, James Urquhart, et al. \"A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models.\", ICML 2023.\n\nQUESTIONS:\n1. Regarding performance, I find DN's improvement margins on CLIP to be smaller than on other models. Any idea what would cause this discrepancy between different contrastively-trained models?\n\nLIMITATIONS:\n1. As mentioned by the author, as a limitation of this work, DN requires access to multiple test samples and assumes they are all from the same distribution. Such an assumption may not hold in practical settings.\n2. Many existing works have explored The idea of using test data for normalization at test time [2, 3, 4]. The authors should consider citing them and discussing their connections and differences. In addition, there is another work also analyzes the loss functions for test-time adaptation [5]. \n\n[3]. Schneider, Steffen, et al. \"Improving robustness against common corruptions by covariate shift adaptation.\" NeurIPS 2020.    \n[4]. Wang, Dequan, et al. \"Tent: Fully test-time adaptation by entropy minimization.\", ICLR 2021.    \n[5]. Goyal, Sachin, et al. \"Test-time adaptation via conjugate pseudo-labels.\", NeurIPS 2022.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCLIP is trained using InfoNCE loss where positive and negative pair alignment is done during training. However, at inference, we simply take the dot product with text embeddings which is not the optimal/similar to the way pretraining was done. Authors propose to rectify this inference and training objective misalignment, based on a first order approximation of InfoNCE loss. Specifically, they subtract the mean representation (calculated over a validation set) from the test image and label/caption representation before applying the dot product. The empirical evaluations of the proposed approach however do not give strong enough gains, especially on CLIP.\n\nSTRENGTHS:\n- Approach is easy to implement.\n- Paper writing is generally smooth and clear. However, section around Eqn 8 (L130 to L133) should be improved and is not clear.\n\nWEAKNESSES:\n- Any reason why the proposed approach does not work well on MSCOCO retrieval image to text. \n- Empirical gains are pretty nominal, especially for the CLIP models. The authors average their gains over CLIP and other not widely used models (TCL and ALBEF). However, the gains seems to be much lower on CLIP itself. \n- Moreover, as mentioned in L252, it seems that there are larger gains with models pretrained on smaller datasets. This suggests that as the pretraining corpus size is increased, say we use CLIP pretrained on LAION5B, there may not be any gains. Can the reviewers test their approach on CLIP pretrained on larger datasets like DataComp XL ( https://huggingface.co/laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K ) or LAION5B.\n- Authors should test their approach on larger and more standard CLIP architecture like ViT-B-16 and ViT-L, which have higher zeroshot accuracies. \n- Table 2, authors report 61.0 as the zeroshot accuracy of CLIP ViT-B-32, which is however 63.2 (https://arxiv.org/pdf/2103.00020.pdf). Can the authors please clarify this.\n- Can the authors provide finetuning experiments on image classification datasets as well like ImageNet or StanfordCars/Flowers/etc. considered in Table 2. \n- (typo) Eq 5, D_s should be D_t in expectation.\n- The supplementary material is supposed to be submitted as a separate PDF and not part of the main paper, to the best of my knowledge.\n\nQUESTIONS:\nPlease see the weakness section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of retrieval and classification accuracy using vision-language representation.  The authors claimed that there is a mismatch between training objectives and inference-time operation. Specifically, InfoNCE used negative information but test time only use positive similarity scores. The authors presented findings that rectifying such misalignment can boost performance consistently across a variety of downstream tasks. They argued that it is sufficient to use the first-order approximation of InfoNCE loss, i.e. just subtracting the mean in a test batch before doing dot product at test time, and called it Distribution Normalization (DN). They applied their test-time augmentation to a number of existing approaches such as CLIP, ALBEF, and TCL, and show certain improvements over the baselines on common classification and retrieval benchmarks such as MSCOCO and Flickr30K.\n\nSTRENGTHS:\nThe paper is clearly written and the proposed approach and statement are simple and easy to understand. The experiments show consistent improvements over baselines that don't use test-time augmentation. \nThe authors applied their approach to some of the top performers and tested them on the various common benchmarks. They also provided a decent set of ablation studies on how test-time batch size affects performance and how this simple normalization help in the fine-tuning setting.\n\nWEAKNESSES:\nOn line 110, the claim that 0.1 is small is not very convincing, since in practice, the value of the referred expressions is typically much smaller, especially after L2 normalization. Such oversimplification as this makes the final outcome, I.e. DN method, looks simplistic.\n\nThe approach in this paper required test-time augmentation, and when compared to other test-time augmentation approaches such as TTA of [54], the result was even inferior in some cases. Only when in combination with TTA, there was marginal improvements (over TTA[54])\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper focus on the misalignment of training and testing of CLIP model. Specifically, CLIP is trained with an InfoNCE	loss containing both positive and negative samples, while tested lack negative samples information. They reveal that the common downstream practice of taking a dot product is only a zeroth-order approximation of the optimization goal, and propose distribution normalization to narrow the gap at test time considering both effectiveness and efficiency. Experimental results show some improvements on different datasets.

STRENGTHS:
-This paper propose an interesting perspective for the difference of the optimization goal between train- and test-time.
-The analysis of InfoNCE Loss is easy to understand.
-The paper is well written, and easy to follow.

WEAKNESSES:
-There is no relevant introduction about test-time tasks in the first paragraph, and the necessity of performing this task at test-time is questionable.
-The proposed distribution normalization requires prior knowledge of the entire test set, which is often not met in practice, making it unable to handle sustained data.
-The performance gain is limited. Is it the limitation of the first-order approximation?

QUESTIONS:
see weakness.

LIMITATIONS:
The limitations are adequately addressed in the paper and seem reasonable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07