Title: Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP
Venue: NeurIPS 2023
Accepted: yes
Keywords: zero-shot; clip; proxy learning
Abstract: Vision-language pre-training methods, e.g., CLIP, demonstrate an impressive zero-shot performance on visual categorizations with the class proxy from the text embedding of the class name. However, the modality gap between the text and vision space can result in a sub-optimal performance. We theoretically show that the gap cannot be reduced sufficiently by minimizing the contrastive loss in CLIP and the optimal proxy for vision tasks may reside only in the vision space. Therefore, given unlabeled target vision data, we propose to learn the vision proxy directly with the help from the text proxy for zero-shot transfer. Moreover, according to our theoretical analysis, strategies are developed to further refine the pseudo label obtained by the text proxy to facilitate the intra-modal proxy learning (InMaP) for vision. Experiments on extensive downstream tasks confirm the effectiveness and efficiency of our proposal. Concretely, InMaP can obtain the vision proxy within one minute on a single GPU while improving the zero-shot accuracy from $77.02\%$ to $80.21\%$ on ImageNet with ViT-L/14@336 pre-trained by CLIP.

Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP

Abstract

Vision-language pre-training methods, e.g., CLIP, demonstrate an impressive zeroshot performance on visual categorizations with the class proxy from the text embedding of the class name. However, the modality gap between the text and vision space can result in a sub-optimal performance. We theoretically show that the gap cannot be reduced sufficiently by minimizing the contrastive loss in CLIP and the optimal proxy for vision tasks may reside only in the vision space. Therefore, given unlabeled target vision data, we propose to learn the vision proxy directly with the help from the text proxy for zero-shot transfer. Moreover, according to our theoretical analysis, strategies are developed to further refine the pseudo label obtained by the text proxy to facilitate the intra-modal proxy learning (InMaP) for vision. Experiments on extensive downstream tasks confirm the effectiveness and efficiency of our proposal. Concretely, InMaP can obtain the vision proxy within one minute on a single GPU while improving the zero-shot accuracy from 77 . 02% to 80 . 21% on ImageNet with ViT-L/14@336 pre-trained by CLIP.

1 Introduction

Vision-language pre-training in CLIP aims to align image-text pairs to facilitate multi-modal understanding for downstream tasks [26]. By optimizing the contrastive loss consisting of images and their corresponding text descriptions, CLIP demonstrates an impressive zero-shot transfer of the pre-trained model to generic tasks. Concretely, given a downstream data set, each class has a proxy obtained from the text embedding about the corresponding class name, and then images can be categorized by finding the nearest class proxy. With class names only, zero-shot accuracy on ImageNet [28] can achieve 77 . 02% with ViT-L/14@336 [9] pre-trained by CLIP.

After the success of CLIP, many research efforts have been devoted to improving the transfer performance on downstream tasks. Most of developed methods can be cast into two categories, that is, few-shot refinement and zero-shot enhancement. Few-shot methods require a small amount of labeled data from the target task to refine the prediction [12, 41, 42]. While these methods show a better performance than zero-shot learning, labeled examples from the target domain may not be available in real-world applications. Therefore, many methods focus on improving the zero-shot learning itself by exploring side information. On one hand, many pre-trained large language models (LLMs), e.g., GPT-3 [4], can be leveraged to obtain better text proxy for classification [20]. On the other hand, unlabeled data from the target domain can be adopted to fine-tune an adapter network or input prompts [23, 30] for alignment. In this work, we focus on understanding the behavior of CLIP by exploiting the unlabeled data without external LLMs or additional architectures for zero-shot categorization.

When considering pre-trained CLIP models and excluding any auxiliary networks for zero-shot transfer, many existing methods try to optimize input text prompts to align images and the text of class names better [30, 33, 42]. However, recent observations demonstrate that the modality gap between text (i.e., language) and vision is still significant after the optimization of CLIP [18]. Consequently, the efforts on improving the text proxy can be ineffective for the vision task.

To mitigate the problem from the gap between the text and vision space, we propose to obtain proxies of classes in the vision space directly. First, our theoretical analysis in Proposition 1 shows that the modality gap will be preserved due to a small temperature parameter in the contrastive loss of CLIP, which confirms the empirical observation in [18]. Furthermore, Proposition 2 indicates that the optimal proxy for the vision task can only be obtained from the vision space. Therefore, with the gap between the text and vision space, directly using the text proxy makes the zero-shot transfer sub-optimal. To recover the optimal vision proxy, we develop an intra-modal proxy learning (InMaP) method to obtain visual proxies with pseudo labels from text proxies. The proposed method is illustrated in Fig. 1. With fixed features extracted by CLIP encoders, the optimization problem in InMaP is convex without encoders in the loop, which can be solved efficiently with gradient descent [3]. Our main contributions can be summarized as follows.

Modality

gap

Text

space

Vision

space

Text proxy

Vision proxy

Recover

We theoretically show that the modality gap cannot be reduced sufficiently by minimizing the contrastive loss in CLIP. While the optimal proxy for image classification is in the vision space, the gap degenerates the zero-shot performance of the text proxy as shown in Theorem 1.

We propose a novel algorithm to obtain the intra-modal proxy directly from the vision space. By recovering the vision proxy with help from the text proxy and an appropriate temperature suggested by Proposition 3 for calibration, effective proxies can be observed from unlabeled data.

Our Theorem 2 indicates that the accuracy of pseudo labels from the text proxy is essential to recover the proxy in the vision space. Therefore, we further develop strategies to refine pseudo labels according to the data distribution by optimal transport (OT) [34].

Experiments on 14 downstream vision tasks demonstrate both the effectiveness and efficiency of our proposal. Concretely, InMaP can achieve 80 . 21% accuracy on ImageNet with ViT-L/14@336 using a single GPU within one minute.

2 Related Work

Before CLIP, zero-shot learning that aims to identify examples of novel classes without any labeled training data has attracted much attention since [22]. However, most of existing works are only able to discover new classes closely related to the training classes [1, 5, 6, 11, 22, 38], where they share the similar attributes, and have to train an individual model for each task. On the contrary, pre-training on large-scale data with the contrastive loss aligns visual and language features and makes a single CLIP model applicable for diverse downstream tasks in a straightforward way. Compared with conventional zero-shot methods, the pre-training data in CLIP may be overlapped with downstream tasks, which can result in data leak for evaluation. While the issue can be addressed by eliminating the overlapping data, the influence on the performance is negligible as discussed in [26].

After pre-training with image-text pairs, the obtained CLIP models can be transferred to various downstream tasks with few-shot or zero-shot strategies. We briefly review these two related directions as follows.

Few-shot transfer CLIP demonstrates that an ensemble of input text prompts for the text encoder can improve the zero-shot performance by a significant margin. Therefore, many methods try to

further refine the prompts. Given a small set of labeled data from the target vision tasks, CoOp [42] optimizes input prompts for the text encoder with labels. By formulating the prompts as learnable variables, it can obtain task-dependent prompts without any hand-crafted designs. On the contrary, CLIP-Adapter [12] optimizes obtained output features from both encoders. However, it adds additional bottleneck layers for training, which increases the complexity of optimization. To reduce the efforts of training with encoders, Tip-Adapter [41] demonstrates a training-free method that incorporates the prediction from a cache model consisting of the labeled image data and that from the text proxy. In this work, we also consider to minimize the training efforts by excluding encoders in optimization and the intra-modal proxy can be obtained from extracted features.

Zero-shot transfer Since labeled data are not always available, many efforts are devoted to improving zero-shot transfer. Some methods explore other large pre-trained models as the supplementary for CLIP. VisDesc [20] leverages GPT-3 to generate rich context descriptions for given class names, and thus shows superior performance over the simple prompt in CLIP. SuS-X [33] includes additional images to mitigate the modality gap. Concretely, auxiliary images can be obtained either from a large data set (e.g., LAION [29]) or a text-to-image generation model (e.g., stable diffusion [27]). The additional large models will inevitably increase the inference time, hence, recent methods consider to leverage the unlabeled target data for fine-tuning. UPL [15] and TPT [30] utilize the unlabeled data to optimize learnable input text prompts. SVL-Adapter [23] trains an additional encoder from the unlabeled data with self-supervised learning for ensemble. Unlike existing methods, our proposal does not need any additional model and we learn the proxy for each class from the vision space directly.

3 The Proposed Method

In this section, we first analyze the modality gap in CLIP, and then elaborate the proposed method that recovers the proxy in the vision space to reduce the influence from the modality gap.

3.1 Cross-Modal Learning in CLIP

By pre-training paired images and text, a vision-language model, e.g., CLIP [26], is capable of classifying unlabeled images in the zero-shot manner. Let f ( · ) and g ( · ) denote the encoder for images and text in the pre-trained CLIP model, respectively. Given a set of unlabeled image data { x i } n i =1 and the unique class names for the set { z j } C j =1 , their corresponding visual and text representations can be extracted as

where x i ∈ R d and z j ∈ R d have the same dimension and a unit norm. d is the dimension of features. z j can be considered as the proxy for the j -th class.

With the features of the i -th image and all text proxies, the image can be classified as

where y i is the prediction from the zero-shot classification. The capacity of zero-shot prediction is from the alignment between vision and language that is optimized by the pre-training objective in CLIP. Given image-text pairs as ( x i , t i ) , two encoders are learned by minimizing the contrastive loss

where m is the mini-batch size, t i = g ( t i ) is the extracted text representation of t i with a unit norm, and τ is the temperature for the normalized cross entropy loss.

While CLIP demonstrates an impressive zero-shot performance, the cross entropy loss focuses on cross-modal ranking and can be sub-optimal for intra-modal vision categorization. It is because that cross entropy loss is closely related to a triplet loss [25] as

which mainly aims to rank the multi-modal positive pair over negative pairs rather than to mix the text and vision space for modality gap reduction. Moreover, a small temperature, e.g., 0 . 01 in CLIP, will

amplify the difference between pairs and preserve the distance between modalities. The phenomenon can be stated as follows. The detailed proof can be found in the appendix.

Proposition 1. Assuming exp( x ⊤ i t i /τ ) ∑ j m exp( x ⊤ i t j /τ ) = δ , we have

̸

where k denotes the nearest negative pair as k = arg max j : j = i x ⊤ i t j . c 1 and c 2 are constants. c 1 > 0 when δ > 1 / 2 and c 2 > 0 when δ > 1 /m .

Remark Proposition 1 implies that when δ is sufficiently large, the relevant text has a higher rank than the irrelevant text, which is the key for zero-shot classification. Moreover, it shows that with the same prediction δ , the absolute distance between modalities, i.e., ∥ x i -t i ∥ 2 2 = 2 -2 x ⊤ i t i , partially depends on the value of temperature, i.e., ∥ x i -t i ∥ 2 2 ≥ 2 -2 x ⊤ i t k -2 c 2 τ . The analysis indicates that optimizing cross entropy loss with a small τ will not pull the text and vision space together.

Some recent work empirically observed that inter-modal distribution is significantly different from the intra-modal distribution in CLIP [18, 33], which confirms our analysis. Consequently, the proxies of class names from the text space may not be sufficient to capture the distribution of vision space, which results in the degenerated zero-shot performance. In lieu of using the text proxy directly for zero-shot visual categorization, we propose to obtain the class proxy in vision space as follows.

3.2 Intra-Modal Proxy Learning

With ground-truth labels { y i } , the standard supervised learning in vision space can be written as

where w j is the learnable proxy for the j -th class and W = [ w 1 , . . . , w C ] ∈ R d × C . τ I is the temperature for the optimization with vision data. With an equivalent form, it is easy to show that w j is in the vision space spanned by { x i } .

Proposition 2. Let W ∗ be the optimal solution to

which is equivalent to Eqn. 1 if w j as well as x i has a unit norm. Then, we have

̸

̸

where p i,j = exp( -∥ x i -w ∗ j ∥ 2 2 / (2 τ I )) ∑ C k exp( -∥ x i -w ∗ k ∥ 2 2 / (2 τ I )) and Π ∥ w ∥ 2 =1 projects the vector to be with a unit norm.

Proof. It is directly from K.K.T condition [3].

Remark Proposition 2 demonstrates that the optimal proxy for vision categorization should be in the vision space. While proxies from the text space can work as the substitute for vision proxies in CLIP, the distance between the text and vision space (i.e., the modality gap) as illustrated in Proposition 1 will degenerate the performance for vision tasks.

To further demonstrate the challenge from the gap between modalities and simplify the analysis, we decompose the proxy of class names as

where z x j is from the vision space spanned by { x i } and z ⊥ j shows the component from the orthogonal subspace such that z x ⊤ j z ⊥ j = 0 . Both z x j and z ⊥ j have the unit norm. z ⊥ j can be considered as encoding the specific information for the text modality. With the decomposition, we can observe that it is possible to recover the optimal prediction if the vision space is covered by the text space as follows.

Proposition 3. Let p i,j and p ′ i,j denote the prediction probability obtained with vision proxies { w ∗ j } and text proxies { z j } , respectively. τ T and τ I denote the temperature in CLIP and that in vision proxy learning, respectively. If z x j = w ∗ j and 0 < a < 1 , we have p ′ i,j = p i,j when τ I = τ T / √ a .

Remark Proposition 3 shows that if the text space contains the whole vision space, the intramodal prediction can be recovered from the cross-modal prediction using a larger temperature as τ I = τ T / √ a , where a measures the overlap between modalities.

However, recent observations indicate that the text and vision space learned by CLIP are distinct with a clear margin [18] and it is hard for the text space to cover the vision space. The difference between the text and vision proxy can be lower-bounded in the following theorem.

Theorem 1. Let Z and W ∗ denote the text proxies and optimal vision proxies, where Z = √ aZ x + √ 1 -aZ ⊥ and Z x consists of a rank r approximation of W ∗ as Z x = U r A ⊤ . W ∗ = U Σ V ⊤ = ∑ d ′ i s i u i v ⊤ i , where d ′ = min { d, C } and s 1 ≥ · · · ≥ s d ′ ≥ 0 . Then, we have

Remark Theorem 1 shows that the gap between text and vision proxy comes from two parts. The former term indicates the distance to the irrelevant text space as 1 - √ a . The latter one depicts the approximation loss from the low-rank overlapping between text space and vision space with r < d ′ . Due to the inherent difference between these two modalities, the distance between Z and W is hard to be minimized.

Therefore, to improve the zero-shot performance in the vision space, we consider to obtain the vision proxy in lieu of directly using the text proxy. The main challenge comes from the fact that the label { y i } is unavailable in zero-shot classification. Fortunately, the intra-modal proxy can be learned by mimicking the proxy from the other modal. Concretely, we propose to minimize the KL divergence between distributions from the text and vision proxy as

where P ′ i and P i indicate the distribution estimated by the text proxy Z and the learnable vision proxy W , respectively.

Proposition 3 suggests that τ I for P i should be larger than τ T in P ′ i to calibrate the magnitude of the distribution and our ablation study confirms the analysis.

Unlike prompt learning methods [30], the problem in Eqn. 2 is defined with extracted features, which does not require the backward operation with large encoder networks for optimization. Moreover, it is convex and the optimal vision proxy can be obtained efficiently by the standard gradient descent [3].

By learning with the pseudo label predicted from the text proxy, we can recover the optimal proxy in the vision space as shown in the following theorem.

Theorem 2. If simplifying the loss function as L ( P ′ , W ) = -∑ i,j P ′ i,j log( P i,j ) and assuming that it is µ -strongly convex in W and letting

where Y i is the ground-truth label distribution for x i , we have

Remark Theorem 2 shows that the distance between the recovered vision proxy and the optimal vision proxy is bounded by the gap between pseudo labels predicted by text proxy and the groundtruth label distribution. After pre-training CLIP models on a large scale data set, it can approximate the distribution of real data, which enables learning of the intra-modal proxy.

Algorithm 1 In traM od a l P roxy Learning (InMaP)

1: Input: Unlabeled image set { x i } , class names { z j } , pre-trained CLIP encoders ( f, g ) , iterations T w , T p , temperature τ T , τ I , threshold α

2: Extract features { x i } and { z j } using pre-trained encoders f and g , respectively.

3: Obtain pseudo labels by solving Eqn. 3 with T p iterations.

4: Refine pseudo labels with the threshold as in Eqn. 5.

5: Obtain vision proxies W by solving Eqn. 2 with T w iterations.

6: return y i = arg max j x ⊤ i w j

3.3 Pseudo Label Refinement

According to Theorem 2, the accuracy of pseudo label is essential for recovering the appropriate vision proxy. Therefore, we develop strategies to improve the quality of pseudo labels.

First, since we have the whole test set rather than a single example for zero-shot learning, the labels obtained from the text proxy can be refined according to the geometry of the target vision data. Given the logits matrix M ∈ R n × C with M i,j = x ⊤ i z j , the original pseudo label can be obtained by solving the problem

where H ( P ) computes the entropy of P and the obtained results should be multiplied by n as the predicted labels. By incorporating a reference distribution q ∈ R C over classes, the problem can be rewritten as

which is known as Sinkhorn distance [8] and is an approximation of optimal transport distance. The refined pseudo label can be obtained by an efficient iterative method [8].

The last challenge is from observing the appropriate reference distribution. Without any prior knowledge, a balanced distribution q = 1 /C is a popular selection in practice. By investigating the prediction from the text proxy, it implies a distribution as ˆ q j = ∑ i P ′ i,j /n . Then, a reference distribution can be obtained by further smoothing ˆ q as

where γ ≤ 1 is the smoothing parameter and γ = 1 implies the original distribution from the text proxy.

Moreover, inspired by the semi-supervised learning [31], soft label with high confidence can be converted into one-hot label to eliminate the influence from irrelevant classes. Let α be the threshold. After obtaining the pseudo label from Eqn. 3, we have

where e k ∈ { 0 , 1 } C is the one-hot vector and the k -th element is 1 .

The proposed intra-modal proxy learning method (InMaP) is summarized in Alg. 1. Pre-trained CLIP models are only applied to extract features as in Step 2, which is the inevitable cost of conventional zero-shot learning. After obtaining features, the computational overhead of our proposed optimizations in Steps 3-5 is mild as shown in the analysis of running time.

4 Experiments

To evaluate the proposed method, we follow the common practice and conduct experiments on ImageNet [28] and 13 diverse downstream vision tasks for zero-shot transfer. Text prompts are

important for obtaining appropriate text proxies. We have the selected 7 prompts in [33] as templates for ensemble, which shows better performance than a single prompt or multiple prompts in CLIP. For the proposed method, the temperature for obtaining pseudo labels with the text proxy τ T is set to 0 . 01 as obtained by CLIP. The temperature for recovering the visual proxy τ I is set to 0 . 04 for all experiments. The intra-modal proxy is learned by standard projected gradient descent, where the initial learning rate is 10 and the number of iterations is 2 , 000 for sufficient training. The learning rate will be decayed by 2 when the norm of gradient increases. Sinkhorn distance is optimized by 20 iterations for refining pseudo labels. All experiments are conducted on a single V100 GPU.

4.1 Ablation Study

First, we have the ablation experiments on ImageNet to demonstrate the efficacy of our method InMaP. Pre-trained ResNet-50 [13] is adopted as the vision encoder and the vanilla zero-shot method in CLIP is denoted as 'Baseline'.

Temperature τ I τ I is used to recover the proxy in the vision space as in Eqn. 2. According to our analysis in Proposition 3, τ I should be larger than τ T in CLIP to leverage the overlap between the text and vision space. Table 1 shows the result when varying τ I in { 0 . 01 , 0 . 02 , 0 . 03 , 0 . 04 , 0 . 05 } . 'Sim' computes the mean similarity between each example and its nearest proxy, depicting the gap (lower the similarly, higher the gap).

First, when τ I = 0 . 01 , the gain is mainly from learning the vision proxy with refined labels and is already 2 . 59% better than the baseline with text proxy. By increasing τ I , the similarity between examples and proxies also increases, which confirms our analysis in Proposition 1. With an appropriate τ I , an additional gain of 0 . 83% can be observed with the similarity of 0 . 39 . According to Proposition 3, a large τ I is crucial to recover the optimal vision proxy with the guidance from the text proxy. Thus, we fix τ I = 0 . 04 for the main comparison.

Pseudo label threshold α Since the SoftMax operator can incur the over-smoothing issue for output logits, converting the soft label with high confidence to one-hot is an effective strategy in semi-supervised learning. We adopt and evaluate the performance with the threshold α in Eqn. 5.

Table 2 summarizes the results with different thresholds. When α = 1 , the soft label after Sinkhorn distance optimization is adopted for optimization, which already surpasses the baseline by a clear margin of 3 . 17% . When reducing α , more soft labels will be converted to one-hot labels and the performance can be further improved by eliminating noise from the SoftMax operator. When letting α = 0 , all pseudo labels become one-hot, which amplifies the noise from low-confidence prediction and degenerates the performance. We find that our method is insensitive to the parameter and fix it as 0 . 6 in the remaining experiments if not specified.

Pseudo label smoothing γ While the validation set of ImageNet is well-balanced with 50 examples for each class, we conduct the experiments with different γ in Eqn. 4 for demonstration.

Table 3 shows the accuracy on ImageNet with varying γ . γ = 1 follows the distribution from the text prediction and degenerates to the original pseudo label without Sinkhorn distance refinement. By balancing the distribution of the class assignment with a small γ , the accuracy can be increased with refined pseudo labels. When γ is sufficiently small, the constraint in Sinkhorn distance degenerates to the balanced constraint and shows the desired performance due to the consistency with the groundtruth distribution. We will have the balanced prior distribution as γ = 0 for other data sets if not specified.

Components in InMaP Finally, we show the gain from two components in InMaP in Table 4. Let 'InMaP 0 . 25 ' denote the proxy learning without refined labels and 'InMaP 0 . 5 ' be the variant optimized by solely applying the threshold α to pseudo labels from the text proxy. 'Sinkhorn' denotes the pseudo labels refined by Sinkhorn distance without proxy learning. We can observe that with the labels predicted from the text proxy, the recovered vision proxy of InMaP 0 . 25 can improve the baseline by 0 . 51% with the reduction in the modality gap. Then, simply thresholding the prediction from the text proxy can further improve the performance by 0 . 12% as in InMaP 0 . 5 . On the other hand, refining the labels via Sinkhorn distance can substantially outperform the baseline by 2 . 21% . Learning with better pseudo labels, InMaP shows an additional gain of 1 . 21% over refined labels. It confirms our analysis in Theorem 2 that the proposed intra-modal proxy learning can benefit from appropriate pseudo labels.

Number of text prompts An ensemble of text prompts shows the superior performance over the single prompt on obtaining text proxy for zero-shot classification [26]. We compare these two strategies on our method and summarize the accuracy on ImageNet in Table 5.

First, we can observe that the ensemble improves the performance of baseline with a clear margin of 2% with different vision encoders. It shows that the vanilla zero-shot transfer strategy is sensitive to the selection of text prompts and confirms the motivation of prompt learning methods. Second, our method outperforms the baseline with the single or ensemble of text prompts for text proxy, which demonstrates the effectiveness of our proposed intra-modal vision proxy. Moreover, the gap between the single and ensemble prompts shrinks on InMaP to less than 1% , which can help reduce the efforts of text prompt tuning.

Running time InMaP consists of two optimization problems, that is, pseudo label optimization in Eqn. 3 and proxy learning in Eqn. 2. The former one can be formulated as optimizing the Sinkhorn distance that can be solved efficiently by the standard iterative method in [8]. With 20 iterations, the running time is only 0 . 002 second, which is negligible. The latter problem is convex and defined with only extracted features. Unlike prompt learning, the expensive encoders are not included in optimization. Even with 2 , 000 iterations for gradient descent, the running time is only about 30 seconds on a single GPU and the cost can be further reduced to 6 seconds with FP16. In addition, many efficient methods are available for solving convex problems as in Eqn. 2 and can be explored if needed [3].

4.2 Comparison on ImageNet

In this subsection, we compare InMaP to state-of-the-art methods on ImageNet with different vision encoders provided by CLIP. Table 6 summarizes the results. Besides the vanilla zero-shot prediction in CLIP, TPT [30], SuS-X [33] and VisDesc [20] are included in the comparison, where TPT leverages test data as ours but for prompt learning and the other two employ external large models that are marked in grey color. Results for these methods are directly borrowed from their original papers.

First, we can observe that our method InMaP 0 . 5 outperforms the baseline and TPT with both ResNet50 and ViT-B/16 as the vision encoder. Unlike the baseline and TPT that optimizes text prompts for images, we propose to reconstruct the proxy in vision space, which reduces the influence from the modality gap and obtains an appropriate intra-modal classifier as demonstrated in Theorem 2. In addition, the running time of proxy learning mainly depends on the dimension of extracted features and that with features from ViT-L/14@336 runs faster than ResNet-50, which guarantees the efficiency with different encoders. By further improving pseudo labels with Sinkhorn distance, InMaP shows the best performance among all methods and even surpasses those ones with additional large models, which confirms the effectiveness of our method. Moreover, InMaP has a steady improvement over InMaP 0 . 5 using different encoders. The observation indicates that label refinement

is complementary to the intra-modal proxy learning, which is consistent with Theorem 2. Finally, with the largest backbone provided by CLIP, InMaP achieves 80 . 21% accuracy on ImageNet, which shows the potential of large pre-trained vision-language models.

After the optimization with the target data set, we demonstrate the performance of InMaP when the target data arrives in a streaming manner but an unlabeled set of relevant data is available. Concretely, the vision proxy is constructed from the unlabeled training set in lieu of the target validation set while the learned proxy is evaluated on the validation set. Table 7 shows the results of our method learned with unlabeled training set ('InMaP train ') or validation set ('InMaP val ') of ImageNet. For InMaP train , τ I and α are reduced to 0 . 03 and 0 . 4 , respectively, while other parameters remain unchanged.

We can observe that even without the target data for recovering the vision proxy, an appropriate unlabeled data set is sufficient to obtain effective proxy for zero-shot classification. By thresholding the original pseudo labels, InMaP 0 . 5 train outperforms InMaP 0 . 5 val over all vision encoders, where the benefit may come from the large size of the training set. Then, if refining pseudo labels with Sinkhorn distance, the performance of proxies obtained with different data becomes similar and is better than the variant without optimized pseudo labels, which confirms the efficacy of our strategy for improving labels. Finally, this experiment demonstrates that the proposed method is applicable for different scenarios in real-world applications.

4.3 Comparison on 13 Downstream Tasks

Then, we evaluate the proposed method on 13 diverse downstream tasks, including Aircraft [19], Caltech101 [10], Stanford Cars [16], CIFAR-10 [17], CIFAR-100 [17], CUB200-2011 [35], Describable Textures Dataset (DTD) [7], EuroSAT [14], Flowers [21], Food101 [2], Oxford-IIIT Pet (Pets) [24], Sun397 [39], and UCF101 [32]. These data sets cover a large range of classification tasks such as fine-grained visual categorization, texture recognition, scene categorization, classification with low-resolution images, etc. The same parameters used for ImageNet are applied for most of them except certain parameters for refining labels. Concretely, Caltech and Flowers can benefit from the appropriate reference distribution for generating pseudo labels and we set γ in Sinkhorn distance as γ = 0 . 9 for ResNet and γ = { 1 , 0 . 5 } for ViT, respectively for the two data sets. Moreover, the threshold α is reduced to 0 . 3 on EuroSAT. Other parameters remain unchanged.

Table 8 summarizes the comparison. First, we observe that with the same parameters for all data sets, InMaP 0 . 5 shows the better performance than the baseline consistently and obtains 1% improvement on average by ResNet-50. It demonstrates that recovering the proxy in vision space can benefit different tasks for zero-shot transfer with pre-trained CLIP models. Second, by incorporating refined labels from optimizing Sinkhorn distance, InMaP can further improve InMaP 0 . 5 by more than 4% on average. The superior performance confirms that appropriate labels can facilitate the intra-modal

proxy learning to obtain better proxy reconstruction in the target space. In addition, our method is the best among all methods on 8 out of 13 data sets with ResNet-50, which shows that leveraging unlabeled target data can be more effective than external models for zero-shot learning. Moreover, the same parameters are shared by different tasks for InMaP, while fine-tuning them by a validation set for each data set can further improve the performance.

With ViT-B as the vision backbone, the accuracy of the baseline increases by 10% , which shows the potential of large models. The proposed method can also benefit from the large models and achieve the accuracy of 71 . 4% on average, which is more than 5% better than the baseline. Compared with SuS-X with external models, InMaP outperforms it by 1 . 74% that confirms the effectiveness of intra-modal proxy learning. Moreover, for the data set with a high zero-shot accuracy as Caltech, the predicted distribution can approximate the ground-truth well and the label refinement with Sinkhorn distance can be skipped by letting γ = 1 . Finally, both InMaP 0 . 5 and InMaP improves over the baseline on diverse tasks with ResNet and ViT, which illustrates that the proposed method is applicable for different vision encoders.

5 Conclusion

In this work, we focus on improving the zero-shot transfer with CLIP. First, our theoretical analysis indicates that the modality gap between the text and vision space obtained by CLIP will be preserved, which can degenerate the performance of zero-shot transfer. To mitigate the challenge, we propose to recover the proxy of each class in the vision space with the help from the text proxy. Concretely, by leveraging the unlabeled target data and refining the prediction from the text proxy, we can obtain the vision proxy via learning with unlabeled data and the corresponding pseudo label. Experiments on diverse data sets demonstrate that our method can improve the zero-shot accuracy consistently with different vision encoders. Exploring our method with other pre-trained models rather than CLIP can be our future work. Besides, recent work [33] shows that other large pre-trained models, i.e., GPT-3, can help CLIP for zero-shot prediction. Combining our method with existing large models is also an interesting future direction. Finally, our analysis for the efficacy of the text proxy is from the perspective of approximating the supervised learning for images as in Eqn. 1. Text information can be complementary to the knowledge in images [36] and investigating the potential benefit of text for vision tasks can help understand vision-language models better.

Limitations

While our analysis is for the modality gap in contrastive pre-training with the architecture of dual encoders, it cannot be applied directly for other pre-training paradigms with different architectures. For example, BEiT-3 [37] has masked data modeling for pre-training with the shared backbone for different modalities and CoCa [40] has encoder-decoder captioning for pre-training. The analysis for the modality gap in these methods has been less investigated and can be inspired by our results as a future direction.

References

Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid. Label-embedding for image classification. IEEE Trans. Pattern Anal. Mach. Intell. , 38(7):1425-1438, 2016.

L. Bossard, M. Guillaumin, and L. V. Gool. Food-101 - mining discriminative components with random forests. In ECCV , volume 8694 of Lecture Notes in Computer Science , pages 446-461. Springer, 2014.

S. P. Boyd and L. Vandenberghe. Convex Optimization . Cambridge University Press, 2014.

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In NeurIPS , 2020.

S. Chen, Z. Hong, G. Xie, J. Zhao, H. Li, X. You, S. Yan, and L. Shao. Transzero++: Cross attribute-guided transformer for zero-shot learning. CoRR , abs/2112.08643, 2021.

S. Chen, G. Xie, Y. Liu, Q. Peng, B. Sun, H. Li, X. You, and L. Shao. HSV A: hierarchical semantic-visual adaptation for zero-shot learning. In NeurIPS , pages 16622-16634, 2021.

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR , pages 3606-3613, 2014.

M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NeurIPS , pages 2292-2300, 2013.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR . OpenReview.net, 2021.

L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR workshop , pages 178-178. IEEE, 2004.

Z. Fu, T. Xiang, E. Kodirov, and S. Gong. Zero-shot learning on semantic class prototype graph. IEEE Trans. Pattern Anal. Mach. Intell. , 40(8):2009-2022, 2018.

P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao. Clip-adapter: Better visionlanguage models with feature adapters. CoRR , abs/2110.04544, 2021.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , pages 770-778. IEEE Computer Society, 2016.

P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens. , 12(7):2217-2226, 2019.

T. Huang, J. Chu, and F. Wei. Unsupervised prompt learning for vision-language models. CoRR , abs/2204.03649, 2022.

J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops , pages 554-561, 2013.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS , 2022.

S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

S. Menon and C. Vondrick. Visual classification via description from large language models. CoRR , abs/2210.07183, 2022.

M. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In ICVGIP , pages 722-729. IEEE Computer Society, 2008.

M. Palatucci, D. Pomerleau, G. E. Hinton, and T. M. Mitchell. Zero-shot learning with semantic output codes. In NeurIPS , pages 1410-1418. Curran Associates, Inc., 2009.

O. Pantazis, G. J. Brostow, K. E. Jones, and O. M. Aodha. Svl-adapter: Self-supervised adapter for vision-language pretrained models. In BMVC , page 580. BMVA Press, 2022.

O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In CVPR , pages 3498-3505. IEEE, 2012.

Q. Qian, L. Shang, B. Sun, J. Hu, T. Tacoma, H. Li, and R. Jin. Softtriple loss: Deep metric learning without triplet sampling. In ICCV , pages 6449-6457. IEEE, 2019.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICML , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 2021.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , pages 10674-10685. IEEE, 2022.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. , 115(3):211-252, 2015.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev. LAION-5B: an open large-scale dataset for training next generation image-text models. In NeurIPS , 2022.

M. Shu, W. Nie, D. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS , 2022.

K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. Raffel, E. D. Cubuk, A. Kurakin, and C. Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS , 2020.

K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR , abs/1212.0402, 2012.

V. Udandarao, A. Gupta, and S. Albanie. Sus-x: Training-free name-only transfer of vision-language models. CoRR , abs/2211.16198, 2022.

C. Villani. Optimal transport: old and new , volume 338. Springer, 2009.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

J. Wang, Y. Xu, J. Hu, M. Yan, J. Sang, and Q. Qian. Improved visual fine-tuning with natural language supervision. In ICCV , 2023.

W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, and F. Wei. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. CoRR , abs/2208.10442, 2022.

Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata. Zero-shot learning - A comprehensive evaluation of the good, the bad and the ugly. IEEE Trans. Pattern Anal. Mach. Intell. , 41(9):2251-2265, 2019.

J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR , pages 3485-3492. IEEE Computer Society, 2010.

J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. Trans. Mach. Learn. Res. , 2022, 2022.

R. Zhang, W. Zhang, R. Fang, P. Gao, K. Li, J. Dai, Y. Qiao, and H. Li. Tip-adapter: Training-free adaption of CLIP for few-shot classification. In ECCV , volume 13695 of Lecture Notes in Computer Science , pages 493-510. Springer, 2022.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. Int. J. Comput. Vis. , 130(9):2337-2348, 2022.

A Theoretical Analysis

A.1 Proof of Proposition 1

Proof. According to the definition, we have

̸

Hence

̸

where k = arg max j : j = i x ⊤ i t j . Since logarithm function is monotone, the similarity between positive pair can be bounded by where c 1 = log( δ ) and c 2 = log( δ ( m -1) ) .

A.2 Proof of Proposition 3

Proof. According to the definition, we have

If √ a/τ T = 1 /τ I , the predictions are equivalent.

A.3 Proof of Theorem 1

Proof. According to the definition, we have

To bound the similarity between Z x and W ∗ , we assume Z x = U r A ⊤ where U r ∈ R d × r is a subset of U ∈ R d × d ′ . By solving the problem min A ∥ U r A -W ∗ ∥ 2 F , we have Z x = U r A = U r ( U ⊤ r U r ) -1 U ⊤ r W ∗ = U r U ⊤ r W ∗ that is projecting W ∗ to the subspace spanned by U r . Since U r is a subset from U , we have U r U ⊤ r = ∑ d ′ i r i u i u ⊤ i with ∀ i, r i ∈ { 0 , 1 } and ∑ i r i = r . Then, we have

Therefore

where the last inequality is from keeping the largest singular values for minimizing the approximation loss. The target result is obtained by taking this inequality to Eqn. 6.

A.4 Proof of Theorem 2

Proof. First, we note that L ( P, W ) is a convex function in W . We assume that it is µ -strongly convex in W such that for the arbitrary ( W 1 , W 2 ) , we have

̸

According to the optimality of W ′∗ , we have

The lower-bound can be obtained from the strongly convexity of W as

Combining inequalities in Eqns. 7 and 8, we have

Peer reviews:
comments: SUMMARY:
Authors provide a series of detailed theories to support the conclusion that the inter-modal distance of the CLIP model is large and the gap can be reduced by introducing proxy features in visual feature space. They also propose the intra-modal proxy learning method to improve the performance of CLIP model with only unlabeled test data. Experiments on several datasets and different network architectures validate their proposed method's effectiveness.

STRENGTHS:
1. Motivation is clear and theoretical analysis is very sufficient.
2. The mehtod derived from theoretical analysis is convincing.
3. The paper is well-organized.

WEAKNESSES:
1. Temperature in CLIP paper is a learnable parameter which is initialized as 0.07 and becomes 100 after training. Is Tau_T in submission the same parameter as the temperature t in CLIP? And why set to 0.01.
2. According to Table 2, it seems alpha contributions little to the proposed method. If we ignore the step, the acc will drop from 63.74 to 63.49, not too much for a single experiment.
3. Can authors provide the results on UCF101 and eurosat, which are common datasets used in CLIP tuning methods.
4. Can authors provide the results of default prompt instead of 7.
5. About TPT results, authors copy the results from their paper, resutls of CLIP baseline is obvious higher than their bsaeline, why? If bsaeline uses ensemble prompt, please refer to Weakness(4). Using high baseline results and copied TPT results seems unfair.

QUESTIONS:
Please refer to Weaknesses.

LIMITATIONS:
No potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work theoretically proves that the gap cannot be reduced sufficiently by minimizing the contrastive loss in CLIP and the optimal proxy for vision tasks resides only in the vision space. Two strategies are developed to further refine the pseudo label obtained by the text proxy to facilitate intra-modal proxy learning (InMaP). Experiments on ImageNet and extensive downstream tasks confirm the effectiveness and efficiency of the proposal.\n\nHowever, I have some concerns. Please refer to the comments below.\n\nSTRENGTHS:\n- This paper is easy to understand.\n- To mitigate the problem of the gap between the text and vision space, the authors theorize that the optimal proxy for visual tasks is solely from visual space. This is reasonable and convincing.\n- Extensive experiments are performed on different datasets.\n\nWEAKNESSES:\n- For NeurIPS, the innovation or contribution of this work is limited, especially the technical solution. Proxy learning is not a new technique. Furthermore, pseudo label refinement is a combination of existing methods. \n\n- Related work introduces few-shot transfer rand zero-shot transfer. It seems that this paper does not report the performance of few-shot transfer (this paper uses CLIP as the Baseline, and CLIP performs a few-shot transfer experiment).\n\n- The layout of this paper is not very reasonable.\n\nQUESTIONS:\n- There seem to be some syntax issues, please check carefully. For example, Line 44 and Line 48.\n\n- Please distinguish between ablation studies and hyperparameter analysis. Does this paper include hyperparameter analysis in \"Alation on ImageNet\"? This confuses me.\n\n- For some comparison methods, this paper shows by lighter color, why is this? It doesn't seem to be clearly stated.\n\nLIMITATIONS:\n- This paper does not analyze the reasons for achieving zero-shot performance gains or provide deeper insights into the field.\n\n- See Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
Based on the phenomenon of modality gap, this paper analyzes that the proxy of visual space is better than that of text space, and explains this theoretically. This paper designs a test-time adaptation method for unlabeled test data

STRENGTHS:
1.  The influence of the modality gap on image classification is analyzed from the perspective of the proxy, and the theoretical results are amazing, which is of great significance for CLIP community research.
2. The results of the experiment are very significant, and I think this article touches on the essence of CLIP.

WEAKNESSES:
Some analysis of validity for the textual model is lacking.

QUESTIONS:
I once thought about the role of textual models in downstream image classification.   CoOp and other works try to optimize the textual model and get success, but from the perspective of this paper, textual models seem to play no key role in downstream tasks just as an initialization.   What is the author's opinion?

LIMITATIONS:
The author has not provided any limitations of this article, so I hope you can add some analysis.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

comments: SUMMARY:
The paper aims to improve the zero-shot performance of CLIP, i.e.: the setting in which given a test image, we compute its visual embedding, computes its similarity to a pre-computer set of text query embeddings where each text query corresponds to a different class and finally output the classification results as class corresponding the most similar text query (in CLIP's joint embedding space). 

The paper argues that zero-shot misclassifications are due to the visual-textual modality gap, i.e. the inherent gap in CLIP's embeddings space between embeddings of image and embeddings of their corresponding text queries. The paper argues (and theoretically proves) that this gap cannot be bridged by optimising on the text query and that the optimal proxy lies within the vision space. The paper proposes a novel algorithm that recovers vision proxies for text queries from unlabelled image test data. The paper demonstrates the effectiveness of the method on 12 downstream vision tasks.

STRENGTHS:
* The paper addresses a key limitation of CLIP (and other vision-language models trained using contrastive loss) - the modality gap. The modality gap is an inherent fundamental "built-in" limitation of CLIP which hinders its performance. Tackling such a key problem in a wide spread and popular model (or model family) is of great important). 
* The paper provides both an in-depth theoretical discussion as well as a thorough experimental evaluation demonstrating the effectiveness of the proposed method.

WEAKNESSES:
Nothing in particular that I can point to.

QUESTIONS:
Nothing in particular.

LIMITATIONS:
The authors did not addressed the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-08