Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP
Authors: Qi Qian
Yuanhong Xu
Juhua Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: zero-shot; clip; proxy learning

Abstract: Vision-language pre-training methods, e.g., CLIP, demonstrate an impressive zero-shot performance on visual categorizations with the class proxy from the text embedding of the class name. However, the modality gap between the text and vision space can result in a sub-optimal performance. We theoretically show that the gap cannot be reduced sufficiently by minimizing the contrastive loss in CLIP and the optimal proxy for vision tasks may reside only in the vision space. Therefore, given unlabeled target vision data, we propose to learn the vision proxy directly with the help from the text proxy for zero-shot transfer. Moreover, according to our theoretical analysis, strategies are developed to further refine the pseudo label obtained by the text proxy to facilitate the intra-modal proxy learning (InMaP) for vision. Experiments on extensive downstream tasks confirm the effectiveness and efficiency of our proposal. Concretely, InMaP can obtain the vision proxy within one minute on a single GPU while improving the zero-shot accuracy from $77.02\%$ to $80.21\%$ on ImageNet with ViT-L/14@336 pre-trained by CLIP.

Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP
Juhua Hu 1 Alibaba Group, Bellevue, WA 98004, USA
Qian Yuanhong Xu²
2 Alibaba Group, Hangzhou, China
3 School of Engineering and Technology, University of Washington, Tacoma, WA 98402, USA fqi.qian, yuanhong xuyh) ) Sallilalaaalbaaa ibaba-iaiin com, juhuahYuw.ed edu
Abstract
Vision-language pre-training methods, e.g., CLIP, demonstrate an impressive zero- shot performance on visual categorizations with the class proxy from the text embedding of the class name. However, the modality gap between the text and vision space result in sub-optimal performance. We theoretically show that the gap cannot be reduced sufficiently by minimizing the contrastive loss in CLIP and the optimal proxy for vision tasks may reside only in the vision space. Therefore, given unlabeled target vision data, we propose to learn the vision proxy directly with the help from the text proxy for zero-shot transfer. Moreover, according our theoretical analysis, strategies are developed to further refine the pseudo label obtained by the text proxy facilitate the intra-modal proxy learning (InMaP) for vision. Experiments on extensive downstream tasks confirm the effectiveness and efficiency of our proposal. Concretely, InMaP can obtain the vision proxy within one minute on a single GPU while improving the zero-shot accuracy from 77.02% 80.21% on ImageNet with ViT-L/14@33 336 pre-traine pre-trained by CLIP.
Introduction
Vision lasgualla pre-training in CLIP aims to align image-text pairs to facilitate multi-modal un- derstanding for downstream tasks (26). By optimizing the contrastive loss consisting of images and their corresponding text descriptions, CLIP demonstrates an impressive zero-shot transfer of the pre-trained model to generic tasks. Concretely, given a downstream data set, each class has proxy obtained from the text embedding about the corresponding class name, and then images can be categorized by finding the nearest class proxy. With class names only, zero-shot accuracy on ImageNet (8 can achieve 77. 77.02% with ViT-L/14@336 [ pre-trained by CLIP.
After the success of CLIP, many research efforts have been devoted to improving the transfer performance on downstream tasks. Most of developed methods can be cast into two categories, that is, few-shot refinement and zero-shot enhancement. Few-shot methods require small amount of labeled data from the target task to refine the prediction [12] 41 42). While these methods show
better performance than zero-shot learning, labeled examples from the target domain may not be available in real-world applications. Therefore, many methods focus on improving the zero-shot learning itself by exploring side information. On one hand, many pre-trained large language models (LLMs), e.g., GPT-3 [4), can be leveraged to obtain better text proxy for classification [20). On the other hand, unlabeled data from the target domain can be adopted to fine-tune an adapter network or input prompts [23] 30 for alignment. In this work, focus understanding the behavior of CLIP
by exploiting the unlabeled data without external LLMS or additional architectures for zero-shot categorization.
"Conresondingg author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

When considering vre-trained CLIP models and excluding any auxiliary networks for zero-shot transfer, many existing methods try to optimize imput text prompts to align images and the text of class names better 33 33
However, recent observations demonstrate that the modality gap between text (i.e., language) and vision is still significant after the optimization of CLIP [8] Consequently, the efforts on improving the text proxy can be ineffective for the vision task.
To mitigate the problem from the gap between the text and vision space, we propose to obtain prox- ies of classes the vision space directly. First, our theoretical analysis in Proposition]] I shows that the modality gap will be preserved due to a small tem- perature parameter in the contrastive loss of CLIP, which confirms the empirical observation in [8] Furthermore, Proposition 2 mal proxy for the vision task can only be obtained from the vision space. Therefore, with the gap be- tween the text and vision space, directly using the text proxy makes the zero-shot transfer sub-optimal. To recover the optimal vision proxy, we develop an intra-modas proxy learning (InMaP) method to obtain visual proxies with pseudo labels from text proxies. The proposed method is illustrated in Fig. [T With fixed features extracted by CLIP encoders, the optimization problem in InMaP is convex without encoders in the loop, which can be solved efficiently with gradient descent [3]. Our main contributions can be summarized as follows.
Text space
provy
Modality gap
Recover
indicates that the opti-
Vision proxy A
Vision
space
Illustration of our proposed intra- modal proxy learning (InMaP). Triangles de- note the class proxy while circles denote the vision data. Different classes are denoted with different colors. By recovering the vision proxy, it can align with vision data better (i.e., from solid line as in the text space to dashed line in the vision space).
Figure 1:
We theoretically show that the modality gap cannot be reduced sufficiently by minimizing the contrastive loss CLIP. While the optimal proxy for image classification is in the vision space, the gap degenerates the zero-shot performance of the text proxy shown in Theorem
We propose a novel algorithm to obtain the intra-modal proxy directly from the vision space. By recovering the vision proxy with help from the text proxy and an appropriate temperature suggested by Proposition for calibration, effective proxies can be observed from unlabeled data.
Our Theorem 2 indicates that the accuracy of pseudo labels from the text proxy essential to recover the proxy in the vision space. Therefore, we further develop strategies to refine pseudo labels according to the data distribution by optimal transport (OT) (34]
Experiments on 14 downstream vision tasks demonstrate both the effectiveness and efficiency of our proposal. Concretely, InMaP can achieve 80.21% % accuracy on ImageNet with ViT-L/14@336 using single GPU within one minute.
2 Related Work
Before CLIP, zero-shot learning that aims to identify examples of novel classes without any labeled training data has attracted much attention since 22 However, most of existing works are only able to discover new classes closely related to the training classes [ 5 6
[T] 22 38, where they share the similar attributes, and have train individual model for each task. On the contrary, pre-training on large-scale data with the contrastive loss aligns visual and language features and makes a single CLIP model applicable for diverse downstream tasks in a straightforward way. Compared with conventional zero-shot methods, the pre-training data in CLJP may be overlapped with downstream tasks, which can result in data leak for evaluation. While the issue can be addressed by eliminating the overlapping data, the influence on the performance is negligible as discussed in [26].
After pre-training with image-text pairs, the obtained CLIP models can be transferred various downstream tasks with few-shot or zero-shot strategies. We briefly review these two related directions as follows.
Few-shot transfer CLIP demonstrates that an ensemble of input text prompts for the text encoder can improve the zero-shot performance by significant margin. Therefore, many methods try to

---

further refine the prompis. Given small set of labeled data from the target vision tasks, CoOp 42 optimizes imput prompts for the text encoder with labels. By formulating the prompts as learnable variables, it can obtain task-dependent prompts without any hand-crafted designs. On the contrary, CLIP-Adapter [12] optimizes obtained output features from both encoders. However, it adds ad- ditional bottleneck layers for training, which increases the complexity of optimization. To reduce the efforts training with encoders, Tip-Adaples [41] demonstrates a training-free method that incorporates the prediction from cache model consisting of the labeled image data and that from the text proxy. this work, we also consider to minimize the training efforts by excluding encoders in optimization and the intra-modal proxy can be obtained from extracted features.
Zero-shot transfer Since labeled data are not always available, many efforts are devoted to improv- ing zero-shot transfer. Some methods explore other large pre-trained models as the supplementary for CLIP. VisDese [20] leverages GPT-3 to generate rich context descriptions for given class names, and thus shows superior performance over the simple prompt in CLIP. SuS-X [33 includes additional images to mitigate the modality gap. Concretely, auxiliary images can be obtained either from a large data set (e.g., LAION [29]) or text-to-image generation model (e.g., stable diffusion (27).. The additional large models will inevitably increase the inference time, hence, recent methods consider to leverage the unlabeled target data for fine-tuning. UPL [15] and TPT [30] utilize the unlabeled data to optimize learnable input text prompts. SVL-Adapter [23 trains an additional encoder from the unlabeled data with sule-vupe learning for ensemble. Unlike existing methods, our proposal does not need any additional model and we learn the proxy for each class from the vision space directly.
3
The Proposed Method
In this section, we first analyze the modality gap in CLIP, and then elaborate the proposed method that recovers the proxy in the vision space reduce the influence from the modality gap.
3.1 Cross-Modal Learning in CLIP
By pre-training paired images and text, a vision-language model, e.g.. CLIP (26,, is capable of classifying unlabeled images in the zero-shot manner. Let f(-) and g(-) denote the encoder for images and text in the pre-trained CLIP model, respectively. Given a set of unlabeled image data (2; 1 and the unique class names for the set {2 1C their corresponding visual and text representations can be extracted as
x; = f(x); zj g(z))
where x, E Rd and zj Rd have the same dimension and unit norm. d is the dimension of features. Zj can be considered as the proxy for the j-th class.
With the features of the i-th image and all text proxies, the image can be classified
y/ arg max x, 3j
where y is the prediction from the zero-shot classification. The capacity of zero-shot prediction is from the alignment between vision and language that is optimized by the pre-training objective in CLIP. Given image-text pairs as (I; fx), two encoders are learned by minimizing the contrastive loss
Σ -lg exp(x t,/T) Σn exp(x, (xp(x(t,/T) log
exp(t x//T)
Σn exp(t, x²/7)
where m is the mini-batch size, t; = g(t,) ) is the extracted text representation of with a unit norm, and r the temperature for the normalized cross entropy loss.
While CLIP demonstrates an impressive zero-shot performance, the cross entropy loss focuses on cross-modal ranking and can be sub-optimal for intra-modal vision categorization. It because that cross entropy loss is closely related to a triplet loss (55 as
Vj.
x t, x3 t
xxx, 2
which mainly aims to rank the multi-modal positive pair over negative pairs rather than mix the text and vision space for modality gap reduction. Moreover, small temperature, c.g., 0.01 in CLIP, will

---

amplify the difference between pairs and preserve the distance between modalities. The phenomenon can be stated as follows. The detailed proof can be found in the appendix.
Proposition 1. Assuming
explx; t:/r)
exp(x 6//7] = 8, we have
Σn
x + 5
x tx + C2T
where k denotes the nearest negative pair as l arg max X Lj. ei and C2 are constants. Ci when >> 1/2 and 02 when ò 1/m.
Remark Proposition iimplies that when is sufficiently large, the relevant text has a higher rank than the irrelevant text, which is the key for zero-shot classification. Moreover, it shows that with the same prediction 8, the absolute distance between modalities, i.c., ||x t. 2 - 2x, t,, partially depends on the value of temperature, i.e., | 1 2 2 2x, tx 20gT. The analysis indicates that optimizing cross entropy loss with a small 7 will not pull the text and vision space together.
Some recent work empirically observed that inter-modal distribution is significantly different from the intra-modal distribution in CLIP [18 (3), which confirms our analysis. Consequently, the proxies of class names from the text space may not be sufficient capture the distribution of vision space, which results in the degenerated zero-shot performance. In lieu of using the text proxy directly for zero-shot visual categorization, we propose obtain the class proxy vision space follows.
3.2 Intra-Modal Proxy Learning
With ground-truthe labels {y:}, the standard supervised learning in vision space can be written as exp(x, Wyr/Tr)
min Σ Eloo log(
Σj exp(x wz/T1)
where wj is the learnable proxy for the j-th class and W [Wi,..., wel Rdxe is the temperature for the optimization with vision data. With an equivalent form, it is easy to show that w, in the vision space spanned by {x:}.
Proposition 2. Let W* be the optimal solution to
exp(- Wy (2/())))
min
log(
W
Σj exp(-Ilx: w, 1111/(2TT)
which is equivalent to Eqn.
if w as well as x, has a unil norm. Then, we have
Ein,, Pi,))x; Evegar Pk.
Vj,
II [|llll
Ein= Pi.)) Evem 4 Pk.j
exp( |x w ((77))
where Pi.j
and 11 w projects the vector to be with a unit norm.
ΣE expl | wi (2/(1))
Proof. It is directly from K.K.T condition
3
Remark Proposition demonstrates that the optimal proxy for vision categorization should be in the vision space. While proxies from the text space can work as the substitute for vision proxies in CLIP, the distance between the text and vision space (i.e., the modality gap) as illustrated in Proposition | will degenerate the performance for vision tasks.
To further demonstrate the challenge from the gap between modalities and simplify the analysis, decompose the proxy of class names as
Zj = Vozz + VI vI-az,;
where z is from the vision space spanned by {x: and 2 shows the component from the orthogonal subspace such that TØTZz z O. Both z and 2 have the unit norm. 2 can be considered as encoding the specific information for the text modality. With the decomposition, we can observe that it is possible to recover the optimal prediction if the vision space is covered by the text space as follows.
Proposition 3. Let Pi,j and pha, denote the prediction probability obtained with vision proxies {w)) and text proxies {zz), respectively. Tr and Tı denote the temperature in CLIP and that vision proxy learning, respectively. If z =w and 0 1, we have Pt., Pij when TT/ vr/va.

---

Remark Proposition 3 shows that if the text space contains the whole vision space, the intra- modal prediction can be recovered from the cross-modal prediction using a larger temperature as /v//a, where measures the overlap between modalities.
However, recent observations indicate that the text and vision space learned by CLIP are distinct with a clear margin [18] and it is hard for the text space to cover the vision space. The difference between the text and vision proxy can be lower-bounded in the following theorem.
Theorem 1.
Let Z and W* denote the text proxies and optimal vision proxies, where Z = vaz= VI- aZ' and Z" consists of rank approximation of W as Z U,AT W UEV
ΣS suuuy where d' = minfd, d, Oj and $1 2 ... 2 Sd' 2 0. Then, we have
1 - 1 20 (
Remark Theorem shows that the gap between text and vision proxy comes from two parts. The former term indicates the distance to the irrelevant text space as - Lava. The latter one depicts the approximation loss from the low-rank overlapping between text space and vision space with d'. Due to the inherent difference between these two modalities, the distance between Z and W is hard be minimized.
Therefore, to improve the zero-shot performance in the vision space, we consider to obtain the vision proxy in lieu of directly using the text proxy. The main challenge comes from the fact that the label {y:) is unavailable in zero-shot classification. Fortunately, the intra-modal proxy can be learned by mimicking the proxy from the other modal. Concretely, we propose minimize the KI. divergence between distributions from the text and vision proxy as
min L(P', W) Σ (RI(PP)) W
where Py' and P. indicate the distribution estimated by the text proxy Z and the learnale vision proxy W, respectively.
Pho = exp(x; 25/TT) Σ exp(x; 2t/Tr)
exp(x, w
Pi.j Σ exp(x; w
Proposition suggests that Ty for P. should larger than Tr in P/' calibrate the magnitude the distribution and our ablation study confirms the analysis.
Unlike prompt learning methods (30], the problem in Eqn. 2 defined with extracted features, which does not require the backward operation with large encoder networks for optimization. Moreover, it is convex and the optimal vision proxy can be obtained efficiently by the standard gradient descent (
By learning with the pseudo label predicted from the text proxy, we can recover the optimal proxy in the vision space as shown in the following theorem.
Theorem simplifying the loss function L(P", W) --E Pi, log(Ptg) and assuming that it is p-strongly convex in W and letting
W = arg min ((P,,W);; W = arg min L(Y, W)
W
W
where Y, is the ground-truth label distribution for x4, we have
2 2 1pp 1o--Yin Y log(Pwr.))
| 1
Remark Theorem shows that the distance between the recovered vision proxy and the optimal vision proxy is bounded by the gap between pseudo labels predicted by text proxy and the ground- truth label distribution. After pre-training CLIP models on a large scale data set, can approximate the distribution of real data, which enables learning of the intra-modal proxy.
5

---

Algorithm Intra-Modal Proxy Learning (InMaP)
Input: Unlabeled image {xz). class names { pre-trained CLIP encoders (f. g). iterations
Tw, Tp, temperature Tr. Ti, threshold a
2: Extract features {x,} and {z,} using pre- pre-trained encoders and g, respectively.
3: Obtain pseudo labels by solving Eqn. 33n.3/ with Tい iterations.
4: Refine pseudo labels with the threshold as in Eqn.
5: Obtain vision proxies W by solving Eqn. with T iterations.
6: return yi = arg max, X W
3.3 Pseudo Label Refinement
According to Theorem 2 the accuracy of pseudo label is essential for recovering the appropriate vision proxy. Therefore, we develop strategies to improve the quality of pseudo labels.
First, since we have the whole test set rather than a single example for zero-shot learning, the labels obtained from the text proxy can be refined according the geometry of the target vision data. Given the logits matrix M E Raxo with M, = x Zj. the original pseudo label can be obtained by solving the problem
P' arg vrgmax(P. M) + (P) s.t. Vi, Σ Pag =/n; Vi,,P. 0
where (P) computes the entropy P and the obtained results should be multiplied by n as the predicted labels. By incorporating reference distribution RC over classes, the problem can be rewritten as
max(P, + HP) s.t. Vi, Pas= 1 Vj. P.J = Vi,j 0
(3)
which is known as Sinkhorn distance [8] and is an approximation of optimal transport distance. The refined pseudo label can be obtained by an efficient iterative method (8].
The last challenge is from observing the appropriate reference distribution. Without any prior knowledge, balanced distribution q 1/C is popular selection in practice. By investigating the prediction from the text proxy. it implies a distribution as d = E, P.g//n Then, reference distribution can be obtained by further smoothing q
9
Ij = Σx 9%
where 7 S 1 is the smoothing parameter and 7 = implies the original distribution from the text proxy.
Moreover, inspired by the semi-supervised learning [31. soft label with high confidence can be
converted into one-hot label to eliminate the influence from irrelevant classes. Let a be the threshold. After obtaining the pseudo label from Eqn. 3 we have
eg k= arg Pl,j
PlA > a
P
(5)
P
O.W.
where E (0,1) 1) is the one-hot vector and the k-th element is 1.
The proposed intra-modal proxy learning method (InMaP) is summarized in Alg. Pre-trained CLIP models are only applied to extract features as in Step 2, which is the inevitable cost of conventional zero-shot learning. After obtaining features, the computational overhead of our proposed optimizations in Steps 3-5 is mild as shown the analysis of running time.
4 Experiments
To evaluate the proposed method, we follow the common practice and conduct experiments on ImageNet [28) and 13 diverse downstream vision tasks for zero-shot transfer. Text prompts

---

important for obtaining appropriate text proxies. We have the selected 7 prompts in [33 as templates for ensemble, which shows better performance than a single prompt or multiple prompts in CLIP. For the proposed method, the temperature for obtaining pseudo labels with the text proxy TT is set to 0.01 as obtained by CLIP. The temperature for recovering the visual proxy ту is set to 0.04 for all experiments. The intra-modal proxy is learned by standard projected gradient descent, where the initial learning rate is 10 and the number of iterations is 2, 000 for sufficient training. The learning rate will be decayed by 2 when the norm of gradient increases. Sinkhorn distance is optimized by 20 iterations for refining pseudo labels. All experiments are conducted on a single V100 GPU.
4.1 Ablation Study
First, have the ablation experiments on ImageNet to demonstrate the efficacy of our method InMaP, Pre-trained ResNet-50 [3] is adopted as the vision encoder and the vanilla zero-shot method in CLIP is denoted as "Baseline".
According to analysis in Proposition 3 Ty should be larger than 7r in CLIP to leverage the overlap between the text and vision space. Table shows the result when varying Ty in {0.01, 0.02, 0.03, 0.04, 0.05). "Sim" computes the mean similarity between each example and its nearest proxy, depicting the gap (lower the similarly, higher the gap).
Temperature Tι 7y is used to recover the proxy in the vision space as in Eqn. 2
First, when TΙ 0.01, the gain is mainly from learning the vision proxy with refined labels and already 2.59% better than the baseline with text proxy. By increasing Tı, the similarity between examples and proxies also increases, which confirms our analysis in Proposition [ With an appropriate Tı, an additional gain of 0.83% can be observed with the similarity of 0.39. According to Proposition 3 large is crucial recover the optimal vision proxy with the guidance from the text proxy. Thus, we fix F 0.04 for the main comparison.
Table 1: Effect of Tι-
Table 2: Effect of a.
Table Effect of 7.
Table 4: Components in InMaP.
TI
Acc%
Sim
Acc%
Acc%
Baseline
60.32
0.26
I
63.49
60.95
methods
Acc%
0.01
62.91
0.31
0.9
63.53
0.5
62.98
Baseline
60.32
0.02
63.22
0.32
0.7
63.70
0.1
63.75
InMap0.25
60.83
0.03
63.66
0.35
0.6
63.74
0.05
63.73
InMap0.5
60.95
0.04
63.74
0.39
0.5
63.72
0.01
63.74
Sinkhorn
62.53
0.05
63.29
0.43
0
63.14
63.74
InMaP
63.74
Pseudo label threshold Since the SoftMax operator can incur the over-smoothing issue for output logits, converting the soft label with high confidence to one-hot an effective strategy in semi-supervised learning. We adopt and evaluate the performance with the threshold a in Eqn. [S]
Table Jubleiissmrrizes the results with different thresholds. When a the soft label after Sinkhorn distance optimization is adopted for optimization, which already surpasses the baseline by clear margin of 3.17%. When reducing a, more soft labels will be converted to one-hot labels and the performance can be further improved by eliminating noise from the SoftMax operator. When letting = 0, all pseudo labels become one-hot, which amplifies the noise from low-confidence prediction and degenerates the performance. We find that our method is insensitive to the parameter and fix it as 0.6 the remaining experiments if not specified.
Pseudo label smoothing y While the validation set of ImageNet is well-balanced with 50 examples for each class, we conduct the experiments with different in Eqn. for demonstration.
Table shows the accuracy on ImageNet with varying y. 7 follows the distribution from the text prediction and degenerates to the original pseudo label without Sinkhorn distance refinement. By balancing the distribution of the class assignment with small 7, the accuracy can be increased with refined pseudo labels. When y is sufficiently small, constraint in Sinkhorn distance degenerates to the balanced constraint and shows the desired performance due to the consistency with the ground- truth distribution. We will have the balanced prior distribution as 7 = for other data sets if not specified.
7

---

Components in InMaP Finally, we show the gain from two components in InMaP in Table 4 Let "InMapo.25" denote the proxy learning without refined labels and "InMaP0.5" be the variant optimized by solely applying the threshold to pseudo labels from the text proxy. "Sinkhorn" denotes the pseudo labels refined by Sinkhorn distance without proxy learning. We can observe that with the labels predicted from the text proxy, the recovered vision proxy of In.app..25 can improve the baseline by 0.51% with the reduction in the modality gap. Then, simply thresholding the prediction from the text proxy can further improve the performance by 0.12% in InMap0.5 On the other hand, refining the labels via Sinkhorn distance can substantially outperform the baseline by 2.21%. Learning with better pseudo labels, InMaP shows an additional gain of 1.21% over refined labels. It confirms our analysis in Theorem 2 that the proposed intra-modal proxy learning can benefit from appropriate pseudo labels.
Number of text prompts An ensemble of text prompts shows the superior performance over the single prompt on obtaining text proxy for zero-shot classification [26]. We compare these two strategies on our method and summarize the accuracy on ImageNet in Table
Table 5: Comparison of accuracy(%) on ImageNet with different text prompts. "Single" denotes the application of the single prompt of "a photo of a O.", while "ensemble" contains 7 prompts as suggested by [33.
Vision encoder
Bascline single
Baseline ensemble
InMaP single
InMaP ensemble
ResNet-50
58.15
60.32
63.15
63.74
ViT-B/16
66.72
68.75
71.93
72.55
First, we can observe that the ensemble improves the performance of baseline with clear margin of 2% with different vision encoders. shows that the vanilla zero-shot transfer strategy is sensitive to the selection of text prompts and confirms the motivation of prompt learning methods. Second, our method outperforms the baseline with the single or ensemble of text prompts for text proxy, which demonstrates the effectiveness of our proposed intra-modal vision proxy. Moreover, the gap between the single and ensemble prompts shrinks on InMaP to less than 1%, which can help reduce the efforts of text prompt tuning.
Running time InMaP consists of two optimization problems, that is, pseudo label optimization in aand proxy learning in Eqn. The former one can be formulated as optimizing the Sinkhorn distance that can be solved efficiently by the standard iterative method in [8]. With 20 iterations, the running time is only 0.002 second, which is negligible. The latter problem is convex and defined with only extracted features. Unlike prompt learning, the expensive encoders are not included in optimization. Even with 2, 000 iterations for gradient descent, the running time is only about 30 seconds on a single GPU and the cost can be further reduced to seconds with FP16. In addition, many efficient methods are available for solving convex problems as in Equ. 2 and can be explored if needed (3]
4.2 Comparison on ImageNet
In this subsection, we compare InMaP to ofate-of-the-ar methods on ImageNet with different vision encoders provided by CLIP. Table summarizes the results. Besides the vanilla zero-shot prediction in CLIP, TPT [30], SuS-X (3 and VisDesc (20) are included in the comparison, where TPT leverages test data as ours but for prompt learning and the other two employ external large models that are marked in grey color. Results for these methods are directly borrowed from their original papers.
First, we can observe that our method InMap0.5 outperforms the baseline and TPT with both ResNet- 50 and ViT-B/16 as the vision encoder. Unlike the baseline and TPT that optimizes text prompts for images, we propose to reconstruct the proxy in vision space, which reduces the influence from the modality gap and obtains an appropriate intra-modal classifier as demonstrated in Theorem 2 In addition, the running time of proxy learning mainly depends on the dimension of extracted features and that with features from ViT-L/14 @333636636 runs faster than ResNet-50, which guarantees the efficiency with different encoders. By further improving pseudo labels with Sinkhorn distance, InMaP shows the best performance among all methods and even surpasses those ones with additional large models, which confirms the effectiveness of our method. Moreover, InMaP has a steady improvement over InMap using different encoders. The observation indicates that label refinement
8

---

Table 6: Comparison of accuracy (%) on ImageNet with different vision encoders in CLIP. i in grey indicates the application of external large models. The overall best performance is in bold, while the best performance without any additional model underlined. "_" denotes that the result is unavailable in their original papers.
Vision encoder
Baseline 26
TPT 33
InMapil.
InMaP
VisDesc
21
SuS-X-ICC
33
SuS-X-SD
ResNet-30
60.32
60.74
60.95
63.74
59.68
61.89
61.84
VIT-R/32
63.77
64.82
67.29
62.97
64.73
64.71
VIT-B/16
68.75
68.98
70.17
72.55
68.03
70.00
69.88
ViT-L/14
75.96
77.14
79.29
75.00
VIT-I/14@336
77.02
78.27
80.21
76.16
is complementary to the intra-modal proxy learning, which is consistent with Theorem 2 Finally, with the largest backbone provided by CLIP, InMaP achieves 80.21 % accuracy on ImageNet, which shows the potential of large pre-trained vision-languee models.
After the optimization with the target data set, demonstrate the performance of InMaP when the target data arrives in streaming manner but an unlabeled set of relevant data available. Concretely, the vision proxy is constructed from the unlabeled training in lieu of the target validation set while the learned proxy is evaluated on the validation set. Table [ shows the results of our method learned with unlabeled training ("InMaP train ") validation set ("InMaPval ") of ImageNet. For InMaP train. and a are reduced to 0.03 and 0.4, respectively, while other parameters remain unchanged.
Table 7: Comparison of accuracy (%) on ImageNet. For InMaP, "train" and "val" denote that the vision proxy is obtained from the unlabeled training set and validation set, respectively.
Vision encoder ResNet-50
ViT-B/32
ViT-B/I6
ViT-L/14
ViT-L/I4@333
InMaPusa val
60.95
64.82
70.17
77.14
78.27
InMaP val
63.74
67.29
72.55
79.29
80.21
InMap0.5
61.38
65.12
70.37
77.69
78.68
train
InMaPtrai
63.98
67.37
72.59
79.21
80.13
We can observe that even without the target data for recovering the vision proxy. an appropriate unlabeled data set is sufficient to obtain effective proxy for zero-shot classification. By thresholding the original pseudo labels, InMaP0.5 train outperforms InMap0.5 over all vision encoders, where the benefit may come from the large size of the training set. Then, refining pseudo labels with Sinkhorn distance, the performance of proxies obtained with different data becomes similar and is better than the variant without optimized pseudo labels, which confirms the efficacy of our strategy for improving labels. Finally, this experiment demonstrates that the proposed method is applicable for different scenarios in real-world applications.
4.3 Comparison on 13 Downstream Tasks
Then, we evaluate the proposed method on 13 diverse downstream tasks, including Aircraft [1]]
Caltech101 [10], Stanford Cars [16, CIFAR-10 10 [17) CIFAR-100
[17] CUB200-2011 [35], De-
scribable Textures Dataset (DTD) 7 EuroSAT 14, Flowers 21 Food101 (2) Oxford-IIIT Pet (Pets) [24] Sun397 [39], and UCF101
332 These data sets cover a large range of classification tasks such as fine-grained visual categorization, texture recognition, scene categorization, classification with low-resolution images, etc. The same parameters used for ImageNet are applied for most of them except certain parameters for refining labels. Concretely, Caltech and Flowers can benefit from the appropriate reference distribution for generating pseudo labels and set in Sinkhorn distance as y = 0.9 for ResNet and 7 = {1, 0.5) for ViT, respectively for the two data sets. Moreover, the threshold a is reduced to 0.3 on ETrOSAT. Other parameters remain unchanged.
Table summarizes the comparison. First, we observe that with the same parameters for all data sets, InMa shows the better performance than the baseline consistently and obtains 1% improvement on average by ResNet-50. It demonstrates that recovering the proxy in vision space can benefit different tasks for zero-shot transfer with pre-trained CLIP models. Second, by incorporating refined labels from optimizing Sinkhorn distance, InMaP can further improve InMap0.5 by more than 4% on average. The superior performance confirms that appropriate labels can facilitate the intra-modal

---

Table 8: Comparison of accuracy (%) 13 diverse downstream tasks with ResNet-50 and VIT-B/16. i in grey indicates the application of external large models. The overall best performance is in bold, while the best performance without any additional model is underlined. denotes that the result is unavailable in their original papers.
Aircmuf
Caltech
Cars
Citarto
Cifar100
CUB
DTD
EumSAT
Flowers
Food
Pets
SUN
UCFI01
Avg.
ResNet-50
Bascline
73.15
11.37
11.13
59.25
55.56
55.07
TPT
17.58
87.02
58.46
40.84
28.35
62.69
74.88
84.49
61.46
60.82
InMapas
16.95
86.41
58.11
74.23
40.72
41.82
42.14
27.52
66.29
74.95
82.94
59.95
57.49
56.12
InMaP
18.96
86.73
63.30
78.84
49.26
49.17
44.86
35.83
66.68
78.36
87.93
63.82
63.36
60.55
VisDesel
16.26
88.11
54.76
73.22
39.69
18.31
41.96
37.60
65.37
76.80
32.39
59.84
58.47
57.14
SuS-X-LoI
21.09
89.57
57.17
74.95
44.48
15, 86
49.23
44.25
67.07
77.62
86.59
63.01
61.19
60:41
SuS-X-SDD
19.47
57.27
74.69
50.59
45.57
67.72
62,95
60.46
0Y-8776
Baseline
25.13
93.51
66.29
91.09
67.29
497.56
45.09
50.17
67.64
84.43
87.00
65.68
21
65.84
TPT
47.75
InMapas
23.55
93.59
67.96
92.48
68.37
$1.54
46.28
53.64
69.64
85.76
9.8.7
67.18
67:25
67.34
InMaP
28.35
93.59
73.00
93.27
72.51
$7.87
50..2
63.9%
71.28
87.76
92.94
70,85
72.06
71.40
VisDese!
57.75
45.59
SuS-X-LCi
30.51
93.91
65,90
90
68.66
56.96
55.32
06
73.08
86, 08
91.58
67.85
66. 72
69.66
S0S-X-SIY!
89.08
54.55
73.81
86.08
67.73
proxy learning to obtain better proxy reconstruction in the target space. In addition, our method is the best among all methods on 8 out of 13 data sets with ResNet-50, which shows that leveraging unlabeled target data can be more effective than external models for zero-shot learning. Moreover, the same parameters are shared by different tasks for InMaP, while fine-tuning them by a validation set for each data set can further improve the performance.
With ViT-B as the vision backbone, the accuracy of the baseline increases by 10%, which shows the potential of large models. The proposed method can also benefit from the large models and achieve the accuracy of 71.4% on average, which is more than 5% better than the baseline. Compared with SuS-X with external models, InMaP outperforms it by 1.74% that confirms the effectiveness of intra-modal proxy learning. Moreover, for the data set with a high zero-shot accuracy as Caltech, the predicted distribution approximate the ground-truth well and the label refinement with Sinkhorn distance can be skipped by letting y = 1. Finally, both InMap0.5 and InMaP improves over the baseline on diverse tasks with ResNet and ViT, which illustrates that the proposed method is applicable for different vision encoders.
5 Conclusion
In this work, we focus improving the zero-shot transfer with CLIP. First, our theoretical analysis indicates that the modality gap between the text and vision space obtained by CLIP will be preserved, which can degenerate the performance of zero-shot transfer. To mitigate the challenge, recover the proxy of each class in the vision space with the help from the text proxy. Concretely, by leveraging the unlabeled target data and refining the prediction from the text proxy, we can obtain the vision proxy via learning with unlabeled data and the corresponding pseudo label. Experiments on diverse data sets demonstrate that our method can improve the zero-shot accuracy consistently with different vision encoders. Exploring our method with other pre-trained models rather than CLIP can be our future work. Besides, recent work [33] shows that other large pre-trained models, i.c.. GPT-3, can help CLIP for zero-shot prediction. Combining our method with existing large models is also an interesting future direction. Finally, our analysis for the efficacy of the text proxy from the perspective of approximating the supervised learning for images as in Eqn. T Text information can be complementary to the knowledge in images (36] and investigating the potential benefit of text for vision tasks can help understand rision-language models better.
propose to
Limitations
While our analysis is for the modality gap in contrastive pre-training with the architecture of dual encoders, it cannot be applied directly for other pre-training paradigms with different architectures. For example, BEiT-3 (37 has masked data modeling for pre-training with the shared backbone for different modalities and CoCa (4) has encoder-decoder captioning for pre-training. The analysis for the modality gap in these methods has been less investigated and can be inspired by our results as a future direction.
10

---

References
[1] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid. Label-embedding for image classification. IEEE Trans. Puttern Anal. Mach. Intell., 38(7):1425-1433) 2016.
[2] L. Bossard, Guillaumin, and L. Gool. Food-101- - mining discriminative components with random forests. In ECCV, volume 8694 of Lecture Notes in Computer Science, pages 446 66664.6.12 Springer, 2014,
[3] S. P. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2014.
[4] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry,
A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler,
J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner,
S. McCandlish, Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners, In NewrIPS, 2020.
[5] S. Chen, Z. Hong, G. Xie, Zhao, H. Li, X. You, S. Yan, and L. Shao, Transzero++: Cross attribute-guided transformer for zero-shot learning. CoRR, abs/21 12.08643, 2021.
[6] S. Chen, G. Xie, Y. Liu, Q Peng, B. Sun, H. Li. X. You, and L. Shao. HSVA: hierarchical semantic-visual adaptation for zero-shot learning. In NeurlPS, pages 16622-16634, 2021.
[7] M. Cimpoi, S. Maji, Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, pages 3606-3613, 2014.
[8] Cuturi. Sinkhom distances: Lightspeed computation of optimal transport. In NeurIPS, pages 2292-2300 2013.
[9] A. Dosovitskiy, I., Beyer, A. Kolesnikov, D. Weissenbomn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
[10] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested 101 object categories. In CVPR workshop, pages 178-178. IEEE, 2004.
[11] Z. Fu, T. Xiang. E. Kodirov, and S. Gong. Zero-shot learning on semantic class prototype graph. IEEE Trans. Putlern Anal. Mach. Intell., 40(8):2009-202) 2022, 2018.
[12] P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang. Y. Zhang, H. Li, and Y. Qiao. Clip-adapter: Better vision- language models with feature adapters. CoRR, abs/21 10.04544, 2021.
[13] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770-778. IEEE Computer Suciety, 2016.
[14] P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. JEEE Sel. Top. Appl. Earth Obs. Remote, Sens., 12(7):2217 2226. 2019.
[15] T. Huang. J. Chu, and F. Wei. Unsupervised prompt learning for vision-language models. CoRR. abs/2204.03649. 2022.
[16] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, pages 554 561, 2013.
[17] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[18] W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou. Mind the gap: Understanding the modality gap in multi-modar contrastive representation learning. In NewrIPS, 2022.
[19] S. Maji, E. Rahtu. Kannala, M. Blaschko, and A. Vedaldi. Fine- Fine-grained visual classification of aireraft. arXiv preprint 300
[20] S. Menon and C. Vondrick. Visual classification via description from large language models. CoRR, abs/2210.07 183, 2022.
[21] M. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In ICVGIP. pages 722-729. IEEE Computer Society, 2008.

---

[22] M. Palatucci, D. Pomerleau, G. E. Hinton, and T. M. Mitchell. Zero-shot learning with semantic oulput codes. In NeurlPS, pages 1410-1418. Curran Associates, Inc., 2009.
[23] O. Pantazis, G. J. Brostow, K. E. Jones, and O. M. Aodha. Svl-adapter: Self-supervised adapter for vision-language pretrained models. In BMVC. page 580. BMVA Press, 2022.
[24] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In CVPR, pages 3498-350S. IEEE, 2012.
[25] Q. Qian, I., Shang, B. Sun, J. Hu, T. Tacoma, H. Li, and R. Jin. Softtriple loss: Deep metric learning without triplet sampling. In ICCV, pages 6449-6457. IEEE, 2019.
[26] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR, 2021.
[27] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10674-10685. IEEE, 2022.
[28] O. Russakovsky, J. Deng, H. Su. Krause, S. Satheesh, S. Ma, Z. Huang. A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):211-2553 2015.
[29] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and open large-scale dataset for training generation image-text models. In
J. Jitsev. LAION-50 NeurIPS, 2022.
[30] M. Shu, W. Nic, D. Huang. Z. Yo, T. Goldstein, A. Anandkumar, and C. Xiao. Test-time prompt tuning for zero-shot generalization vision-language models. In NeurIPS, 2022.
[31] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. Raffel, E. D. Cubuk, A. Kurakin, and C. Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurIPS, 2020.
[32] K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/ 12s.0222,, 2012.
[33] V. Udandarao, A. Gupta, and S. Albanie. Sus-x: Training-free name-only transfer of vision-language models. CoRR, abs/221 161198 2022.
[34] C. Villani. Optimal transport: old and new, volume 338. Springer, 2009.
[35] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[36] J. Wang, Y. Xu, J. Hu, M. Yan, J. Sang, and Q Qian. Improved visual fine-tuning with natural language supervision. In /CCV, 2023.
[37] W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, and F. Wei. Image as foreign language: Beit pretraining for all vision and vision-language tasks. CoRR, abs/2208. 10442, 2022.
[38] Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata, Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE Trans. Pattern Anal, Mach. Intell., 4109) 2019.
[39] J. Xiao, Hays, K. Fhinger, A. Oliva, and A. Torralha. SUN database: Large-scale scene recognition from abhey to zoo. In CVPR, pages 3485-3492. IEEE Computer Suciety, 2010.
[40] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners image-text foundation models. Trans, Mach, Learn. Res., 2022, 2022.
[41] R. Zhang, W. Zhang, Fang, Gao, K. L. Dai, Y. Qiao, and H. Li. Tip-adapter: Training-free adaption of CLIP for few-shot classification. ECCV, volume 13695 Lecture Notes in Computer Science, pages 493-510. Springer, 2022.
[42] K. Zhou, J. Yang, C. C. Loy, and 7. Liu. Learning to prompt for vasion-language models. Int. J. Comput. Vis,, 1300)223377-2488, 2022.
12

---

A Theoretical Analysis
A.1 Proof of Proposition
Proof. According to the definition, we have
(1 - 8) =
Σ
exp(x, t//T)
Hence
8
8(m - 1) 1 1-6
exp(x, t//T)
exp(x x t//T)
exp(x; tt/r)
8 pjjt
8 exp(x} tz/T)
exp(x,Tt./ t:/T)
8
where arg maxx:/4 x tj. Since logarithm function is monotone, the similarity between positive pair can be bounded by
x; to + CIT S x t, S x tu + C2T
where = log( ( and C2 log( 5(m-1))
1-8
A.2 Proof of Proposition
Proof. According to the definition, we have
exp(x] 2; ///TT) P/. Σn exp(x. 2T/
exp( /ox, 2 /T/TT)
vax( w; wi/TT)
Ee expl /ax, ( z /(/TT)
Σn expl V /x wz/Ts)
Va/TI = 1/TT,, the predictions are equivalent.
A.3 Proof of Theorem
Proof. According the definition, we have
I| w 2C 2V W 20 - va) + val -W
(6)
To bound the similarity between 2nd and W we assume Z U where U, ddx
subset of U E Rdxd'' By solving the problem min ||U,A W' we have 7" U W U.U, W that is projecting W* to the subspace spanned by U,. Since
U, (U,,UU
is subset from U, we have U,U,- = Σi ruuu, with Vi,r, E [0, and E; = Then, we have
IU,U, - = E(n 1)s 1P E(r: (r+ 1)²s3
Therefore
1 -,,, w
2
where the last inequality is from keeping the largest singular values for minimizing the approximation loss. The target result is obtained by taking this inequality to Eqn. 6
A.4 Proof of Theorem
Proof. First, we note that L(P, W) is a convex function in W. We assume that it μ-strongly convex in W such that for the arbitrary (W1, W2), we have
(P) 2 L(P. Wo) + (Vw, L(P. W. Wa) + H [ Wal
2
13

---

According to the optimality of W we have
L(P", L(P' L(Y, w*) I L(P', w - = (P' - Y, - log(Pw>
(7)
The lower-bound be obtained from the strongly convexity of W as
L(P', W*) I(Y, = I(Y, - I(Y,W" + (P' - Y, 1..-lop (lgg((()--))
2タ W w + P Y. - logí Pw.-)) ))
(8)
2
Combining inequalities in Eqns. and
we have
2 1 -Ylle|l log(Pw.)) [ll/://oo(PP::)--oo/PPP log((ww)) )Ie
||" - 1/ S (P' Y,,og (w..) log(Pw.))) S /
14


Reviews:
[{"review_id": "wcJYNB1903", "reviewer": "Reviewer_eKBJ", "summary": "The paper aims to improve the zero-shot performance of CLIP, i.e.: the setting in which given a test image, we compute its visual embedding, computes its similarity to a pre-computer set of text query embeddings where each text query corresponds to a different class and finally output the classification results as class corresponding the most similar text query (in CLIP's joint embedding space). \n\nThe paper argues that zero-shot misclassifications are due to the visual-textual modality gap, i.e. the inherent gap in CLIP's embeddings space between embeddings of image and embeddings of their corresponding text queries. The paper argues (and theoretically proves) that this gap cannot be bridged by optimising on the text query and that the optimal proxy lies within the vision space. The paper proposes a novel algorithm that recovers vision proxies for text queries from unlabelled image test data. The paper demonstrates the effectiveness of the method on 12 downstream vision tasks.", "questions": "Nothing in particular.", "limitations": "The authors did not addressed the limitations of their work. ", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper addresses a key limitation of CLIP (and other vision-language models trained using contrastive loss) - the modality gap. The modality gap is an inherent fundamental \"built-in\" limitation of CLIP which hinders its performance. Tackling such a key problem in a wide spread and popular model (or model family) is of great important). \n* The paper provides both an in-depth theoretical discussion as well as a thorough experimental evaluation demonstrating the effectiveness of the proposed method.  ", "weaknesses": "Nothing in particular that I can point to. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R-eKBJ-L-A: Thank you for the suggestion. We will add the following discussion.\n\n\nWhile our analysis is for the modality gap in contrastive pre-training with the architecture of dual encoders, it cannot be applied directly for other pre-training paradigms with different architectures. For example, BEiT-3 [1] has masked data modeling for pre-training with the shared backbone for different modalities and CoCa [2] has encoder-decoder captioning for pre-training. The analysis for the modality gap in these methods has been less investigated and can be inspired by our results as a future direction.\n\n[1] Image as a Foreign Language: BEIT Pretraining for All Vision and Vision-Language Tasks. CVPR 2023.  \n[2] CoCa: Contrastive Captioners are Image-Text Foundation Models. TMLR 2022."}, {"review_id": "hshKITq22z", "reviewer": "Reviewer_dkv6", "summary": "Based on the phenomenon of modality gap, this paper analyzes that the proxy of visual space is better than that of text space, and explains this theoretically. This paper designs a test-time adaptation method for unlabeled test data", "questions": "I once thought about the role of textual models in downstream image classification.   CoOp and other works try to optimize the textual model and get success, but from the perspective of this paper, textual models seem to play no key role in downstream tasks just as an initialization.   What is the author's opinion?", "limitations": "The author has not provided any limitations of this article, so I hope you can add some analysis.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.  The influence of the modality gap on image classification is analyzed from the perspective of the proxy, and the theoretical results are amazing, which is of great significance for CLIP community research.\n2. The results of the experiment are very significant, and I think this article touches on the essence of CLIP.", "weaknesses": "Some analysis of validity for the textual model is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R-dkv6-W-A: Thanks for this great suggestion. With the contrastive pre-training, the textual feature will be aligned with the vision feature to enable zero-shot learning. It means that the similarity between the image feature $x_i$ and its corresponding text $t_i$ as in Proposition~1 can be lower bounded as $x_i^\\top t_i\\geq x_i^\\top t_k + \\log(\\delta/(1-\\delta))\\tau$ for $k\\ne i$. It implies that when $\\delta$ is sufficiently large, the relevant text has a higher rank than the irrelevant text, which is the key for zero-shot classification. We will add this analysis in revision.\n\nR-dkv6-Q-A: Textural model provides an applicable classifier only with class names, which is a great property for zero-shot learning. However, when labeled data is available as in few-shot learning, some recent work shows that fine-tuning the classifier initialized by the textural encoder can achieve the comparable or even better performance than CoOp, i.e., Table 7 and Fig. 14 in [1]. \n\nHowever, it should be noted that compared with the vision encoder, the textual encoder contains potential knowledge from text information that may not be covered by the vision data, for which the additional value as the regularization can be explored in the future work. We will add this discussion in the revision.\n\n\nR-dkv6-L-A: Thank you for the suggestion. We will add the following discussion.\n\nWhile our analysis is for the modality gap in contrastive pre-training with the architecture of dual encoders, it cannot be applied directly for other pre-training paradigms with different architectures. For example, BEiT-3 [2] has masked data modeling for pre-training with the shared backbone for different modalities and CoCa [3] has encoder-decoder captioning for pre-training. The analysis for the modality gap in these methods has been less investigated and can be inspired by our results as a future direction.\n\n[1] Robust Fine-tuning of Zero-Shot Models. CVPR 2022.  \n[2] Image as a Foreign Language: BEIT Pretraining for All Vision and Vision-Language Tasks. CVPR 2023.  \n[3] CoCa: Contrastive Captioners are Image-Text Foundation Models. TMLR 2022.\n"}, {"review_id": "aEG4P5Mw6e", "reviewer": "Reviewer_fSSP", "summary": "This work theoretically proves that the gap cannot be reduced sufficiently by minimizing the contrastive loss in CLIP and the optimal proxy for vision tasks resides only in the vision space. Two strategies are developed to further refine the pseudo label obtained by the text proxy to facilitate intra-modal proxy learning (InMaP). Experiments on ImageNet and extensive downstream tasks confirm the effectiveness and efficiency of the proposal.\n\nHowever, I have some concerns. Please refer to the comments below.", "questions": "- There seem to be some syntax issues, please check carefully. For example, Line 44 and Line 48.\n\n- Please distinguish between ablation studies and hyperparameter analysis. Does this paper include hyperparameter analysis in \"Alation on ImageNet\"? This confuses me.\n\n- For some comparison methods, this paper shows by lighter color, why is this? It doesn't seem to be clearly stated.", "limitations": "- This paper does not analyze the reasons for achieving zero-shot performance gains or provide deeper insights into the field.\n\n- See Weaknesses and Questions.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper is easy to understand.\n- To mitigate the problem of the gap between the text and vision space, the authors theorize that the optimal proxy for visual tasks is solely from visual space. This is reasonable and convincing.\n- Extensive experiments are performed on different datasets.", "weaknesses": "- For NeurIPS, the innovation or contribution of this work is limited, especially the technical solution. Proxy learning is not a new technique. Furthermore, pseudo label refinement is a combination of existing methods. \n\n- Related work introduces few-shot transfer rand zero-shot transfer. It seems that this paper does not report the performance of few-shot transfer (this paper uses CLIP as the Baseline, and CLIP performs a few-shot transfer experiment).\n\n- The layout of this paper is not very reasonable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R-fSSP-W1-A: The main contribution of this work is to theoretically understand the modality gap in CLIP, which has also inspired our novel proposal of proxy learning. To the best of our knowledge, this is the first work to explore intra-modal proxy learning for CLIP. Meanwhile, to better recover the appropriate vision proxy as implied by our theoretical analysis, we propose a novel pseudo label refinement strategy based on Sinkhorn distance, where it is not trivial to obtain an appropriate reference distribution for Sinkhorn distance as discussed in Line 210 and 215. Thanks for pointing out this. We will make this clear in revision.\n\nR-fSSP-W2-A: As discussed in Line 32: ``labels from the target domain may not be available'' and stated in Line 39, our work focuses on zero-shot learning. Many existing methods also solely aim to improve the zero-shot performance of CLIP as discussed in the related work. Thanks for pointing out this. We will make this clear in revision.\n\nR-fSSP-W3-A: Thanks for the suggestion. We will improve the organization and make it clear.\n\nR-fSSP-Q1-A: We will thoroughly check the writing as suggested in revision.\n\nR-fSSP-Q2-A: We included the analysis of hyper-parameter in the section of ``Ablation on ImageNet'' and the results can be found in Tables 1-3. Thanks for pointing out this and we will make it clear in revision.\n\nR-fSSP-Q3-A: As mentioned in the caption of Table 5, methods with grey results denoted by $\\dagger$ ``indicates the application of external large models'', which includes GPT-3 [1] and Stable Diffusion [2]. We will make it clear in revision.\n\nR-fSSP-L-A: Please kindly note that the main contribution of this work is to theoretically understand the modality gap in CLIP and derive our novel proposal of intra-modal proxy learning accordingly. The performance gain is demonstrated theoretically in Theorem 1 and 2, and empirically on ImageNet and diverse tasks in Table 5 and 6. This work can help better understand CLIP and inspire further improvement in the field. We will clarify in revision.\n\n[1] Language Models are Few-Shot Learners. NeurIPS 2020.    \n[2] High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022."}, {"review_id": "qLyVcI712e", "reviewer": "Reviewer_DJum", "summary": "Authors provide a series of detailed theories to support the conclusion that the inter-modal distance of the CLIP model is large and the gap can be reduced by introducing proxy features in visual feature space. They also propose the intra-modal proxy learning method to improve the performance of CLIP model with only unlabeled test data. Experiments on several datasets and different network architectures validate their proposed method's effectiveness.\n", "questions": "Please refer to Weaknesses.", "limitations": "No potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Motivation is clear and theoretical analysis is very sufficient.\n2. The mehtod derived from theoretical analysis is convincing.\n3. The paper is well-organized.", "weaknesses": "1. Temperature in CLIP paper is a learnable parameter which is initialized as 0.07 and becomes 100 after training. Is Tau_T in submission the same parameter as the temperature t in CLIP? And why set to 0.01.\n2. According to Table 2, it seems alpha contributions little to the proposed method. If we ignore the step, the acc will drop from 63.74 to 63.49, not too much for a single experiment.\n3. Can authors provide the results on UCF101 and eurosat, which are common datasets used in CLIP tuning methods.\n4. Can authors provide the results of default prompt instead of 7.\n5. About TPT results, authors copy the results from their paper, resutls of CLIP baseline is obvious higher than their bsaeline, why? If bsaeline uses ensemble prompt, please refer to Weakness(4). Using high baseline results and copied TPT results seems unfair.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R-DJum-W1-A: Yes, they are the same. 0.01 was obtained by CLIP and we did not change it. We will make it clear in revision.\n\nR-DJum-W2-A: Please kindly note that the result is obtained on ImageNet, where achieving a gain of 0.25\\% is not trivial. Although the parameter can be eliminated to reduce tuning efforts, we find that it is not sensitive and the same value can be applied for most of data sets.\n\nR-DJum-W3-A: Thanks for the suggestion. The results on UCF101 and EuroSAT are summarized in Table 1 in the attached PDF and we will include them in the revision. With the additional data sets, the main conclusion over the 13 data sets remains the same, where the proposed method achieves the best average performance among all methods.\n\nR-DJum-W4-A: Most of our baselines that relies on handcrafted prompts have 7 prompts, so we keep the same setting for the fair comparison. Note that TPT aims to learn the optimal prompt where the handcrafted prompts are inapplicable. The results with a single prompt of \\``a photo of a ( ).\\'' on ImageNet are provided in Table 2 (i.e., denoted by ``single'') in the attached PDF. It can be observed that our method still demonstrates a consistent improvement over baseline and outperforms TPT with a clear margin of more than 2\\%. Thanks for pointing out this. We will add this discussion in revision.\n\nR-DJum-W5-A: The CLIP baseline reported in our work is with 7 prompts suggested by [26], which is different from the baseline in TPT with a single prompt or an ensemble of 80 prompts. It should be noted that TPT aims to learn the optimal prompt that does not require the ensemble of prompts and is better than the ensemble. Therefore, the comparison is fair and TPT also includes the ensemble as the baseline in their original paper. Finally, we compare TPT with a single prompt in Table 2 in the attached PDF, where our method still shows a better performance by more than 2\\% on ImageNet. We will add this discussion and make it clear."}]

Author rebuttal:
We would like to thank all reviewers for their valuable time and insightful comments. We will answer (A) the weaknesses (W), questions (Q) and limitations (L) for each reviewer. The experiments suggested by Reviewer DJum can be found in the attached PDF.

Meta-review: This paper received accept, weak accept and borderline accept (2x) recommendations. There is a rebuttal.
Reviewer dkv6 requires some analysis of validity for the textual model and the authors provide an explanation in the rebuttal. Reviewer fSSP has various concerns around innovation / contribution and evaluation. Their concerns are mainly addressed in the rebuttal and they remain their borderline accept rating. Similarly, Reviewer Djum remains their rating.
The AC agrees with the reviewers' opinion towards accepting this work. The authors are strongly encouraged to include the additional feedback into the paper, as well as including the results from the rebuttal in the camera-ready version.
Decision: Accept (poster)