Boosting Vision-Language Models with Transduction
Authors: Maxime Zanella
Benoît Gérin
Ismail Ben Ayed
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Vision-Language
zero-shot
transduction
unsupervised learning
few-shot
Primary area: machine_vision

Abstract: Transduction is a powerful paradigm that leverages the structure of unlabeled data to boost predictive accuracy. We present TransCLIP, a novel and computationally efficient transductive approach designed for Vision-Language Models (VLMs). TransCLIP is applicable as a plug-and-play module on top of popular inductive zero- and few-shot models, consistently improving their performances. Our new objective function can be viewed as a regularized maximum-likelihood estimation, constrained by a KL divergence penalty that integrates the text-encoder knowledge and guides the transductive learning process. We further derive an iterative Block Majorize-Minimize (BMM) procedure for optimizing our objective, with guaranteed convergence and decoupled sample-assignment updates, yielding computationally efficient transduction for large-scale datasets. We report comprehensive evaluations, comparisons, and ablation studies that demonstrate: (i) Transduction can greatly enhance the generalization capabilities of inductive pretrained zero- and few-shot VLMs; (ii) TransCLIP substantially outperforms standard transductive few-shot learning methods relying solely on vision features, notably due to the KL-based language constraint.

Boosting Vision-Language Models with Transduction
Maxime Zanella" UCLouvain, UMons
Benoît Gérin" UCLouvain
Ismail Ben Ayed ETS Montréal
Code: http:///ghtaa..cca//aaaaaaal/att
Abstract
Transduction is powerful paradigm that leverages the structure of unlabeled data to boost predictive accuracy. We present TransCLIP, novel and compu- tationally efficient transductive approach designed for Vision-Language Models (VLMs). TransCLIP applicable as a plug-and-play module on top popular inductive zero- and few-shot models, consistently improving their performances. Our new objective function can be viewed as a regularized maximum -ikellhood estimation, constrained by KL divergence penalty that integrates the text-encoder knowledge and guides the transductive learning process. We further derive an iterative Block Majorize-Mo Majorize-Minimize (BMM) procedure for optimizing our objec- tive, with guaranteed convergence and decoupled aasplee updates, yielding computationally efficient transduction for large-scale datasets. We report comprehensive evaluations, comparisons, and ablation studies that demonstrate: Transduction can greatly enhance the generalization capabilities of inductive pretrained zero- and few-shot VLMs; (ii) TransCLIP substantially outperforms standard transductive few-shot learning methods relying solely vision features, notably due to the KL-based language constraint.
1
Introduction
4.3%
Zero-shot
-3.3%
2-shot Prompt Tuning
3.4%
2-shot Adapter
4.9%
TransCLIP improvement
4.00
3.1%
Top- Accuracy
4.3%
+3.1%
2.8%%
4.1%
+ 4.7%
+5.0%
+5.2%
6.6%%
4.6%
65
CLI
CLP ViT-B/16
EVA-CLIP 8B
ResNet-50
ResNet-101
ViT-B/32
ViT-L/14
Figure 1: TransCLIP improves significantly the averaged top-1 accuracy 11 datasets when used on top of inductive zero-shot CLIP, 2-shot CoOp prompt tuning and 2-shot TaskRes adapter for various encoder sizes.
Equal contributions and corresponding authors. {maxime. zanella, benoit, www.bb.ggo.inneeeeeeeeeoooo.eeee surlin)oulouvvin be
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Combining vision and language modalities can greatly enhance expressiveness and reduce ambiguities in the understanding and interpretation of our environment. This principle is central in the develop- ment of Vision- Language Models (VLMs), such as CLIP [50), which learns visual representations through natural-language supervision. In the pre-training phase, an input image x and associated text description are encoded by separate vision and text encoders. This yields feature representations f = (x) and = (c(c), which can be aligned by contrastive learning. Such a joint embedding space for the visual and textual modalities facilitates zero-shot recognition and yields powerful adaptation capabilities for a large variety of tasks. The recent literature on adapting VLMs has
grown substantially, in both the zero-shot and few-shot learning settings [73] 72 16 70, 74 25] 43)
However, so far, these techniques predominantly align with induction, i.e., inference for each test sample is performed independently from the other samples within the target dataset.
In contrast, transduction performs joint inference on all the test samples of a task, leveraging the statistics of the target unlabeled data [58] 21, In the context of standard vision-based classifiers, this has enabled transductive methods to outperform inductive-inference approaches as evidenced by benchmarks over large-scale datasets such as ImageNet [2].
Within the scope of deep learning, transduction has mainly been explored for few-shot learning to address the inherent challenges of training under limited supervision. This recent and quite abundant few-shot literature, e.g.. [ standard based pre wrairrinn models (such as ImageNet pre- treerrininng. However, as we will show in our experiments (Table 4. the direct application of existing transductive few-shot methods to VLMs yields poor performances, sometimes underperforming the inductive zero-shot predictions. This might explain why the transductive paradigm has been overlooked in zero-shot and few-shot learning for VLMs so far. The low performance of current transductive few-shot methods in the context of VLMs could be explained by the fact that the underlying objective functions do not account for the text knowledge. In this new multi-modal paradigm, additional supervision could be leveraged from the textual descriptions of the classes (prompts) [50), e.g., Ca =a photo of [kth class name], along with their corresponding representation t₂ = 0c(cx) derived from the language encoder. We utilize the interleaved representation of text prompts and images with their cosine" similarity f tk, which yields text-based prediction ýk, thereby guiding our transductive optimization procedure with text-encoder knowledge. Our method optimizes a new objective function integrating text-driven penalty. Optimization is carried out efficiently w.r.t the assignment variables associated with the unlabeled samples, which are then used as final predictions.
76 24 751. among others, has focused on adopting
Adapting VLMs has recently attracted wide attention in the literature, predominantly focusing on inductive methods. Motivated by findings in NLP, which indicate that better prompt strategies could enhance performance [53] 26 22), substantial efforts were directed towards prompt tuning (35]
for VLMs, with CoOp [73 standing out as the pioneering work along this line. Following CoOp,
prompt tuning has become the favorite strategy for adapting VLMs in a variety of contexts, including unsupervised [25] 43
15 41 and few-shot [73] 72 40 12
65| 6 74 8 29 30 67 learning.
Mcanwhile, there have been a few efforts towards computationally more efficient adapters 770 48 66).
Our transduction formulation aligns with this initiative. By operating solely on the output embeddings (i.e., in a black-box setting), TransCLIP is computationally efficient and does not make assumptions on the underlying encoder architectures. Still, our method is orthogonal to these design choices and could be applied atop any of the above-mentioned inductive approaches.
(i) We introduce transductive formulation that enhances the zero-shot and few-shot generalization capabilities of VLMs by leveraging the structure of unlabeled data [I.. Our new objective function can be viewed as a regularized maximum- likelihood estimation, constrained by a Hallbackk-eebbler (KL) divergence penalty integrating the text-encoder knowledge and guiding the transductive learning process. We further derive an iterative Block Majorize-Minimizz (BMM) procedure for optimizing our objective, with guaranteed convergence and decoupled sample- assignment updates, yielding computationally efficient transduction for large-scale datasets, such as IetgeNet. (ii) Our method can be used plug-and-p -ad---ddd-lyyy play module on top of current inductive zero-shot models and few-shot learning methods, consistently boosting their performance. Also, (iii) our approach substantially outperforms recent transductive few-shot methods in the literature, notably due to the KL-based language supervision as a critical success factor.
Main contributions.
IIn VLMs, such as CLIP 50). both visual and text embeddings are normalized (i.e., are withing the unit hyper-sphere). Thus, the cosine similarity corresponds the dot product.

---

2 Related Work
Transduction for vision-only classifiers. The use of unlabeled test data at inference time has received attention lately in rapidly emerging subjects, such as few-shot learning and unsupervised test-
time adaptation. Examples include adjusting batch normalization layer statistics [46] and minimizing the entropy of predictions [60], which can be supplemented by pseudo-labeling strategies [36]. In
the few-shot literature solely based on vision models, transduction leverages both the few labeled
samples and unlabeled test data, outperforming inductive methods [76. 5 24 37 75]. One of
the first works introducing transduction in vision-based few-shot learning proposes propagating the labels from the support (labeled) to the query (unlabeled) set with meta- graph [39]. Building on this idea, another work proposes to iteratively augment the support set to improve label propagation (4. LaplacianShot (76) also exploits the inherent structure of the data through a graph- Laplacian clustering, which discourages disparate class predictions for samples with close features, while matching each query set point to the nearest support prototype. Alternative approaches propose directly learning the class prototypes. For instance, Transductive Fine-Tuning (TF) [13 uses the prediction entropy on the query samples as a regularization term, while TIM and its variants [5] 59 employ the mutual information between the query samples and their predictions. BD-CSPN [37 refines the class prototypes by reducing the feature biases between the support set and the most confident query samples. An additional group of methods performs clustering in the feature space, for instance, by solving an optimal transport problem like PT-MAP 24. by projecting features into sub-spaces to facilitate clustering 775, or by revisiting the standard K-means with an additional partition complexn regularizer to control the number of predicted classes
Zero- and few-shot learning in VLMs. Thanks to their extensive pre-training, VLMs exhibit stronger generalization capabilities than vision-only models but may also fail [50] 67] In response, substantial recent efforts have been directed towards using their general knowledge and adapting them on more specific tasks 63 77, (7). Arguably, the most popular strategy is prompt tuning [35] which is explored both in the unsupervised (43
15
41
I and few-shot 73
72
40
12
65
6 74
g
9, (29 30 settings. The pioneering work, CoOp 73 updates input text-prompe tokens by leveraging
the context provided by the few labeled samples (i.e., the support set). Building on this success, various strategies have been developed to enhance this approach, especially through additional regularization. For instance, ProGrad (74) guides the prompts towards the original hand-crafted ones by gradient projection. Prompt tuning has also been explored in the zero-shot setting, e.g., using the predictive confidence to generate pseudo-labels 22 41). Despite its popularity, prompt tuning remains tedious in terms of computations, due to the many back mmck-rropagations through the text encoder. This challenge compounded in the recent developments, which introduce visual tokens 29 30] alongside the text tokens. In contrast, there has been limited efforts so far in developing black-box 66 62, which only access the final embedding states. These methods often rely on the so-called adapters 23 like Tip-Adapter(-P --F) [70], which adds classifier at the output of the vision encoder, in the form of a cache model involving the few-shot samples. Lately, strong baseline based on Gaussian discriminant analysis clustering (62) demonstrates VLMs' adaptation abilities with a Gaussian hypothesis on the embedding space.
methods 48 17
16
Transductive inference in VLMs. Despite the growing interest in unsupervised, zero-shot and paradigm has not been explored so far in this
few-shot learning for VLMs, the waassuutive-infereen new multi-modal context, except for the very recent work in [45), which was deployed for small-size tasks (Ns 102 test samples). However, the method in [45] may not be computationally tractable for large-scale query sets, due expensive inner loops for estimating the Dirichlet distribution's parameters. We provide a computationally efficient solution, which can scale up to large target datasets (such as ImageNet). while being easily amenable as a plug-and-play module on top of state-of-the-art inductive methods. It is worth mentioning that test-time adaptation methods also employ the transduction paradigm, but their settings are very different from those studied in this work. For instance, SwapPrompt [41] has been designed to make batch predictions on-the-fi -ly, and has continual-leame leaminng mechanisms such as an exponential moving average prompt across batches. TPT [3] work on single sample with many data augmentations to train one prompt per image. Both methods require access to model weights for training (i.e., do not operate a black-box setting) and an expensive training procedure. We also note that prompt tuning does not scale well with the model size and is even impractical on very large models such as EVA-CLIP-8B (55) We still report the performances of this class of methods in the Appendix (Table 9)
3

---

3 TransCLIP: Transduction for Vision-Language Models
In this section, describe objective function for transductive inference in vision-language models, and derive a block majorize-minimize (BMM) algorithm for minimizing it, with guaranteed convergence and decoupled sample-ane camplee-ssignment updates. When dealing with a zero-shot classification problem based on vision-language model, such as CLIP, and given a set of K candidate classes, class, e.g., Ck a photo of [kth class name], lo = I..... K. Let t (,(c)) denotes the corresponding normalized (unit hyper-sphere) embedding representation, with 8, representing the language encoder. Similarly, each test image x,, = I..... NN is projected onto normalized embedding space of the same dimension, using visual encoder 0,: f, = (x,). In the standard inductive zero-shot inference, classification of given image x; is done by evaluating the cosine similarity between these two encoded modalities and predicting the class corresponding to the most similar text embedding: k = argmax,, f, ty. Furthermore, one can compute pseudo-labels corresponding to these zero- shot predictions by applying the softmax function with a temperature scaling T, which yields the following pahability-simplle vector for each sample:
one creates textual descriptions, the so-called prompts [38. each corresponding
exp(rf. t&)
y, (Dsx))skSK E AKK Ú1.k exp((TT,Tt
where Ax denotes the probability simplex. Let D (iEN:1sis samples indices of the target dataset, with Q the set of unlabeled query samples indices, i.e., those for which we want make a prediction, and S the set of labeled support samples indices in the few-shot setting.
denotes the
Note that, in the zero-shot setting, S = 0. We define Gaussian Mixture Model-clustering (GMM) term in our objective function by modeling the likelihood of these target data as a balanced mixture of multivariate Gaussian distributions, each representing class i and parameterized by mean vector μg and a diagonal covariance matrix Σ:
Pi.k Pr(fi, k; Mg E) det(E)= exp (- [[(f-Mu) m4)
14x))
Notation Pr,k is introduced here simplify the equations in the sequel. Notice that, unlike standard GMMs, we deploy a common diagonal covariance matrix Σ across all classes. Interestingly. in experiments,
found this simplifying choice improves the performance while reducing the computational load as there are substantially fewer parameters to learn. This is particularly the case when dealing with large numbers of classes in large-scale target datasets such as ImageNet.
3.1 Proposed objective function
Our objective function depends on two types of variables: (i) Sample-to-class assignment variables within the probability simplex: 2; (#L4)ISHKR E Ax. E Q; and (ii) GMM parameters
f = (1+) and Σ.
We propose to minimize the following objective, which integrates a GMM-clustering term, a Laplacian regularizer and Kullback- Leibler (KL) divergence penalty encoding the text-encoder knowledge and guiding the transductive learning process:
LyuBo-sHor (z, 12, 14, Σ)
Σ 2 log(p:) ΣΣ IEDJED
WijZ Zj +
Σ TEQ
(Z,(2,1)5.)
S6 º
GMM clustering
Taplacian reg.
Text knowledge
where p; = (pr.))skSK E A concatenates the GMM probabilities, Wij denotes some measure of affinity between visual embeddings f, and fy. and the sample- wise parameterized KL terms are given by:
(z)(2)l)))) 2,7 log z, Az, logýr i Q
2>0
In the following, we describe the effect of each term in our objective function in 2
2 Note that each CLIP version comes with a temperature scaling factor 7, which is optimized along with the learnable parameters during pre-training.
'Nticee that, for À 1, the expression in corresponds to the KL divergence.

---

GMM-based clustering: This unsupervised-leammin term is akin to the GMM-based muximum-likelimee estimation objective in the standard EM algorithm [3] By taking the negative logarithm, its minimization corresponds to maximizing the likelihood of the data. can also be viewed as probabílistic generalization of the K-means clustering objective (28). Indeed, assuming E is the identity matrix reduces the first term in 2 to the K-means objective.
Laplacian regularization:
The second term (2) is the Laplacian regularizer, widely used in the context of graph/spectral clustering [56] and semi-supervised learning [7]. This term encourages nearby samples in the embedddng space (i.e., pairs of samples with high affinity Wi,;) to have similar z assignments. In our case, we propose to build a positive semi- definite (PSD) affinity matrix based on the cosine similarities as wi f. f (Gram matrix). As we see below, this PSD condition is important to obtain convergent Mlajorize-Miniman optimizer with decoupled (parallel) sample-wise updates for the z-assignments, yielding highly efficient transduction for large-scale target datasets (such as ImageNet).
Text-guided KI. divergence: This term dedicated vision-language models and, will see in our experiments (ablation studies in Tables 4 and 6). has a substantial effect on performance. encourages the prediction not deviate significantly from the zero-shot predictions, thereby providing text supervision to the other two unsupervised-leariin terms. Furthermore, being convex over Z;, E Q. this term facilitates the optimization of the objective w.r.t the assignment variables.
3.2 Extension to the few-shot setting
Our zero-shot formulation naturally extends the few-shot setting. We integrate supervision from the labeled-support samples, in the form of cross-entropy, which corresponds to minimizing the following overall loss:
Lrew sHOT (z, E) =
Σ z log(p:) + CZERO SHot (z, pe, Σ)
(4)
[S]
iES
Note that, in the first term, the z; are fixed, with z; yi, E S and y; the one-hot ground-trun -ruth label associated with the corresponding shot.
3.3 Block Majorize-Minimize (BMM) optimization
As our objective depends on three types of variables (z, μ. E), we proceed with a BMM procedure, alternating three sub-step optimizers. Each sub-step optimizes over one block of variables while the other two are fixed, ensuring the overall objective does not increase. Importantly, the obtained z-updates (Eq. (5) are decoupled, yielding computationally efficient transduction for large-scale datasets. Also, our overall procedure is guaranteed to converge (Theorem I
Majorize-Minimize (MM) with respect to the z-block When la and Σ are fixed, both the GMM- and KL--hased terms are convex w.r.t z; However, the Laplacian term is concave (for (for PSD matrix W.. Therefore, we proceed with inner iterations, each minimizing linear and tight upper bound, the so-called majorizing function in the MM-optimization literature 33 21 [1], which guarantees the overall objective does not increase. To obtain the tight linear bound, let us write the Laplacian term conveniently in the following matrix form: z Vzz, with Y - W @ I, where & denotes the Kronecker product and I is the N N identity matrix. Note that A is negative semi-definite for positive semi-definite W. Therefore, z Lz is concave function with respect to z. and its first-order approximation at current solution z' (1 being the iteration index) gives the following tigh upper bound on the Laplacian term:
z Vz < ( T
(z
Replacing the quadratic Laplacian term by this linear bound yields a majorizing function on our overall objective. Importantly, this majorizing function is a sum of decoupled objectives, each
"This makes the overall sub- sab-problem non-convex and there no closed-form solution. "Tight" means that the upper bound is equal to the original objective at the current solution z'.
5

---

corresponding to one assignment variable z;, yielding highly efficient optimizer for large- scagle target datasets. Indeed, using simplex constraints z, E Ax, Q, and solving the Karush-Ko Kuhn- Tucker (KKT) conditions independently for each 2ú, we obtain the following decoupled update rules for the z-block:
c exp((og(p.) EjED wiyz, (1
(1+1) 2
exp(log(pi) EjEDD W1iz T
Closed-form updates of f and E When both z and E fixed, objective in 4 is convex. can be minimized by setting its gradient w.r.t each My to zero, which yields the following closed-form updates:
1y S Eies u., + O Eieg Z.,.
(6)
Cies 2 + o Eieo 2
Similarly, when both z and μ are fixed, the following closed-form updates minimize the overall objective w.r.t E:
diag(2) [S] Eies Ex (f
121 Eieo
7 +
The complete procedure is summarized in Appendix B Note that, after convergence, we use the sample-to-elass assignment variables 2, as predictions for each sample of the query set Q using the argmax operation for conventional classification.
3.4 Convergence
Our optimizer can viewed as an instance of general Block Majorize-Minimizz paradigm for optimization [51], which optimizes majorizing function for each block of variables. The convergence of general BMM procedures well studied in the optimization community [51]. Indeed,
under certain conditions (such as the strong convexity of the block- wise majorizing functions), we can establish convergence of our procedure using the following result (more details in Appendix A):
Theorem (Convergence of BMM (51) Assume that, for each block, the majorizing function quasi-corver and its first-order behavior is the same as the original objective locally. Furthermore, assume that the sub-problem solved for each block has unique solution. Then, every limit point of the iterates generated by BMM is coordinate -ist minimum of the overall objective.
4 Experiments
Datasets.
Following the setting of previous works 73 33
we assess TransCLIP on ImageNet |||
and ten datasets for fine- fine-grained classification scenes (SUN397 (44), aircraft types (Aireraft 4p ).
satellite imagery (EuroSAT (18), automobiles (Cars B2). food items (Food [4],, pet breeds
(Pets (49]. flowers (Flowers [47] general objects (Caltech101 (4). textures (DTD (IO) and
human actions (UCF101 (54)). We additionally measure performance on four variants of ImageNet
(Adversarial 20 ImageNetV2 [52], Rendition [9) Sketch 61) Numerical results are reported in
terms of the top-1 accuracy with the VIT-B/16 encoder, averaged over three random seeds.
Benchmarks. We aim to show the breadth potential applications of transduction in the context of VLMs. Notably, employing supervised fine-tuning, followed by transduction with TransCIIIP on the unlabeled test samples, emerges as a powerful and efficient solution. This is particularly convenient when the labeled samples (the support set) and/or computational power not accessible at inference (i.c., test) time" To this end, we first study the applicability of our zero-shot formulation TransCLIP- ZS (Eq. (2) across three settings:
(i) on top of inductive zero-shot learning and popular few-shot learning methods; (ii) on top of 16-shot ImageNet pretraining for cross-datars transferability, and
"This application is hardly discussed in the transductive literature. We make all zero-shot- and few-shot text and image embeddings publicly available, to future works without resorting heavy computations.

---

Table 1: TransCLIP atop inductive vision-language zero-shot and popular few-shot methods,
Method
Imagetes
Ansrall
EaesSAAr
Fowtloi
Flowerfff LPC
Caloebitti
DTD
UCr101
CLIP-VI---16
66.6
62.5
24.7
48.3
05.6
85.9
89.1
70.7
41.2
43.5
67.5
65.3
+ Trans
105,33
68.96..3
26.43.2
65
14666
894,388
K7-1
76.7.89
92.7.g.5
49.5.4.
74/46699
70.3
CoOp eoy
65.7
66.9
20.7
56.4
67.6
84.3
90.2
78.2
92.5
20.1
712
67.6
+ TTaasCLIPg2
683,36
11.54.6
23.8.31
65.3,.9
71.46.3
663,20
01.--18
81.8.11.5
938,1.3
35..5.4
77.7,6.8
724,48
065
672
76.8
678
671
85.8
10.6
K37
44.0
51.
154
200
TrassCLIP-2 25
720,25
713,46
3..71.9
769.011
71.03.
869
92.8.9.1
935
57.7.61
60.
PLOT
669
67.0
68.
$4.9
91.9
51.
44.0
74.7
71.3
TranaCl
158,89
M,3,33
28 a.
7K8.6.
760
K3.3
91.1
03.2.11.4
567. 567.39
8144677
750,87
Tacokes
01.6
TratsCLIP- 2P
12.022
12.5,44
314
7..7..11
71562.4
865,30
91.6.1.5
20.7.0.1
59.5.6/0
764.44
145.37
FroGrad
67.0
67.0
57.0
68.2
50.8
03.5
69.5
IP-22
7165,66
30.5,1.8
-199
72.3,61
565.166
92.7.14
91.5.10.7
661.667
57.9.6.1
79.3.6.1
143,48
CoOp axy
68.1
92.5
92.2
94.5
59.4
74.0
26
13,3,5
42.3
772.25
56.
1.11
05.8.3.1
10.6
63.
81.8,43
MIP-M secerit in
707
708
35.7
70.8
24.1
56.5
41.9
94.
78.1
75.6
TrassCLIP-25
127.19
144,35
36.
793729
674.009
9..2.1.3
95.5,33
64.0.4
83,3,32
77..2.3
PLOT maces
700
718
34.8
84.7
76.6
55.5
92.8
93.2
94.9
61.0
79.7
76.6
+ TTaast"3 IP-2s
712,72
35. a.
188
636,222
95.8
9.
650
813,3.6
180,1.4
Tacskes aven
710
72M
33.3
733
76.1
16.1
v1.9
55.0
94.9
56.7
75.5
74.0
* TrassCLIP-2 CLIP-2
750.22
153,23
34441.1
761144
77.3.1.1
67.3.1.2
93.0..11
02.4.7.4
95
6433466
79.2,3.7
77.3,277
FroGrad + TTaas 1P-25
702
21.7
34.0
00.5
75.0
85.4
92.0
91.1
94.4
50.3
77.9
74.6
225
250,33
35.
24.5..3
27.92.
467.1.7
05:3,4.
95.
64%.5.1
81.2.5.4
77.
CoOp axy
719
74.9
43.3
8500
6288
64.2
91.9
96.8
95.8
09.7
53.1
79.9
1 'TrassCLIP 25
153.14
76
42%
150.10
K3.
86.3
9,.2,12
97.5.03
.5.
71.3.17
85.4,23
8L..1.1
733
76.0
15.9
x2.3
86.8
42.0
v6.7
K39
M0.7
4 TranSCLIP22
142.99
763,08
449,8.3
65.2a.
K2.740.4
674.00
93.3.0.9
96..
95....1
692.4
85.6,1.7
PLOT "TraaSCIIP
723
36.0
$4.6
85.6
92.5
96.0
548
81.7
2788,53
750.10
4.9
84.5.5
k59.0.2
v7.3.a.1
9.0.10
68.7.44
85.7,09
803.14
Dasokes ever
63.5
14
769.au
4.2
80.5g.3
123.a7
67.5.m6
.84
97.6.a.3
96..0.1
7.22.
86.2.28
60.3.a.1
ProGrad
72.1
$2.9
85.8
56.6
05.9
68.9
79.9
TTrasCCPPP
735.14
76.,17
428.g0
83.7.02
83.3,02
874,13
93..,48
97.4.08
961,0.1
714.25
86.134
SL3.1.
Table Cross-Datans transferability evaluation. Few-shot learning methods are trained on 16-shot ImageNet and evaluate on the ten other fine-grat ine-grained datasets. Average excludes ImageNet.
Thegs
Method
Nos
SUN
EaroAAT
StafoutCars
Foddini
Peli
Fowwar 102
Caleehi11)
DTD
UEFI01
Avarage
CoOp mu
39.5
64.1
62.0
73.3.14
67.45544
11
$4..999
66-
K6.1,20
89--1.11
14/2,7.2
48.4.07
42.1
69%
.4,1
Cross Datsee
CoCoUp sooe
71.1
67.9
22.7
44.5
64.9
86.2
90.7
31.6
91.9
45.2
68.8
60.5
163,5.
696,2.7
2.
392
67.0/11
8988
790077
963,003
505.54
145,5.2
692,
MaPLE poocs
70.5
67.3
26.4
45.5
65.7
86.4
90.4
12.0
93.7
46.3
65.7
66.1
- 11P55
76/.61
60.,2.5
24..002
59
66 66..1.2
89.7
78.0,60
94 3406
404.3.1
744,56
692
ProGrad
38.9
64.0
35.9
673
92.9
45.2
65.9
634
TTanCLIPPSS
647
22.7.11
552 16.4
67.9
670,
91.
13.9 6.1
94.0 ,1.1
466.34
13.5
68 466
Promphhit -TrassCLJPPS
769,5.5
699,2.
249.03
594,344
676,20
0.7
76776.2
24.4
51:1.5
769,70
694.37
(iii) on top of 16-shot ImageNet pretraining for domain generalization on the four ImageNet variants. Secondly, we compare our few-shot extension TransCLIPP (Eq. (4) to transductive few-shot learning methods. As for TransCLIP--S,, we operate a black-box setting (i.e., using only the output embeddings, without training the model parameters).
Implementation details. The main component of our transductive formulation is the text- text-guided KL divergence penalty. We fix A 1 for all our zero-shot experiments (see ablation study Table o and X = 0.5 in all the few-shot experiments to reduce the impact of the text-driven regularization. Another component of our optimization problem is the Laplacian regularization, which enforces consistent predictions for close instances. We truncate the affinity matrix to the 3 nearest-neighbor making sparse. μ is initialized with the top-8 most confident samples of each class for the zero-shot setting. For the few-shot setting, we use the class- wise average over the shot embeddings.
4.1 Main results
Transduction improvements. Table [ and 2 demonstrate the advantages of our transductive approach in zero-shot, few-shot, and cross-dataset transferability. TransCLIP enhances the zero-shot 1 accuracy by over 5% and popular few-shot methods by 4% (1-shot) on average, without the need for additional labels. Ta highlights that TransCLIP can applied on top of prompt tuning and adapter fine-tuning solutions, enhancing performance for both in-domain and domain
7

---

Table 3: Domain Generalization evaluation with improved manual prompting strategy (custom templates are given in Table 24b 16-shot prompt-tuning and 16-shot adapter.
Target
Method
ImageNett
Adivensrial
ImageNetV2
Rendition
Sketch
Average
Average OOD
CLIP-VET-BII o/ a phoko of
66.6
47.9
6016
25.8
46.0
59.0
57.1
G-hont
+ TrasCLIP-ZS
7033337
18.9
62.3,1.7
75.0.1.
40.7.3.7
614,,24
39.2.2.1
tomplates
62.3
77.8
48.4
T1.742.7
02.1,1,4
63. 63.4,111
16.1.0.3
61.1,2,7
63,2,1.6
1,13
CLIP-ViT-W16 w/ prompi timing (CoOp)
71.9
49.4
64.1
75.1
47.2
61.5
59.0
Domaia
TransCLIP
13.3,1.4
5..8.1.4
64.60.5
60.3.3.1
610.1.5
00:4,14
CLI--VII-B166 adapeer (oashllss
73.0
50.3
65.6
77.8
49.2
63.2
60.7
TrnsCLIPPZS
74.41.1
11,6,
66. 65.4.02
78 18.6
51,6,2,4
6433,11
61 6118.11
Table 4: Transductive few-shot learning evaluation. w/o text denotes X = 0 in Eq.
Shots
Method
ImageNar
SUNS37
Airenstt
BunSAT
Suaffrdd las
FosdU0)
Part
Flowors102
Catesb1011
DTT
UCF101
Average
CLIP-VE-B/I6
66.6
62.5
24.7
48.3
65.6
85.9
89.1
70.7
93.2
43.5
67.5
65.3
TF [3
38.1
32.5
43.5
67.8
75.5
31.6
48.8
42.8
RD-CSPN
42.0
83.5
ShplacianShtt
41.1
РТTMAP X
40.1
$2.6
23.8
39.7
484
61.4
61.8
69
S6.1
41.8
63.5
$2.7
TMM [
37.5
48.3
22.8
48.3
44.8
65.7
$3.9
86.4
75.1
35
62.7
$2.8
TransCLIP-ES wdlo deat
30.2
43.4
23.7
56.6
41.0
50.9
$4.3
83.5
77.7
36.9
54.5
50.2
TrasCLIPPSS
69.8
70.6
29.9
72.5
70.9
87.9
93.8
84.8
93.1
53.3
78.4
73.2
TF
56.8
53.7
64.1
BD.CSPN
ShelacianSht
62.5
29.6
74.3
58.5
92.8
92.0
$2.7
71.7
PT-MAP
$7.6
68.1
31.2
74.9
63.1
81.1
79.5
76.2
60.2
$8.4
73.9
65.8
TIM [
$7.4
67.0
32.8
79.3
65.8
83.5
K2.3
93.4
88.5
$8.1
76.5
71.3
TresSCLIPPSS
$3.9
63.8
34,2
79.4
63.5
76.7
76.7
93.3
92.8
57.0
74.8
69.6
LIPSSCIPPES
70.3
74.0
94,0
61.1
79.1
75.9
RD&CSPN
70.7
76.0
LaplacianShot 70
00.9
68.3
36.1
78.1
69.2
81.2
81.7
94.8
95.1
58.6
76.3
72.6
16
PT-APP [4]
64.0
72.0
37.4
75.6
72.0
82.7
96.1
78.5
63.7
63.7
76.3
70.2
TIM X
67.8
73.6
40.6
X3.6
79.5
84.9
88.7
95.4
92.4
67.5
82.1
77.8
TransLIPPFS wef
65.9
$1.1
77.0
86.1
95.2
94.6
65.3
80.0
76.6
TresCLIPPPS
generalization tasks. However, we observe in Table 1 that transductive gains sometimes decrease with the number of shots, presumably because data structure information can be partially captured in the shots. These results underline the value of considering the structure of the unlabeled test samples during prediction, especially on top of zero- and low-shot models or when facing domain shifts, an aspect not leveraged by the current zero- and few-shot VLM literature. More detailed results for five different backbone architectures and comparisons with unsupervised non-transductive methods are provided Appendix I I for the zero-shot setting, in Appendix C.2 for TransCIIIP on top of popular few-shot methods, in Appendix C.3 for cross-datase transferability and in Appendix CC4 for domain generalization. With its
unchanged, TransCLIP exhibits strong generalization from convolutional networks transformer-based models, as also depicted in Figure
Transductive few-shot learning. We compare TransCLIP-FS. TransCLIP-F'S without text regularization (i.e., = 0) and state-of-the-ard transductive few-shot methods. It is important to note that these few-shot methods were primarily developed for vision-centric tasks. Hence, they rely on visual information, omitting the textual elements. This allows us to study the impact of our text-based regularization term. Table 4 shows that incorporating language in the transductive paradigm boosts the performance over vision-only methods. Especially for the 1- to 4-shot settings, our language-driven KL penalty enhances the performance by large margin on many tasks (c.g., ImageNet, SUN397, StanfordCars, DTD). As the number of shots increases, the text-driven penalty becomes less useful, especially for the datasets capitalizing on the visual shots rather than the text-encoder knowledge (e.g., EuroSat and Flowers). This points to promising future directions involving more flexible text regularization (e.g., an adaptable taking into account the number of shots and the quality of the text embeddings). Detailed results for five different encoder architectures are provided in Appendix C.5 consistently showing similar conclusions.
regulari

---

Table 5: Performance and runtime comparison between TransCLIP and prompt learning solutions on average over ImageNet and the 10 fine- ene-grained classification datasets. UPL" is transductive adaptation of the original unsupervised procedure
25 more details in Appendices C.1 and
(a) Zero-shot setting.
(b) Few-shot setting (4-shot).
Performance
Runtime
Performance
Runtime
UPL. TransCLIP--S
>150
CuOp+UPL* TransCLIP-FS 75.9
74.4
>12h
70.3
14.4 sec
35.3 sec
Table 6: Analysis on the components and sensitivity hyper-parameters of TransCLIP-ZS.
(a) Components of the procedure.
(b) Text regularization hyper-parameter X.
Update Update
Lapl.
ImageNet
SUN397
Aireralt
EuroSAT
ImageNot
SUN397
Aireraft
EaroSAT
69.7
67.5
25.5
63.9
0.1
56.3
58.6
26.0
63,5
68.7
25.1
51.6
0.5
69.8
69.3
26.6
65.6
x
69.9
68.8
27.0
64.5
2
69.5
67.6
26.2
64.1
70.3
68.9
26.9
65.1
5
68.2
65.2
25,2
51,2
(c) Number of nearest-neighbors
(d) Impact of an isotropic E.
neighhors
ImageNet
SUN397
Aircraft
EuroSAT
ImageNet
SUN397
Aireraft
EumSAT
70.3
68.9
26.9
65.1
Σ (ours)
70.3
68.9
26.9
65,1
5
70.3
68.9
26.8
65.1
isotropic
69.4
68.0
64.1
10
70.2
68.8
26,9
65.2
-0.9
-0.9
-0.5
-1.0
Comparison with prompt learning. Following current VLMs literature, adapting the input prompt instead of GMM parameters could be seen as more straightforward solution. For fair comparison, adapt Unsupervised Prompt Learning (UPL) 25 for the transductive setting and reevaluate its main syper-parameter (see Appendix C.I). T shows clearly that TransCLIP outperforms UPL while being two three orders of magnitude faster. Additional details on runtime provided in Table of the Appendix.
4.2 Ablation studies
Components of TransCLIP. We study the impact of the principal components involved in the TransCLIP procedure over four diverse datasets. Table 6a shows that updating μ and Σ allows to significantly boost TransCLIP's performance. This indicates the importance of having dynamic parametric model instead a fixed one. Table Table demonsstsss the critical role of text-driven penalty for TransCIIP in the zero-shot setting. Additional results the sensitivity of X in the few-shot setting are depicted in Figure of the Appendix. Alongside the prior findings from Table it is evident that incorporating text information is key the success of TransCLIP and its wide applicability across the zero- and few-shot learning scenarios. The number of nearest- neighbss considered in the Laplacian term (Eq. (2) does make a significant difference in TransCLIP's performance
suggested by Table 6c However, removing the Laplacian regularization (Table 6a) leads to inferior results on some datasets such as ImageNet and EuroSAT. We choose to consider nearest-neighhors to make the affinity matrix W sparse and reduce memory consumption. We also investigate the diagonal covariance matrix design by restricting it to be isotropic (i.e., o² La with Ia the identity matrix). Table 6d shows that a non-isotropic Σ performs better without significantly increasing the amount of trainable parameters.
Scaling larger VLMs. We report TransCLIP-ZS performance on EVA-CLIP 8 billion parameter It is worth mentioning that
version [55] (approximately 42 times larger than the CLIP-ViT-BI TransCLIP is easily applicable to multi-biliion parameter models since it does not necessitate gradient computation model parameter training (i.e., it only requires the memory needed for single-sample inference because the whole dataset processing can performed one sample a time). Table shows that transduction also bring significant improvements to larger models (details in Appendix C.I

---

Table 7: Performance of TransCLIP-ZS for increasingly large VLMs. Relative A is the improvement
normalized by the zero-shot emror. (ACCTRANsCLI)) ACCzezo-Snnl
(100
ImageNet
Avenge (l1 datasets)
#Params
Zero-shot
TransCLIP-SS
relative A
Zero-sbot
TransCLLP--S
relative
RIT
177M
66.6
70.7
+11 %
65.3
70.3,5.0
+14 %
CIIP-VIT-/I4
427M
72.9
77,244.3
+16 %
72.5
77.4,4.
+18 %
EVA-CLIP-SB
7.58
82.5
84.6,2.1
+12 %
81.5
85.8,4.3
+23 %6
5 Conclusion
In this work, we studied the transductive paradigm in the context Vision-Language Models and proposed the TransCLIP method. Our algorithm is highly efficient, it operates solely in the output embedding space (i.e., black-box setting), making it suitable for a wide range of models, including very large ones. This also enables TransCLIP to be compatible with models that are accessible only through APIs. We first showed how TransCLIP can bring transduction to the inductive zero- shot setting, achieving consistent gains without additional supervision. Then, proposed setting that applies transduction on top of popular few-shot methods, offering a convenient strategy combine computationally intensive supervised fine-tuning with efficient test-time transduction. Finally, we highlighted the limitations of current transductive few-shot methods and proposed a simple extension of TransCLIPP incorporate labeled samples. In all our experiments, TransCLII'ss text- text-guided KL divergence term appears as a key factor in its success. Future work may focus on further enhancing this regularization term, for example, by making more resilient (e.g., with adaptive class-wise weighting) when text prompts are less reliable.
Acknowledgments
M. Zanella and B. Gérin are funded by the Walloon region under grant No. 2010235 (ARIAC by DIGITALWALLONIAA.N The present research benefited from computational resources made available on Lucia, infrastructure funded by the Walloon Region under grant No. 1910247.
10

---

References
[1] Jameel Abdul Samadh, Mohammad Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muhammad Muzammal Naseer, Fahad Shahbaz Khan, and Salman H Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. Advances in Neural Information Processing Systems, 36, 2024.
[2] Omer Belhasin, Guy Bar-Shalom, and Ran El- Transboost: Improving the best imagenet performance using deep transduction. Advances in Neural Information Processing Systems, 35: 2022.
[3] Christopher M Bishop. Pattern recognition and machine learning. Springer google schola, 5 43, 2006.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-minn mond-inn discriminative components with random forests. In Computer EsiionEECCY 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
[5] Malik Boudiaf, Imtiaz Ziko, Jérôme Rony, José Dolz, Pablo Piantanida, and Ismail Ben Ayed. Information maximization for few-shot learning. Advances in Neural Information Processing Systems, 23:2445-22457, 2020.
[6] Adrian Bulat and Georgios Tzimiropoulos, Lasp: Text-to-text optimization for language-aware soft prompting of vision & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23232-23244 June 2023.
[7] Olivier Chapelle, Bernhard Schlkopf. and Alexander Zien. Semi Suni-Supervised Learning. The MIT Press, 1st edition, 2010. ISBN 0262514123
[8] Guangyi Chen, Weiran Yao, Xiangehen Song, Xinyue Li, Yongming Rao, and Kun Zhang Plot: Prompt learning with optimal transport for vision-language models. In The Eleventh International Conference on Learning Representations, 2022.
[9] Eulrang Cho, Jooyeon Kim, and Hyunwoo J Kim. Distribution-wware prompt tuning for vision- language models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22004-2201 2023.
[10] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10.1 2009..266450555
[12] Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor Guilherme Turrisi da Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Variational prompt tuning improves generalization of rision-languag models. arXiv preprint arXiv:2210.02.390. 2022.
[13] Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for sew-shot image classification. In International Conference on Learning Representations, 2019.
[14] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and puttern recognition workshop. pages 178-178. IEEF, 2004.
[15] Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmen- tation with diffusions for effective test-time prompt tuning. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 2704-2714, 2023.

---

[16] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter. Better rision-languaae models with feature adapters. International Journal of Computer Vision, pages 1-15, 2023.
[17] Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. Calip: Zero-shot enhancement of clip with parameter-free attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 746-754, 2023.
[18) Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226 2019.
[19] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli. Mike Guo, et al. The many faces of robustness: critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8340-8349, 2021.
[20] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardr, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15262-15271, 2021.
[21] Mingyi Hong, Xiangfeng Wang, Meisam Razaviyayn, and Zhi-Quan Luo. Iteration complexity analysis of block coordinate descent methods. Mathematical Programming, 163:85- 2017.
[22] 7. hong, D Friedman, and D) Chen. Factual probing [mask]: Learning vs. learning to recall. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
[23] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference Machine Learning, pages
PMLR, 2019.
[24] Yuqing Hu, Vincent Gripon, and Stéphane Pateux. Leveraging the feature distribution in transfer-based few-shot learning. In International Conference on Artificial Neural Networks, pages 487-499, 2021.
[25] Tony Huang, Jack Chu, and Fangyun Wei, Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.00 036499,,,99. 2022.
[26] Z Jiang, F Xu, J Araki, and G Neubig. How can we know what language models know. In Association for Computational Linguisties (ACL), 2020.
[27] Thorsten Joachims. Transductive inference for text classification using support vector machines. In Proceedings of the Sixteenth International Conference on Machine Learning, pages 200-209,
1 1999.
[28] Michael Kearns, Yishay Mansour, and Andrew Y Ng. An information-theorele analysis of hard and soft assignment methods for clustering. Learning in graphical models, pages 495-520, 1998.
[29] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[30] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15100- 2023.
[31] Philipp Krähenbühl and Vladlen Koltun. Parameter learning and convergent inference for dense random fields. In International conference on machine learning, pages 513-521. PMLR, 2013.
[32] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013,
12

---

[33] Kenneth Lange, David R Hunter, and Ilsoon Yang. Optimization transfer using surrogate objective functions. Journal of computational and graphical statistics, 9(1):1-20, 2000.
[34] Michalis Lazarou, Tania Stathaki, and Yannis Avrithis. Iterative label cleaning for transduc- tive and semi-supervised few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3751-8760, 2021.
[35] Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-effiiient prompt tuning. arXiv preprint arXiv:2 00.00661,,1... 2021.
[36] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning, pages 6028-6028 2228.603..111 PMLR, 2020.
[37] Jinlu Liu, Liang Song, and Yongqiang Qin. Prototype rectification for few-shot learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 1 16, pages 741-756 Springer, 2020.
[38] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
[39] Y Liu, J Lee, M Park, S Kim, E Yang, SJ Hwang, and Y Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
[40] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[41] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision- languale models. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
[42] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.515/ 2013.
[43] Shu Manli, Nie Weili, Huang De-An, Yu Zhiding, Goldstein Tom, Anandkumar Anima, and Xiao Chaowei. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurlPS, 2022.
[44] Ségolène Martin, Malik Boudiaf, Emilie Chouzenoux, fean-Christophe Pesquet, and Ismail Ayed. Towards practical few-shot query sets: Transductive minimum description length infer- ence. Advances in Neural Information Processing Systems, 35:34677-34666 2022.
[45] Ségolène Martin, Yunshi Huang, Fereshteh Shakeri, -hrritthtt Pesquet, and Ismail Ben Ayed. Transductive zero-shot and few-shot clip. In CVPR 2024-IEEE Conference on Computer Vision and Pattern Recognition, 2024.
[46] Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' Amour, Balaji Lakshminarayanan, and Jasper Snock. Evaluating prediction- time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
[47] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 729. IEEE, 2008
[48] Yassine Ouali, Adrian Bulat, Brais Matinez, and Georgios Tzimiropoulos. Black box few- shot adaptation for vision-language models. In Proceedings of the JEFE/CVF International Conference Computer Vision, pages 15534-15544 2023.
[49] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pages 3498-3505 IEEE, 2012.
13

---

[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[51] Meisam Razaviyayn, Mingyi Hong, and Zhi-Quan Luo. unified convergence analysis of block successive minimization methods for nonsmooth optimization. SIAM Journal on Optimizaation, 23(2):1 126-1153, 2013. doi: 10.1137/12083 20899 1009. httpss:tt::/// http:///oi..o////////// 1137/120891099
[52] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference machine learning, pages 5389-5400. PMLR, 2019.
[53] Shin, Logan R. L. IV Razeghi, Y, E Wallace, and S Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods Natural Language Processing (EMNLP), 2020.
[54] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint 12 2012.
[55] Quan Sun, Jinsheng Wang. Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. Eva-clip-188 188: Scaling clip to 18 billion parameters, 2024.
[56] Meng Tang, Dmitrii Marin, Ismail Ben Ayed, and Yuri Boykov. Kernel cuts: Kernel and spectral clustering meet regularization. International Journal of Computer Vision, 127:477-511, 2019.
[57] Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip HS Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No "zero-shot" without exponential data: Pretraining con- cept frequency determines multimodal model performance. arXiv preprint arXiv:2404.0m 04125, 2024.
[58] V.N. Vapnik. An overview of statistical learning theory. IEEE Transactions on Neural Networks, (0(5):9889999 1999. doi: 10. 10.1109772.788640.
[59] Olivier Veilleux, Malik Boudiaf, Pablo Piantanida, and Ismail Ben Ayed. Realistic evaluation of transductive few-shot learning. Advances in Neural Information Processing Systems, 34: 9302, 2021.
[60] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021. URL htttps:// http:///ppppeeeeeeee.eee/rrrrrra
[61] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[62] Zhengbo Wang, Jian Liang, Lijun Sheng, Ran He, Zilei Wang, and Tieniu Tan. A hard-to-beat baseline for training-free free clip-based adaptation. In The Twelfth International Conference on Learning Representations, 2023.
[63] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes. Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7971, 2022.
[64] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485 3492. IEEE, 2010.
[65] Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge- guided context optimization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6757-6767, 2023.
14

---

[66] Tao Yu. Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning vision- language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10899-10909, 2023.
[67] Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 1593-1603 2024.
[68] Maxime Zanella and Ismail Ben Ayed. On the test-time zero-shot generalization of vision- language models: Do we really need prompt learning? In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 23783-23793, 2024.
[69] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey, arXiv preprint arXiv:2304.00685, 2023.
[70] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free ffren adaption of clip for few-shot classification. In European Conference on Computer Vision, pages 493-510. Springer, 2022.
[71] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in neural information processing systems, 16, 2003.
[72] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learn- ing for vision-language models. In JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[73] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV), 2022.
[74] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang, Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 15659-15669, 2023.
[75] Hao Zhu and Piotr Koniusz. Ease: Unsupervised discriminant subspace learning for transductive few-shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9078-9088, June 2022.
[76] Imtiaz Ziko, Jose Dolz, Eric Granger, and Ismail Ben Ayed. Laplacian regularized few-shot learning. In International conference on machine learning, pages 11600-11670, PMLR, 2020.
15

---

A More details on convergence
As mentioned in the main paper, our derived block-wise optimization procedure Eqs. (5) (6)
and 7 can be viewed non-convex optimization, also referred to as the Block Successive Minimization (BSUM) method (51) We update each block of variables, with the other blocks fixed, by minimizing tight upper bound (majorizing function), thereby guaranteeing the overall objective does not increase at each step. In the steps with respect to μ and Σ. we optimize directly the objective in closed-form which could be also viewed as a particular case of optimizing a tight upper bound. The convergence of the general BSUM procedure well studied in the optimization community [51]. Indeed, under the following assumptions for each block of variables, one can establish convergence results for the application of BSUM non-convex problems [51]
instance of the the general Block Majorize-Minininn paradigm for
A1: The majorizing function is a tight upper bound, i.e., equal to the objective at the current solution.
A2: The first-order behavior of the majorizing function is the same as the original objective locally.
Indeed, when assumptions A1 and A2 are verified for each block, we have the result in Theorem [1
As for our case of alternating Eqs. 5). and 77) it is straightforward to verify that Assumptions A and A2 are satisfied for each block of variables. Furthermore, the majorizing functions convex and thus quasi-convex. Also, the sub-problem solved for each block has a unique solution. In particular, for the z-updates, the majorizing function is the sum of linear and a strongly convex function (the negative entropy). Therefore. it is strongly convex. As for the μ-- and E-updates, the solutions are obtained closed form (hence unique).
Algorithm TransCLIP
Require: A set of image embeddings (f))gggN,, a set of textual class embeddings (t+)ıgkK,, the temperature of the CLIP model.
wa, t f[TF, Vijj
Affinity measure, truncated with top-3 values Initial predictions, yp the softmax function D Class centroids initialization Covariance matrix initialization, d is the emb. dim.
2: Y; + ((F, t)) Vi
3: Ms mean s.t = k S
k
4: diag(E) < 1
5: 2; < ýi Vi
Initial assignments
6: while (1), (2) and (3) not converged do
>> Block-wise updates loop
7: while (I) not converged do Sr exp(lo((p).))) ++ ++ en (irds.))
> z-update loop
21, + Ev expllog(p,, w ) ( 9: end while
8:
Vivk
(l) z-step
10: Pa <- J Exeo kf R Ereo s.
4
(2) μ-step
3 Eies 2,,t 15 Eieo
11: diag(E) +- 1 EiEn Ee mull (20--m))22- +w +5 Eieo Ee (f.
b (3) E-step
7+1
end while
return (rg
Prediction with assignment variables
B Further details on TransCLIP implementation
This section aims to provide additional pseudo-algorithm supplement Section as well as more details on TransCLIP hyper-parameters presented Section Our code is available at https:: //github. com/ rithm summarizes the main steps of the TransCLIP algorithm.
and a pseudo-code in Algo-
For the zero-shot setting, we use the embedding of top-8 most confident initial predictions for each class as explained Section 4
16

---

Hardware. All our experiments were conducted on single A100-40 GB. In terms of memory. TransCLIP consumes 16.9 GB when inferring on ImageNet, and can therefore process large datasets on smaller 24 GB GPU.
Hyper-parameters. In practice, TransCLIP performs 10 iterations of z, μ.. E block-wird wise updates. For each z-update, we perform 5 iterations, as we found it sufficient for convergence. In the zero-shot setting, À 1 and y 0.5 and search for the value of y in {0.002, 0.01, 0.02, 0.2). The number of validation shots is set at min #shots), and we build 1-nearest neighbor classifier with the query samples and their final class assignment to predict the class of each validation sample.
(as there
support samples). In the few-shot setting, we
Prompt templates. We employ the prompt templates detailed in Table 24a for all our experiments zero-shot setting unless otherwise explicitly specified. Only when specified, we utilize the custom template ensembling for ImageNet in 700 (see Table 24b).
C Additional results.
We provide detailed results for all the studied vision backbones of CLIP over the 11 datasets to support the transferability of TransCLIP across both convolutional networks and transformer-based models. We additionally report other methods that do not fit into the transductive setting.
C.1 Zero-shot
In Table 9 We report performances of CLIP encoders well as the 8 billion parameter EVA-
CLIP 55. We compare TransCLIP-ZS to unsupervised methods namely TPT 43 MTA [68),
SwapPrompt 4T), and UPL 25 Note that TPT and MTA two test-time augmentation methods working on a single image at a time, thus they differ from our transductive setting, still we report their performance for informational purposes.
As mentioned in Section we slightly modify UPL to apply to the test set in a transductive manner (transductive UPL is denoted UPL Indeed, UPL relies the generation of N 16 hard pseudo-labels per class from training set, after what cross-entropy loss function on soft tokens is minimized. Instead, UPL* generates the pscudo-labels directly from the test set. For fairness, reevaluated the number of pseudo-labels to select and still found that 16 per class yields the best results average, in Table 23
UPL'.
C.2 TransCLIP-ZS on top of few-shot methods
Tables TO and report the performance TransCLIP-ZS on top of CoOp [x] Eip-AdpterrF [70]. PLOT TaskRes [66] and ProGrad [74] for five encoders. The results consistent with the main findings of Section and indicate their generalization for several encoder architectures.
C.3
Cross-Dataset transferability
In Table 15
We report the performance of TransCLIP-ZS on top of CoOp 73 CoCoOp [72).
ProGrad 74). PromptSRC [30] and MaPLE We additionally report PromptAlign [Л. which
working on single image at time and thus differs from our transductive setting. Note that
PromptSRC and MaPLE introduce learnable vision tokens, and are therefore not compatible with convolutional based encoders. The results are similar to those of Section
C.4 Domain Generalization
We extend the results from Table to five encoders. These results support those of Section and show that TransCLIP can improve both zero- and few-shot model generalization for various encoders.
In Tables T6 and 17
17

---

C,5 Transductive few-shot learning
In Tables 18 19 20 and 22 we implemented transductive methods from the traditional few- shot literature that align the most with work in terms of computational efficiency and wide applicability: TIM [5], LaplacianShot [76], BD-CSPN [37], TF [13], and PT-MAP [24]. Additionally, due the lack of transductive methods in Vision-Language and ensure more comprehensive comparisons, we introduce a hybrid method named CoOp+UPL. This method combines prompt learning with both labeled shots and selected pseudo-labels following the methodology of UPL [25]. More details on each method and their validation procedure are outlined below. Methods with tunable pper-pprameters are fine-tuned using the validation split provided with each dataset. line with other work [48], validation is performed for each dataset and for every shot number, setting the number of validation shots min(4, #shots). Hyper-pen prper-parameters are then optimized through grid search to maximize accuracy on the validation set. Note that we only search for y across 4 values for TransCLIP. More details on the grid search for each method given below. Detailed results for the five architectures studied in this paper are available in Table 18
Now we describe the implementation details for each reported transductive few-shot methods.
19 20
21
22
Transductive Fine-Tuning We follow the original implementation of Transductive Fine- Tuning [13]. The authors kept the hyper-parameters fixed for all datasets since the goal was to 1 and the number of training steps 25. How- ever, they pointed out possible improvements if the hhper-paramene Therefore, we search for the optimal temperature value by validation in {0.25, 0.5, 1, 1, 2, 4} and the number of iterations in (10, 15, 20, 25, 30, 35, 40}.
propose simple baseline, with temperature
were tuned for each dataset.
BD-CSPN. We follow the original implementation of BD-CSPN [37]. Regarding the hyper- parameters, this method generates Z pseudo-labels per class from the query set augment the support set and to build the K prototype vectors. They also introduce a temperature scaling parameter for the computation of the prototype vectors. The authors set Z to 8 and the temperature scaling to 10. We search for the value of in {0, 2, 8, 9, 10) and E in f2.5, 5, 10, 20, 403 by validation.
LaplacianShot. We follow the original implementation of LaplacianShot [76]. They balanced the Laplacian regularization term with a factor and used k-nearest neighbors consistency. We follow the proposed ranges to find the hyper-parameter values by validation, with À in {0.1, 0.3, 0.7, 0.8, 1, 1.2, 1.5} and the number of neighbors to consider {3, 5, 10).
PT-MAP. We follow the original implementation of PT-MAP 24 In their work, the authors show a small performance sensitivity to the learning rate a used to update the class prototypes through iterative adaptation. Following their discussion, search a in (0.2, 0.4).
TIM. We follow the original implementation of TIM (5] The authors proposed two solvers to find the solution to the minimization problem: gradient-descent TIM (TIM-GD) and alternating- direction method (TIM-ADM). We decide focus the second approach since there are fewer hyper-parameter to tune. They set the weighting factors of the cross-entropy, the marginal entropy, and the conditional entropy terms to 0.1, 1 and 0.1, respectively. They also introduced temperature parameter 7 in their classifier and set it to 15. We search for the values of the cross-entropy and the conditional entropy factors in {0.05, 0.1, 0.4, 0.7, 1) and the temperature in (5, 10, 15, 30, 60) by validation.
CoOp+UPL. We implement a natural extension of CoOp to include the pseudo-labels proposed by UPL. As in UPL., N = 16 hard pseudo-labels per class are generated according to the prediction's 's confidence. Pseudo-labet from the query set C Q and labeled shots from S are unified into single learning set S P. To separate the contribution of the pseudo-labels from the labeled shots,
18

---

we split the cross- eross-yntropy loss function into two terms:
Lsup (x = |||
Loour (V/xg)
(8)
jES
+(1 - 3)
Cur (Vxx)), B [0,
(P]
JEP
Where V denotes the vector of learnable context token embeddings. Despite increased computational needs, search for the value of [ in {0.1,0.3 0.5, 0.7, 0.9} by validation for the sake of fairness. The number of epochs, the learning rate and its schedule, the optimizer and the context tokens initialization follow exactly the CoOp implementation.
D Limitations
As discussed in Section the gain of TransCLIP-ZS on top of few-shot methods tends to decrease when the number of shots high (e.g., 16 shots) and future works may investigate this aspect.
Secondly, as TransCLIP's performance relies greatly on its text-regularization term, TransCLIP subject some biases. One notable bias pertains to the quality of text embeddings within each class. Recent literature has highlighted that these embeddings exhibit a preference for more frequently occurring concepts [57]. However, this issue may be mitigated through proposed few-shot extension (e.g., introducing labels for more challenging classes).
80
Top-1 accuracy
70-
Zero-shot
1-shot
4-shot
16-4hot
X
TranSLIIP
50
0.0
0.1
0.2
0.4
0.5
086
0.7
0.5
0.9
1.0
1.1
12
13
14
15
16
17
18
19
2.0
Lambda
Figure 2: Sensitivity analysis A. Lower values reduce penalty towards zero-shot prediction and are more appropriate for higher number of shots. Top-1 accuracy averaged over datasets is reported.
Table 8: Runtime and performance comparison between TransCL.PP and zero-shot prompt learning.
UPL* is transductive adaptation the original unsupervised procedure (25. "Prediction" refers to similarity measurement for CLIP and UPL*, and to the iterative procedure for TransCLIP-ZS.
Daraset
#samples
Training
Images Texts encoding
Prediction
Total
Top-1 accuracy
CLIP UPL'
58.7
~0 see
58.7 sec
666.6
ImageNet
50,000
151 min
58.7 see
~0 see
152 min
69.6
TransCLIP-ZS
58.7
14.4 sec
73.1 see
70.3
49.2 see
see
49.2 see
62.5
SUN397
19,850
UPL'
39 min
49.2
min
67.4
TransCLIPPS
49.2 sec
2.6 see
51.8 see
68.9
CLIP
20.5 sec
~0 sec
20,5 sec
65.6
StanfordCars
8.041
UPL.
min
20.5 sec
~0 see
min
71.1
TranCCIP--S
20.5
21.2
69.4
4.8 sec
4.8
Flowers
2.463
UPL
min
4.8 sec
~0 sec
min
73.5
TransCLIP-ZS
4.8 sec
0.2 sec
5.0 sec
76.7
19

---

Table 9: Adaptation of CLIP on 11 classification datasets with zero-shot methods.
Methol
ImageNaa
SUN399
Alreail
Banol AT
Stesfhed
Food
Floweri122
Calechi00
DTD
UCPI
Average
CLIP ResNet-50
38.0
58.8
17.0
36.2
55.7
77.4
85.8
66.0
85.7
61.9
58.7
TTnnCLIP 25
60.8.2.8
16.0.0.4
59.8.13.4
37...3.
39.3.3.6
12.2.6.3
88.6,3.0
T/ photo
17.6
58.5
84.5
62.7
57.9
63.1
16.7
92.1
63.1
78.0
89.1
3
85.8
62.9
Stophhmmpt
63.9
18.0
46.6
59.6
75.1
89.1
70.32
89.0
65.7
62.5
CLIP-RoNGt-101
60.6
59.0
17.
3277
63.2
80.7
87.0
64.4
80.9
37.2
61.1
90.4
TI-2CLIP-2
64.8.4.2
651.660
192,1.3
59.3
686.54
$1.9.1.2
19.8.27
72.6..2
93.0,3.1
429,6.7
68%
660,66
UPL
63.7
633
80.9
67.3
CLIP-VIT-B22
61.9
62.1
19.3
452
60.2
30.4
87.4
66.5
91.5
42.7
63.6
61.9
TRISCLIP 25
64.9.3.0
67.6.55
20.3
2..3
19.0
6333
$1.5 $1.5.1.1
59.0.17
74.4..9
91.8,0.3
504,
68.7.5.
66.3,46
UPL.
19.1
30.3
64.
89.3
67.8
65.3
CLIP-VIT--16
66.6
62.5
24.7
48.3
65.6
85.9
89.1
70.7
98.2
43.5
67.5
65.3
70.3
68.96..3
26.9.2.2
65:1
094.3.8
6914
67.1
92.6.335
76.7.5.9
92.7
49.5
74.4,
70.3 70.3,5.1
IFT
69.0
65.3
24.8
42.4
66.9
84.7
87.8
69.0
68.0
65.3
MTA o/ photo
65.0
25.
38.7
68.1
85.0
86.2
68.3
68.1
65.1
UPL"
67.4
24.7
69.5
21.1
85.8
92.4
73.5
73.7
69.8
CLI-NIT-L/14
72.9
62.7
32.6
6033
76.9
90.9
93.5
79.5
95.2
24.9
72.5
ThanCCIPPP
77.2
735_5.9
759_155
290_2.1
91.91.
85.3 85.3.5.8
07-4
6000
81.7.67
774,49
UPL.
76.6
72.1
35.1
61.7
826
90.9
95.2
83.7
94.9
57.2
80.1
75.5
EVA-CLIP-SH
570
94.8
96.3
86.8
98.0
63.6
84.4
TIaNCCIPPZS
$4
303.3,7
194.1.8
19.4
350.0.2
93.9.04
96.3.0.0
91.8.5.0
98.3.0.3
686,88
93.699.
858,43
Table 10: TransCLIP atop inductive vision-language zero-shot and popular few-shot methods for ResNet-50 vision encoder.
Image
SUNY9T
Almmat
FaotAr
SuaftenrCars
Plover1122
Catecch
D7T0
UCFI0
Method
CLIP.ReeNeesS0
58.0
58.8
17.0
.6.2
55.7
71.4
95.8
66.0
85.7
42.9
61.9
%8.7
TLasCLIPSS
60.8
64:2.54
16.6 0.4
39.6,33.4
57.9.
78.0.0.6
19.3,3.6
7222,6.2
88.6
478,50
68%
64.0
CaOp
57.4
60.0
8.5
49.4
55.8
74.2
85.9
69.0
87.3
45.1
62.9
99.5
TrSSUUPP
60.2.g.
65.3_8.3
9.3-.
1-.1
58.8_3.0
71.0.2.1
$6.9.11
9116,124
88.7.1.4
53.2,82
69.
64.3
61.
62.1
186
66.2
59.
77.1
36.3
76.1
56.3
47.6
64.7
61.0
TrasCUPP 25
62.3.
66.5
192.0
6644,163
66
60.3.1.1
11..007
892,29
89
$9.1,11.
88.9
88.900.6
$28.5.2
71.0
07.614.6
614
62.0
20.9
59.4
34.
88.5
49.6
64.5
63.7
TUSSCLPPS
62.4..10
66.444
204.g
6944.9.6
60.2...9
13.1.333
67.1.2.7
84.5.9.0
56887.2
69.3.48
67.4.37
Paniral
60.9
56.1
64.1
695...7
16.0.5.1
184
694.144
00.5..
71...13
879.00
833, 11.6
88..00
518.84
713.9,7
67.0
CoUp
29.3
63.
203
71.3
62.9
73.8
37.0
85.7
393
54.0
67.6
66.5
6..7.19
18.446
213.13
24.4..1
6.00
16.9
88%.19
91.9
00.5
390.55
758
201
TIP Maplerr
62.6
65.6
25.4
70.5
63.4
77.9
367
81.5
91.1
55.4
70.9
64.5
61.0.0,4
W8.5
346
209.0.
69.2.
76
1..3
800
924
90.3
59.
767
206
TackRes
62.
66.7
25.1
66.3
36.
90.6
57.4
67.9
68.0
TrasUUPPS
M...35
217..1
61.5 1.8
88,
609,35
14...6
69.9..9
ProGrad
62.5
25.1
66.1
90.3
59.8
76.2
70.9
67.5.12
M33.33
14.
1.
806,12
917.774
11..1.
598,53
762.7,4
209.29
04Op
69.4
91.8
74.4
73.2
63.5.9.
199.
m..53.1
77.5.9
885,20
94...63
914.
657.24
794.50
74.0...8
TIP Anglee-
26.2
91.2
31.
K1.3
74.1
18.9
39.0
91.7
92.5
66.0
44.9
TrUSLLPPS
64.5
71.0 91
33.3
805
12.9
784
894.
94.1
1.4
91.0 1.6
65.5 9.
799
74.6
Toolles
644
703
291
75.5
69.5
1866
$9.3
94.
50.5
64.7
7944
73.3
TranCL
644.2
708.04
291.39
75.54.2
60.8..0
78.6.0.1
803..8
947..6.
90.
647.24
794..36
73.3.
923
75.4
73.8
TruSCLIPSS
63.5.
11.2.13
2900 28
29
3.7
784.14
892.10
943,00
1.
657,20
799
73....1
20

---

Table 11: TransCLIP atop inductive vision-language zero-shot and popular few-shot methods for ResNet- 101 vision encoder.
Method
Inagetee
SUN3397
Alcomil
BareAAT
StantaedC.nn Can
Foodt01
Pet
Calechim
DTP
UCPI0
Average
CLI--ReeNet-101
60.6
59.0
17.9
52.7
80.7
64.4
89.9
61.1
59.4
643,43
65.
1922,1.3
08/6,5.4
89.8.2.7
72.
930,
4274
089,7.
66.6
CoUp
51.0
65.7
ThaiCLIP-ZS
643.3.
663.49
163.166
582.7.2
702.6.7
798,29
89.1.2.6
10.611.1
92.5.28
49.95.5
727.69
67.3.5.0
61.6
77.5
91.7
46.3
65.
64.0
TrasSCLIP
6644.28
65.
2..0..8
10.65.7
$12,1.
903..1
59.4.0..9
923-111
51.7.54
73.3.33
699.5.9
TakRe
63.6
62.6
22.3
53.9
66.4
76.2
66.3
74.8
91.2
49.2
67.3
65.0
TraiCLIP--S
66.6,3.
67.9.83
23.3.0.8
64.0.11.1
70.3.3.
60.7,2.4
89.7.3.3
$54.10.6
92.1.088
53,5,4,3
74.,66
69.8,4.8
CoUp
630
65.9
26.5
67.4
70.3
77.8
87.4
85.5
92.3
55.5
72.3
09.5
- TTnnCUUPPS
66.0.30
27.7..0
71
235
900
91
914
938.1.
59
99.7441
17.3
724
Adaper F
15.0
64.6
70.9
81.2
90.
93.0
59.3
74.1
71.1
674.34
698.44
208,2
7400331
$32,1.0
63.5,45
995-00
79248.1
758,277
21.
TraSCLIP-ZS
67.7.2,
713,35
254.1.1
66,
7.9.2.5
$1,8,1,4
22,8
,69,
903,009
616.,5
78557755
728,35
CoOp
05.1
03.5
66.1
78.1
75.9
68.5-2.
730,20
34.9.0.1
83.0..
79.3.08
$13.233
812.a.1
77...14
TIP Mapeer F
68.3
72.8
36.1
82.0
80.5
81.
21122
959
77.0
TTansCLPPP 2s
092...9
755,077
26.00
8000..0
824
03
777.0.7
TakRes
67.6
72.1
15.5
74.9
30.6
31.9
89.5
945
92.6
64.
793
76.3
68.
44.8
75. 1.0
80x.02
9009
89
833,33
770.0.
Table 12: TransCLIP atop inductive vision-language zero-shot and popular few-shot methods for ViT-B/32 vision encoder.
Imagotaa
SUND9
Atronif
EamoAAT
Sunfonikian
Predtmi
P0zee
Callesri11
DTD
UKPPm
Anany
Method
CLIPAIT-RO2
61.9
62.1
19.1
45.2
60.2
K7.4
91.5
47.7
63.6
51.9
649.30
67.8.5.5
20.31.3
99,0,338
63,,33
81.5,11
89.0.,.7
7444.79
913,,03
504,7,7
687.551
66.5.46
CaOp
60.8
69.3
19.6
$1.9
59.5
75.7
87.7
71.5
91.8
43.1
66.0
67.8
639.31
68.3.5.
177-22
649.130
634, 634,33
188.30
8922.15
84
843.128
02:2404
$3.
71.9
68.0
IIP-AdagterFF
64.
65.4
22.3
30.7
61.
80.4
$7.5
02.4
50.9
66.5
66.5
TrasCLIP-2S
663.11
69.
23.3.1.1
71.8.1211
648.377
814.10
885110
89.6.8.
9233.03
559
72.5.g
7077.11
65.3
23.8
62.4
91.2
S1.7
667.211
68.8..5
23.9.0.1
134.11.
64
807.1.
88.2.35
866..899
57.0,43
12..g..
205
Prxirdd
61.0
84.8
21.1
515
605
752
81.9
91.5
51.1
6..
64.7
TrusCLIP-S
649.22
692.444
23.2.111
63,3,98
63,633
802,20
39.6,1.7
8644,120
92.1,06
558,47
72.0.54
68.0
CoOp
63.2
67.1
24.1
67.8
65.4
75.5
85.8
$7.6
92.9
35.1
74.9
69.4
* 2u
65772.6
70773.7
25.3.1.
37
374..0
694.30
788,33
90.5.17
92.0
934405
59.344
79.3.44
72.9.35
63.8
68.3
28.
71.5
67.6
86.9
85.6
18.5
94.6
35.0
75.1
71.6
675.1.7
22.0
28.
168,33
68.
$17.08
902.
91.5
33.
958
621
621.4.1
783.33
738.
TaidRes
66.1
70.1
25.3
65.8
69.5
80.4
87.3
93.9
57.9
71.7
70.2
TrISCLIP
67..1.
12.7.2.6
37.1,83
2033.08
$166.11
90.0.2.
88.3.6.
942,0.
619.40
76.
73.2.3.0
PaKiral
65.2
60.6
66.5
93.4
36.1
60.9
67.1,1.
127.3.0
25.6.0.
14.0,110
69.5.29
8.....7
9111.17
92.7.8.0
92.....
612,g81
780
73.2.3,4
CaDy
66.8
72.3
32.8
82.4
76.1
76.6
88.8
95.5
94.9
64.9
78.5
75.6
TIP Adupter F
* JrnnCLI
TrasCLPP 25
69.0...5
6K4,16
65.4
14....77
142.1.9
74.1
33.0.
34.8
43.4
M1.10
71.0
507.22
220
81.
902
100
95.5
94.
95 95440
95.1
6733,2
65.0
82.5220
8122.27
80.5
77.6.8.
71.0..4
77.1
TaddRc
6922..1
63.2
14.01.0
73.5
37.0
050
100
631
803
71..0.5
16.7
73.2
33.2
900
762
854
950
60.3
800
73.9
TranSCLIP
6K4.1.
148.1.
33..000
823,23
316,14
904.10
9.3.04
673,25
827.28
71..1.3
21

---

Table 13: TransCLIP atop inductive vision-language zero-shot and popular few-shot methods for ViT-B/16 vision encoder.
Mathel
Imagakie
SUN
Altrnatt
FumdAr
SanfonsCan
Feedfii
Flever
Calech101
DTD
LKP100
CLIP-IIT-B16
66.6
82.5
24.7
48.3
65.6
85.9
89.1
70.7
93.2
43.5
67.5
65.3
TranCUIPP2S
70.3,377
66.3.6.3
26.222.
1661,
69 69,8
87.1,1.2
92.
927..05
49.5660
74/8.69
N0%
CoUp
65.7
M6.9
20.7
26.4
67.5
84.5
90.2
78.2
92.5
2011
71.3
67.6
* TrISCLIPSS
69.33,6
11.5.4.
23.5.3.1
653,,8
71.9
863,20
91.9
89.8
93,,1.3
554454
17.7.6.5
72.4
TIP IIR-Acgpte-F
66.5
67.2
36.8
678
67.1
85.6
90.6
94.0
51.6
73.4
70.9
14hot
+ TransCLIN2S
120,15
71.8..6
769,9.1
110,30
869.1.
93 931.24
92.8.9.1
93.5.0
57746.1
52.6
800.67
75.
71.3
158,8,9
3,33
0.8
188.60
2.
911 08
9122-114
94.0 0.0
567,39
25.0.37
TaicRes
09.6
68.1
31.2
65.0
69.1
84.5
40.2
$1.6
95.6
53.4
71.8
70.5
TranCLI
72.0.8
72.5.4.
737.8.1
11
907....1
942.04
594.60
764.46
745.3.7
Pnnind
61.0
67.0
28.7
67.
66.2
84.9
91.4
60.8
95.5
51.8
71.5
06.3
+ TranCTi1s7
701331
21/6,4.6
1191139
86.5
86.5,1.
22.,1.44
91.5
91.5-1077
96.1
$79,81
79.3,6.
24.5,48
Calp
68.8
60.7
10.8
69.6
74.4
84.5
92.5
92.2
94.5
904
71.5
44.0
TraSCLIP 25
113
13,3.3.
33.1
774.75
11.7
865,19
936.3.1
95.3
95.1
818.
TIP-käglar-FF
70.7
70.8
35.7
74.
94.8
30.8
75.1
75.6
4-bott
TrasCLIP-S
127.1.9
74.4.3.5
36...0.
197.29
75..1.8
874.00
93.2.1.
33.3
640,4.
83.3.52
77.9.2.3
PLOT
7000
21.8
34.8
84.7
76.6
83.5
01.8
94.9
61.0
79.7
26.6
TranC
772,7,2
818.9
758.8
856,22
92.5
95....6
948.0.
63,,26
833.36
78.0...4
Tasckes
71.0
22.8
15.3
25.8
26.1
86.1
91.9
85.0
94.9
29.7
75.5
74.5
+ TrasCLIP-ZS
750
15.3.
34.41.1
78.1,4
272
87.3.1.2
93.0,11
92.4.7.4
95
64.3
39.2.37
77.2.2.7
Panirad
702
21.7
14.0
69.5
75.0
85.4
92.0
91.1
94.4
86.8
71.9
74.6
TranCLIP22
723,2.1
75.0.33
35.5.1.
149,53
77%,2.
$7.,115
937,1.7
95.3.4.
95
648,81
8322.44
77.7.3.1
71.9
74.9
43.3
82.8
842
05.8
00.
63.1
79.9
TO zs
153.14
76.0.18
42..
850.02
86.32211
9522,,2
90.5.08
.0.
85.9.2.
8L1.1.1
16thot
TIP-Alagtr.FF
TrusCLIP-ZS
142.89
73.3
768...8
76.0
449.0.3
852.00
85.0
a27.044
82.
$74.06
96.8
93.5.09
969.0.7
7...
95.7
022.15
7088
85...17
K3.9
81.1.0.4
50.7
PLOT
72.3
26.0
463
92.1
84.5
85.6
92.5
02.1
96.0
71.1
84.8
61.7
+ TTaCLLIZs
178.8.3
75.0...
41.5.49
84..7
79.6.9
s59,002
92.2...
95.0.0
657.24
857...9
B0.3
TakRes
13.0
76.0
44.8
80.7
813
86.9
92.5
97.3
95.0
70.9
63.4
50.5
TTSCLPP 25
741.10
76.
43.8..2
803.033
x28.07
$7.5.066
9.9.0.4
90000
862.2.
808.03
Parkial
72.1
25.1
428
83.6
82.9
059
66.9
61.6
70.9
TTaSCLIPZ
735.14
768.1.7
42.5..0
8377,02
872.113
937.08
974.08
96940L
714,25
161.344
8L..1.1
Table 14: TransCLIP atop inductive vision-language zhoo-shot and popular few-shot methods for ViT-L/14 vision encoder.
Mechod
ImageNia
SUN197
Alroan
EmeuSAY
Foodiiii
Pas
Flowrrt2
Calechi11
DTD
UC2101
CLI--VI-LA14
12.9
87.7
32.6
50.3
76.9
90.9
93.5
79.5
95.2
53.5
74.9
72.5
+ TranSCT
77,2,4,
73.5.5,9
35.
35.5,2,7
75.9.15.6
79.00..1
91.91.0
94
94/721.2
85.1,58
974,2,3
60.
81/7,6.7
7.4...9
CoOp
21.5
68.9
16.9
68.4
08.8
89.0
94.0
87.2
95.0
586
78.7
95.2
TIICCLIP
75.9.4.
74.3.5,4
38.0.1.0
80.4
81.5.2.8
91.0,21
9533,14
950,73
96.3
,33
64.1.55
53.5
79.6
Aaapor
39.5
2..
15.5.45
2.4
15.5.77
1.3
919..09
94.1 ,099
97.4.65
116
649.56
$3.8,59
80.0
Tookfees
76.3
31.4
19.6
89.8
91.5
95.0
27.3
56.6
75.9.4.5
41.2,16
83.0
81
915,16
949.1.
94.7.22
062-12
687.56
815.61
00.5..9
PmiGrnd
15.6
71.1
38.4
71.4
50.0
90.5
94.4
39.0
95.7
5838
50.2
16.6
+ 2I
769,3,3
15.8.47
41.8
78777.3
81.1.11
91.7,1,2
95.61,2
975,855
9644,07
668,7,
840..38
80.4.38
CoOp
749
73.1
4366
76.2
88.6
96.6
95.9
96.7
66.1
830
79.5
17.9.3.0
76.9.3.8
4..0...
84.0.0.7
912,24
958,13
9733.4
97.4
477.0.7
67.7.3
85.9.3
81.
Adaptr
77.0
53.9
THIKLIPP
19.0.2.
71.2.311
47.6.0.
83.0.1.6
829.0.6
91%.07
968,08
98..,3
975.1.
890.46
87.7.37
K3.6.19
TaskRes
77.1
74.9
42.5
77.3
$3.5
90.6
94.4
90.1
96.6
65.1
$0.0
79.3
+ TranCLIP zs
794...2
78.5.3.6
4...2.4
81.44.11
83.2.g.
918.1.1
95.41.3
96.5.6.4
97.7.1.1
68.0.29
86.1.6.1
k2.1_2.8
Prnfinal
76.
74.9
44.5
70.3
63.9
90.6
94.8
95.6
96.7
66.1
$3.0
50.6
TTINKCLPP 2s
788,2,3
78.2,3,2
46882,
826.3.3
84.0.a.1
918.,.2
958,1.1
979.23
974.04
10.3.4
87/7.3
82.8.2.3
782
77.5
554
67.2
97.2
74.1
67.2
64.5
2I
1955.13
19..2.3
905-34
88.0g.1
915.1.7
95 15.
994..04
96.1.
183.1.3
19.0..
83.6.1.1
16-8ort
Adaptre
79.3
79.6
55.5
91.6
98.3
74.0
57.4
TrInKLIPP
10...0
8.00..4
560..0
67.4..7
919.0.
957,1.
979.04
139
59.8.14
$5._g.7
TacRRes
78.1
76.7
$5.0
$7.6
01.5
94.6
977
97.2
74.2
56.
$3.9
* TTunCLIPZ
19.8.17
294.2.7
$5.3...
854.1
978406
126.45
589.2.
$4....66
Protinal
784
78.3
S5A
88.5
68.
608
94.8
00.8
67.
75.
47.
$4.
79.6.11
801.18
$42.1.
87.3..4
91%.1.
958,10
9944,0
978,03
751.13
000.34
856-0.8
22

---

Table 15: Cross-Datasel transferability evaluation for five encoders. Few-shot learning methods are trained on 16-shot ImageNet and evaluate on the ten other fine-grained datasets. Average excludes ImageNet.
Source
Tage
Imagohia
SUNU
Ainraft
Sratfee
Pautmii
Plwertt
Caleeiiii
TCPH01
Method
CoOp
63.0
94.2
56.7
Ree 80
63.5.8.5
624.58
142.04
165.16.8
762,2.
$47..05
57A-17
36.3377
80.5.56
S87.49
CoCoOp
88.2
65.5
35.6
39.0
58.2
6.2.1.
0.
5.3-13
147 3.5
983
453337
161,19
PrxCrad
614
85.0
63.56.2
633344
141,88
332,
551233
77:4,13
57:5-16
700.3.2
59.522.
4231,0
525,49
393
CaUp
66.
960
60.4
86.5
08.52.0
637.53
15:2.10
209.5.6
65.2.677
81.041.
$69.09
097.9.4
90.3.19
37.9.37
587.73
064.43
CoLoUp
65.2
629
13.8
2588
62.8
81.2
87.2
64.0
91.3
193
61.1
59.4
TranCLIPP25
734,81
65.6.27
178
452,
67.3.44
7..9..5
67.1
900.44
400,03
674.6.3
6333339
CoUp
0688
00.6
14.
318
56.9
78.8
85.0
58.9
90.3
35.9
61.5
575
684.166
657_44
149.63
604
9.4...5
$6.5.09
929.26
404.48
678.88
62.6661
CoCoOp
66.0
64.6
13.5
40.
99.6
30.5
88.2
65.4
92.1
427
64.9
61.7
674
123.00
134
85....5
759
924.6,3
47.8,51
11.66
650:344
ProGrad
66.9
61.9
13.5
35.4
56.3
79.6
96.3
66.3
91.4
38.0
62.5
55.4
+ -TansCIPPS
684.8
665.45
142.60
51.7.844
598.33
80....3
$6.0.n
70.9
70.9.00.1
92.3.1.
42.5.45
678.5.
6314.5.
MiPLE
65.7
65.0
15.1
41.0
60.6
80.8
59.4
65.5
91.5
42.3
63.5
61.
- -TRNSCTIPS
71 114
677
183.04
604
18.7
85.4
46.
68
MaPLE m/ hompiAlgn
188
19.7
63.5
$2.1
88.4
66.1
92.1
42.5
65.5
62.5
Colp
51.9
60.0
1877
46.6
60.1
$4.5
18.1
60.1
91.7
19.5
64.1
620
TraSCLPPP
13.
674.54
17.1.14
54.5,99
658.48
96.3.2.0
99.4.11
242,1,2
93.4.67
42.1.26
66.1,4.1
CoLoUp
71.1
67.0
22.7
44.6
649
16.2
10377
71.6
95.9
45.2
68.8
55.6
+ TTanCLIP-SS
163,57
69
22
592.144
67.0.2.1
65.
89.8.g
1900.7.
94.3...1
508.5.
14.5.577
0.3.366
PaxGral
711
63.9
21.6
35.0
64.0
$5.9
90.2
67.5
92.9
43.2
65.9
63.4
+ T.TnsCTT 1897
66
22.
55.
557.16.4
6T.
91.%.11
159
940,
46.8,34
75% 176
PrepiSBC
31.4
67.3
24.1
45.0
63.6
wi.1
26.5
93.6
46.2
68.9
65.8
TraKCLPPZ zs
159
699
249.00
594
5944
67.0.20
15.3.12
8..4.
167.622
942.0.4
51.1.5
160.7a
05.4.37
McPLE
N65
24.4
15.8
66.7
w3.4
71.0
43.7
46.3
66.1
766461
243..61
19.5.3.3
658,12
65....0
89.
75.0.6.
943...6
4944.11
744.88
09.1...
Ma PremptAlion
CoOp
78.2
64.9
21.6
51.4
755
89.3
91.0
68.9
93.6
43.6
68.8
669
N8.14
106.57
127
1900.34
18
w
467.46
18
183
71
Co0a0p
718
70.
31.0
47.
77.9
61.4
94.1
16.3
91.1
503
241
311
738.30
$32,331
253
787.08
900.08
90
97 97....1
45.5
192.5.1
760049
ProGral
76.4
458
583
NA,12
15...55
187228
184.64
338
121
816,444
741.25
324.26
152.196
783.10
90..08
630
974.13
562,24
610.6.1
N63,42
Table 16: Domain Generalization evaluation for five encoders with manual prompting strategies.
Source
Target
Method
ImageNet
Adversarial
ImageNetV2
Readition
Sketch
Average
Average OOD
VIT-1/14 VIT-R/16 ViT-R/32 ResNet-101 ReaNct-50
w/ photo
22.0
33.3
40.7
+ TransCLIP-2 ZS
60.8,2,8
21.5 0.4
5144..1
5288 3.3
35.1,
44.3,002
40.2 0.5
w/ custom templates
60.3
23.8
53.4
60.5
35.5
46.7
43.3
IransCLIP-SS
61.7,1.4
23.4.0.5
52.6.0.8
56.4.4.2
36.6,..1
46.1.0.6
42.2.1.1
a photo of
28.2
38.0
46.2
+ TTansCLIPP7S
64.8,4,2
29.2,1.0
56.2,19
65.1.1.0
42,2,4.3
51.5,2.5
48.2,
w/ custom tempiates
62.5
29.8
56.1
67.7
40.6
$1.4
48.6
TransCLIP-ZS
65,6,3,0
30.6,0.8
57.0.0.9
68.240.5
44.0,,3.4
53.1,1.7
49.9.1.4
w/ photo of
61.9
29.9
54.7
66.8
40.8
50.8
48.1
TIansCLIP-7S
64.9,3.0
30.5.0.6
55.7
67.0.002
43,6,288
52.4.
49.2,
w/ custom tomplates
63.8
32.1
56.3
69.5
42.1
$2.8
50.0
TransCLIP zs
66,2,2,5
32.4,0.3
56.6,0.2
69.2.
44.3,
53.7,
50.6
w/ photo of
66.6
47.9
60.6
73.8
46.0
59.0
$7.1
TransCIIP-2S
70.3,3.7
49.5,1.7
62.3
75,0,1.3
49.7.3.7
61442,
59.1
w/ custom templates
68.8
50.6
62.3
77.8
48.4
61.6
59.8
TransCLIP-ZS
71,5.2.7
52.1,14
6344,.11
78.140.2
51.1
63.241.6
61.2,144
w/ photo of
72.9
68.4
67.2
85.3
$7.4
70.2
69.6
TranCLIP-ZS ZS
77.2,4.3
7144,00
69.1,1.8
87.1,
60.0,2.6
72.9.2.7
71,9,23
w/ castom tempiates
75.9
70.9
70.2
87.8
59.7
72.9
72.2
78.6,2.7
73.6,2.7
70.8.0.5
89.0,
61.922.
74.8,118
73.8
23

---

Table 17: Domain Generalization evaluation for five encoders. Few-shot learning methods trained on 16-shot ImageNet and evaluated on the other variants.
Source
Target
Method
ImageNet
Adversarial
ImageNerV2
Rendition
Sketch
Average
Average OOD
VIT-L/14 ViT-B/16 VIT-1/22 RosNet-101 ReaNet-50
CuOp
22.0
45.5
41.2
+ TransCLIP-ZS
63.5+0.5
21.0
53.6.1.4
52.3
34.8,2.0
45.0 0.5
40.4.0.8
laskRes
64.6
22,9
56.4
60.8
35.9
48.1
44.0
TransCLIP.SS
64,4.0,2
21.7.1.2
54.8.
56.2.4.6
36.9,
1.3
42.4. 1.6
CoOp
66.5
29.5
58.3
63.6
39.0
51.4
47.6
TransCLIP-ZS
68.5,2.0
29.9,0.5
58.6,0.2
64,8,1.2
42.3,3.3
52,8,114
48.9,
TaskRes
67.6
30.0
59.6
68.4
41.8
53.5
49.9
+ TransCLIP-ZS
69.3
30.2,0.2
59.3.0.4
68.8,0.4
44.6,2.9
54.4.1.0
50.7
CoDp
66.8
31.2
58.5
65.2
40.1
52.3
48.7
+ TransCLIP-ZS
6.64.16
31.3+0.1
58.3
65.5
42.7,2.6
53.2+0.9
49.4.0.7
laskRes
68.2
31.3
59.3
69.5
42.5
54.2
50.6
TransCLIP
69.2,
31.3,0.1
59.1.0.2
69.3.
44.9,
54.8,0.6
51.2,0.5
CoOp
49.4
64.1
61.5
58.9
TransCLIP
73.3,
50.8,
64.6,0.4
75.7.007
50.3,
62.9,
60.4,
TaskRes
73.0
50.3
65.6
77.8
49.2
63.2
60.7
TransCLIP-ZS
74.1
51,9116
65.4.0.2
78.400.6
51.6,244
64.3
61.8,1.1
CoOp
78.2
69.4
70.8
85.4
57.5
72.3
70.8
TransCLIP-ZS
79.5
71,9.26
71.
86.941.5
60.0,2,5
73,9,1.6
72.5.
TaskRes
78.1
71.3
71.6
87.9
60.1
73.8
72.7
+ TransCLIP-ZS
79.8
74.243.0
718
88.9+1.1
62.0.1.9
75.4,1.6
74.2
Table 18: Detailed results of transductive methods in the few-shot setting for the 11 datasets with ResNet-50 as visual backbone.
Shots
Method
ImaguNet
SUN397
Almaall
EamSAT AT
StanfockKirr
Fodi01
Pat
Flooro1022
Caltsch001
DTD
LCFF10
Avarge
TF
20.6
31.2
13.1
39.0
21.8
28.3
27.2
53.6
66.1
27.7
38.1
33.3
BD-CSPY
24.7
36.9
13.9
40.3
27.2
34.1
34.1
66.7
74.3
32.8
41.4
38.9
LaplacianShot
23.8
35.5
14.0
42.3
27.0
34.7
37.3
66.6
72.4
32.8
43.2
39.1
PE-MP
42.9
15.7
48.0
44.8
56.5
61.4
46.9
38.6
52.2
42.7
TIM
4000
13.4
CaOp
17.5
73.5
TrISCLIPeS
63.5
2006
70.3
56.2
77.2
83.7
87.4
$1.3
70.7
65.8
TF
29.6
43.1
16.6
$7.2
32.3
41.4
40.1
68.4
77.5
41.4
$1.3
45.4
BDEN
33.2
48.1
13.8
58.6
36.2
47.4
20.0
77.0
80.7
43.2
54.1
49.7
SaplacianShot
33.1
47.8
17.7
60.0
36.1
48.7
50.4
77.5
$1.0
43.3
55.2
50.1
PT-MPP
54.6
19.3
61.4
60.1
68.9
$1.5
50.4
61.9
$2.5
52.1
CaOp UPL
59.8
64.0
19.3
62.9
59.2
74.8
$1.2
80.5
88.1
49.5
68.0
64.3
TraISCLIPPFS
59.5
66.2
20.3
71.5
S9.7
77..2
86.0
87..1
87.88
72.8
67..5
TF
38.5
$3.1
20.4
64.9
42.8
$2.5
49.3
80.7
83.6
48.4
19.3
$4.0
BD CSPN
40.7
54.9
20.2
65.4
43.4
56.6
54.3
83.7
84,0
59.8
55.6
SaplacianShot
TIM
43.3
59.1
22.9
71.0
49.6
61.0
58.8
87.6
79.1
65.88
59.5
CaOp UPL
60.5
65.7
23.3
71.0
63.0
75.8
$3.6
87.3
88.0
69.1
67.5
TrasCLIPPS
59.3
66.3
25.0
73.8
61.4
76.6
$.166
88.4
88.2
57.6
73.3
68.4
TF
45.1
59.7
24.1
66.8
51.2
61.1
61.7
86.4
86.3
55.9
65.1
601.
BD-CSPN
22.9
62.4
LaplacianShot
69.1
87.6
PT-MP
50.6
64.2
23.4
66.7
69.6
76.9
72.9
$4.8
70.6
60.5
TIM
49.9
63.4
25.0
69.5
59.7
70.0
71.8
89.9
82.9
59
70.8
64.7
CuOp UPL
60.9
67.0
26.0
71.7
66.5
75.5
$2.7
91.2
B8.3
59.0
71.4
69.1
TransCLIPPSS
59.9
68.3
28.0
74..5
67.6
76.9
86.6
90.4
8K.7
62.1
76.
70.8
TF
63.2
26.6
71.8
66.1
66.4
90.3
64.2
LaplacianSher
34.6
71.5
66.7
89.5
67.5
PI-MAP
54.1
66.1
25.6
68.1
61.1
70.6
79.0
75.2
57.0
71.0
62.7
TIM
55.5
60.8
3038
81.6
68.0
72.4
75.0
88.9
85.7
63.
74.4
69.3
CoOp UPL.
61.9
60.4
31.6
7800
71.4
76.2
83.5
93.6
89.1
62.8
73.5
71.8
TraSSCIPPPSS
64.4

---

Table 19; Detailed results of transductive methods in the few-shot setting for the 11 datasets with ResNet-101 as visual backbone.
Shots
Methoo
InapoÑat
SUN397
Alconft
EaroSAT AT
SuaffrrCCon
Fealffi
Pars
Flavon122
Caltes1011
DTD
UCP101
Avange
TF
24.9
33.3
16.0
38.5
29.4
34.3
37.1
57.0
71.6
29.7
43.5
57.8
CDES
29.9
40.2
16.8
39.5
35.1
42.6
51.0
70.0
79.8
32.1
51.8
44.4
LaplacianShot
30,0
40.0
17.1
40.6
37.2
43.8
51.8
71.7
79.4
34.9
52.1
45.3
PT-MAP
34.5
462
18.1
49.3
53.0
69.5
65.0
48.1
CaOp
62.7
208
63.6
77.8
72.8
89.6
47.0
64.9
TranSCLIPPPS
64.3
6666
19.6
67.2
70.0
82.9
91.5
80.4
91.2
47.0
70.1
68.3
TF
34,8
16.6
19.6
53.7
41.2
49.1
$1.1
73.8
83.1
42.3
56.3
50.1
BD-CSPN
39.9
51.7
20.3
56,2
46.7
57.7
60.4
80.8
85.5
45.55
59.4
54.7
LaplocianShot
39.9
5188
20.9
59.5
39.0
63.2
81,9
45.5
59.8
55.8
TIM
42.4
35.6
19.9
63.5
50.2
69.2
67.3
85.5
81.5
49.0
62.6
58.8
CaOp UHL
63.0
65.4
23.6
66.4
06.6
77.8
$5.2
81.2
89.4
51.4
70.9
67..
TranSCLPPPSS
64.8
67.2
22.7
68.3
70.7
80.8
$9.1
85.2
91.5
49.8
72.8
699.
TF
44.9
$6.9
23.7
62.8
53.4
61.6
61.1
83.7
87.5
51.5
65.4
59.3
BD-CSPN
58.8
54.4
70.1
SaplacianShot
PT-MPP
51.7
25.5
68.0
71.6
79.9
74.6
56.4
71.0
61.8
TIM
51.2
63.2
25.1
73.6
61.4
75.8
70.8
87.0
87.8
71.7
66.3
CuOp+ UPL
63.9
67.1
25.4
70.8
69.3
79.5
$5..5
67.4
90.3
55
73.2
69.2
TransCLIPPSS
65.1
68.7
26.2
73.7
71.6
81.3
90.1
88.6
91.7
56.4
73.2
71.5
TF
62.9
25.1
63.3
69.D
72.3
89.1
70.2
65.0
23.3
89.5
LaplacianSter
7K88
71.7
77.3
89.6
70.3
65.0
PE-MP
55.3
66.5
28.1
67.0
64.6
73.7
8A.6
76.6
59.4
61
72.2
64.5
TIM
56.6
67.1
28.1
74,3
70.0
77.0
85.3
91.5
88.6
60.5
71.7
70.1
CaOp+ UPI.
64.6
69.0
78.1
77.9
73.5
79.5
85.88
92.1
90.7
61
75.8
72.6
FrasSCLIPS
69.6
27.9
71.2
81.5
90.3
89.0
76.1
72.6
72.6
91.8
72.4
LaplocianShot
56.0
63.5
29.4
71.2
65.8
74.4
78.6
91.7
90.2
58.8
72.3
68.5
16
РТ-MAP
58.6
68.3
30.9
69.5
69.2
75.3
85.3
78.2
61.5
62.9
73.4
65.5
TIM
61.4
70.6
34.6
79.2
75.8
78.8
84.4
91.8
88.9
67.2
76.4
73.6
UPL
34.9
82.1
79.5
94.0
74.9
TrasSCLIPPSS
Table 20: Detailed results of transductive methods in the few-shot setting for the 11 datasets with ViT-B/32 as visual backbone.
Shots
Method
InsgpoNot
SUNNw97
Airoratt
EamSAAT AT
StanfondCors
Frotf01
Pats
Foseni102
Caltshhill
DTTD
UCF101
Averggo
TF
25.1
36,1
14.6
44,4
26.7
34.4
23.3
60.0
74,4
29.0
46.4
38.6
BD CSPN
30.1
42.9
16.2
15.7
13.8
41.2
43.9
73.1
80.2
30.8
$2.6
41.5
SaplacianShox
29.2
41.7
16.1
48.6
33.2
43.1
43.8
73.3
80.6
32.7
52.9
45.0
PT-MAP
33.1
48.8
13.0
54,8
38.6
49.8
62.4
52.5
37.9
57.0
45.7
CaOp
21.0
78.8
TrasCLIPPPS
64.3
63.9
22.7
63.5
03.7
82.2
90.
83.2
92.2
69.3
68.4
TF
34.7
49.5
19.3
56.5
37.4
48.7
47.4
75.1
83.9
-4..5
57.7
$0.4
BD-CSPN
39.2
53.1
20.7
57.2
42.1
55.5
55.2
82.4
86.8
45
61.6
$4.55
SaplacianShot
39.1
53.9
20.1
58.3
42.4
57.7
57.3
62.5
86.7
4..9
62.6
55.2
TIM
41.1
50.0
31.1
68.9
44
66.2
60.1
86.5
81.5
58.7
CoOp UPL
63.4
66.6
22.8
71.9
60.8
78.5
$5.0
$1.0
90.1
53.5
70.2
67.6
TransCLIPPSS
61.8
69.5
22.9
76.9
63.8
81.2
$9.9
85:4
92.1
52.9
71.0
70.0
TF
40.5
59.4
23.2
62.1
48.6
60.8
$7.9
85.2
89.1
$2.6
65.22
59,0
BD-CSPN
61.1
23.4
65.3
64.8
87.2
67.0
61.0
SaplacianShot
6864
87.6
61.6
68.9
70.3
73.5
70..
59.7
TIM
50.4
65.0
21.7
70.0
56.
73.0
74.4
90.5
88.7
71.8
65.5
CuOp UPL
63.9
68.8
26.6
72,6
63.7
78.2
85.2
88.8
90.1
55
73.1
69.7
TIINSCLIPPSS
64.7
70.1
26.4
78.0
66.5
90X3
87..2
88.7
92.2
$8.0
74.3
71.5
TF
50.9
64.7
27.1
67.6
68.5
68.0
$9.4
90.5
58.2
70.7
64.8
27.4
LaplocianShot
70.0
73.7
70.88
PEMMP
53.7
63.3
27.4
70.9
58.5
72.8
75.4
75.2
59.7
71.5
63.0
TIM
56.2
69.0
78.4
75.8
65.
76.1
7906
92.3
87.4
63.3
75.4
69.9
CoOp UPL.
64.8
69.7
30.0
79.6
68.9
79.3
85.5
91.6
91.8
62.
73.9
72.5
TransCLIPPSS
65.5
71.3
28.0
78.2
70.8
$1.0
89.4
90.0
92.3
61.1
77.0
73.2
73.3
Laplacian LaplacianShet
51.8
66.7
28.4
71.2
60.9
73.2
75.3
91.3
91.3
72.9
67.7
16
PT-MAP
$6.9
69.9
29.3
71.3
63.1
74.1
78.7
77.1
60.7
61.9
72.9
65.1
TIM
60.5
71.3
33.0
79.4
72.2
78L1
$5.0
92.8
88.4
66.6
79.1
73.3
CoOp UPL
71.9
34.1
84.3
79.0
85.8
95,2
92.4
78.3
74.8
TroSSCLIPPPS
25

---

Table 21: Detailed results of transductive methods in the few-shot setting for the 11 datasets with ViT-B/16 as visual backbone.
Shots
Method
InapaNet
SUN397
Alconit
EaroSAT AT
StanforCCon
Fealffit
Pars
Plavon112
Caltss1011
DTD
UCP101
Avange
TF
29.7
38.1
19.2
46.0
32.5
43.5
38.2
67.8
75.5
31.6
48.8
42.8
CDEN
35.4
45.7
22.0
45.7
42.0
54.2
52.9
82.9
83.5
34.7
58.0
50.5
LaplacianShot
34,9
44.5
22.1
52.1
53.0
52.2
83.1
83.4
35.8
57.3
50.9
PT-MAP
40.1
52.6
23.8
39.7
48.4
64.4
61.8
69.4
63.5
52.7
65.1
75.1
CaOp
27.3
70.0
81.7
92.7
73.1
70.6
TranSCLIPPPS
69.8
7006
29.9
72.5
70.9
87.9
93.8
84.8
95.1
78.4
75.2
TF
40.5
51.6
25.3
63.1
45.1
58.8
54.8
83.2
67.0
47.3
59.4
56.0
BD-CSPN
46.1
56.1
26.7
6:..7
50.7
67.5
64.6
89.6
89.6
48.9
64.0
60.8
LaplocianShot
55.9
23.1
68.2
68.2
66.0
89.7
65.1
61.4
TIM
87.9
60.7
28.1
75.8
95.7
78.7
70.6
91.4
86.5
64.9
CaOp UHL
69.2
09:2
30.1
73.4
71.0
83.8
$8..4
87.9
93.3
$3.9
75.8
72.4
TranSCLPPES
70.3
70.9
30.0
77.1
71.7
87.0
91.7
90.5
93.5
55.1
78.5
74.2
TF
51.1
61.0
30.3
64.9
56.8
71.0
65.9
90.9
91.5
53.7
67.9
64.1
BD-CSPN
62.5
30.5
64.8
75.3
70.9
65.9
SaplacianShot
PT-MPP
68.1
31.2
74.9
81.1
79.5
76.2
60.2
73.9
65.8
TIM
57.4
67.0
32.8
79.3
65.8
83.5
$2.3
93.4
88.5
76.5
71.3
CuOp+ UPL
69.7
71.1
32.6
74,0
74.6
83.8
91.3
92.1
93.2
58.9
76.9
74.4
TransCLIPPSS
70.3
71.9
34.0
79,4
74.0
86,4
91.6
93.6
94.00
61.1
79.1
75,9
TF
66.8
34.1
68.5
65.4
77.4
74.3
95.8
92.4
73.8
69.5
74.2
LaplacianSter
65.9
33.4
73.2
79.3
79.3
74.6
70.0
PE-MP
61.0
70.6
34.1
75.0
08.5
82.D
84.5
77.2
62.1
75.6
68.5
TIM
62.6
71.3
35.9
79.8
74.4
84.3
$7.4
94.0
90.7
63
80.2
74,9
CaOp UPI.
70.5
72.8
38.6
79.1
78.3
84.5
90.4
94.4
93.3
60.6
79.66
76.6
FrasCCLPPS
70.5
73.2
36.1
79.7
86.7
93,9
94.2
65.7
81.5
77.3
76.0
72.7
LaplocianShot
60.9
63.3
36.1
78.1
09:2
81.2
$1.77
94.8
93.1
$8.6
76.3
72.6
16
РТ-MAP
64.0
72.0
37.4
75.6
72.0
82.7
$6.1
78.5
63.7
63.7
76.3
70.2
TIM
67.8
73.6
40.6
83.6
79.5
84.9
83.7
95.4
92.4
67.5
82.1
77.8
UPL
75.1
43.2
83.0
95.8
80.4
79.1
78.
Table 22: Detailed results of transductive methods in the few-shot setting for the 11 datasets with ViT-L/14 as visual backbone.
Shots
Method
InsgpNNot
SUNVw97
Airoraft
EamsSA7 AT
StanfondCirs
FootfO1
Pats
Foseni102
Caltssholl
DTD
UCF101
Averggo
TF
36.6
41.2
26.3
49.8
45.2
53.9
45.8
81.8
79.7
58.3
50.4
BD CSEN
45.3
50.5
25.9
53.3
57.5
67.3
66.7
93.1
88.4
39.6
61.2
59.8
SaplacianShox
43.5
48.1
30.9
56.6
56.
69.3
65.8
93.3
87.9
40.1
59.8
PT-MAP
58.1
33.1
65.6
60.6
80.1
75.2
58.5
61.3
71.5
CaOp
72.6
72.7
79.2
89.5
86.8
76.6
TrasCLIPPPS
75.9
74.3
37.9
71.4
78.8
92.2
95.4
95.9
95.6
61
83.33
78.9
TF
50.1
56.6
33.5
71.7
58.3
71.6
65.7
93.0
90.5
.9.8
69.4
64.6
BD-CSPN
57.0
61.2
35.6
72.6
65.
79.9
77.2
95.7
92.8
$2.5
74.7
69.5
SaplacianShot
56.5
61.3
35.9
76.8
80.3
77.4
96.2
93.3
52.4
741.8
70.0
78.4
77.9
75.22
TIM
67.6
35.4
82.2
87.4
95.1
91.4
73.2
CaOp UPL
76.1
73.4
39.9
72.3
$1.4
90.3
92.5
94.0
94.7
62.0
8.2.2
78.1
TransCLIPPSS
76.8
75.1
10.0
82.1
79.9
91.8
95.0
95.5
95.9
62.6
83.22
79.9
TF
61.6
66.5
40.6
71.4
69.6
$1.9
79.0
96.4
94.4
58.5
77.5
72.5
BD-CSPN
64.5
67.8
40.6
21.4
84.7
$2.8
96.7
75.8
SaplacianShot
78.9
97.3
74.7
41.7
77.4
73.8
8ALO
89.9
78.5
72.1
TIM
68.9
72.7
42.0
78.4
77.8
90.0
92.3
97.4
91.1
63.5
83.7
78,0
CuOp UPL
76.5
75.1
44.1
79.3
83.
90.1
92.6
95.2
95.3
65
83.9
80.1
TIINSCLIPPES
76.9
76.2
45.9
81.5
81.2
91.4
944.3
98.2
96.1
66.8
944
81.2
TF
67.4
72.0
45.6
76.1
76.5
96.2
$5.1
97.2
95.1
81.5
77.1
CD-C
LaplocianShot
70.4
80.8
76.8
PEMMP
70.7
74.6
44.1
78.4
77.1
89.2
91.3
79.5
64.5
79..7
74.0
TIM
75.1
76.4
46.7
86.8
83.2
89.5
92.7
96.9
94.4
70.2.
81.3
81.0
CoOp UPL.
76.9
75.8
49.6
81.7
85.5
90.1
93.2
95.9
95.3
65.6
$4.0
81.2
TransCLPPPS
77.2
77.3
50.0
82.6
91.6
94..5
98.5
97.0
70.7
86.0
82.7
74.9
Laplacian
78.1
LaplacianShet
69.8
72.7
47.0
81.7
50.2
85.0
90.
95.0
95.7
82.8
79.0
16
PT-MAP
72.9
75.9
48.1
79.1
79.9
89.1
92.0
80.5
66.0
65.6
80.5
75.4
TIM
76.4
78.7
$2.5
89.4
86.5
91.0
92.0
98.2
94.5
73.22
84.8
83.4
CoOp UPL
76.9
77.2
54.1
85.9
90.6
93.2
97.1
95.6
83.4
TIISSCLIPPPS
$3.0
84/4
86.5
26

---

Table 23: UPL* top-1 accuracy ImageNet for 16 and 32 top-confidence pseudo-labels drawn from the test set.
Architecture N
N 16
N 32
ResNet-50
60.60
61.60
59.66
ViT-B/16
68.92
69.62
68.87
Table 24: Prompt templates for each dataset.
(a) Prompt templates used in the experiments unless otherwise specified.
(b) Custom prompt templates for ImageNet dataset [50
Dataset
Prompt template
"itap of []"" "
ImageNet
"s photo of D."
bad photo of the []." "a origami []" photo of the large []." "
SUN397
photo
Airemftt
photo a [], a typa of aircraft.
EuroSAT
centered satellite photo of D.'. photo
D.". aa photo of [], a type of food.".
"a D video game." "art
FoodiDI
[]."
Pets
"a photo [],
of pet.".
"a photo of the small O)""
photo la phota of DD.".
type flover.".
Caltech101
DTD
"[] texture.".
UCF101
photo person doing D...
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claim i) from the abstract is discussed and supported by results Tables T
and B and claim (ii) is discussed and supported by results in 4 The contributions announced
in the introduction: (i) is presented in Section 3] (ii) is supported by Tables []] 2 and [3]
and (iii) is supported by Table 4 More results are also available in the Appendix for five encoders.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations are presented in Appendix D
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
28

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The convergence of our algorithm is proved and assumptions are enumerated in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The hyper-parameter values are stated in Section 4 and in Appendix B A complete pseudo-code is provided in Appendix with further empeettntaioon details.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
29

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code is provided at the following link: https://github.
Maz2aneella/crannsdciiioo-----
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. www.cc/pwwpcppccccccceoeooooeeee htte://wiissseeeeeeeeee www.eeeuiisssoonnooooooooooooe
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: For the datasets used in experiments. we follow the settings of previous works, as stated in Section 4 The implementation transductive methods detailed in Appendix C.5 If not specified, the exact hyper-parameters from the initial implementations are used for other cited methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Every numerical accuracy an average over three random seeds. Our experi- ments cover 15 datasets and 6 encoder architectures.
Guidelines:
The answer NA means that the paper does not include experiments.
30

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The hardware used for the experiments presented in Appendix and
runtime is discussed in Section 4 (Table
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics stttps://neurr: wwweeuurpssccppulic//eeeeeeeeeee cc/ppublccEthicccu.eeel...
Answer: [Yes]
Justification:
Our work aligns with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper is not tied to specific application.
31

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
All models and methods are credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
32

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new asset is released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing nor research with human subject was involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsoureing nor research with human subject was involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
33

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "k1vCdycObE", "reviewer": "Reviewer_NaRY", "summary": "In this paper, the authors explore fine-tuning VLMs to specific unlabelled/partially-labelled datasets in a transduction setting. The authors propose an objective function to carry out joint inference of labels for all the test samples simultaneously. The authors then propose an iterative block Majorie minimize procedure to arrive at a local optimum. Empirical results on multiple datasets show clear superiority of the proposed method over other prompt-based fine-tuning and inductive zero-shot and few-shot methods.", "questions": "See weaknesses above.", "limitations": "Authors have adequately discussed limitations in the paper.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is generally well-written, easy-to-follow and intuitive. Experiments clearly show big improvement over current state-of-the-art. I believe this method could be very useful for downstream applications of VLMs in different domains.", "weaknesses": "Some aspects of the writing should be made more clear.\n\n1. Is Eq 2 did you mean zi^Tz_j since both are vectors? \n\n2. How is the final prediction done, for the ith sample? Is it the argmax of z_i?\n\n3. Design choice: The experiments show great results, but is there apriori any expectation for using a GMM model for the classes? Perhaps this is related to the linear manifold hypothesis for these VLMs and LLMs. Some more discussion on this will be good. As it is written it comes as a random choice which seems to work well with no intuition behind it.\n\n4. Line 167. why is the max function needed for W to be PSD? Even without the max operator, W will be a gram matrix and thus PSD. Am I missing something here?\n\n5. Why does the laplacian regularization enforce visually similar points to have the same assignment? I understand this when z is one-hot, but here z is any vector in the simplex. Is this still true when z belongs to the simplex?\n\n6. One of the baselines used in the paper is a modification of UPL, but this modification is not clearly mentioned in the paper or the appendix. How is the modification done in the transduction setting? Also a small description explaining UPL would be good in the appendix for added context to the readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Eq 2.**\n\n It's an omission on our side; indeed, it should be $\\mathbf{z}_i^\\top \\mathbf{z}_j$. Thanks for pointing that out! We will update the objective function accordingly.\n\n\n**W2: Final prediction.** \n\nYes, we take the argmax for the final prediction. We should have made it clearer at the end of Section 3.3 as follows: `'Note that, after convergence, we use the sample-to-class assignment variables $\\mathbf{z}_i$ as predictions for each sample $i$ of the query set $\\cal Q$, *using the argmax operation for conventional classification*.\". We will also update Algorithm 1 in the Appendix to further clarify this (line 13: return argmax($\\mathbf{z}$)). \n\n\n**W3: Design choice (using a GMM).**\n\nWe agree that further discussions / illustrations on this choice would be helpful to readers. The most basic unsupervised clustering objective would be K-means. This could be viewed as an extension of inductive methods (e.g., CoOp, which also search for new class centroids via prompt tuning) to unlabeled test data. However, K-means makes the assumption that the clusters are 'spherical', which may not be the case for the embedding vectors ensuing from VLMs. Indeed, K-means corresponds to a particular (simplified) case of our formulation, in which the covariance matrix is fixed to the identity matrix, as pointed out in lines 160-161. \n\nThe ablation studies in Table 6a show TransCLIP performance when $\\boldsymbol{\\Sigma}$ is not updated (note that the other regularization terms are present), resulting in a large performance drop. This suggests that the spherical-cluster assumption (i.e., identity covariance matrix), as in K-means, may not be valid for VLM embeddings. Table 6d also shows the impact of having an isotropic $\\boldsymbol{\\Sigma}$, which could be seen as an intermediate solution between K-means (with fixed $\\boldsymbol{\\Sigma}$) and the TransCLIP objective function (with learnable $\\boldsymbol{\\Sigma}$), demonstrating the importance of learning $\\boldsymbol{\\Sigma}$. \n\nTo provide further insights, we added Figures 2a and 2b to this rebuttal (please refer to the attached PDF document), which show that feature variance varies across the dimensions and that our learnable $\\boldsymbol{\\Sigma}$ tends to fit such 'elliptical' cluster distributions. Additionally, the introduction of a learnable $\\boldsymbol{\\Sigma}$ can also be seen as learning the weighting factors of the Mahalanobis distance between the class centroids and the data points (as pointed out by Reviewer aJGT).\n\n\n**W4: Affinity matrix.** \n\nYou are right; our affinity matrix W can be rewritten as $F^\\top F$ and is indeed a Gram matrix, hence PSD. Therefore, the max operator is not necessary to ensure the PSD condition. We will remove this statement. Thanks!\n\n\n**W5: Effect of Laplacian regularization beyond the simplex vertices.** \n\nYes, the effect of Laplacian regularization holds within the simplex, beyond the vertices. Indeed, for a given, non-vertex $\\mathbf{z}_i$ lying within the simplex, the vector $\\mathbf{z}_j$ that maximizes $\\mathbf{z}_i^\\top \\mathbf{z}_j$ is the one-hot (vertex) vector where the entry corresponding to the argmax of $\\mathbf{z}_i$ is 1 (e.g, for $\\mathbf{z}_i = [0.7, 0.2, 0.1]$, $\\mathbf{z}_j = [1, 0, 0]$ maximizes the dot product). This function is convex in $\\mathbf{z}_j$ over the simplex, and any deviation of $\\mathbf{z}_j$ from the maximizing vertex can only cause a decrease in the dot product. \nOf course, this is an intuitive example, but the general behavior of the Laplacian term in our objective function is more complex. Multiple points are linked to each other, with potentially different predictions (and different most confident classes), combined with the KL term penalizing divergence from the zero-shot prediction (potentially deviating from the one-hot solutions).\n\n\n**W6: Modification of UPL.** \n\nAs suggested, we will add a small description of UPL and UPL$^*$ in the Appendix (in C.1 Zero-shot, line 570) for better clarity.\n\nUPL: Unsupervised Prompt Learning [25] where N = 16 hard pseudo-labels per class are generated from a training set. Then a cross-entropy loss function on soft tokens is minimized (see UPL$^*$ below).\n\nUPL$^*$: We implement a natural extension of UPL. N = 16 hard pseudo-labels per class are generated from the query (test) set ${\\cal P}\\subseteq {\\cal Q}$ according to the prediction’s confidence. For fairness, we reevaluated the number of pseudo-labels to select and still found that 16 per class yields the best results on average, as seen in Table 22. The following cross-entropy loss function is minimized: \n$$\n\\mathcal{L}(\\overline{\\text{V}}| \\\\{ \\mathbf{x}_i \\\\} _{i=1} ^{|{\\cal P}|})) = \\frac{1}{|{\\cal P}|} {\\sum} _{j \\in {\\cal P}} \\mathcal{L} _{\\tiny{UPL}} (\\overline{\\text{V}}|\\mathbf{x}_j)\n$$\nwhere $\\overline{\\text{V}}$ denotes the vector of learnable context token embeddings. At inference, the learned tokens are used to generate the text embedding of each class.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DYmfd7TfDG", "reviewer": "Reviewer_aJGT", "summary": "The paper proposes a method named TransCLIP that performs transductive inference to boost classification performance of Zero-Shot &  Pre-trained Few-Shot CLIP models. The proposed methodology is an extension of [1], but for VLMs. TransCLIP proposes to learn the class prototypes, in contrast to fixed-prototypes considered in [1] and adds a language guidance term to penalize predictions that stray from the CLIP pseudo-label. Following [1], the paper uses a Majorize-Minimize framework to solve for the prototypes and class predictions in closed form. Experiments are shown on 11 standard datasets, showing that TransCLIP improves on top of existing few-shot fine-tuning methods.\n\n[1] Imtiaz Ziko, Jose Dolz, Eric Granger, and Ismail Ben Ayed. Laplacian regularized few-shot\nlearning. In International conference on machine learning, pages 11660–11670. PMLR, 2020.  \n[2] Ma, Xiaosong et al. “SwapPrompt: Test-Time Prompt Adaptation for Vision-Language Models.” Neural Information Processing Systems (2023).  \n[3] Shu Manli et al. “Test-time prompt tuning for zero-shot generalization in vision-language models.” Neural Information Processing Systems (2022).", "questions": "1. As discussed above, the GMM angle is unclear, but maybe it boils down to learning class prototypes in the absence of few-shot data.  \n2. Existing test-time methods show strong empirical evidence of the power of transduction for VLMs. A discussion of the scope of these methods is necessary.  \n3. Since the primary novelty of the paper is to suggest that language-guidance can greatly improve transduction, the appropriate ablation is necessary. Setting $\\lambda=0$ in the proposed KLDiv parametrization only removes the cross-entropy term, and is not the desired ablation.", "limitations": "The authors have addressed some limitations of the work. There is no potential negative societal impact from this work", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\\+ The paper proposes to use transduction to improve CLIP’s downstream performance. This is a promising direction which has seen recent efforts [2,3]. The considered formulation is a scalable alternative to existing test-time methods. The solution is well motivated, and the paper is easy to follow.  \n\\+ TransCLIP builds on top of, and modifies [1] for VLMs. TransCLIP makes two major design choices  i) The class prototypes are learnable ii) A language-guidance term is added to regularize updates. Both design choices lead to improved empirical performance on CLIP.  \n\\+ The experiments are comprehensive . The proposed language guidance term and other design choices used in TransCLIP lead to improved accuracy over zero-shot CLIP and few-shot finetuned CLIP.", "weaknesses": "\\- Discussion on Test-Time methods. Recently many test time methods (a type of transductive learning) have been proposed to improve VLM performance. A discussion of the tradeoffs between the proposed approach and existing test-time adaptation (TTA) methods needs to be discussed. For instance, TransCLIP requires the entire test batch, while TTA methods require fewer test samples [2,3].  \n\\- Issues with missed references. Important references are missing in Section 3.3. The entire section follows similar arguments from [1], but re-derived by adding the language-guidance term.  \n\\- The GMM argument is unclear to me. However, the GMM clustering term in eq 2 is minimized when label assignments are given by the closest prototype. The loss is reminiscent of $\\mathcal{N}$ in [1], but the paper proposes to use a Mahalanobis distance instead of euclidean distance. I am unsure of the insight provided by posing it as GMM clustering instead of drawing parallels with [1].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 and Q2: Discussion on Test-Time methods**.\n\nWe agree that the mentioned test-time methods also employ the transduction paradigm. However, their settings are in fact very different from the ones studied in our work (zero- and few-shot adaptation of CLIP, improving inductive few-shot methods). We still report their performance in Table 8 of the Appendix but we agree that a more complete discussion is needed. We will add the following discussions: \n\n- *Design*: SwapPrompt [40] has been designed to make batch predictions on-the-fly, and has continual-learning related mechanisms such as an exponential moving average prompt across batches. The setting of TPT [42] also differs from ours; it has been designed to work on a single sample with many data augmentations to train one prompt per image.\n\n- *Resource*: Both methods require access to model weights for training (i.e., do not operate in a black-box setting). We also note that prompt tuning does not scale well with the model size and is even impractical on very large models such as EVA-CLIP-8B.\n\n- *Computation overhead*: TPT takes nearly 5 hours for ImageNet. As SwapPrompt relies on prompt learning for several epochs and requires data augmentation, its running time is above the one of UPL$^*$ (please see Table 1 in the attached document). In comparison, TransCLIP takes only 73 seconds for the whole pipeline (images and texts encoding + transduction; please refer to Table 1 in the attached document).\n\n- *Batch size*: It is true that TransCLIP first requires a test batch to work on. However, we would like to highlight our answer Q1.b for Reviewer AvCq, which discusses and assesses how online predictions for new (single) samples can be processed after running TransCLIP.\n\n**W2: Issues with missed references. All Section 3.3 follows similar arguments from [1], but re-derived by adding the language term.**\n\nPlease note that we mention [1] (LaplacianShot, reference [73] in the paper) in related work (lines 85-87), and provide direct experimental comparisons to it in Table 4 in the paper, as well as in Tables 17 to 21 in the Appendix. We agree with the reviewer that we could emphasize that the derivation of [73] (i.e., linearization of the Laplacian term) is a sub-step of our approach but without the text term (this could be done in sub-section Majorize-Minimize with respect to the z-block in section 3.3). \n\nWe clarify, however, that [73] does not have a Block Majorize-Minimize (BMM) optimization procedure (i.e., the additional $\\boldsymbol{\\mu}$ (prototypes) and $\\boldsymbol{\\Sigma}$ (covariance) steps in Eqs. (6) and (7) in our section 3.3). The objective function in [73] is quadratic (the term using the fixed prototypes being linear) and has only assignment variables, whereas our objective function is higher-order with three blocks of intertwined variables (prototypes, covariance and assignments). Interestingly, Table 4 shows that, even without the text term, our objective function brings significant improvements over the LaplacianShot one when the number of shots increases (due to learning prototypes $\\boldsymbol{\\mu}$ and covariance $\\boldsymbol{\\Sigma}$ from both unlabeled and labeled data).   \n\nWe would also like note that the general principle of linearizing concave quadratic terms (which provides a majorizing function on the Laplacian term in our case) is quite common in the Majorize-Minimize optimization literature, and has been used in other works in machine learning, even much earlier than LaplacianShot, e.g., the following work in the context of conditional random fields (CRFs): Krahenbuhl and Koltun, Parameter Learning and Convergent Inference for Dense Random Fields, ICML 2013. \nWe will add this reference and connect to the LaplacianShot procedure ([73]) in Section 3.3 (e.g., at line 193).\n\n\n**W3 and Q1: GMM argument.**\n\nThey are two major differences with the linear term $\\mathcal{N}(Y)$ in LaplacianShot [73], in which the class prototypes are fixed (computed from the labeled shots only) and the assignment vector is the only block of variables. First, as mentioned by the reviewer, our formulation enables to learn class prototypes $\\boldsymbol{\\mu}$ from both the unlabeled and labeled data (both in zero- and few-shot). \nThis claim is supported by the results in lines 1 and 4 of Table 6a in the main paper, which show that performance tends to decrease when the class prototypes are fixed.\n\nSecond, by introducing a learnable covariance matrix $\\boldsymbol{\\Sigma}$, we enable TransCLIP to better model the feature variance, which may be unevenly spread across the dimensions of the embedding space. To further support this claim, we add Figures 2a and 2b in the attached PDF document, which show that feature variance varies across dimensions, and that our learnable $\\boldsymbol{\\Sigma}$ tends to fit this particular distribution. You are correct that learning a covariance matrix is equivalent to learning the weights of the Mahalanobis distance between the data points and their class prototypes, and we will add this discussion to Sec. 3.1, in the GMM-based clustering part. Thank you for pointing that out! \nInterestingly, the learned GMM parameters can then be used to make predictions on new incoming samples, as discussed in Q1.b for Reviewer AvCq.\n\n**Q3: KL Div parametrization.**\n\nWe would like to point to Table 6b in the main paper, which studies the impact of $\\lambda$ for the zero-shot setting.\n\nHowever, this Table studies only the zero-shot case. Therefore, we provide here a more thorough study (please refer to Figure 1 in the attached document) for a wider range of settings (zero-, 1-, 4-, and 16-shot), which clearly shows both the importance of this term and its stability around the selected value. Indeed, the average zero-shot performance does not vary significantly between $\\lambda=0.5$ and $\\lambda=1.6$, and $\\lambda=0.5$ is an acceptable value for all few-shot settings. We will add this extension to our current ablation studies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RO4kir8IV2", "reviewer": "Reviewer_AvCq", "summary": "The paper proposes a transductive method to boost the performance of existing vision-language models by assuming that all unlabeled test samples are available during the training stage. Specifically, the paper models a Gaussian mixture model, where each class is represented by a Gaussian distribution. The method refines predictions for each test sample by constraining the similarity between predictions of similar points and the similarity between the final prediction and the prediction based on text features. Experiments are conducted on standard datasets.", "questions": "See weaknesses.", "limitations": "Copy from Q1: The paper does not discuss a significant limitation of the proposed method. Specifically, when the test dataset arrives online, one by one, the method may not be able to make predictions for individual data points immediately. Instead, it may require collecting a sufficient number of data points before training the model and making predictions. Furthermore, if a new data point arrives after the model has been trained, does the model need to be retrained to make a prediction for the new point? Is there an efficient way to handle new data points?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a less explored area of transductive learning on vision-language models, filling a gap in the existing research.\n2. The motivation for the method is clear, the modeling is reasonable, and an appropriate optimization method is provided.\n3. The proposed method is quite general and can be applied in both zero-shot and few-shot scenarios. It is also compatible with other methods.", "weaknesses": "1. The paper does not discuss a significant limitation of the proposed method. Specifically, when the test dataset arrives online, one by one, the method may not be able to make predictions for individual data points immediately. Instead, it may require collecting a sufficient number of data points before training the model and making predictions. Furthermore, if a new data point arrives after the model has been trained, does the model need to be retrained to make a prediction for the new point? Is there an efficient way to handle new data points?\n2. Although the authors provide an analysis of training times in Table 5, the testing speed is more critical for users. An analysis of the time complexity during testing is also needed.\n3. Considering that the method requires using all unlabeled test data for training, it resembles unsupervised learning (e.g., [25]). It is recommended to discuss the differences between the proposed method and existing unsupervised learning methods in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.a: Online setting.**\n\nThis is indeed a general limitation of transductive-inference approaches. Still, we believe this transductive setting is useful in a breadth of application domains and real scenarios, as pointed out by Reviewers NaRY and uSPt and evidenced by an abundant and recent literature on transductive few-shot learning; actually, in the recent years, the transductive setting has been very popular in the vision-only few-shot learning literature (see, for instance, [5, 13, 33, 36, 43, 73, 24, 72], among others). Use cases where batches of test imaging data are available occur frequently in practice, e.g., in video-stream sequences, or in smart-device photos taken and stored every day. The setting is also very appealing in various application domains, such as remote sensing or histopathology where large images are divided into patches, or more generally, to analyze existing databases or large corpora of web-scraped unlabeled images (e.g., to create pseudo-labels for large unlabeled image sets taken and stored every day). \n\nAs correctly mentioned by the reviewer, if data points are coming sequentially, we would need to wait to collect a batch of them before conducting TransCLIP (while still using the zero-shot ability of the VLM if on-the-fly prediction is needed). We will add this discussion to the limitation section, as well as to the conclusion, as this could be an interesting future work to broaden the application range of TransCLIP.\n\n**Q1.b: Prediction for new samples.**\n\nThis is, indeed, a very interesting point and extension that we missed in our first version.\nIn fact, after running TransCLIP, the GMM parameters could be stored to make a prediction on a new individual sample, resembling the inductive paradigm:\n\n$$\n\\mathbf{z}_{i} = \\frac{ \\mathbf{p} _{i} }{ \\mathbf{p} _{i}^\\top \\mathbf{1}_K }\n$$\n\nThis can be seen as a *Maximum Likelihood* predictor for sample $i$. We denote this naive prediction rule $\\textbf{\\textit{MLH}}$.\n\nMore interestingly, Update Eq. 5 can be directly adapted. Let's assume we don't have access to other data points (e.g., very low resource application or for privacy concerns), we remove the Laplacian term and get the following prediction rule:\n\n$$\n\\mathbf{z}_{i} = \\frac{\\hat{\\mathbf{y}} _{i} ^ {\\lambda} \\odot \\exp ( \\log ( \\mathbf{p} _{i}))}{(\\hat{\\mathbf{y}} _{i} ^ {\\lambda} \\odot \\exp (\\log (\\mathbf{p} _{i}))) ^ \\top \\mathbf{1}_K} = \\frac{\\hat{\\mathbf{y}} _{i} ^ {\\lambda} \\odot \\mathbf{p} _{i}}{(\\hat{\\mathbf{y}} _{i} ^ {\\lambda} \\odot \\mathbf{p} _{i}) ^ \\top \\mathbf{1}_K}\n$$\n\nThis can be seen as a *Maximum A Posteriori* predictor for sample $i$ since it is the class probability likelihood $\\mathbf{p}_i$ weighted by the initial pseudo-label $\\hat{\\mathbf{y}} _{i}^{\\lambda}$ (prior knowledge). We denote this prediction rule $\\textbf{\\textit{MAP}}$.\n\nFollowing this very interesting comment by the reviewer, we conduct new experiments (average over 100 random seeds): We split the test set, keeping randomly 10% of the data points as a held-out set, and infer TransCLIP on the remaining 90% of the samples. We then apply the prediction rule on each sample of the held-out set (independent predictions, as in the inductive setting). The Table below summarizes the performance and shows that our $\\textbf{\\textit{MAP}}$ predictor rule for new unseen data points still brings a significant gain, validating the feasibility of this approach.\n\n| Model                         | ImageNet | SUN397 | Aircraft | EuroSAT | Average |\n|-------------------------------|----------|--------|----------|---------|---------|\n| CLIP (*on held-out*)       | 66.6     | 62.5   | 24.7     | 48.3    | 65.3    |\n| $\\textbf{\\textit{MLH}}$ (*on held-out*)              | 67.5     | 67.2   | 24.6     | 64.1    | 68.0    |\n| $\\textbf{\\textit{MAP}}$ (*on held-out*)             | **69.9**     | **68.6**   | **26.6**     | **64.7**    |**70.1**    |\n|                               |          |        |          |         |         |\n| TransCLIP-ZS (*transductive*)   | 70.3     | 68.9   | 26.9     | 65.1    | 70.3\n\n**Q2: Precision on runtime.**\n\nWe provide more details on the runtime of each step in the attached document (see Table 1). On ImageNet, UPL$^*$ takes 151 minutes for prompt tuning, followed by 59 seconds for images+texts encoding during the inference step, resulting in a total of 152 minutes. In contrast, TransCLIP has no training stage and requires 14 seconds to run, which is only a fraction of the 59 seconds needed for images+texts encoding. We will extend Table 5 of the main paper to include these clarifications.\n\n**Q3: Comparison to unsupervised learning.**\n\nWe refer to Table 8 in the Appendix, which contains a direct comparison of TransCLIP with UPL [25]. Please note that UPL was initially designed to find pseudo-labels in a unlabeled training set, followed by a training stage before inferring on the test set, making UPL an inductive method. Therefore, we extended UPL to UPL$^*$, drawing pseudo-labels directly from the test set, hence making it a transductive method (please refer to the details in lines 569-572). In conjunction with our response to W6 by Reviewer 1 (NaRY), we will add a description of UPL and UPL$^*$ in the Appendix for better clarity.\n\nWe also compare TransCLIP to other unsupervised methods (in Table 8 of the Appendix) such as TPT [42] and MTA [65] (two test-time augmentation techniques) and SwapPrompt [40]. We draw attention to the differences between their settings and ours (lines 567-569), which will be extended by discussion \"W1 and Q2\" with Reviewer aJGT.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v5q30T5Cl4", "reviewer": "Reviewer_uSPt", "summary": "The paper introduces TransCLIP, a transductive learning method designed to enhance the performance of vision-language models (VLMs) for zero-shot and few-shot learning scenarios. By incorporating a novel objective function constrained by Kullback-Leibler divergence, TransCLIP not only improves prediction accuracy but also ensures computational efficiency. This method operates as a plug-and-play module on existing VLMs, leveraging unlabeled data to boost the model's predictive capabilities significantly across various datasets.", "questions": "See weaknesses section.", "limitations": "See weaknesses section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "TransCLIP integrates transductive learning into VLMs effectively, which is traditionally challenging due to the complexity of multimodal data integration.\n\nThe method consistently outperforms standard inductive and transductive approaches by leveraging text-encoder knowledge, which guides the learning process and significantly boosts performance.\n\nThrough the iterative Block Majorize-Minimize optimization procedure, TransCLIP ensures efficient computation, making it scalable for large-scale applications.\n\nThe method's ability to function atop various pre-existing models without requiring modifications to the underlying architectures enhances its applicability in real-world scenarios.", "weaknesses": "The proposed objective function of TransCLIP is composed of multiple terms, including a Gaussian Mixture Model-clustering term, a Laplacian regularizer, and a Kullback-Leibler divergence penalty. While this multifaceted approach aims to integrate various aspects of the data, it raises concerns about potential conflicts or trade-offs between these terms. The interaction and balance among these components are crucial, as overemphasis on one could undermine the effectiveness of others, potentially leading to suboptimal learning outcomes.\n\nAlthough the paper addresses computational efficiency through an iterative Block Majorize-Minimize (BMM) optimization procedure, a detailed comparison of computation costs with other methodologies is lacking. Understanding how TransCLIP's computational demands stack up against alternative approaches, particularly in terms of time complexity and resource usage, is essential for evaluating its practical applicability and efficiency.\n\nThe scalability and performance of TransCLIP when applied to very large datasets remain uncertain. While the method is claimed to be computationally efficient, the real-world effectiveness and efficiency on datasets significantly larger than those tested (like datasets beyond the scale of ImageNet) need thorough investigation. This includes an assessment of whether the benefits observed on smaller or moderate-sized datasets consistently translate to much larger scales.\n\nThe paper provides theoretical convergence guarantees for the optimization procedure. However, the robustness of these proofs and their assumptions in practical, real-world scenarios could be questioned. Specifically, the conditions under which convergence is guaranteed should be scrutinized to ensure they are not overly restrictive or detached from practical applications. This scrutiny is vital to validate the method's theoretical foundation and to ensure its reliability across various deployment contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Balance of terms.**\n\nWe understand your concern. The terms in our objective function are not necessarily in conflict; they could be seen as a clustering (driven by the GMM term), which is regularized by the Laplacian term propagating the labels between neighboring samples, and by the KL-divergence term that incorporates the text knowledge of the VLM.\n\nRegarding the utility of the two regularization terms, we show in Table 6a (line 3) of the main paper that the Laplacian term helps improving performance. In Table 4 of the main paper, we demonstrate the importance of the KL text regularization (TransCLIP vs. TransCLIP *w/o text*).\n\nRegarding the sensitivity to hyper-parameters, we emphasize that ***the text-regularization weighting factor $\\lambda$ is set to the same value for all the experiments that involve TransCLIP-ZS, regardless of the dataset or the few-shot method***. We evaluate *a posteriori* the impact of varying $\\lambda$ and show that there is only a minor sensitivity as long as $\\lambda$ does not tend towards $0$ (please refer to Table 6b of the main paper, and to Figure 1 that we added in this rebuttal, in the attached PDF document). We also evaluate the sensitivity to the number of neighbors considered in the Laplacian term, and show again that it does not significantly impact the performances (please refer to Table 6c of the main paper).\n\n**W2: Computation costs.**\n\n - *Time complexity*: The Block Majorize-Minimize procedure detailed in Section 3.3 presents a decoupled $\\mathbf{z}$ update equation (Eq. 5) and closed-form update equations for the GMM parameters $\\boldsymbol{\\mu}$ and $\\boldsymbol{\\Sigma}$ (Eqs. 6 and 7). These enable our algorithm to achieve rapid convergence compared to traditional prompt tuning methods such as UPL [25], which require extensive forward-backward propagation over multiple epochs. \nWe provide runtime comparisons in Tables 5a and 5b of the main paper, and a more detailed analysis in Table 1 of the attached document. We will incorporate these new results into Tables 5a and 5b of the main paper to clarify the speed and efficiency advantages of TransCLIP. \n\n- *Resource usage*: we discuss the hardware requirements of TransCLIP in Appendix B, noting that TransCLIP consumes 16.9 GB of memory when inferring on ImageNet, making it feasible to run on a single 24 GB GPU for large datasets. \n\n**W3: Scalability to even larger datasets.**\n\nAs discussed in Section 4.2, TransCLIP is easily applicable to multi-billion parameter models and large datasets since it does not necessitate gradient computation or model parameter training (i.e., it only requires the memory needed for single-sample inference, because the whole dataset processing can be performed one sample at a time). Results in the Table below support the scalability of TransCLIP in effectively managing more than one million images in a few minutes (on ImageNet training set which contains 1,281,167 images categorized into 1,000 classes).\n\n| Model     | Zero-shot | TransCLIP-ZS | Runtime   |\n|-----------|-----------|--------------|-----------|\n| ViT-B/16  | 68.09     | 71.02        | 284 sec.  |\n| ViT-L/14  | 74.82     | 77.95        | 396 sec.  |\n\n**W4: Theoretical convergence guarantees.**\n\nAs stated in Section 3.4, our optimizer can be viewed as an instance of the general Block Majorize-Minimize (BMM) paradigm, and we establish convergence of our procedure. The technical conditions that guarantee convergence stem from the mathematical properties of our objective\nand its block-wise majorizing functions used in the BMM optimizer (such as the strong convexity of the block-wise majorizing functions). Hence, these conditions do not restrict any practical application aspects.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GaeKdZ6eHj", "reviewer": "Reviewer_bXXg", "summary": "Summary:\n\nThe paper under review investigates the integration of contrastive vision-language pretraining (CLIP) with transductive learning methodologies. The authors are motivated by the efficacy of transductive learning in utilizing unlabeled data to enhance the performance of conventional supervised learning frameworks. They propose to extend this approach to the domain of vision-language pretraining. The key innovation lies in modeling the distribution of language data using a Gaussian mixture model (GMM). The authors introduce three specific learning objectives: GMM clustering, Laplacian regularization, and text knowledge preservation. These objectives collectively aim to regularize the distribution of unlabeled data while maintaining the integrity of text-guided vision-language pretraining.\nThe complexity of optimizing three intertwined variables necessitates an advanced optimization strategy. To address this, the authors propose the Block Majority-Minimize optimization technique, which iteratively fixes two variables while updating the third. This approach ensures the overall minimization of all three variables, providing a robust convergence guarantee. Through comprehensive quantitative experiments, the authors demonstrate that their proposed method can serve as a versatile framework, enhancing the performance of various backbone methods across different scenarios.", "questions": "Please see the weaknesses part.", "limitations": "The authors have discussed the limitation in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n\n- Organization and Clarity:\n\nThe paper is meticulously organized, with a clear and coherent presentation of concepts. The structured formulation facilitates a straightforward understanding of the core ideas. The writing is lucid, allowing readers to seamlessly follow the logical progression of the methodology.\n\n- Experimental Rigor:\n\nThe experimental evaluation is thorough and extensive. The authors employ a diverse array of datasets and baseline methods, lending credibility and robustness to their findings. The significant performance improvements observed in experiments underscore the practical utility and contribution of the proposed method.\n\n- Theoretical Foundations:\n\nA detailed convergence analysis is provided, offering a solid theoretical underpinning for the optimization process. This analysis enhances the credibility of the proposed optimization strategy and reassures readers of its reliability.", "weaknesses": "Weaknesses:\n\n- Unclear Motivation:\n\nThe motivation for leveraging transductive learning in this context is somewhat ambiguous. The assertion that transductive learning can enhance the handling of unlabeled data, while valid in general, seems less compelling here. In the CLIP framework, vision and language data are inherently paired and labeled, diminishing the applicability of transductive learning, which traditionally targets unlabeled data.\n\n- Mismatch with Zero-Shot Learning:\n\nThere is a conceptual misalignment between transductive learning and zero-shot learning paradigms. Transductive learning focuses on knowledge propagation from observed training data to observed test data. In contrast, zero-shot learning aims to generalize to unseen test data. This fundamental difference makes the application of transductive learning to zero-shot scenarios appear somewhat forced and unconvincing.\n\n- Lack of Intuitive Justification:\n\nThe introduction of the GMM and Laplacian terms lacks intuitive explanation. While the quantitative results are impressive, there is an absence of qualitative analysis to elucidate why these terms specifically enhance learning performance. A deeper exploration of the underlying reasons for their effectiveness would strengthen the paper's contributions.\n\n- Computational Overhead:\n\nThe integration of GMM and Laplacian regularization is computationally intensive. The paper would benefit from experimental validation of the method's efficiency, demonstrating that the performance gains justify the additional computational costs. Providing benchmarks or comparative analyses regarding computational efficiency would address potential concerns about scalability and practicality.\n\nOverall Assessment:\n\nIn conclusion, this paper presents a novel approach to enhancing vision-language pretraining by incorporating transductive learning principles. Despite some motivational and conceptual ambiguities, the paper's methodological rigor, comprehensive experiments, and theoretical contributions make it a valuable addition to the field. Addressing the highlighted weaknesses through additional qualitative analyses and computational efficiency evaluations would further solidify the paper's impact and applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Unclear motivation.**\n\nWhile VLMs such as CLIP enable zero-shot predictions, they are far from being perfect (please refer to the zero-shot performances of CLIP for 6 different model sizes in Table 8 in the Appendix). Please note that CLIP receives unlabeled samples and makes class predictions by comparing the images to several candidate text prompts. Therefore, CLIP yields pseudo-labels that could be used as a noisy supervision in transduction without any label.\n\nWhat we propose in our work is to improve zero-shot CLIP and inductive few-shot methods based on CLIP as pre-training, by incorporating the transduction paradigm during inference.\n\n**W2: Mismatch with Zero-Shot Learning.**\n\nIt is true that many transductive, vision-only methods (e.g., [5, 13, 33, 36, 43, 73, 24 , 72], among others) propose to propagate knowledge from observed training (labeled) data to unlabeled test data. However, in this work, we show that, in the context of VLMs, transduction could also be conducted without any label, by leveraging the pseudo-labels generated from the text embeddings, as a form of noisy supervision.\n \nIndeed, we make zero-shot predictions by employing text-based noisy supervision (the KL term in Eq. 2) and the structure of the unlabeled data in the feature space (GMM clustering and Laplacian regularization terms), which effectively corresponds to the transductive paradigm. Please note that we propose an extension of TransCLIP, named TransCLIP-FS (the objective function in Eq. 4), for cases where labeled data are available, which combines noisy supervision from the zero-shot predictions and supervision from the few-shot samples, as in the above-mentioned transductive methods.\n\n**W3: Lack of Intuitive Justification.**\n\nThank you for giving us the opportunity to clarify. The GMM term could be viewed as a maximum-likelihood estimation objective, like in the Expectation-Maximization algorithm. It could also be viewed as a probabilistic generalization of the K-means clustering objective (lines 157-162). The principal difference with the standard K-means clustering lies in the capability of TransCLIP to learn a covariance matrix (variable $\\boldsymbol{\\Sigma}$). \n\nAs shown in Figures 2a and 2b, which we added for this rebuttal (please refer to the attached PDF document), the learned covariance matrix enables fitting the unevenly spread feature distribution across the dimensions of the embedding space. We also show that fixing the covariance matrix (which corresponds to K-means if we omit the Laplacian and KL terms) results in a performance drop (line 2 in Table 6a of the main paper). \n\n**W4: Computational Overhead.**\n\nThe Block Majorize-Minimize procedure presented and detailed in Section 3.3 provides a decoupled $\\mathbf{z}$-update equation (Eq. 5), as well as closed-form updates for GMM parameters $\\boldsymbol{\\mu}$ and $\\boldsymbol{\\Sigma}$ (Eqs. 6 and 7). These decoupled updates enable the algorithm to run substantially faster than popular prompt-tuning approaches (such as UPL [25]), which require costly forward-backward propagation for multiple epochs. We report run-time comparisons in Tables 5a and 5b in the main paper. \n\nFurthermore, we add Table 1 in this rebuttal (please refer to the attached PDF document), which provides a more detailed analysis, showing: (i) TransCLIP's prediction time is a fraction of the time required for feature encoding, which \nmakes its total runtime (encoding + prediction) comparable to the total runtime of the inductive zero-shot CLIP baseline; (ii) The total runtime of UPL is an order-of-magnitude slower, due to the training overhead. We will add these detailed runtimes to Tables 5a and 5b in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We greatly appreciate the reviewers' insightful and constructive comments, and are pleased that four out of the five reviewers voted towards acceptance. 

We are also glad that the reviewers found the method useful for various domains and real-world scenarios (Reviewers NaRY and uSPt), the experiments comprehensive and convincing (Reviewers aJGT, bXXg and NaRY), and the work to fill a gap in the literature (Reviewer AvCq). The reviewers also highlighted the interest of our iterative block majorize-minimize (BMM) procedure, which tackles three blocks of intertwined variables (prototypes, covariance and assignments) while guaranteeing convergence.

Below, we address all the questions raised by the reviewers. In particular, we provide additional ablation studies and clarifications, as requested. For the reviewers' convenience, we have included additional illustrations in the attached document.

Meta-review: The paper targets a new setting based on transductive learning for CLIP-based models. The paper received five reviews in total and all reviewers voted acceptance (borderline accept or above). The paper is strong in several notable aspects: it is well-written, well-structured, and easy-to-follow; it fills a gap in the literature as this topic is under-studied; the method achieved big improvement over prior SOTA and could be very useful for downstream applications of CLIP; the experiments are comprehensive. In terms of weaknesses, the reviewers pointed out that some concepts are ambiguous and more discussions need to be added for test-time methods. The rebuttal has well addressed most concerns raised by the reviewers. Given the novelty and strong results of the paper, the AC recommends acceptance.
Decision: Accept (spotlight)