Boosting Vision-Language Models with Transduction

Authors: maxime  zanella
benoît  uclouvain
  gérin
uclouvain  ismail
ben  ayed
éts  montréal
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:42:09

Abstract: Transduction is a powerful paradigm that leverages the structure of unlabeled data to boost predictive accuracy. We present TransCLIP, a novel and computationally efficient transductive approach designed for Vision-Language Models (VLMs). TransCLIP is applicable as a plug-and-play module on top of popular inductive zero- and few-shot models, consistently improving their performances. Our new objective function can be viewed as a regularized maximum-likelihood estimation, constrained by a KL divergence penalty that integrates the text-encoder knowledge and guides the transductive learning process. We further derive an iterative Block Majorize-Minimize (BMM) procedure for optimizing our objective, with guaranteed convergence and decoupled sample-assignment updates, yielding computationally efficient transduction for large-scale datasets. We report comprehensive evaluations, comparisons, and ablation studies that demonstrate: (i) Transduction can greatly enhance the generalization capabilities of inductive pretrained zero- and few-shot VLMs; (ii) TransCLIP substantially outperforms standard transductive few-shot learning methods relying solely on vision features, notably due to the KL-based language constraint.

Main paper: 
 related work: 
 Transduction for vision-only classifiers. The use of unlabeled test data at inference time has received attention lately in rapidly emerging subjects, such as few-shot learning and unsupervised testtime adaptation. Examples include adjusting batch normalization layer statistics [46] and minimizing the entropy of predictions [60] , which can be supplemented by pseudo-labeling strategies [36] . In the few-shot literature solely based on vision models, transduction leverages both the few labeled samples and unlabeled test data, outperforming inductive methods [76, 5, 24, 37, 75] . One of the first works introducing transduction in vision-based few-shot learning proposes propagating the labels from the support (labeled) to the query (unlabeled) set with a meta-learned graph [39] . Building on this idea, another work proposes to iteratively augment the support set to improve label propagation [34] . LaplacianShot [76] also exploits the inherent structure of the data through a graph-Laplacian clustering, which discourages disparate class predictions for samples with close features, while matching each query set point to the nearest support prototype. Alternative approaches propose directly learning the class prototypes. For instance, Transductive Fine-Tuning (TF) [13] uses the prediction entropy on the query samples as a regularization term, while TIM and its variants [5, 59] employ the mutual information between the query samples and their predictions. BD-CSPN [37] refines the class prototypes by reducing the feature biases between the support set and the most confident query samples. An additional group of methods performs clustering in the feature space, for instance, by solving an optimal transport problem like PT-MAP [24] , by projecting features into sub-spaces to facilitate clustering [75] , or by revisiting the standard K-means with an additional partition-complexity regularizer to control the number of predicted classes [44] .Zero-and few-shot learning in VLMs. Thanks to their extensive pre-training, VLMs exhibit stronger generalization capabilities than vision-only models but may also fail [50, 69, 57] . In response, substantial recent efforts have been directed towards using their general knowledge and adapting them on more specific tasks [63, 73, 70] . Arguably, the most popular strategy is prompt tuning [35] , which is explored both in the unsupervised [43, 15, 41, 1] and few-shot [73, 72, 40, 12, 65, 6, 74, 8, 9, 29, 30] settings. The pioneering work, CoOp [73] , updates input text-prompt tokens by leveraging the context provided by the few labeled samples (i.e., the support set). Building on this success, various strategies have been developed to enhance this approach, especially through additional regularization. For instance, ProGrad [74] guides the prompts towards the original hand-crafted ones by gradient projection. Prompt tuning has also been explored in the zero-shot setting, e.g., using the predictive confidence to generate pseudo-labels [25, 41] . Despite its popularity, prompt tuning remains tedious in terms of computations, due to the many back-propagations through the text encoder. This challenge is compounded in the recent developments, which introduce visual tokens [29, 30] alongside the text tokens. In contrast, there has been limited efforts so far in developing black-box methods [48, 17, 16, 66, 62] , which only access the final embedding states. These methods often rely on the so-called adapters [23] , like Tip-Adapter(-F) [70] , which adds a classifier at the output of the vision encoder, in the form of a cache model involving the few-shot samples. Lately, a strong baseline based on Gaussian discriminant analysis clustering [62] demonstrates VLMs' adaptation abilities with a Gaussian hypothesis on the embedding space.Transductive inference in VLMs. Despite the growing interest in unsupervised, zero-shot and few-shot learning for VLMs, the transductive-inference paradigm has not been explored so far in this new multi-modal context, except for the very recent work in [45] , which was deployed for small-size tasks (≈ 10 2 test samples). However, the method in [45] may not be computationally tractable for large-scale query sets, due to expensive inner loops for estimating the Dirichlet distribution's parameters. We provide a computationally efficient solution, which can scale up to large target datasets (such as ImageNet), while being easily amenable as a plug-and-play module on top of state-of-the-art inductive methods. It is worth mentioning that test-time adaptation methods also employ the transduction paradigm, but their settings are very different from those studied in this work. For instance, SwapPrompt [41] has been designed to make batch predictions on-the-fly, and has continual-learning mechanisms such as an exponential moving average prompt across batches.TPT [43] work on a single sample with many data augmentations to train one prompt per image. Both methods require access to model weights for training (i.e., do not operate in a black-box setting) and an expensive training procedure. We also note that prompt tuning does not scale well with the model size and is even impractical on very large models such as EVA-CLIP-8B [55] . We still report the performances of this class of methods in the Appendix (Table 9 ).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Limitations are presented in Appendix D.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
transclip: transduction for vision-language models: 
 In this section, we describe our objective function for transductive inference in vision-language models, and derive a block majorize-minimize (BMM) algorithm for minimizing it, with guaranteed convergence and decoupled sample-assignment updates. When dealing with a zero-shot classification problem based on a vision-language model, such as CLIP, and given a set of K candidate classes, one creates textual descriptions, the so-called prompts [38] , each corresponding to a class, e.g., c k = a photo of a [kth class name], k = 1, . . . , K. Let t k = θ t (c k ) denotes the corresponding normalized (unit hyper-sphere) embedding representation, with θ t representing the language encoder. Similarly, each test image x i , i = 1, . . . , N , is projected onto a normalized embedding space of the same dimension, using visual encoder θ v : f i = θ v (x i ). In the standard inductive zero-shot inference, classification of a given image x i is done by evaluating the cosine similarity between these two encoded modalities and predicting the class corresponding to the most similar text embedding: k = argmax k f ⊤ i t k . Furthermore, one can compute pseudo-labels corresponding to these zeroshot predictions by applying the softmax function with a temperature scaling 3 τ , which yields the following probability-simplex vector for each sample:EQUATIONwhere ∆ K denotes the probability simplex. Let D = {i ∈ N : 1 ≤ i ≤ N } = S ∪ Q denotes the samples indices of the target dataset, with Q the set of unlabeled query samples indices, i.e., those for which we want to make a prediction, and S the set of labeled support samples indices in the few-shot setting.Note that, in the zero-shot setting, S = ∅. We define a Gaussian Mixture Model-clustering (GMM) term in our objective function by modeling the likelihood of these target data as a balanced mixture of multivariate Gaussian distributions, each representing a class k and parameterized by mean vector µ k and a diagonal covariance matrix Σ:p i,k = Pr(f i , k; µ k , Σ) ∝ det(Σ) -1 2 exp - 1 2 (f i -µ k ) ⊤ Σ -1 (f i -µ k )Notation p i,k is introduced here to simplify the equations in the sequel. Notice that, unlike standard GMMs, we deploy a common diagonal covariance matrix Σ across all classes. Interestingly, in our experiments, we found this simplifying choice improves the performance while reducing the computational load as there are substantially fewer parameters to learn. This is particularly the case when dealing with large numbers of classes as in large-scale target datasets such as ImageNet.Our objective function depends on two types of variables: (i) Sample-to-class assignment variables within the probability simplex: z i = (z i,k ) 1≤k≤K ∈ ∆ K , i ∈ Q; and (ii) GMM parameters µ = (µ k ) 1≤k≤K and Σ.We propose to minimize the following objective, which integrates a GMM-clustering term, a Laplacian regularizer and a Kullback-Leibler (KL) divergence penalty encoding the text-encoder knowledge and guiding the transductive learning process:EQUATIONwhere p i = (p i,k ) 1≤k≤K ∈ ∆ K concatenates the GMM probabilities, w ij denotes some measure of affinity between visual embeddings f i and f j , and the sample-wise parameterized 4 KL terms are given by: KL λ (zEQUATION) In the following, we describe the effect of each term in our objective function in (2):• GMM-based clustering: This unsupervised-learning term is akin to the GMM-based maximum-likelihood estimation objective in the standard EM algorithm [3] . By taking the negative logarithm, its minimization corresponds to maximizing the likelihood of the data. It can also be viewed as a probabilistic generalization of the K-means clustering objective [28] . Indeed, assuming Σ is the identity matrix reduces the first term in (2) to the K-means objective. • Laplacian regularization: The second term in (2) is the Laplacian regularizer, widely used in the context of graph/spectral clustering [56] and semi-supervised learning [7] . This term encourages nearby samples in the visual-embedding space (i.e., pairs of samples with high affinity w i,j ) to have similar z assignments. In our case, we propose to build a positive semidefinite (PSD) affinity matrix based on the cosine similarities as w ij = f ⊤ i f j (Gram matrix). As we see below, this PSD condition is important to obtain a convergent Majorize-Minimize optimizer with decoupled (parallel) sample-wise updates for the z-assignments, yielding a highly efficient transduction for large-scale target datasets (such as ImageNet).• Text-guided KL divergence: This term is dedicated to vision-language models and, as we will see in our experiments (ablation studies in Tables 4 and 6 ), has a substantial effect on performance. It encourages the prediction not to deviate significantly from the zero-shot predictions, thereby providing text supervision to the other two unsupervised-learning terms. Furthermore, being convex over z i , i ∈ Q, this term facilitates the optimization of the objective w.r.t the assignment variables.Our zero-shot formulation naturally extends to the few-shot setting. We integrate supervision from the labeled-support samples, in the form of a cross-entropy, which corresponds to minimizing the following overall loss:EQUATIONNote that, in the first term, the z i are fixed, with z i = y i , i ∈ S and y i the one-hot ground-truth label associated with the corresponding shot.As our objective depends on three types of variables (z, µ, Σ), we proceed with a BMM procedure, alternating three sub-step optimizers. Each sub-step optimizes over one block of variables while the other two are fixed, ensuring the overall objective does not increase. Importantly, the obtained z-updates (Eq. ( 5)) are decoupled, yielding computationally efficient transduction for large-scale datasets. Also, our overall procedure is guaranteed to converge (Theorem 1).Majorize-Minimize (MM) with respect to the z-block When µ and Σ are fixed, both the GMMand KL-based terms are convex w.r.t z i . However, the Laplacian term is concave 5 (for PSD matrix W ). Therefore, we proceed with inner iterations, each minimizing a linear and tight upper bound, the so-called majorizing function in the MM-optimization literature [33, 21, 31] , which guarantees the overall objective does not increase. To obtain the tight linear bound, let us write the Laplacian term conveniently in the following matrix form: z ⊤ Ψz, with Ψ = -W ⊗ I, where ⊗ denotes the Kronecker product and I is the N × N identity matrix. Note that Ψ is negative semi-definite for a positive semi-definite W . Therefore, z ⊤ Ψz is a concave function with respect to z, and its first-order approximation at current solution z l (l being the iteration index) gives the following tight 6 upper bound on the Laplacian term:z ⊤ Ψz ≤ (z l ) ⊤ Ψz l + (Ψz l ) ⊤ (z -z l )Replacing the quadratic Laplacian term by this linear bound yields a majorizing function on our overall objective. Importantly, this majorizing function is a sum of decoupled objectives, each corresponding to one assignment variable z i , yielding a highly efficient optimizer for large-scale target datasets. Indeed, using simplex constraints z i ∈ ∆ K , i ∈ Q, and solving the Karush-Kuhn-Tucker (KKT) conditions independently for each z i , we obtain the following decoupled update rules for the z-block:EQUATIONClosed-form updates of µ and Σ When both z and Σ are fixed, our objective in ( 4) is convex. It can be minimized by setting its gradient w.r.t each µ k to zero, which yields the following closed-form updates:EQUATIONSimilarly, when both z and µ are fixed, the following closed-form updates minimize the overall objective w.r.t Σ:EQUATIONThe complete procedure is summarized in Appendix B. Note that, after convergence, we use the sample-to-class assignment variables z i as predictions for each sample i of the query set Q using the argmax operation for conventional classification.Our optimizer can be viewed as an instance of the general Block Majorize-Minimize paradigm for optimization [51] , which optimizes a majorizing function for each block of variables. The convergence of general BMM procedures is well studied in the optimization community [51] . Indeed, under certain conditions (such as the strong convexity of the block-wise majorizing functions), we can establish convergence of our procedure using the following result (more details in Appendix A):Theorem 1 (Convergence of BMM [51] ) Assume that, for each block, the majorizing function is quasi-convex, and its first-order behavior is the same as the original objective locally. Furthermore, assume that the sub-problem solved for each block has a unique solution. Then, every limit point of the iterates generated by BMM is a coordinate-wise minimum of the overall objective.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The hardware used for the experiments is presented in Appendix B. and runtime is discussed in Section 4 (Table 5 ).Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiments: 
 Datasets. Following the setting of previous works [73, 43] , we assess TransCLIP on ImageNet [11] and ten datasets for fine-grained classification of scenes (SUN397 [64] ), aircraft types (Aircraft [42] ), satellite imagery (EuroSAT [18] ), automobiles (Cars [32] ), food items (Food [4] ), pet breeds (Pets [49] ), flowers (Flowers [47] ), general objects (Caltech101 [14] ), textures (DTD [10] ) and human actions (UCF101 [54] ). We additionally measure performance on four variants of ImageNet (Adversarial [20] , ImageNetV2 [52] , Rendition [19] , Sketch [61] ). Numerical results are reported in terms of the top-1 accuracy with the ViT-B/16 encoder, averaged over three random seeds.Benchmarks. We aim to show the breadth of potential applications of transduction in the context of VLMs. Notably, employing supervised fine-tuning, followed by transduction with TransCLIP on the unlabeled test samples, emerges as a powerful and efficient solution. This is particularly convenient when the labeled samples (the support set) and/or computational power are not accessible at inference (i.e., test) time 7 . To this end, we first study the applicability of our zero-shot formulation TransCLIP-ZS (Eq. ( 2)) across three settings: (i) on top of inductive zero-shot learning and popular few-shot learning methods; (ii) on top of 16-shot ImageNet pretraining for cross-dataset transferability, and Implementation details. The main component of our transductive formulation is the text-guided KL divergence penalty. We fix λ = 1 for all our zero-shot experiments (see ablation study in Table 6 ), and λ = 0.5 in all the few-shot experiments to reduce the impact of the text-driven regularization. Another component of our optimization problem is the Laplacian regularization, which enforces consistent predictions for close instances. We truncate the affinity matrix to the 3 nearest-neighbors, making it sparse. µ is initialized with the top-8 most confident samples of each class for the zero-shot setting. For the few-shot setting, we use the class-wise average over the shot embeddings.Transduction improvements. Table 1 and 2 demonstrate the advantages of our transductive approach in zero-shot, few-shot, and cross-dataset transferability. TransCLIP enhances the zero-shot top-1 accuracy by over 5% and popular few-shot methods by 4% (1-shot) on average, without the need for additional labels. Table 3 further highlights that TransCLIP can be applied on top of prompt tuning and adapter fine-tuning solutions, enhancing performance for both in-domain and domain generalization tasks. However, we observe in Table 1 that transductive gains sometimes decrease with the number of shots, presumably because data structure information can be partially captured in the shots. These results underline the value of considering the structure of the unlabeled test samples during prediction, especially on top of zero-and low-shot models or when facing domain shifts, an aspect not leveraged by the current zero-and few-shot VLM literature. More detailed results for five different backbone architectures and comparisons with unsupervised non-transductive methods are provided in Appendix C.1 for the zero-shot setting, in Appendix C.2 for TransCLIP on top of popular few-shot methods, in Appendix C.3 for cross-dataset transferability and in Appendix C.4 for domain generalization. With its hyper-parameters unchanged, TransCLIP exhibits strong generalization from convolutional networks to transformer-based models, as also depicted in Figure 1 .Transductive few-shot learning. We compare TransCLIP-FS, TransCLIP-FS without text regularization (i.e., λ = 0) and state-of-the-art transductive few-shot methods. It is important to note that these few-shot methods were primarily developed for vision-centric tasks. Hence, they rely on visual information, omitting the textual elements. This allows us to study the impact of our text-based regularization term. Table 4 shows that incorporating language in the transductive paradigm boosts the performance over vision-only methods. Especially for the 1-to 4-shot settings, our language-driven KL penalty enhances the performance by a large margin on many tasks (e.g., ImageNet, SUN397, StanfordCars, DTD). As the number of shots increases, the text-driven penalty becomes less useful, especially for the datasets capitalizing on the visual shots rather than the text-encoder knowledge (e.g., EuroSat and Flowers). This points to promising future directions involving more flexible text regularization (e.g., an adaptable λ taking into account the number of shots and the quality of the text embeddings). Detailed results for five different encoder architectures are provided in Appendix C.5, consistently showing similar conclusions. Comparison with prompt learning. Following current VLMs literature, adapting the input prompt instead of GMM parameters could be seen as a more straightforward solution. For a fair comparison, we adapt Unsupervised Prompt Learning (UPL) [25] for the transductive setting and reevaluate its main hyper-parameter (see Appendix C.1). Table 5 shows clearly that TransCLIP outperforms UPL while being two to three orders of magnitude faster. Additional details on runtime are provided in Table 8 of the Appendix.Components of TransCLIP. We study the impact of the principal components involved in the TransCLIP procedure over four diverse datasets. Table 6a shows that updating µ and Σ allows to significantly boost TransCLIP's performance. This indicates the importance of having a dynamic parametric model instead of a fixed one. Table 6b demonstrates the critical role of text-driven penalty for TransCLIP in the zero-shot setting. Additional results on the sensitivity of λ in the few-shot setting are depicted in Figure 2 of the Appendix. Alongside the prior findings from Table 4 , it is evident that incorporating text information is key to the success of TransCLIP and its wide applicability across the zero-and few-shot learning scenarios. The number of nearest-neighbors considered in the Laplacian term (Eq. ( 2)) does not make a significant difference in TransCLIP's performance as suggested by Table 6c . However, removing the Laplacian regularization (Table 6a ) leads to inferior results on some datasets such as ImageNet and EuroSAT. We choose to consider 3 nearest-neighbors to make the affinity matrix W sparse and reduce memory consumption. We also investigate the diagonal covariance matrix design by restricting it to be isotropic (i.e., Σ = σ 2 I d with I d the identity matrix).Table 6d shows that a non-isotropic Σ performs better without significantly increasing the amount of trainable parameters.Scaling to larger VLMs. We report TransCLIP-ZS performance on EVA-CLIP 8 billion parameter version [55] (approximately 42 times larger than the CLIP-ViT-B/16). It is worth mentioning that TransCLIP is easily applicable to multi-billion parameter models since it does not necessitate gradient computation or model parameter training (i.e., it only requires the memory needed for single-sample inference because the whole dataset processing can be performed one sample at a time). Table 7 shows that transduction can also bring significant improvements to larger models (details in Appendix C.1).  
conclusion: 
 In this work, we studied the transductive paradigm in the context of Vision-Language Models and proposed the TransCLIP method. Our algorithm is highly efficient, as it operates solely in the output embedding space (i.e., black-box setting), making it suitable for a wide range of models, including very large ones. This also enables TransCLIP to be compatible with models that are accessible only through APIs. We first showed how TransCLIP can bring transduction to the inductive zeroshot setting, achieving consistent gains without additional supervision. Then, we proposed a new setting that applies transduction on top of popular few-shot methods, offering a convenient strategy to combine computationally intensive supervised fine-tuning with efficient test-time transduction. Finally, we highlighted the limitations of current transductive few-shot methods and proposed a simple extension of TransCLIP to incorporate labeled samples. In all our experiments, TransCLIP's text-guided KL divergence term appears as a key factor in its success. Future work may focus on further enhancing this regularization term, for example, by making it more resilient (e.g., with adaptive class-wise weighting) when text prompts are less reliable.  
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: The paper is not tied to specific application.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: No new asset is released.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No crowdsourcing nor research with human subject was involved.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: No crowdsourcing nor research with human subject was involved.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: As mentioned in the main paper, our derived block-wise optimization procedure in Eqs. ( 5), ( 6) and ( 7) can be viewed as an instance of the the general Block Majorize-Minimize paradigm for non-convex optimization, also referred to as the Block Successive Minimization (BSUM) method [51] .We update each block of variables, with the other blocks fixed, by minimizing a tight upper bound (majorizing function), thereby guaranteeing the overall objective does not increase at each step. In the steps with respect to µ and Σ, we optimize directly the objective in closed-form, which could be also viewed as a particular case of optimizing a tight upper bound. The convergence of the general BSUM procedure is well studied in the optimization community [51] . Indeed, under the following assumptions for each block of variables, one can establish convergence results for the application of BSUM to non-convex problems [51] :• A1: The majorizing function is a tight upper bound, i.e., equal to the objective at the current solution.• A2: The first-order behavior of the majorizing function is the same as the original objective locally.Indeed, when assumptions A1 and A2 are verified for each block, we have the result in Theorem 1 [51] .As for our case of alternating Eqs. ( 5), ( 6) and ( 7), it is straightforward to verify that Assumptions A1 and A2 are satisfied for each block of variables. Furthermore, the majorizing functions are convex and thus quasi-convex. Also, the sub-problem solved for each block has a unique solution. In particular, for the z-updates, the majorizing function is the sum of a linear and a strongly convex function (the negative entropy). Therefore, it is strongly convex. As for the µand Σ-updates, the solutions are obtained in closed form (hence unique).Require: A set of image embeddings (f i ) 1≤i≤N , a set of textual class embeddings (t k ) 1≤k≤K , τ the temperature of the CLIP model. 1: w i,j ← f ⊤ i f j ∀ i, j ▷ Affinity measure, truncated with top-3 values 2: ŷi ← φ(τ f ⊤ i t)) ∀ i ▷ Initial predictions, φ the softmax function 3: µ k ← mean{f i s.t y = k, i ∈ S} 8 ∀ k ▷ Class centroids initialization 4: diag(Σ) ← 1 1 d ▷ Covariance matrix initialization, d is the emb. dim. 5: z i ← ŷi ∀ i ▷ Initialz i,k ← ŷλ i,k exp(log(p i,k )+ j∈D wij z j,k ) k ′ ŷλ i,k ′ exp(log(p i,k ′ )+ j∈D wij z j,k ′ ) ∀ i ∀ k ▷ (1) z-step 9:end while 10:µ k ← γ |S| i∈S z i,k fi+ 1 |Q| i∈Q z i,k fi γ |S| i∈S z i,k + 1 |Q| i∈Q z i,k ∀ k ▷ (2) µ-step 11: diag(Σ) ← γ |S| i∈S k z i,k (fi-µ k ) 2 + 1 |Q| i∈Q k z i,k (fi-µ k ) 2 γ+1 ▷ (3) Σ-step 12: end while 13: return argmax k (z)▷ Prediction with assignment variablesThis section aims to provide an additional pseudo-algorithm to supplement Section 3 as well as more details on TransCLIP hyper-parameters presented in Section 4. Our code is available at https://github.com/MaxZanella/transduction-for-vlms and a pseudo-code in Algorithm 1 summarizes the main steps of the TransCLIP algorithm.Hardware. All our experiments were conducted on a single A100-40 GB. In terms of memory, TransCLIP consumes 16.9 GB when inferring on ImageNet, and can therefore process large datasets on a smaller 24 GB GPU.Hyper-parameters. In practice, TransCLIP performs 10 iterations of z, µ, Σ block-wise updates.For each z-update, we perform 5 iterations, as we found it sufficient for convergence. In the zero-shot setting, we set λ = 1 and γ = 0 (as there are no support samples). In the few-shot setting, we set λ = 0.5 and search for the value of γ in {0.002, 0.01, 0.02, 0.2}. The number of validation shots is set at min(4, #shots), and we build a 1-nearest neighbor classifier with the query samples and their final class assignment to predict the class of each validation sample.Prompt templates. We employ the prompt templates detailed in Table 24a for all our experiments in zero-shot setting unless otherwise explicitly specified. Only when specified, we utilize the custom template ensembling for ImageNet as in [70] (see Table 24b ).We provide detailed results for all the studied vision backbones of CLIP over the 11 datasets to support the transferability of TransCLIP across both convolutional networks and transformer-based models. We additionally report other methods that do not fit into the transductive setting.In Table 9 . We report performances of 5 CLIP encoders as well as the 8 billion parameter EVA-CLIP [55] . We compare TransCLIP-ZS to unsupervised methods namely TPT [43] , MTA [68] , SwapPrompt [41] , and UPL [25] . Note that TPT and MTA are two test-time augmentation methods working on a single image at a time, thus they differ from our transductive setting, still we report their performance for informational purposes.UPL * . As mentioned in Section 4, we slightly modify UPL to apply it to the test set in a transductive manner (transductive UPL is denoted UPL * ). Indeed, UPL relies on the generation of N = 16 hard pseudo-labels per class from a training set, after what a cross-entropy loss function on soft tokens is minimized. Instead, UPL * generates the pseudo-labels directly from the test set. For fairness, we reevaluated the number of pseudo-labels to select and still found that 16 per class yields the best results on average, as seen in Table 23 .In Tables 10, 11 , 12, 13 and 14. We report the performance of TransCLIP-ZS on top of CoOp [73] , Tip-Adapter-F [70] , PLOT [8] , TaskRes [66] and ProGrad [74] for five encoders. The results are consistent with the main findings of Section 4 and indicate their generalization for several encoder architectures.In Table 15 . We report the performance of TransCLIP-ZS on top of CoOp [73] , CoCoOp [72] , ProGrad [74] , PromptSRC [30] and MaPLE [29] . We additionally report PromptAlign [1] , which is working on a single image at a time and thus differs from our transductive setting. Note that PromptSRC and MaPLE introduce learnable vision tokens, and are therefore not compatible with convolutional-based encoders. The results are similar to those of Section 4.In Tables 16 and 17 . We extend the results from Table 3 to five encoders. These results support those of Section 4 and show that TransCLIP can improve both zero-and few-shot model generalization for various encoders.In Tables 18, 19 , 20, 21 and 22, we implemented transductive methods from the traditional fewshot literature that align the most with our work in terms of computational efficiency and wide applicability: TIM [5] , LaplacianShot [76] , BD-CSPN [37] , TF [13] , and PT-MAP [24] . Additionally, due to the lack of transductive methods in Vision-Language and to ensure more comprehensive comparisons, we introduce a hybrid method named CoOp+UPL. This method combines prompt learning with both labeled shots and selected pseudo-labels following the methodology of UPL [25] . More details on each method and their validation procedure are outlined below. Methods with tunable hyper-parameters are fine-tuned using the validation split provided with each dataset. In line with other work [48] , validation is performed for each dataset and for every shot number, setting the number of validation shots at min(4, #shots). Hyper-parameters are then optimized through a grid search to maximize accuracy on the validation set. Note that we only search for γ across 4 values for TransCLIP. More details on the grid search for each method is given below. Detailed results for the five architectures studied in this paper are available in Table 18 , 19, 20, 21, 22. Now we describe the implementation details for each reported transductive few-shot methods.Transductive Fine-Tuning. We follow the original implementation of Transductive Fine-Tuning [13] . The authors kept the hyper-parameters fixed for all datasets since the goal was to propose a simple baseline, with a temperature set to 1 and the number of training steps to 25. However, they pointed out possible improvements if the hyper-parameters were tuned for each dataset. Therefore, we search for the optimal temperature value by validation in {0.25, 0.5, 1, 2, 4} and the number of iterations in {10, 15, 20, 25, 30, 35, 40}.BD-CSPN. We follow the original implementation of BD-CSPN [37] . Regarding the hyperparameters, this method generates Z pseudo-labels per class from the query set to augment the support set and to build the K prototype vectors. They also introduce a temperature scaling parameter ε for the computation of the prototype vectors. The authors set Z to 8 and the temperature scaling ε to 10. We search for the value of Z in {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10} and ε in {2.5, 5, 10, 20, 40} by validation.LaplacianShot. We follow the original implementation of LaplacianShot [76] . They balanced the Laplacian regularization term with a factor λ and used k-nearest neighbors consistency.We follow the proposed ranges to find the hyper-parameter values by validation, with λ in {0.1, 0.3, 0.5, 0.7, 0.8, 1, 1.2, 1.5} and the number of neighbors to consider k in {3, 5, 10}.PT-MAP. We follow the original implementation of PT-MAP [24] . In their work, the authors show a small performance sensitivity to the learning rate α used to update the class prototypes through iterative adaptation. Following their discussion, we search α in {0.2, 0.4}.TIM. We follow the original implementation of TIM [5] . The authors proposed two solvers to find the solution to the minimization problem: gradient-descent TIM (TIM-GD) and alternatingdirection method (TIM-ADM). We decide to focus on the second approach since there are fewer hyper-parameters to tune. They set the weighting factors of the cross-entropy, the marginal entropy, and the conditional entropy terms to 0.1, 1 and 0.1, respectively. They also introduced a temperature parameter τ in their classifier and set it to 15. We search for the values of the cross-entropy and the conditional entropy factors in {0.05, 0.1, 0.4, 0.7, 1} and the temperature in {5, 10, 15, 30, 60} by validation.CoOp+UPL. We implement a natural extension of CoOp to include the pseudo-labels proposed by UPL. As in UPL, N = 16 hard pseudo-labels per class are generated according to the prediction's confidence. Pseudo-labels from the query set P ⊆ Q and labeled shots from S are unified into a single learning set S ∪ P. To separate the contribution of the pseudo-labels from the labeled shots, we split the cross-entropy loss function into two terms:EQUATION+(1 -β) 1 |P| j∈P L UPL (V|x j ) , β ∈ [0, 1]Where V denotes the vector of learnable context token embeddings. Despite increased computational needs, we search for the value of β in {0.1, 0.3, 0.5, 0.7, 0.9} by validation for the sake of fairness.The number of epochs, the learning rate and its schedule, the optimizer and the context tokens initialization follow exactly the CoOp implementation.As discussed in Section 4, the gain of TransCLIP-ZS on top of few-shot methods tends to decrease when the number of shots is high (e.g., 16 shots) and future works may investigate this aspect.Secondly, as TransCLIP's performance relies greatly on its text-regularization term, TransCLIP is subject to some biases. One notable bias pertains to the quality of text embeddings within each class.Recent literature has highlighted that these embeddings exhibit a preference for more frequently occurring concepts [57] . However, this issue may be mitigated through our proposed few-shot extension (e.g., introducing labels for more challenging classes).. Justification: Claim i) from the abstract is discussed and supported by results in Tables 1, 2 and 3, and claim (ii) is discussed and supported by results in 4. The contributions announced in the introduction: (i) is presented in Section 3, (ii) is supported by Tables 1, 2 and 3, and (iii) is supported by Table 4 . More results are also available in the Appendix for five encoders.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: There is no such risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All models and methods are credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.In VLMs, such as CLIP[50], both visual and text embeddings are normalized (i.e., are withing the unit hyper-sphere). Thus, the cosine similarity corresponds to the dot product.Note that each CLIP version comes with a temperature scaling factor τ , which is optimized along with the learnable parameters during pre-training.Notice that, for λ = 1, the expression in (3) corresponds to the KL divergence.This makes the overall sub-problem non-convex and there is no closed-form solution.''Tight" means that the upper bound is equal to the original objective at the current solution z l .This application is hardly discussed in the transductive literature. We make all zero-shot-and few-shot text and image embeddings publicly available, to ease future works without resorting to heavy computations.For the zero-shot setting, we use the embedding of top-8 most confident initial predictions for each class as explained in Section 4. 


Decision: The paper targets a new setting based on transductive learning for CLIP-based models. The paper received five reviews in total and all reviewers voted acceptance (borderline accept or above). The paper is strong in several notable aspects: it is well-written, well-structured, and easy-to-follow; it fills a gap in the literature as this topic is under-studied; the method achieved big improvement over prior SOTA and could be very useful for downstream applications of CLIP; the experiments are comprehensive. In terms of weaknesses, the reviewers pointed out that some concepts are ambiguous and more discussions need to be added for test-time methods. The rebuttal has well addressed most concerns raised by the reviewers. Given the novelty and strong results of the paper, the AC recommends acceptance.

Reviews: review: paper_summary: paper_summary: In this paper, the authors explore fine-tuning VLMs to specific unlabelled/partially-labelled datasets in a transduction setting. The authors propose an objective function to carry out joint inference of labels for all the test samples simultaneously. The authors then propose an iterative block Majorie minimize procedure to arrive at a local optimum. Empirical results on multiple datasets show clear superiority of the proposed method over other prompt-based fine-tuning and inductive zero-shot and few-shot methods.
strength_weakness: strength_weakness: Some aspects of the writing should be made more clear.

1. Is Eq 2 did you mean zi^Tz_j since both are vectors? 

2. How is the final prediction done, for the ith sample? Is it the argmax of z_i?

3. Design choice: The experiments show great results, but is there apriori any expectation for using a GMM model for the classes? Perhaps this is related to the linear manifold hypothesis for these VLMs and LLMs. Some more discussion on this will be good. As it is written it comes as a random choice which seems to work well with no intuition behind it.

4. Line 167. why is the max function needed for W to be PSD? Even without the max operator, W will be a gram matrix and thus PSD. Am I missing something here?

5. Why does the laplacian regularization enforce visually similar points to have the same assignment? I understand this when z is one-hot, but here z is any vector in the simplex. Is this still true when z belongs to the simplex?

6. One of the baselines used in the paper is a modification of UPL, but this modification is not clearly mentioned in the paper or the appendix. How is the modification done in the transduction setting? Also a small description explaining UPL would be good in the appendix for added context to the readers.
questions: questions: See weaknesses above.
limitations: limitations: Authors have adequately discussed limitations in the paper.
paper_summary: paper_summary: The paper proposes a method named TransCLIP that performs transductive inference to boost classification performance of Zero-Shot &  Pre-trained Few-Shot CLIP models. The proposed methodology is an extension of [1], but for VLMs. TransCLIP proposes to learn the class prototypes, in contrast to fixed-prototypes considered in [1] and adds a language guidance term to penalize predictions that stray from the CLIP pseudo-label. Following [1], the paper uses a Majorize-Minimize framework to solve for the prototypes and class predictions in closed form. Experiments are shown on 11 standard datasets, showing that TransCLIP improves on top of existing few-shot fine-tuning methods.

[1] Imtiaz Ziko, Jose Dolz, Eric Granger, and Ismail Ben Ayed. Laplacian regularized few-shot
learning. In International conference on machine learning, pages 11660–11670. PMLR, 2020.  
[2] Ma, Xiaosong et al. “SwapPrompt: Test-Time Prompt Adaptation for Vision-Language Models.” Neural Information Processing Systems (2023).  
[3] Shu Manli et al. “Test-time prompt tuning for zero-shot generalization in vision-language models.” Neural Information Processing Systems (2022).
strength_weakness: strength_weakness: \- Discussion on Test-Time methods. Recently many test time methods (a type of transductive learning) have been proposed to improve VLM performance. A discussion of the tradeoffs between the proposed approach and existing test-time adaptation (TTA) methods needs to be discussed. For instance, TransCLIP requires the entire test batch, while TTA methods require fewer test samples [2,3].  
\- Issues with missed references. Important references are missing in Section 3.3. The entire section follows similar arguments from [1], but re-derived by adding the language-guidance term.  
\- The GMM argument is unclear to me. However, the GMM clustering term in eq 2 is minimized when label assignments are given by the closest prototype. The loss is reminiscent of $\mathcal{N}$ in [1], but the paper proposes to use a Mahalanobis distance instead of euclidean distance. I am unsure of the insight provided by posing it as GMM clustering instead of drawing parallels with [1].
questions: questions: 1. As discussed above, the GMM angle is unclear, but maybe it boils down to learning class prototypes in the absence of few-shot data.  
2. Existing test-time methods show strong empirical evidence of the power of transduction for VLMs. A discussion of the scope of these methods is necessary.  
3. Since the primary novelty of the paper is to suggest that language-guidance can greatly improve transduction, the appropriate ablation is necessary. Setting $\lambda=0$ in the proposed KLDiv parametrization only removes the cross-entropy term, and is not the desired ablation.
limitations: limitations: The authors have addressed some limitations of the work. There is no potential negative societal impact from this work
paper_summary: paper_summary: The paper proposes a transductive method to boost the performance of existing vision-language models by assuming that all unlabeled test samples are available during the training stage. Specifically, the paper models a Gaussian mixture model, where each class is represented by a Gaussian distribution. The method refines predictions for each test sample by constraining the similarity between predictions of similar points and the similarity between the final prediction and the prediction based on text features. Experiments are conducted on standard datasets.
strength_weakness: strength_weakness: 1. The paper does not discuss a significant limitation of the proposed method. Specifically, when the test dataset arrives online, one by one, the method may not be able to make predictions for individual data points immediately. Instead, it may require collecting a sufficient number of data points before training the model and making predictions. Furthermore, if a new data point arrives after the model has been trained, does the model need to be retrained to make a prediction for the new point? Is there an efficient way to handle new data points?
2. Although the authors provide an analysis of training times in Table 5, the testing speed is more critical for users. An analysis of the time complexity during testing is also needed.
3. Considering that the method requires using all unlabeled test data for training, it resembles unsupervised learning (e.g., [25]). It is recommended to discuss the differences between the proposed method and existing unsupervised learning methods in detail.
questions: questions: See weaknesses.
limitations: limitations: Copy from Q1: The paper does not discuss a significant limitation of the proposed method. Specifically, when the test dataset arrives online, one by one, the method may not be able to make predictions for individual data points immediately. Instead, it may require collecting a sufficient number of data points before training the model and making predictions. Furthermore, if a new data point arrives after the model has been trained, does the model need to be retrained to make a prediction for the new point? Is there an efficient way to handle new data points?
paper_summary: paper_summary: The paper introduces TransCLIP, a transductive learning method designed to enhance the performance of vision-language models (VLMs) for zero-shot and few-shot learning scenarios. By incorporating a novel objective function constrained by Kullback-Leibler divergence, TransCLIP not only improves prediction accuracy but also ensures computational efficiency. This method operates as a plug-and-play module on existing VLMs, leveraging unlabeled data to boost the model's predictive capabilities significantly across various datasets.
strength_weakness: strength_weakness: The proposed objective function of TransCLIP is composed of multiple terms, including a Gaussian Mixture Model-clustering term, a Laplacian regularizer, and a Kullback-Leibler divergence penalty. While this multifaceted approach aims to integrate various aspects of the data, it raises concerns about potential conflicts or trade-offs between these terms. The interaction and balance among these components are crucial, as overemphasis on one could undermine the effectiveness of others, potentially leading to suboptimal learning outcomes.

Although the paper addresses computational efficiency through an iterative Block Majorize-Minimize (BMM) optimization procedure, a detailed comparison of computation costs with other methodologies is lacking. Understanding how TransCLIP's computational demands stack up against alternative approaches, particularly in terms of time complexity and resource usage, is essential for evaluating its practical applicability and efficiency.

The scalability and performance of TransCLIP when applied to very large datasets remain uncertain. While the method is claimed to be computationally efficient, the real-world effectiveness and efficiency on datasets significantly larger than those tested (like datasets beyond the scale of ImageNet) need thorough investigation. This includes an assessment of whether the benefits observed on smaller or moderate-sized datasets consistently translate to much larger scales.

The paper provides theoretical convergence guarantees for the optimization procedure. However, the robustness of these proofs and their assumptions in practical, real-world scenarios could be questioned. Specifically, the conditions under which convergence is guaranteed should be scrutinized to ensure they are not overly restrictive or detached from practical applications. This scrutiny is vital to validate the method's theoretical foundation and to ensure its reliability across various deployment contexts.
questions: questions: See weaknesses section.
limitations: limitations: See weaknesses section.
paper_summary: paper_summary: Summary:

The paper under review investigates the integration of contrastive vision-language pretraining (CLIP) with transductive learning methodologies. The authors are motivated by the efficacy of transductive learning in utilizing unlabeled data to enhance the performance of conventional supervised learning frameworks. They propose to extend this approach to the domain of vision-language pretraining. The key innovation lies in modeling the distribution of language data using a Gaussian mixture model (GMM). The authors introduce three specific learning objectives: GMM clustering, Laplacian regularization, and text knowledge preservation. These objectives collectively aim to regularize the distribution of unlabeled data while maintaining the integrity of text-guided vision-language pretraining.
The complexity of optimizing three intertwined variables necessitates an advanced optimization strategy. To address this, the authors propose the Block Majority-Minimize optimization technique, which iteratively fixes two variables while updating the third. This approach ensures the overall minimization of all three variables, providing a robust convergence guarantee. Through comprehensive quantitative experiments, the authors demonstrate that their proposed method can serve as a versatile framework, enhancing the performance of various backbone methods across different scenarios.
strength_weakness: strength_weakness: Weaknesses:

- Unclear Motivation:

The motivation for leveraging transductive learning in this context is somewhat ambiguous. The assertion that transductive learning can enhance the handling of unlabeled data, while valid in general, seems less compelling here. In the CLIP framework, vision and language data are inherently paired and labeled, diminishing the applicability of transductive learning, which traditionally targets unlabeled data.

- Mismatch with Zero-Shot Learning:

There is a conceptual misalignment between transductive learning and zero-shot learning paradigms. Transductive learning focuses on knowledge propagation from observed training data to observed test data. In contrast, zero-shot learning aims to generalize to unseen test data. This fundamental difference makes the application of transductive learning to zero-shot scenarios appear somewhat forced and unconvincing.

- Lack of Intuitive Justification:

The introduction of the GMM and Laplacian terms lacks intuitive explanation. While the quantitative results are impressive, there is an absence of qualitative analysis to elucidate why these terms specifically enhance learning performance. A deeper exploration of the underlying reasons for their effectiveness would strengthen the paper's contributions.

- Computational Overhead:

The integration of GMM and Laplacian regularization is computationally intensive. The paper would benefit from experimental validation of the method's efficiency, demonstrating that the performance gains justify the additional computational costs. Providing benchmarks or comparative analyses regarding computational efficiency would address potential concerns about scalability and practicality.

Overall Assessment:

In conclusion, this paper presents a novel approach to enhancing vision-language pretraining by incorporating transductive learning principles. Despite some motivational and conceptual ambiguities, the paper's methodological rigor, comprehensive experiments, and theoretical contributions make it a valuable addition to the field. Addressing the highlighted weaknesses through additional qualitative analyses and computational efficiency evaluations would further solidify the paper's impact and applicability.
questions: questions: Please see the weaknesses part.
limitations: limitations: The authors have discussed the limitation in the paper.

Comments: comment: Thank you for the clarifications. Since my concerns have been addressed, I have increased my score to 6. I recommend incorporating the discussions on test-time methods into the main paper.
Thanks for providing further comments to address my concerns. No further concerns remained from my side, so I would like to keep my current score.
Thanks for your response. Most of my concerns have been addressed. Considering the limitations in the online setting, I slightly increased my score to 5.