Towards Robustness Prompt Tuning with Fully Test-Time Adaptation for CLIP's Zero-Shot Generalization

Authors: ran  wang
zhen  fang
jie  lu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 08:58:37

Abstract: In the field of Vision-Language Models (VLM), the Contrastive Language-Image Pretraining (CLIP) model has yielded outstanding performance on many downstream tasks through prompt tuning. By integrating image and text representations, CLIP exhibits zero-shot generalization capabilities on unseen data. However, when new categories and distribution shifts occur, the pretrained text embeddings in CLIP may not align well with unseen images, potentially leading to a decrease in CLIP's zero-shot generalization performance. To address this issue, many existing methods use test samples to update the CLIP model during testing through a process known as Test-Time Adaptation (TTA). Previous TTA techniques, such as image augmentation, can lead to overfitting given outlying samples, while methods based on teacher-student distillation can increase memory use. Further, these methods significantly increase inference time, which is a crucial factor in the testing phase. To improve robustness, mitigate overfitting, and reduce bias toward outlying samples, we propose a novel method: Self-Text Distillation with Conjugate Pseudo-labels (SCP), designed to enhance CLIP's zero-shot generalization. SCP uses gradient information from conjugate pseudo-labels to enhance the model’s robustness toward distribution shifts. It also innovates by using a fixed prompt list to distil learnable prompts from within the same model, acting as a self-regulation mechanism that minimizes overfitting. Additionally, SCP is a fully test-time adaptation method that does not require retraining. It directly improves CLIP's zero-shot generalization at test time without increasing either memory overheads or inference time. In fact, in evaluations across three zero-shot generalization scenarios, SCP surpasses existing state-of-the-art methods in performance and significantly reduces inference time.

Main paper: 
 introduction: 
 Within the field of large-scale vision-language models [11, 16] , the Contrastive Language-Image Pre-training model (CLIP [29] ) has demonstrated superior performance over traditional convolutional neural network models across a range of tasks, including few-shot classification and zero-shot generalization tasks [32] . The main idea of CLIP is an innovative integration of textual data with visual input through a dual-encoder framework. As such, CLIP consists of two encoders: an image encoder and a text encoder. It generates image features and text features to match a vast corpus of imagetext pairs through a contrastive learning scheme. By this method, CLIP efficiently aligns visual and textual representations, allowing the model to interpret the visual content of images and to capture their associated textual representations. This distinctive strategy means CLIP is able to capture the intricate dynamics between visual and textual elements, significantly broadening its applicability and efficacy across diverse scenarios [46] .Recent research has integrated the concept of prompts into CLIP's text encoder. Taken from the domain of natural language processing (NLP [6] ) this approach, known as prompt tuning, uses embedded prompts [21] , such as "a photo of a" to let CLIP swiftly adapt to downstream tasks without the need to fine-tune the encoder. As a result, computation time is significantly reduced [34] . Further, to avoid the time-consuming task of manually crafting fixed prompts, analysts can use learnable prompts that dynamically change depending on the requirements of the task [49, 50] .This adaptability means CLIP can more effectively leverage its pretrained knowledge, optimizing performance across a diverse range of tasks with minimal additional computational overhead.However, while learnable prompt tuning enhances the capabilities of CLIP, this model still faces challenges with zero-shot generalization not only due to distribution shifts within unseen test data but also when confronted with entirely new labels or categories [44] . More specifically, images can contain drastically different content with huge variations in features. As an example, consider the differences between a sunny park and a snowy mountain landscape. Not only will the two images contain very different backgrounds, but unusual animal species and culturallyspecific items will likely be encountered as well. These conditions represent both new category and new distribution shifts that can lead to a misalignment of image-text pairs. Overall, the result for CLIP will be diminished zero-shot generalization performance in new scenarios [42] . To address this challenge, previous methods have leveraged test data, using them to update the model during test-time through a process known as test-time adaptation (TTA).While existing test-time adaptation methods can mitigate the effects of distribution shifts by using information from test data, these methods face challenges of their own. For example, most updates during test time depend on pseudo-labeling [24, 37] , but pseudolabeling can lead to a bias towards outlier samples when the distribution changes, diminishing the model's performance in changing environments. A more reliable method is image augmentation. This approach aims to minimize marginal entropy to achieve more certain results [34, 47] . However, its lack of adequate regularization can bias outcomes and lead to overfitting. Conversely, the teacherstudent distillation method [8, 40] generally yields smoother results through distillation. The downside is that this strategy does increase memory use, particularly with large-scale vision-language models. It also substantially increases the inference time, both of which are critical considerations during testing.To address these issues, we introduce a novel method: Self text distillation with Conjugate Pseudo-labels (SCP) for CLIP's zeroshot generalization. SCP employs conjugate pseudo-label to reduce the bias on outlier samples and self-text distillation to reduce overfitting. Specifically, instead of using only entropy information as is the case with traditional pseudo-labels, SCP uses conjugate pseudolabels which are optimized based on both gradient and entropy information. This steers the model toward optimization, while reducing any bias toward outlying samples in the face of distribution shifts. The overall result is increased robustness. Further, to reduce overfitting, SCP implements self-text distillation as a form of self-regularization. It uses teacher-student distillation within the same frozen CLIP text encoder, which uses less memory than loading two separate CLIP models. The learnable prompt mechanism acts as the student model, and a fixed prompt list serves as the teacher model. This distillation process imposes constraints on the learnable prompt, preventing it from deviating too far during updates. Through self-text distillation, the process not only maintains low memory use but also reduces inference time during the testing phase. Additionally, SCP involves a fully test-time adaptation method, improving robustness directly during the testing phase without the need for retraining.Overall, the contributions of our work can be summarized as:• Our method employs conjugate pseudo-labels at test-time, using gradient information to effectively mitigate bias toward outlying samples while improving robustness to distribution shifts. • Our method implements a novel self-text distillation strategy within the same model, acting as self-regularization to reduce overfitting. • Our method is a fully test-time adaptation method that eliminates the need for retraining and directly improves the zero-shot generalization of CLIP by reducing both memory use and inference time. 
related works: 
 In this section, we present the study of vision-langauge models with prompt tuning and the methods of test-time adaptation.Vision-Language Pre-trained Models [22, 33] (VLMs) serve as a bridge to unify the understanding of visual and textual data, achieving this through extensive pre-training that integrates both image and text inputs. Of these models, the CLIP model [29] stands out for its robust performance across a variety of tasks, including fewshot classification, zero-shot classification, and generalization tasks. CLIP is equipped with separate encoders for text and images and learns via self-supervised techniques using contrastive loss. It essentially aligns the representations of both textual and visual content, allowing the model to "understand" and interpret visual material in tandem with text descriptions. For this reason, CLIP boasts a great deal of utility in a range of settings and has proven to be quite effective.One of the more recent developments in the realm of visionlanguage models is the application of prompt tuning. Prompt tuning emerged from the field of NLP as a means of effectively tailoring large-scale models for downstream tasks [7, 30] . This approach leverages textual prompts to direct the model's attention towards particular tasks, employing prompts as cues for the model to generate task-relevant responses. For instance, CLIP uses prompts such as "a photo of a [object]" to assess the match between an image and possible text descriptions, enhancing its classification capabilities based on the most fitting description.Most early studies involved manually setting prompts. But finding appropriate prompts can be a time-consuming and challenging task. Hence, recent research has focused on enabling the model to learn more suitable prompts independently [18] . For example, the CoOp [50] method was the first approach to introduce learnable prompts for CLIP. The goal was to improve the model's ability to adapt to a task using a small set of training samples. While a significant advancement in theory, the actual mechanism often resulted in a model overfit for specific tasks. To address this, Co-CoOp [49] seeks to maintain CLIP's generalization performance across a range of tasks by applying meta-learning techniques that optimize the variation in prompts. By contrast, PLOT [2] employs optimal transportation to manage multiple prompts. Meanwhile, Maple [17] integrates learnable weights into both the image and text encoders, representing a step forward in enhancing CLIP's learning capabilities. However, what all these methods have in common is that they require the CLIP model to be retrained. Plus, they only provide a modest boost to zero-shot generalization performance. Conversely, our approach with SCP is designed to directly enhance CLIP's zero-shot generalization potential during testing, eliminating the extensive retraining process. SCP focuses on exploiting the pre-trained model's inherent properties, enabling CLIP to effectively adapt to changing data.Test-time adaptation refers to updating the model during the testing phase [26, 37, 47] , as distinct from traditional domain adaptation methods that require data from both the source and target domains [41, 45] . This approach does not need access to source data nor does it require retraining. Instead, it updates the model using information from the test data to mitigate the effects of distribution shift. Previous methods include entropy minimization, image augmentation, teacher-student model distillation, and the use of pseudo-labels. Entropy minimization [26, 37] aims to enhance the certainty of model predictions by minimizing the entropy of the test data, but it may lead to overfitting on specific high certainty test samples. Image augmentation techniques employ extensive augmentations to minimize marginal entropy [34, 47] . This approach seeks more stable results at the cost of an increased computational overhead and reduced inference speed, especially with larger-scale vision-language models. Teacher-student model distillation updates two models simultaneously to mitigate overfitting [8, 40] . But, again, this significantly increases the demand on computational memory. Pseudo-labeling [35, 39] , which is the method SCP uses, involves updating the model with predictions as pseudo-labels. However, inaccurate pseudo-labels risk biasing the model toward outlying samples. Hence, to minimize this bias while reducing overfitting and maintaining efficiency, we adopted an innovative self-text distillation method. This approach, which only requires one model that performs both the teacher and the student roles, uses conjugate pseudo-labels to do the optimization based on both gradient and entropy information. 
preliminaries: 
 In this section, we first introduce the structure of CLIP and the settings of the learnable prompt, followed by an explanation of the concepts of zero-shot generalization and test-time adaptation.The CLIP model [29] consists of two encoders, one is an image encoder 𝑓 (•), which is based on either ResNet [12] or ViT [36] , and the other is a text encoder 𝑔(•), which is based on Transformer. CLIP pre-trains the contrast loss between image and text to get a better correspondence between image and text. To apply CLIP to specific downstream image classification tasks, a straightforward and effective method involves embedding text prompts, such as "a photo of a", prior to the text encoder. These prompts, along with the [CLASS] token, are inputted to the text encoder. Consequently, the text encoder of CLIP is guided to generate a semantic representation that aligns with the corresponding input image. The inference process of CLIP can be expressed as:𝑃 ( ŷ = 𝑖 |x) = exp(cos(w 𝑖 , f (x))/𝜏) 𝐾 𝑗=1 exp(cos(w 𝑗 , f (x))/𝜏) (1)where 𝑃 ( ŷ = 𝑖 |x) represents the probability of the input x belonging to class 𝑖, cos(w 𝑖 , f (x)) is the cosine similarity between the class embedding w 𝑖 and the image feature f (x) from image encoder 𝑓 (•), 𝜏 is a temperature parameter, and 𝐾 is the total number of classes.The CoOp framework introduces a novel method that incorporates learnable prompts into the CLIP model while freezing the remaining parameters of the encoders [50] . This is achieved by formulating the prompts provided to the text encoder 𝑔(•) as follows:EQUATIONwhere each [V 𝑚 ], for 𝑚 = 1 to 𝑀, represents a learnable vector with the same dimensionality as the word embeddings used in CLIP.The variable 𝑀 determines the number of contextual tokens within the prompt, and [CLASS] is the class name token.For an image-text pair (x, t), the image feature f (x) is obtained from the image encoder 𝑓 (•), and the text feature g(t) is obtained from the text encoder 𝑔(•). The prediction is then calculated by:EQUATIONIn this expression, for each individual prompt t 𝑖 , the class token [CLASS] is substituted with the respective word embedding vector corresponding to the name of the 𝑖-th class.The concept of zero-shot generalization refers to the ability of a model to correctly categorize new, unseen data [10] . Specifically, let X 𝑆 = {(x 𝑠 , 𝑦 𝑠 )} 𝑁 𝑠 𝑠=1 represent the dataset of seen classes, where x 𝑠 is a feature vector that can be obtained using a pre-trained deep learning model such as ResNet. The labels 𝑦 𝑠 are the corresponding annotations for these feature vectors. Y 𝑆 = {𝑦 𝑠1 , 𝑦 𝑠2 , . . . , 𝑦 𝑠𝐶 𝑠 } indicates the label set of seen classes within the label space Y, where 𝐶 𝑠 is the number of seen classes. Similarly, let X 𝑇 = {(x 𝑡 , 𝑦 𝑡 )} 𝑁 𝑡 𝑡 =1 represent the dataset of unseen classes, with Y 𝑇 = {𝑦 𝑡 1 , 𝑦 𝑡 2 , . . . , 𝑦 𝑡𝐶 𝑡 } indicating the label set of unseen classes, where 𝐶 𝑡 is the number of unseen classes. Y = Y 𝑆 ∪ Y 𝑇 denotes the union of both the seen and unseen classes, ensuring Y 𝑆 ∩ Y 𝑇 = ∅. The objective in zero-shot generalization is to train a model 𝑓 (𝜃 ) on X 𝑆 and apply 𝑓 (𝜃 ) : X 𝑇 → Y 𝑇 to classify the test samples of unseen classes [28, 43] .Test-time Adaptation (TTA) is the process of using test data information to update the model during test time. The aim is to adapt a pre-trained model to a target domain by adjusting the model parameters 𝜃 at test time [3, 26] . This process refines the model's generalization capability, improving performance on samples from different distributions. Fully test-time adaptation avoids the need to retrain the original model [20] and relies solely on using the test data for access to source data. It is particularly advantageous when access to source domain data is limited by privacy or storage considerations. TTA is achieved by minimizing an unsupervised  
methodology: 
 In this section, we first provide an overview of the SCP framework. Then, the two core steps of the SCP algorithm are discussedthese being the conjugate pseudo-labelling process and self-text distillation.Figure 1 provides an overview of the SCP method. SCP is based on a CLIP model, which extracts image and text features and uses these features to construct a similarity matrix that the student model then uses to make predictions. Next, the conjugate pseudo label loss is calculated based on the gradient information and the pseudo-labels generated from the entropy information of the predictions. Teacher predictions are made based on a fixed prompt list, while knowledge is distilled through KL divergence with the student predictions to result in the self-text distillation loss.More specifically, the target data are divided into image patches, which are then input into the image encoder to obtain the image features. Then, learnable prompts generate learnable text patches, which are input into the text encoder to produce text features for the student model. These features are used to calculate the similarity matrix upon which the student predictions are based. The predictions from the student model with entropy values below a set threshold of 𝛼 are then selected as the pseudo-labels. The loss is calculated from both the pseudo-labels and the gradient loss from the gradient information. As such, the overall conjugate pseudo label loss comprises the pseudo label loss and the gradient loss. This approach, which leverages the gradient information between the model's current output and the desired output, ensures that the generated pseudo-labels will more effectively guide the model's learning process.Concurrently, a fixed prompt list generates an assembled, fixedtext patch via the averaging method to generate the teacher model's prediction. Next, the student prediction and the teacher prediction are aligned using KL divergence, with the self-text distillation process aiming to minimize this divergence. This configuration helps the model learn stable knowledge representations from fixed prompts while exploring new or task-specific knowledge through adaptable prompts. Further, the process of self-regulation through text distillation maintains the model's regularity without modifications to the original CLIP architecture and without adding computational overhead.In this approach, CLIP's image and text encoders are pre-trained, denoted as 𝑓 (•) and 𝑔(•), respectively, and both are kept constant during the analysis. Given a specific target image x 𝑡 , the corresponding image and text features are extracted via the learnable prompts described in Equation ( 2). This process is formalized as follows: 𝑓 (x 𝑡 ) = [𝑖 1 , 𝑖 2 , . . . , 𝑖 𝑛 ] and 𝑔(t) = [𝑡 1 , 𝑡 2 , . . . , 𝑡 𝑛 ]. Then, the probability is computed as detailed in Equation ( 3).The next two sections formally describe the process of selecting conjugate pseudo-labels and the self-text distillation in detail.In Fully TTA situations where the model is updated only during the testing phase without access to source data, the accuracy of pseudo-labels is crucial for ensuring the effectiveness of the model. Previous pseudo-label methods only consider the prediction output from the model [24] , which may enhance the original bias of the model, especially when the model is overconfident in its prediction.Therefore, to enhance the reliability of pseudo-labels in fully TTA, we employ the conjugate pseudo-labels technique 𝑦 conjugate 𝑡 . This method considers both gradient information and entropy certainty. It initially involves entropy filtering to identify outputs with lower entropy, leading to more credible predictions from the model. Mathematically, we define the entropy of the model's output probabilities p as:EQUATIONFor each sample x 𝑡 in the target domain X 𝑇 , we calculate the entropy 𝐻 (p(x 𝑡 )) of the model's output. Samples with entropy lower than a threshold 𝛼, i.e., {x 𝑡 |𝐻 (p(x 𝑡 )) < 𝛼 }, are selected for generating more reliable pseudo-labels. The selected pseudolabels 𝑦 pseudo 𝑡 are then used to compute the cross-entropy loss for updating the model with the model's predictions ŷ𝑡,𝑖 :EQUATIONFor the selected pseudo-label samples, we compute the conjugate pseudo-labels based on the gradient of a performance measure with respect to the model's parameters. In the CLIP model, the prediction is based on the cosine similarity between the image features and corresponding text features. The learnable parameters of the model include the prompt embeddings associated with the text input t.Given a pair of image and text (x 𝑡 , t), the cosine similarity in the CLIP model is defined as:EQUATIONwhere f (x 𝑡 ) represents the image features and g(t) represents the text features extracted by the respective encoders in the CLIP model. Consequently, the conjugate pseudo-label for an image x 𝑡 is determined by the gradient of the cosine similarity with respect to the image features f (x 𝑡 ):EQUATIONThis gradient provides direction for the model update, aiming to increase the cosine similarity for the correct image-text pairs, thereby improving the model's accuracy on the target task. The gradient of the cosine similarity with respect to the image features f (x 𝑡 ) is calculated as:EQUATIONThis takes both the alignment and magnitude of the image features into account, directing the updates to not only match the direction of the text features but also to adjust the scale of the image features. Therefore, the conjugate loss is then:EQUATIONwhere 𝑧 𝑖 are the logits. 𝜖 is a scaling factor to adjust the contribution of the similarity term in the loss function. 𝛼 is a hyperparameter to control the influence of the conjugate gradient term in the loss.For the total conjugate pseudo-label method, the parameters are updated using a combination of the cross-entropy loss and gradient loss:L conj = L grad + L ce (10) By applying these steps, our goal is to utilize a more accurate pseudo-label to guide updates in the distribution shift scenario, adapting the pre-trained model to the target domain more effectively.The conjugate pseudo-labels provide some additional update information from the gradient information, optimizing the robustness of the pseudo-labels at test-time. Furthermore, in addition to this process, a self-regularization method also constrains the optimization direction, further enhancing the robustness of the model's generalization capabilities. Existing test-time adaptation methods commonly employ prior knowledge and use two models in a teacher-student model distillation setup. However, for largescale vision-language models, this approach significantly increases memory use and inference time. Minimizing these two overheads is often of critical practical importance in real-world applications.Hence, SCP employs a novel self-text distillation strategy as a self-regularization mechanism. A fixed prompt list in CLIP's text encoder acts as the "teacher" and a learnable prompt mechanism in the same encoder acts as the "student". This method explicitly guides the training trajectory by imposing constraints, with the aim of maximizing mutual consistency between the prompt features and the frozen CLIP features. Ultimately, the goal is to mitigate the problem of overfitting. Formally, given a learnable prompt t, and a set of fixed text templates {𝑇 1 ,𝑇 2 , . . . ,𝑇 𝑛 }, the text embedding process through the CLIP text encoder 𝑔(•) and the computation of fixed text feature embeddings v fixed can be represented as follows:EQUATIONThe probability distributions 𝑝 (•) and 𝑞(•), derived from the feature vectors v and v fixed through a softmax function, are given:EQUATIONwhere w 𝑖 represents the weight vector associated with the 𝑖-th category.The self-text distillation loss, calculated using the KL divergence, is expressed as:L text = ∑︁ 𝑖 𝑝 (𝑖 |v) log 𝑝 (𝑖 |v) 𝑞(𝑖 |v fixed ) (14)The self-text distillation method focuses on aligning the learnable and fixed embeddings across the dataset. Consequently, the calculation of L text , which aims to average the losses, may need to be reconsidered based on the specific context of self-text distillation implementation. This approach not only conserves memory, which would otherwise be consumed by employing a teacher-student model simultaneously, but it also ensures more stable updates to the prompts.Additionally, to enhance prediction stability and further reduce bias towards extreme samples, our methodology integrates a weighted averaging mechanism for updating the learnable prompts. This method accumulates the influence of past prompts, ensuring that recent ones are adjusted in the context of their predecessors. The cumulative weighted prompt is computed as follows:EQUATION)where the weight 𝑤 𝑗 = 𝑒 -(𝑛-( 𝑗+1) ) corresponds to the Gaussianbased weight for the 𝑗-th prompt in the sequence, 𝑛 is the total number of prompts, and T represents the updated prompt after applying the weighted average. In this scheme, the weights are configured to give greater influence to the more recent prompts, achieving a balance between embracing new updates and retaining the historical context, leading to a prompt that evolves smoothly over time and is both stable and robust.In our model, the overall optimization method is:EQUATION)where, 𝜆 is weight coefficient. We provide the algorithm pseudo code in Appendix A.1. 
experiments: 
 In this section, we first present the experimental settings. Then, we provide the results of a comparison on memory usage and the inference time. In addition, we present an ablation study, analyzing the impact of each component of the model.Benchmarks. We conduct zero-shot generalization evaluations on three benchmarks• Wild world distribution shift [26] This evaluation is designed to assess the model's robustness to changes in wild-world distributions using 15 differently corrupted ImageNet variants. These are considered to be indistinguishable from real-world scenarios. • Cross-domain generalization [29] . This evaluation assesses the generalization of the model to changes in fine-grained categorical datasets on 10 different categorical datasets. • Natural distribution shift [34] This evaluation assesses the model's robustness to natural distribution shifts on four ImageNet variants. These data are considered out-of-distribution data for ImageNet.Baselines. We compare the SCP with various test-time adaptation methods, all of which are based on comparisons using the CLIP model to ensure fairness. Each method qualifies as a full test-time adaptation approach, with no access to source data and no retraining involved and updates are only made at test time. We strictly adhered to the settings of these methods, reporting all results based on five runs. The comparison included:• TENT [37] , the first method to employ test-time entropy minimization. • Pseudo-label [24] , which updates the model using generated pseudo-labels.• MEMO [47] , the inaugural image augmentation method for testtime adaptation. • CoTTA [40] , which ustilizes a teacher-student model with image augmentation.• RMT [8] , a robust teacher-student model employing symmetric cross-entropy.• SAR [26] , a sharpness-aware and reliable entropy minimization method.• TPT [34] , the first test-time adaptation method integrating CLIP prompt tuning with minimization of marginal entropy through image enhancement.Datasets. For the wild world distribution shift benchmark [26] , we evaluate on the ImageNet-C dataset [15] , which encompasses 15 types of image corruptions, including noise (Gaussian, shot, impulse), blur (defocus, glass, motion, zoom), weather (snow, frost, fog), and digital effects (brightness, contrast, elastic transformation, pixelation, JPEG compression). Subsequently, for the cross-domain generalization benchmark [29] , we assess the performance on 10 distinct datasets. They include OxfordPets [27] , StanfordCars [19] , Caltech101 [9] , DTD [4] , EuroSAT [13] , FGVCAircraft [23] , Flow-ers102 [25] , Food101 [1] . For the natural distribution shift benchmark [34] , we evaluate on Imagenet [5] , ImageNet-A [48] , ImageNet-R [14] , ImageNet-Sketch [38] , and ImageNetV2 [31] .Implementation Details. We test baselines following the experimental setups and implementation of TENT, Pseudo Label, MEMO, CoTTA, RMT, SAR and TPT. For our method, SCP, we initiate with the 4-token prompt "a photo of a" and selected samples with entropy less than 1 for pseudo-label, setting 𝜆 1 = 10, 𝜆 2 = 1000, and 𝜆 3 = 5. For cross-domain evaluations, we set the batch size to 1. For the other two benchmarks, the batch size is set to 100. The learning rate is 0.0025. The length of fixed prompt list is 10 prompts.In this section, we present the experimental results for the three benchmarks. Addtionally, we provide the evaluations of memory usage and inference time. For all benchmark tests, we compare each method separately based on the CLIP's ResNet50 and ViT-B/16 models. All methods require no retraining, with updates conducted only during testing.Wild world distribution shift. In Table 1 , we compare the performance of the various methods in 15 ImageNet corruption scenarios. When using the ResNet50 and ViT-B/16 models, the results from TENT and pseudo label suggest that relying solely on entropy minimization and pseudo-labels may reduce CLIP's generalization performance. Although TPT may show some improvement, it relies on image augmentation based on noise transformation, which may limit its effectiveness in corruption scenarios. However, our method, SCP, outperforms all other methods in terms of average accuracy and demonstrates improvements in almost all datasets. This is because SCP utilizes gradient information to obtain more accurate pseudo-labels and incorporates self-regulation to address distribution shifts.Cross-domain generalization. In Table 2 , we evaluate the generalization performance of various models across 10 cross-domain datasets. These datasets feature high-resolution images with a wide range of unseen categories, from animals to vehicles such as cars and airplanes. When using the ResNet50 and ViT-B/16 models, entropy minimization methods, such as TENT and SAR, become unstable. They solely rely on certainty, but the unseen categories SCP balances efficiency and accuracy, surpassing entropy minimization in speed and maintaining high accuracy, indicating effective management of distribution shifts with lower computational demands. For detailed results, please refer to the Appendix A.3.In this section, we discuss the ablation study, focusing on the impact of self-text distillation and conjugate pseudo-labels, as well as the hyperparameters, including 𝜆1, 𝜆2, 𝜆3. In addition, we compare the experiments on prompt length, averaging methods, fixed prompt list length and threshold 𝛼. Table 4 presents the contribution of the self-text distillation (L text ) and conjugate pseudo-labels (L conj ) techniques toward enhancing the CLIP model's performance. As observed, applying conjugate pseudo-labels improves the average accuracy, which shows the robustness of using both the gradients and entropy to optimize the direction of the model updates. Next, integrating the self-text distillation further increases the average accuracy. This shows that consistency in the model is instrumental in boosting its generalization capabilities. Notably, in all scenarios, the combination of the two techniques yielded the highest average accuracy. Hence, we can conclude that this combined strategy does enhance the model's ability to generalize across diverse distribution shifts.In the ablation study presented in Figure 2 , we explore the impact of different hyperparameters on the performance of SCP. For 𝜆 2 set to 100, we observe that an increase in 𝜆 1 tends to initially improve accuracy but then shows a decrease, suggesting a sweet spot for the trade-off between these two loss components. As 𝜆 2 increases to 1000, the pattern becomes less clear, potentially indicating a different dynamic at higher scales of gradient loss. Furthermore, varying 𝜆 3 , which controls the self-text distillation KL divergence, shows a non-monotonic relationship with accuracy.Wild Besides, for Prompt token length, a prompt length of 4 tokens provides the highest accuracy. Regarding Averaging methods, Gaussian weighting methods outperform both Exponential Moving Average (EMA) and Basic Moving Average (BMA). The threshold 𝛼, equal to 1, provides the highest performance. For a more detailed results, please refer to Appendix A.2. 
conclusion and future work: 
 In this study, we introduce a self-text distillation with conjugate pseudo-labels method (SCP) specifically designed for CLIP, aimed at enhancing its zero-shot generalization capability when dealing with changes in the distribution through test-time updates. SCP significantly enhances CLIP's zero-shot generalization performance across wild world distribution shifts, natural distribution shifts, and cross-domain generalization, while also accelerating inference speed and effectively mitigates the overfitting issues associated with previous test-time adaptation methods.However, the current research focuses solely on individual datasets and a single type of distribution shift. In the future, we aim to investigate the impacts under continuous distribution changes and mixed distributions to further stabilize and robustify CLIP's zeroshot generalization. 
 
 Appendix:  


Decision: This submission focuses on improving the Contrastive Language-Image Pretraining (CLIP) model’s zero-shot generalization capabilities, which are challenged by new categories and distribution shifts. The proposed method, Self-Text Distillation with Conjugate Pseudo-labels (SCP), enhances CLIP’s robustness during test-time adaptation, outperforming existing methods without increasing memory overhead or inference time.

The initial comments are positive, with three borderline accepts. Reviewer 5htV requests clarification on SCP’s dependence on CLIP and addressing overfitting. Reviewer WHP5 raises concerns about computational cost and hyperparameters. Another reviewer asks for clarifications on sensitivity analysis and the mechanism of conjugate pseudo-labels. All reviewers acknowledge SCP’s simplicity and good performance. 

After carefully reviewing the paper, comments, and rebuttal, AC finds that most concerns have been addressed, except for the computational cost of gradients and threshold choices. AC tends to give an acceptance recommendation. Please prepare for the camera-ready version carefully and ensure the suggested changes are incorporated into the revision. This recommendation has been discussed with SAC.

Reviews: review: paper_summary: paper_summary: This paper addresses enhancing the zero-shot generalization capabilities of Vision-Language Models (VLMs), specifically the CLIP model, through a novel method called Self-Text Distillation with Conjugate Pseudo-labels (SCP). This method aims to improve robustness against distribution shifts at test time without retraining, focusing on reducing memory overhead and inference time.
strength_weakness: strength_weakness: - It seems that SCP outperforms existing state-of-the-art methods in zero-shot generalization scenarios across several benchmarks.
- Unlike other test-time adaptation techniques that increase inference time and memory usage, SCP is designed to be lightweight and fast
limitations: limitations: - It would be great if the authors could elaborate on potential trade-offs or limitations associated with using SCP.
- The method is dependent on CLIP, which may limit its applicability to other VLMs without substantial modifications. Can the principles of SCP be adapted for other types of neural network models or architectures beyond CLIP and VLMs?
- While SCP aims to reduce overfitting through self-regulation, the discussions on how it prevents overfitting are not sufficient to me.
paper_summary: paper_summary: The paper, "Towards Robustness Prompt Tuning with Fully Test-Time Adaptation for CLIP's Zero-Shot Generalization", presents a novel method called Self-Text Distillation with Conjugate Pseudo-labels (SCP). This method is designed to improve the zero-shot generalization capabilities of the CLIP (Contrastive Language-Image Pre-training) model by enhancing its robustness against distribution shifts and new categories encountered during test time. The SCP method employs conjugate pseudo-labels to mitigate bias toward outlying samples and uses self-text distillation to reduce overfitting. The key advantage of SCP is that it operates fully at test time, eliminating the need for retraining and ensuring minimal memory and inference time overheads.
strength_weakness: strength_weakness: 1. SCP improves the zero-shot generalization performance without requiring retraining. This is highly beneficial in practical applications where retraining is not feasible due to time or computational constraints.
2. The paper includes a thorough evaluation across three benchmarks: wild world distribution shifts, cross-domain generalization, and natural distribution shifts. The results demonstrate that SCP outperforms existing state-of-the-art methods in both accuracy and inference speed.
3. SCP maintains low memory usage and inference time compared to other methods, making it practical for real-world applications.
limitations: limitations: 1. The computational cost of calculating and usng gradient information for conjugate pseudo-labels might be non-trivial for larger models and datasets, potentially offsetting the benefits of reduced memory and inference time.
2. The method involves several hyperparameters, which may require fine-tuning for different tasks and datasets. This can be a tedious process and might limit the generalizability of the approach.
paper_summary: paper_summary: This paper proposes a method to improve zero-shot CLIP’s zero-shot generalization, called Self-text distillation with Conjugate Pseudo-labels (SCP). SCP employs conjugate pseudo-label at test-time, using gradient information to reduce the bias on outlier samples. And SCP includes a self-text distillation strategy within the same model, to reduce over-fitting.
strength_weakness: strength_weakness: 1， The paper writing is clear and easy to understand.

2， Using self-distillation to avoid memory burden is applicable.

3， The proposed method is a fully test-time adaptation method that can eliminate the need for retraining.
limitations: limitations: 1，The improvement seems to be limited. Especially for dataset zoom, Impul, DTD etc.

2， How to choose threshold? Sensitivity analysis.

3, Why $L_{conj}$ brings negative impact on improvement for some dataset, as shown in Tab 4.

4, It is not clear to me SCP employs conjugate pseudo-label to reduce the bias on outlier samples.