Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models

Authors: ce  zhang
simon  stepputtis
katia  sycara
yaqi  xie
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 09:16:25

Abstract: Test-time adaptation, which enables models to generalize to diverse data with unlabeled test samples, holds significant value in real-world scenarios. Recently, researchers have applied this setting to advanced pre-trained vision-language models (VLMs), developing approaches such as test-time prompt tuning to further extend their practical applicability. However, these methods typically focus solely on adapting VLMs from a single modality and fail to accumulate task-specific knowledge as more samples are processed. To address this, we introduce Dual Prototype Evolving (DPE), a novel test-time adaptation approach for VLMs that effectively accumulates task-specific knowledge from multi-modalities. Specifically, we create and evolve two sets of prototypes—textual and visual—to progressively capture more accurate multi-modal representations for target classes during test time. Moreover, to promote consistent multi-modal representations, we introduce and optimize learnable residuals for each test sample to align the prototypes from both modalities. Extensive experimental results on 15 benchmark datasets demonstrate that our proposed DPE consistently outperforms previous state-of-the-art methods while also exhibiting competitive computational efficiency.

Main paper: 
 introduction: 
 Although deep learning models have achieved great success in various machine learning tasks [47, 48] , they often suffer from significant performance degradation due to distribution shifts between the training data from the source domain and the testing data from the target domain [33, 63, 14] . To address this challenge, a number of works [21, 57, 64] adopt the transductive learning principle, assuming access to both labeled source data and unlabeled target data-a scenario known as the domain adaptation setting. However, this setting contrasts with most practical scenarios, where we only have access to a well-trained model and cannot re-access the source data due to privacy or data retention policies. In response, researchers have proposed test-time adaptation, which leverages only the unlabeled target data stream to adapt the model to out-of-distribution domains [78, 59, 61] .Recently, large-scale vision-language models (VLMs), such as CLIP [45] and ALIGN [24] , have garnered increasing attention in the research community. These models, pre-trained on massive webscale datasets, exhibit remarkable zero-shot capabilities and open-world visual understanding [45, 69, 73, 31] . While the large-scale pre-trained (source) datasets like LAION-5B [49] are accessible, it is impractical for individuals to train on them due to their immense size. Consequently, adapting VLMs to downstream tasks via efficient fine-tuning with limited annotated samples from the target domain has become a focus of recent research [84, 83, 80, 70] . However, although these methods have proven effective, they pose a significant limitation: they assume the availability of annotated samples from the target domain, which is often not practical in real-world scenarios. This constraint hinders the broader deployment of VLMs in diverse and dynamic environments [19, 20, 46, 74] .Figure 1 : Comparison of our DPE with zero-shot CLIP [45] , TPT [53] , and TDA [26] . We denote CLIP's parallel textual and visual encoders as E t and E v , respectively. While previous methods solely adapt the CLIP model from a single modality, we design our DPE to evolve prototypes from both textual and visual modalities to progressively capture more accurate multi-modal representations for target classes during test time.To address the label scarcity problem in practice, a number of approaches apply the test-time adaptation setting to the domain of adapting VLMs to downstream tasks, as shown in Figure 1 . Specifically, Shu et al. [53] propose test-time prompt tuning to learn an adaptive prompt for each individual sample in the test data stream to enhance CLIP's zero-shot generalizability to out-of-distribution domains. Building on TPT, DiffTPT [12] incorporates diffusion-based data augmentations to facilitate more effective prompt tuning during test time. More recently, Karmanov et al. [26] propose an alternative training-free dynamic adapter approach to establish dynamic visual caches with the unlabeled test samples.However, we recognize that existing works overlook the following inherent properties of test-time adaptation in VLMs: (1) Cumulative. We expect that with more seen samples, the performance should improve as task-specific knowledge accumulates [39, 56] . However, test-time prompt tuning methods [53, 12] treat each test instance independently, resetting to the original model for each new sample, failing to extract historical knowledge from previous test samples. (2) Multi-modal. Effective adaptation of VLMs benefits from leveraging knowledge from both textual and visual modalities [27, 34] . However, previous works only capture domain-specific knowledge from a single modality, adapting CLIP based solely on textual [53, 12] or visual [26] feature refinement.To this end, we propose Dual Prototype Evolving (DPE), a novel test-time VLM adaptation approach that effectively accumulates task-specific knowledge from multi-modalities, as illustrated in Figure 1 . Unlike previous methods that focus on adapting VLMs from a single modality, we create and evolve two sets of prototypes-textual and visual-progressively capturing more accurate multi-modal representations for target classes during test time. To extract historical knowledge from previous test samples, we update these two sets of prototypes online using cumulative average and priority queue strategies, respectively. We further optimize these multi-modal prototypes by introducing learnable residual parameters for each individual test sample to enhance the zero-shot generalization capability of our model. Specifically, rather than solely relying on the entropy minimization objective [61, 78] , our DPE also accounts for the alignment between multi-modal prototypes to ensure consistent multi-modal representations. Notably, our DPE requires only the optimization of multi-modal prototypes in the embedding space during test time, eliminating the need to backpropagate gradients through the textual encoder of CLIP, as required in TPT [53] and DiffTPT [12] .The test-time generalization capabilities of our proposed DPE method are extensively evaluated across 15 diverse recognition datasets in two scenarios: natural distribution shifts and cross-dataset generalization. The experimental results validate the superior performance of our DPE, which achieves an average improvement of 3.55% and 4.30% over the state-of-the-art TPT [53] method in these scenarios. Moreover, our proposed DPE achieves this performance while also exhibiting 5× and over 10× test-time efficiency compared to TPT [53] and DiffTPT [12] , respectively. The contributions of this paper are summarized as follows:• We propose dual prototype evolving (DPE), a novel test-time adaptation method for VLMs that progressively captures more accurate multi-modal representations for target classes during test time.• To promote consistent multi-modal representations, we introduce and optimize learnable residuals for each test sample to align the prototypes across modalities.• Experimental evaluations demonstrate that our DPE consistently outperforms current state-of-theart methods across 15 diverse datasets while maintaining competitive computational efficiency. 
related work: 
 Vision-Language Models. Leveraging vast image-text pairs from the Internet, recent large-scale vision-language models (VLMs), such as CLIP [45] and ALIGN [24] , have shown remarkable and transferable visual knowledge through natural language supervision [77, 10] . These VLMs enable a "pre-train, fine-tune" paradigm for performing downstream visual tasks, such as image recognition [45, 16, 35] , pbject detection [66, 65] , and depth estimation [79, 23, 72] .To effectively transfer VLMs to these downstream tasks, researchers have developed two primary methods for adapting the model with few-shot data: prompt learning methods [84, 83, 27, 50, 85, 5] and adapter-based methods [80, 13, 75, 70, 30] . Specifically, prompt learning methods, such as CoOp [84] and CoCoOp [83] , focus on learning input prompts with few-shot supervision from downstream data. On the other hand, adapter-based methods, like Tip-Adapter [80] and TaskRes [70] , modify the extracted visual or textual representations directly to enhance model performance. However, these approaches often assume the availability of labeled samples from the target domain, which can limit their effectiveness in real-world scenarios. In this work, we address the challenge of test-time adaptation, where the model is required to adapt solely at test time without access to any training samples or ground-truth labels from the target domain. This setting is crucial for real-world deployment, as it allows for robust performance in novel and unseen environments where labeled data cannot be obtained in advance.Test-Time Adaptation. To effectively transfer a model trained on the source domain to the target domain, test-time adaptation methods [61, 78, 59, 3, 58] aim to adjust the model online using a stream of unlabeled test samples. These methods enable the deployment of well-trained models in various out-of-distribution scenarios, thereby enhancing the applicability and reliability of machine learning models in real-world applications [33, 28, 41] . Researchers have applied test-time adaptation techniques successfully across various machine learning tasks, including semantic segmentation [22, 51, 82] , human pose estimation [32, 25] , and image super-resolution [52, 8] .Recently, increasing research efforts have focused on adapting large-scale VLMs during test time [37, 55, 1, 71, 81, 76] . As the seminal work, Shu et al. [53] firstly propose test-time prompt tuning (TPT), which enforces consistency across different augmented views of each test sample. Building on this approach, several subsequent studies have sought to further enhance TPT. For instance, DiffTPT [53] utilizes diffusion-based augmentations to increase the diversity of augmented views, while C-TPT [68] addresses the rise in calibration error during test time prompt tuning. Unlike these approaches, which treat each test sample independently, TDA [26] establishes positive and negative visual caches during test time, enhancing model performance as more samples are processed. Similarly, recent DMN [81] utilizes a dynamic memory to gather information from historical test data. However, these methods solely adapt the model from a single modality perspective, limiting their effectiveness in capturing task-specific knowledge from out-of-distribution domains. Given this, we design DPE to evolve two sets of prototypes from both textual and visual modalities to progressively capture more accurate multi-modal representations for target classes during test time. 
method: 
 We introduce Dual Prototype Evolving (DPE) as illustrated in Figure 2 , to enhance CLIP's zero-shot generalization capabilities across diverse distributions during test time. Unlike previous methods that focus solely on one modality, we design two sets of prototypes, textual and visual, which are progressively updated using the unlabeled test dataset D test .Figure 2 : An overview of our DPE method. We introduce prototypes from both textual and visual modalities and enable prototype-based inference with CLIP. For each test sample, we optimize both prototypes using learnable residual parameters with alignment loss L align and self-entropy loss L aug . These prototypes are also progressively evolved over time to capture more accurate and discriminative multi-modal representations for target classes.Zero-Shot CLIP. CLIP [45] utilizes two pre-trained parallel encoders: a visual encoder E v (•) and a textual encoder E t (•), which embed images and text descriptions into a shared embedding space R d . For a C-class classification task, CLIP performs zero-shot predictions by computing the similarities between the extracted image feature and the C candidate text features, written asEQUATIONwhere X test ∈ D test denotes the input test image, and T c represents the the class-specific description input for class y c . The pairwise similarities sim(•, •) are calculated using cosine similarity, and t represents the temperature parameter in the softmax function.Test-Time Prompt Tuning. To enhance the zero-shot generalizability of CLIP, TPT [53] proposes learning an adaptive prompt using the test stream samples. Specifically, for each test sample X test , TPT generates N augmented views {A n (X test )} N n=1 and averages the top ρ-percentile confident predictions based on an entropy threshold τ to obtain the final prediction:EQUATIONHere, H(p) = -C i=1 p i log p i calculates the self-entropy of the prediction p. The objective of TPT is to optimize the learnable prompt to minimize the self-entropy of the final prediction, i.e., min H(P TPT (X test )).In our DPE method, we construct and iteratively evolve two sets of class-specific prototypes from both visual and textual modalities to achieve a more precise representation of each class over time.Textual Prototype Evolution. In this work, we follow CLIP [45] to use multiple context prompt templates for prompt ensembling. Specifically, for each class c, we generate a total of S text descriptions, denoted as{T (i) c } S i=1. The prototypes of these descriptions in the embedding space are calculated as t c = 1 S i E t (T (i) c ). To further improve the quality of these prototypes over time, we design them to be updated online through a cumulative average with each individual sample X test in the test stream. The update rule is given by:EQUATIONwheret = [t 1 t 2 • • • t C ] ⊤ ∈ R C×dis the online updated prototype set, and t * ∈ R C×d is the optimized textual prototypes for each individual sample X test in Equation (10) . To ensure stable online updates, we set an entropy threshold τ t to filter out low-confidence samples (for which H(P CLIP (X test )) < τ t ) from updating the online prototypes, and maintain a counter k for tracking confident samples.Visual Prototype Evolution. Inspired by TDA [26] , we recognize that the historical image features of test images can also be utilized to enhance CLIP's discrimination capability. Therefore, we design a priority queue strategy to store the top-M image features for each class and symmetrically compute a set of visual prototypes that evolve over time. Note that since we cannot access the labels of the test samples, we assign the image features to the queue according to their predicted pseudo-labels. The priority queue for each class c is initialized as empty, denoted as q c = ∅. As test samples arrive, we store the image features f c and the corresponding self-entropy h c in the priority queue, represented as q c = {(fEQUATIONc )} m . The elements are sorted by self-entropy h(m) c such that h (m) c < h (>m) c. Using this priority queue, the class-specific visual prototype is obtained by:v c = 1 Sc m f (m) c, where S c ≤ M denotes the total number of image features stored in the queue.The priority queues are updated during testing by replacing low-confidence image features with high-confidence ones. Specifically, for each individual test sample X test , we first predict the pseudo-label ℓ and compute the corresponding self-entropy h as:EQUATIONFigure 3 : t-SNE [60] visualizations of the stored image features in the priority queues. With more samples getting in, the selected image features from each class become more clustered, leading to more representative visual prototypes.Then, we consider the following two scenarios to iteratively update the priority queue q ℓ for class ℓ: (1) If the priority queue is not full, we directly add the pair (E v (X test ), h) to the queue; (2) If the priority queue is full and the entropy h of the new sample is lower than the highest entropy value (of the last element) currently in the queue, we replace the highest-entropy element with the new feature and self-entropy (E v (X test ), h). If f is not lower, we discard the new sample and leave the queue unchanged. After each update, we re-sort the priority queue based on the self-entropy values and re-compute the visual prototypesv = [v 1 v 2 • • • v C ] ⊤ ∈ R C×d for all classes.In Figure 3 , we present the t-SNE [60] visualizations of the stored image features in the priority queues (with queue size M = 6) after updating with 1500 samples (left) and 15000 samples (right) on the Food101 [2] dataset. We highlight the stored features from 25 random classes using different colors while marking the others in gray. These visualizations illustrate that our priority queue strategy effectively accumulates high-confidence samples, progressively refining the representativeness of the visual prototypes over time.Prototype-Based Inference. Based on our two sets of multi-modal prototypes {t c } C c=1 and {v c } C c=1 , the final prediction for input X is given byEQUATIONHere, t represents the temperature parameter in the softmax function, ⊤ denotes the matrix transpose, andA(x) = α exp (-β (1 -x))is the affinity function, where α is a balance hyperparameter and β is a sharpness ratio. In Appendix A.3, we conduct a sensitivity analysis of these two hyperparameters to evaluate their impact on the overall performance of DPE.To further improve the zero-shot generalizability of our method, we introduce prototype residual learning, which optimizes multi-modal prototypes for each test sample. Unlike previous prompt tuning approaches [53, 12] that require backpropagating gradients through the text encoder to update input prompts, our method directly updates the prototype sets in the embedding space.Specifically, after being evolved with the last test sample, the dual sets of multi-modal prototypes,denoted as t = [t 1 t 2 • • • t C ] ⊤ ∈ R C×d and v = [v 1 v 2 • • • v C ] ⊤ ∈ R C×d, are considered as the initialization for updating with the current test sample. We further introduce learnable residualparameters t = [ t1 t2 • • • tC ] ⊤ ∈ R C×d and v = [v 1 v2 • • • vC ] ⊤ ∈ R C×d .These parameters are initialized to zero and are used to optimize the prototypes for each given test input X test , denoted asEQUATION)Similar to Equation ( 2), we optimize these residual parameters to promote consistent predictions across a total of N different augmented views of the given test image X test using the unsupervised entropy minimization objective:EQUATIONwhereEQUATIONHowever, researchers have shown that focusing solely on reducing entropy can lead the model to make overconfident predictions [68] . To address this, we apply an additional constraint to align the multimodal prototypes during optimization, explicitly enforcing consistent multi-modal representations between dual sets of prototypes. Specifically, we introduce a self-supervised alignment loss that utilizes the contrastive InfoNCE loss [42] to bring prototypes from the same class closer together while pushing prototypes from different classes further apart:EQUATIONIn summary, the final objective for optimizing the multi-modal prototypes t, v isEQUATIONwhere λ is a scale factor to balance the contribution of the alignment loss. Note that t * and v * are obtained from a single update step.After optimizing the prototypes for each test sample, we evolve the online textual prototypes t as described in Equation ( 3), and also update the priority queues to re-compute the visual prototypes v. The evolved prototype sets then serve as the initialization for the next test sample, progressively enhancing generalization capability during test-time adaptation. 
experiments: 
 In this section, we evaluate our proposed method on robustness to natural distribution shifts and crossdatasets generalization across 15 various datasets. Moreover, we also compare the test-time efficiency of our DPE with existing methods. Finally, we provide ablation experiments to systematically analyze the effects of different algorithm components and design choices.Datasets. We follow previous work [53, 12] to evaluate our method on two benchmarking scenarios, namely, robustness to natural distribution shifts and cross-datasets generalization. (1) For the evaluation of robustness to natural distribution shifts, we assess the performance of our method using the ImageNet [7] dataset alongside its variant out-of-distribution datasets, including ImageNet-A [20] , ImageNet-V2 [46] , ImageNet-R [18] , and ImageNet-Sketch [62] .(2) For cross-datasets generalization tasks, we conduct comprehensive assessments across 10 diverse recognition datasets, including FGVCAircraft [38] , Caltech101 [11] , StandfordCars [29] , DTD [6] , EuroSAT [17] , Flowers102 [40] , Food101 [2] , OxfordPets [43] , SUN397 [67] , and UCF101 [54] . These datasets offer a comprehensive benchmark for evaluating the robustness of various methods across different distributional variations.Implementation Details. We follow previous works [53, 12] to adopt ResNet-50 [15] and ViT-B/16 [9] backbones as the visual encoder of CLIP. In Appendix C.2, we detail the specific hand-crafted prompts utilized for each dataset. Following TPT [53] , we generate 63 augmented views for each test image using random resized cropping to create a batch of 64 images. We learn the prototype residual parameters using AdamW [36] optimizer with a learning rate of 0.0005 for a single step. In default, the scale factor λ in Equation ( 10) is set to 0.5, the normalized entropy threshold τ t is set to 0.1, and the queue size M is set to 3. For the affinity function in Equation ( 5), we set α = 6.0 and β = 5.0, respectively. All experiments are conducted on a single 48GB NVIDIA RTX 6000 Ada GPU. To ensure the reliability of our results, we perform each experiment three times using different initialization seeds and report the mean accuracy achieved.Baselines. We compare our method with established test-time adaptation approaches for CLIP: (1) TPT [53] , a prompt tuning method that aims to minimize self-entropy across predictions of multiple augmented views; (2) DiffTPT [12] , an enhanced version of TPT that utilizes diffusion-based augmentations to optimize prompts; (3) TDA [26] , a training-free, adapter-based method which constructs positive and negative caches during test time. (4) TPS [55] , an efficient approach that dynamically learns shift vectors for per-class prototypes based solely on the given test sample;(5) DMN-ZS [81] , a backpropagation-free method that utilizes a dynamic memory to aggregate information from historical test data. Additionally, we present the zero-shot performance of CLIP using the simple prompt "a photo of a {CLASS}" as well as the results from prompt ensembling to show the absolute performance improvements. We also report the performance of CoOp [84] , a train-time adaptation method, using 16-shot annotated samples per class on ImageNet. For a fair comparison, we directly report the results of these baselines from their respective original papers. Note that in the DiffTPT [12] paper, the results are based on a subset of the datasets containing 1,000 test samples. This limited sample size may introduce potential imprecision in the reported results. Robustness to Natural Distribution Shifts. In Table 1 , we first compare the performance of our method with other state-of-the-art methods on in-domain ImageNet and its 4 out-of-distribution variants. Due to domain shifts, zero-shot CLIP [45] underperforms in out-of-distribution scenarios.As shown in the table, adopting prompt ensembling and prompt learning methods like CoOp [84] can enhance CLIP's generalizability. However, it is important to note that CoOp is a train-time adaptation method that requires an annotated training set, limiting its effectiveness in real-world settings. Despite this, our method still exhibits significant performance gains of 4.20% and 4.21% on average across two different backbones compared to CoOp, indicating the superiority of our DPE in enhancing generalization capability on out-of-distribution domains.Focusing on test-time adaptation methods, the experimental results demonstrate that our method achieves superior zero-shot generalization performance across various out-of-distribution datasets compared to other approaches. Specifically, our method outperforms existing state-of-the-art prompt tuning methods, surpasses TPT [53] by 3.55% and 3.49% and DiffTPT [12] by 2.10% and 3.65% on average when using ResNet-50 and ViT-B/16 backbones, respectively. Moreover, our method also outperforms cache-based TDA [26] by margins of 1.23% and 0.92% across two different backbones, indicating the effectiveness of our DPE approach. Moreover, our DPE demonstrates performance advantages over the recent TPS [55] and DMN-ZS [81] approaches, outperforming them by 1.43% and 0.84% on average across 5 datasets using the ResNet-50 backbone, further highlighting the superiority of our method. We also demonstrate that our DPE can also be effectively applied to prompts learned using CoOp [84] with a 16-shot ImageNet setup. We compare the performance with other methods on the same 5 datasets in Appendix A.1, where our method consistently demonstrates competitive performance. These results highlight the general effectiveness of our proposed test-time adaptation method in both in-domain and out-of-distribution scenarios.Cross-Datasets Generalization. In Table 2 , we further assess the generalizability of our proposed method against other state-of-the-art methods on 10 fine-grained recognition datasets. Given the significant distributional differences, methods may exhibit variable performance across these datasets. Notably, our method, which is not trained on any annotated data, significantly outperforms CoOp [84] by average margins of 5.75% and 5.52% on two respective backbones. When compared to other test-time adaptation methods, DPE exhibits average performance gains of 2.08% to 0.90% compared to DiffTPT and TDA, respectively. On the more advanced ViT-B/16 backbone, DPE continues to outperform existing approaches on 7 out of 10 datasets, with average improvements ranging from 1.87% to 4.30%. These results demonstrate the superior robustness and adaptability of our method in transferring to diverse domains during test time, which is crucial for real-world deployment scenarios. Our proposed method shows improved computational efficiency compared to other prompt tuning methods, for example, 5× faster than TPT [53] and over 10× faster than DiffTPT [12] , as it requires only learning the prototype residues without the need to backpropagate gradients through the textual encoder. While our method is less efficient than TDA [26] and TPS [55] , as we still backpropagate gradients to update multi-modal prototypes, it offers notable performance advantages. Different Textual Prototype Evolution Rules. In Table 4 , we report the performance on ImageNet [7] using different textual prototype evolution rules. We have the following key observations: (1) Fully updating our textual prototypes t to the optimized prototypes t * for each individual test image results in collapsed performance;(2) Compared to not evolving the textual prototypes, using an exponential moving average update rule with a decay rate of 0.99 leads to a slight performance improvement of 0.18%; however, setting a lower decay rate of 0.95 decreases the performance by 0.36%. (3) Our cumulative average update rule yields the highest performance, achieving a 0.48% improvement compared to no update on ImageNet [7] .Hyperparameters for Dual Prototype Evolution. We provide a sensitivity analysis for the hyperparameters τ t and M on the Caltech101 [11] dataset in Figure 4 (Left). Specifically, τ t represents the normalized entropy threshold for evolving our textual prototypes. When τ t = 0, our method does not evolve the textual prototypes, leading to a significant performance decrease, as shown in Figure 4 (Left). Moreover, setting τ t = 0.1 results in the highest performance, whereas a higher threshold leads to a slight decrease in performance. Additionally, the queue size M acts as a soft threshold hyperparameter for evolving the visual prototypes. Our setting of M = 3 consistently yields the highest performance. Lowering M causes the visual prototypes to fail in capturing the diversity of test samples from the same class, while increasing M introduces additional low-confidence noisy samples that hinder discrimination among target classes. Notably, our DPE method consistently outperforms other approaches across a reasonable range of hyperparameter settings: all combinations of entropy threshold τ t ≥ 0.1 and queue size M > 3 achieve over 90.3% accuracy on Caltech101, whereas TPT [53] and TDA [26] only achieve 87.02% and 89.70%, respectively.Effects of Different Learnable Modules. Recall that in our DPE method, we optimize our multimodal prototypes by introducing two sets of learnable residual parameters t and v for each individual test image. In Figure 4 (Middle), we ablate the effects of each set of learnable residual parameters and report the performance across three datasets. Specifically, on ImageNet [7] , optimizing only the textual prototypes for individual samples results in a 1.40% improvement, while optimizing only the visual prototypes yields a non-trivial 0.36% improvement, compared to keeping both t and v fixed.Optimizing both sets of residual parameters leads to a further performance increase, e.g., by 1.52% on ImageNet [7] . This indicates both learnable modules contribute to the overall effectiveness of DPE. Scaling the Alignment Loss. Finally, we ablate the effect of the alignment loss by varying the scale factor λ in Figure 4 (Right). Compared to optimizing solely using entropy minimization loss (i.e., λ = 0) during test-time adaptation, applying the additional alignment loss results in a performance improvement of 0.23% to 1.07% across three different datasets. However, there is a trade-off between prototype alignment and self-entropy minimization: setting λ too high leads to a performance drop.Our experiments show that our setting of λ = 0.5 yields the highest performance. Impact of Varying Update Steps. In Equation (10) , we update the multi-modal prototypes with a single update step for each test instance.To evaluate the impact of different numbers of update steps on overall performance, we conduct ablation experiments by varying the number of update steps from 1 to 5 and report the resulting performance on ImageNet. As shown in Table 5 , the number of update steps does not significantly influence performance (within a range of 0.2%). While increasing the update steps to 2 yields a slight performance gain of 0.04%, it also leads to a proportional decrease in inference efficiency. Given this trade-off, we adopt the single-step update as the default for balancing efficiency and performance.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We believe that we have clearly introduced the proposed DPE method in the main texts for reproduction. We have provided all the training details, computational resources, and hyper-parameter settings in the implementation details in Section 4.1. Also, please be assured that we will make our source code publicly available upon acceptance. 
conclusion: 
 In this work, we introduce Dual Prototype Evolving (DPE), a novel and effective approach for enhancing the zero-shot generalizability of VLMs during test time. Unlike previous methods that only focus on adapting the VLMs from one modality, we create and evolve two sets of prototypes-textual and visual-progressively capturing more accurate multi-modal representations for target classes during test time. Moreover, we also introduce prototype residual learning to optimize the dual prototype sets for each individual test sample, which further enhances the test-time generalization capabilities of VLMs. Through comprehensive experiments, we demonstrate that our proposed DPE achieves state-of-the-art performance while also exhibiting competitive test-time efficiency.Limitations. While our proposed DPE method effectively adapts CLIP to out-of-distribution domains during test time, we identify two potential limitations: (1) It still requires gradient backpropagation to optimize the multi-modal prototypes. This optimization process introduces additional computational complexity compared to zero-shot CLIP [45] , which may affect its real-time performance in practical deployment scenarios. (2) Since DPE needs to maintain priority queues to evolve the visual prototypes, it increases the memory cost during inference.Broader Impacts. In this work, we aim to build more reliable machine learning systems by leveraging the extensive knowledge of current foundational models, specifically CLIP [45] . Specifically, we follow TPT [53] to apply the test-time adaptation setting to vision-language models to align with realworld scenarios. By employing our DPE approach, the CLIP model can adapt itself to diverse domains during test time, which enhances its practical applicability in real-world deployment scenarios. We hope this work inspires future studies to focus on the generalization and robustness of pre-trained large-scale foundation models.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: To the best of our knowledge, this paper does not pose such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have properly cited all the assets we used in our paper. We also list the license information of the used datasets and code in Appendix D. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: This paper does not introduce new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Table A1 : Performance comparisons on robustness to natural distribution shifts. We present top-1 accuracy (%) results for all evaluated methods employing both ResNet-50 and ViT-B/16 visual backbones of CLIP. Additionally, we assess the performance using prompts learned by CoOp [84] with 16-shot training data per class on ImageNet [7] . The best results are highlighted in bold. In Figure 4 (Left), we provide a sensitivity analysis of queue size M on the Caltech101 dataset. We further analyze the impact of hyperparameter M on larger-scale ImageNet in Table A4 . Similar to the results on Caltech101, we observe that the performance increases by 0.5% when adjusting M from 1 to 3 but exhibits a slight decrease of 0.2% when further increasing to 7. We speculate that initially increasing the value of M allows our priority queue to collect more diverse features and obtain representative prototypes. However, further increasing it leads to the inclusion of more low-confidence noisy samples, which has adverse effects.We conduct additional ablation experiments to analyze the individual effect of each component in Table A5 . In the table, VPE, TPE, and PRL refer to visual prototype evolution, textual prototype evolution, and prototype residual learning, respectively. Note that Experiment #3 is invalid since TPE requires optimized textual prototypes t * from PRL. As shown, VPE is the most influential component, providing a ∼2% improvement over zero-shot CLIP. The other two components also contribute significantly to the overall performance.The alignment loss L align acts from a global perspective by promoting consistent multi-modal prototypes, ensuring that the representations are aligned for all subsequent test samples. The self-entropy loss L aug , in contrast, greedily targets on improving individual sample predictions by penalizing high-entropy predictions across augmented views. To provide a clearer understanding, we analyze the effects of the two loss terms on ImageNet using the ResNet-50 backbone and report the performance in Table A6 . We can observe that while the alignment loss alone improves the performance by 0.56%, the self-entropy loss provides a greater performance gain of 1.28%.Combining both loss terms further enhances performance by an additional 0.23%.We acknowledge that our DPE method shares some high-level ideas with DMN-ZS, TPS, TaskRes and MaPLe. However, there are some key distinctions. Here, we discuss the differences between our method and these approaches, respectively:• DMN [81] . While DMN(-ZS) also utilizes historical test samples to enhance the test-time generalizability of VLMs, it only updates the visual memory online while keeping the textual features/classifier unchanged. Therefore, we consider DMN similar to TDA, as both methods adapt CLIP only from a uni-modal (visual) perspective. In contrast, our DPE is designed to progressively capture more accurate multi-modal representations on the fly with test samples.• TPS [55] . Similarly, since TPS only updates the textual prototypes during testing, we categorize it with TPT and DiffTPT, which also account only for uni-modal (textual) adaptation. Moreover, TPS has similar limitations to TPT, as discussed in Lines 46-49, where it treats each test instance independently, resetting to the original model for each new sample. In contrast, our DPE can accumulate task-specific knowledge as more test samples are processed.• TaskRes [70] and MaPLe [27] . While our method shares some similarities in method details (e.g., multi-modal prototype residuals), we focus on a completely different test-time adaptation setting. Specifically, TaskRes and MaPLe aim to adapt CLIP using labeled few-shot samples, whereas our proposed DPE approach leverages only the unlabeled target data stream to adapt the model to out-of-distribution domains. Moreover, we innovatively propose textual/visual prototype evolution, which enables our method to progressively capture more accurate multi-modal representations during test time. The two works mentioned above, while effective in learning from few-shot samples, do not incorporate such knowledge accumulation techniques.In Table C7 , we present the detailed statistics of each dataset we used in our experiments, including the number of classes, the sizes of training, validation and testing sets, and their original tasks. C8 : Textual prompts used in experiments. In addition to these prompts, we also employ CuPL [44] prompts to further enhance performance."itap of a {CLASS}." ImageNet [7] "a bad photo of the {CLASS}." ImageNet-V2 [46] "a origami {CLASS}." ImageNet-Sketch [62] "a photo of the large {CLASS}." ImageNet-A [20] "a {CLASS} in a video game." ImageNet-R [18] "art of the {CLASS}." "a photo of the small {CLASS}."Caltech101 [11] "a photo of a {CLASS}." DTD [6] "{CLASS} texture." EuroSAT [17] "a centered satellite photo of {CLASS}." FGVCAircraft [38] "a photo of a {CLASS}, a type of aircraft." Flowers102 [40] "a photo of a {CLASS}, a type of flower." Food101 [2] "a photo of {CLASS}, a type of food." OxfordPets [43] "a photo of a {CLASS}, a type of pet." StanfordCars [29] "a photo of a {CLASS}." SUN397 [67] "a photo of a {CLASS}." UCF101 [54] "a photo of a person doing {CLASS}."In Table C8 , we detail the specific hand-crafted prompts utilized for each dataset.Datasets. We list the known license information for the datasets below:• MIT License: ImageNet-A [20] , ImageNet-V2 [46] , ImageNet-R [18] , and ImageNet-Sketch [62] .• CC BY-SA 4.0 License: OxfordPets [43] .• Research purposes only: ImageNet [7] , StandfordCars [29] , DTD [6] , FGVCAircraft [38] , SUN397 [67] .Code. In this work, we also use some code implementations from existing codebase: CLIP [45] , CoOp [84] , TPT [53] , and TDA [26] . The code used in this paper are all under the MIT License.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: The paper introduces a novel method for test-time generalization in vision-language models (VLMs), specifically applied to CLIP. The method, DPE, evolves both textual and visual prototypes during test time to capture accurate multi-modal representations. The method aims to improve performance on out-of-distribution tasks by progressively accumulating knowledge from unlabeled test samples. The approach is evaluated on 15 benchmark datasets, demonstrating superior results compared to existing methods.

While the performance improvement is marginal and the complexity increased, the method introduces a novel and technically solid approach to improving VLMs in test-time adaptation settings. The comprehensive experimental evaluation and potential to inspire future research in test-time generalization make the paper a worthwhile contribution to the field, justifying acceptance to NeurIPS.

Reviews: review: paper_summary: paper_summary: 1. This paper proposes a novel test-time adaptation method (DPE) for VLMs that captures multi-modal representations for target classes during test time.
2. This paper introduces and optimizes learnable residuals for each test sample to align the prototypes across modalities.
3. The results of this paper are promising while maintaining competitive computational efficiency.
strength_weakness: strength_weakness: 1. The core idea is not novel enough. Many concurrent works with TDA which is DPE's main comparison method have shown a similar idea, such as DMN-ZS[1] actually utilizes the information of two modalities while not demanding any backpropagation, and TPS[2] which has been cited in the paper proposes that residual prototype is useful for test-time prompt learning. Though the specific designs of DPE are different from them, the core idea is somewhat similar.
2. More analysis of hyperparameters is needed. Though the introduction of visual information is helpful intuitively, hyperparameters such as (top-)M may affect the performance obviously, which should be explored comprehensively.
3. More analysis of loss functions is needed. Alignment loss is helpful for performance improvement, but too much alignment leads to a performance drop, the reason should be discussed.
4. Efficiency comparison is not comprehensive. Though DPE performs faster than TPT and DiffTPT, the efficiency comparison with backpropagation-free methods (such as TDA,TPS, and DMN) should be provided.

[1] Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models.
[2] Just shift it: Test-time prototype shifting for zero-shot generalization with vision-language models.
questions: questions: 1. Many concurrent works (such as DMN-ZS, TPS) have shown similar ideas, and comparisons with them including the core idea design and performance should be provided.
2. A detailed analysis of hyperparameter M should be provided, and how to choose the M to obtain robust performance in the test-time scenarios also should be discussed.
3. A detailed analysis of loss functions should be provided. As shown in the paper, more alignment brings performance drop, the reason should be discussed. By the way, should the text prototype be aligned in the direction of the visual prototype? Or does the visual prototype only need to be aligned in the direction of the text? Are there any related experiments?
4. Efficiency comparison with backpropagation-free methods (such as TDA,TPS, andDMN) should be provided.
limitations: limitations: 1. The core idea is not novel enough, and it seems to be a combination of existing several ideas.
2. Detailed analysis of hyperparameters and loss functions is missing.
3. Efficiency comparison is not comprehensive.
paper_summary: paper_summary: The paper proposes a novel test-time adaptation method for CLIP models, drawing inspiration from previous works on prototype learning and CLIP-based adaptors.

For each test sample, both textual and visual prototypes are optimized using learnable residual parameters based on alignment loss and self-entropy loss. These prototypes are progressively updated to better utilize the stream of test samples.

Experimental results on common benchmarks demonstrate the proposed method's effectiveness over recent baselines. Additionally, the authors provide comprehensive ablation studies to highlight the impact of hyperparameters.
strength_weakness: strength_weakness: - The method combines multiple existing techniques, each of which has been explored in previous works. For instance, learnable residuals have been studied in [1], where task residuals were used for CLIP-based adaptation, and learnable multi-modal representations were used in [2] with a similar motivation for CLIP-based adaptation.

- The performance gain appears marginal  (Table 1), especially given the extensive hyperparameter tuning required. The results are sensitive to hyperparameters (Section 4.3). This may raise concerns about whether the method's efficacy is inherent or merely a result of meticulous hyperparameter tuning. Note that influential factors in the proposed method include (1) the entropy threshold, (2) queue size, (3) weight scale factor, and (4) prototype update rules.



- There seems to be an overstatement of the method's applicability. While the title and introduction suggest broad applicability to "Vision-Language Models," the method is only tested on CLIP models, with no experiments conducted beyond CLIP.

[1] Yu et al., Task Residual for Tuning Vision-Language Models, CVPR 2023.

[2] Khattak et al., MaPLe: Multi-modal Prompt Learning, CVPR 2023.
questions: questions: - Is the proposed method applicable to non-CLIP models?

- Can a simplified version of the method be presented by removing certain components? Specifically, which components are the most influential? From the ablation studies, it appears that each component functions as an add-on.
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduces a novel test-time adaptation approach for vision-language models (VLMs) called Dual Prototype Evolving (DPE). The method effectively accumulates task-specific knowledge from multi-modalities by creating and evolving two sets of prototypes—textual and visual—during test time. This approach ensures more accurate multi-modal representations for target classes and promotes consistent representations by optimizing learnable residuals. Extensive experiments on 15 benchmark datasets demonstrate that DPE consistently outperforms previous state-of-the-art methods in both performance and computational efficiency.
strength_weakness: strength_weakness: -Complexity: The introduction of learnable residuals and the dual prototype evolution mechanism adds complexity to the model, which might pose implementation challenges for practitioners.
-Computational Cost: While the paper claims competitive computational efficiency, the requirement to maintain and update priority queues for visual prototypes can increase memory and computational costs, particularly for large-scale datasets.
-Generalization to Other VLMs: The paper primarily focuses on CLIP, and it is unclear how well the proposed method generalizes to other vision-language models or tasks beyond those evaluated.
questions: questions: 1. Can you provide more details on the computational overhead introduced by the dual prototype evolution mechanism compared to baseline methods?
2. How does the proposed method perform when applied to vision-language models other than CLIP? Have you conducted any preliminary experiments in this direction?
3. Could you elaborate on the potential limitations of the prototype residual learning approach and how it might affect the model's performance in different scenarios?
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impacts of their work. The paper mentions two primary limitations: the additional computational complexity introduced by gradient back-propagation for optimizing multi-modal prototypes and the increased memory cost due to maintaining priority queues. The authors provide constructive suggestions for future work to address these issues, emphasizing the need for more efficient optimization techniques and memory management strategies.
paper_summary: paper_summary: This paper proposes a novel method for test-time generalization of vision-language models. Specifically, the paper updates two prototypes, textual and visual prompts, online using test samples. Additionally, the authors learn task residuals by aligning the two different modality prototypes, further improving performance. Experiments are conducted on standard benchmarks.
strength_weakness: strength_weakness: 1. More ablation studies are expected: What are the different impacts of the two loss terms in Eq 10?
2. Clarification is needed: Are t and v in Eq 10 achieved after loss convergence, or are they obtained from a single update step? Is there a significant difference between the two?
3. Discussion and comparison with a closely related method (DMN [a]) are expected. Both methods maintain the online queue/memory and weight cached features to obtain new prototypes. The difference is that this paper averages cached features to obtain prototypes, while DMN uses an attention mechanism. The authors should discuss both methods and the differences in prototype generation.

[a] Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models, CVPR2024
questions: questions: See weaknesses.
limitations: limitations: Na

Comments: title: Official Comment by Reviewer Sqqz
Many thanks for your follow-up feedback
Official Comment by Reviewer Sqqz
Thanks for your positive recommendation
comment: Thanks for your response. 

Most of my concerns about the experiment have been addressed, but the performance improvement (+0.9%/+0.4%) is not very promising considering the increase in inference time (+100%, nearly double).

Meanwhile, I maintain my opinion on novelty, so I choose to maintain my score as Borderline accept.
Dear Reviewer Sqqz,

Thank you for providing further detailed comments. 

---

**Comment (a)**: “*Which model (DMN-ZS or DMN-ZS\*) in response (4) corresponds to DMN-ZS in response (1)?*”

**Response (a)**: DMN-ZS* refers to the DMN-ZS mentioned in **Response (1)**. We evaluated its performance and efficiency using the official code implementation of DMN-ZS, where the pseudo-label predictions are by default enhanced by $N=64$ augmented views and confidence selection. For the DMN-ZS variant in **Response (4)**, we manually set $N=1$ and reported the results correspondingly.

---

**Comment (b)**: “*Can you supplement performance comparison based on CLIP-ViT-B/16? You can just combine existing results into one table.*”

**Response (b)**: We apologize for not including the full results in the rebuttal post due to character limit. Please find the complete comparisons below:

| Method         | ImageNet  | ImageNet-A | ImageNet-V2 | ImageNet-R | ImageNet-S |  Average  | OOD Average |
| -------------- | :-------: | :--------: | :---------: | :--------: | :--------: | :-------: | :---------: |
| CLIP-RN50      |   58.16   |   21.83    |    51.41    |   56.15    |   33.37    |   44.18   |    40.69    |
| TPT            |   60.74   |   26.67    |    54.70    |   59.11    |   35.09    |   47.26   |    43.89    |
| DiffTPT        |   60.80   | **31.06**  |    55.80    |   58.80    |   37.10    |   48.71   |    45.69    |
| TDA            |   61.35   |   30.29    |    55.54    |   62.58    |   38.12    |   49.58   |    46.63    |
| TPS            |   61.47   |   30.48    |    54.96    |   62.87    |   37.14    |   49.38   |    46.36    |
| DMN-ZS         | **63.87** |   28.57    |    56.12    |   61.44    |   39.84    |   49.97   |    46.49    |
| **DPE (Ours)** |   63.41   |   30.15    |  **56.72**  | **63.72**  | **40.03**  | **50.81** |  **47.66**  |

| Method         | ImageNet  | ImageNet-A | ImageNet-V2 | ImageNet-R | ImageNet-S |  Average  | OOD Average |
| -------------- | :-------: | :--------: | :---------: | :--------: | :--------: | :-------: | :---------: |
| CLIP-ViT/B-16  |   66.73   |   47.87    |    60.86    |   73.98    |   46.09    |   59.11   |    57.20    |
| TPT            |   68.98   |   54.77    |    63.45    |   77.06    |   47.94    |   62.44   |    60.81    |
| DiffTPT        |   70.30   |   55.68    |    65.10    |   75.00    |   46.80    |   62.28   |    60.52    |
| TDA            |   69.51   | **60.11**  |    64.67    |   80.24    |   50.54    |   65.01   |    63.89    |
| TPS            |   70.19   |   60.08    |    64.73    |   80.27    |   49.95    |   65.04   |    63.76    |
| DMN-ZS         | **72.25** |   58.28    |    65.17    |   78.55    | **53.20**  |   65.49   |    63.80    |
| **DPE (Ours)** |   71.91   |   59.63    |  **65.44**  | **80.40**  |   52.26    | **65.93** |  **64.43**  |

As shown, our proposed DPE still outperforms TPS and DMN-ZS by 0.89% and 0.44% on average across 5 datasets using the ViT-B/16 backbone of CLIP. We have updated the results in Table 1 of the revised manuscript. For more performance comparisons, please also kindly consider referring to Table B in the attached one-page PDF in the general response, where we further compare our method with TDA using the OpenCLIP ViT-L/14 backbone, and observe consistent performance improvements.

---

Finally, we would like to summarize the key novelties of this work:

- To the best of our knowledge, our work is the first to capture domain-specific knowledge from a **multi-modal** perspective for test-time adaptation of VLMs. Specifically, we achieve this by evolving two sets of prototypes from both textual and visual modalities to progressively capture more accurate multi-modal representations for target classes during test time.

- We proposed textual and visual prototype evolution to extract historical knowledge from previous test samples, enabling **effective accumulation** of knowledge over time.

- We further introduced prototype residual learning with an **alignment constraint** to enhance **consistent** multi-modal representations and ensure alignment of prototypes across modalities.

- Our proposed DPE achieves state-of-the-art performance across 15 various datasets while also exhibiting competitive test-time efficiency.

---

We hope that our responses have addressed your concerns. Please kindly let us know if you have any further questions.

Best,

Authors
Thanks for your detailed response. The novelty of the core idea is still my primary concern, and I have a few questions about the response.

(a) Which model (DMN-ZS or DMN-ZS*) in response (4) corresponds to DMN-ZS in response (1)?

(b) Can you supplement performance comparison based on CLIP-ViT-B/16? You can just combine existing results into one table.
Dear Reviewer Sqqz,

We greatly appreciate the time you have dedicated and the valuable feedback you have provided. As the discussion period draws to a close (Tue, August 13), please kindly let us know if there are any remaining questions. We will be more than happy to provide any further details or clarifications.

Best,

Authors
Dear Reviewer owvr,

Thank you for your encouraging positive recommendation. We greatly appreciate your recognition of the technical contributions of our work. We will incorporate these additional experiments in our revision carefully. Thank you again for your valuable feedback on our paper.

Best,

Authors
I appreciate the authors for the detailed response and additional experiments. It would be great to include some of these results in the main paper. My primary concern remains the marginal improvement of the proposed approach given its complexity. Despite the performance, I still lean towards recommending acceptance of the paper for its technical contribution.
Dear Reviewer vqJd,

Thank you for your insightful review of our paper. We greatly appreciate your positive recommendation!

Best,

Authors
Thanks for your response. My concern has been addressed. I will increase my score to 6.
Dear Reviewer SVvN,

We sincerely appreciate the time and effort you invested in reviewing our manuscript.  We greatly appreciate your positive recommendation!

Best,

Authors
Thank you for thoroughly addressing my concerns. After reviewing the comments and responses for the other reviewers, I see that their concerns have also been resolved. The authors have provided clear definitions of terms for better understanding, conducted additional experiments to further evaluate the effectiveness of the proposed method, and offered more in-depth analysis of how the proposed method works in various settings.

Overall, the rebuttal enhances my confidence in this paper. With careful consideration, I believe this paper with revision is worthy of NeurIPS and will significantly impact the test-time generalization field. My final decision is “accept.”