Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models
Authors: Ce Zhang
Simon Stepputtis
Katia P. Sycara
Yaqi Xie
Venue: NeurIPS 2024 poster (2024)
Keywords: Test-Time Adaptation
Vision-Language Models
CLIP
Transfer Learning
Primary area: machine_vision
TL;DR: We introduce Dual Prototype Evolving (DPE), a novel test-time adaptation approach for VLMs that effectively accumulates task-specific knowledge from multi-modalities.

Abstract: Test-time adaptation, which enables models to generalize to diverse data with unlabeled test samples, holds significant value in real-world scenarios. Recently, researchers have applied this setting to advanced pre-trained vision-language models (VLMs), developing approaches such as test-time prompt tuning to further extend their practical applicability. However, these methods typically focus solely on adapting VLMs from a single modality and fail to accumulate task-specific knowledge as more samples are processed. To address this, we introduce Dual Prototype Evolving (DPE), a novel test-time adaptation approach for VLMs that effectively accumulates task-specific knowledge from multi-modalities. Specifically, we create and evolve two sets of prototypes—textual and visual—to progressively capture more accurate multi-modal representations for target classes during test time. Moreover, to promote consistent multi-modal representations, we introduce and optimize learnable residuals for each test sample to align the prototypes from both modalities. Extensive experimental results on 15 benchmark datasets demonstrate that our proposed DPE consistently outperforms previous state-of-the-art methods while also exhibiting competitive computational efficiency.

Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models
Ce Zhang Simon Stepputtis Katia Sycara Yaqi Xie School of Computer Science, Carnegie Mellon University {cezhang, sstepput, katia, yaqix)@cs. cmu. edu
Abstract
Test-time adaptation, which enables models to generalize to diverse data with unlabeled test samples, holds significant value in real-world scenarios. Recently, researchers have applied this setting to advanced pre-trained vision-language mod- els (VLMs), developing approaches such as test-time prompt tuning further extend their practical applicability. However, these methods typically focus solely on adapting VI.Ms from single modality and fail to accumulate task-specific knowledge as more samples are processed. To address this, we introduce Dual Prototype Evolving (DPE), novel test-time adaptation approach for VLMS that fectively accumulates task-specific knowledge from mulii-modalities. Specifically, we create and evolve two sets of prototypes-textue and visual-to progressively capture more accurate multi-modal representations for target classes during test time. Moreover, to promote consistent multi-modal representations, we intro- duce and optimize learnable residuals for each test sample to align the prototypes from both modalities. Extensive experimental results on 15 benchmark datasets demonstrate that our proposed DPE consistently outperforms previous state-of-the- art methods while also exhibiting competitive computational efficiency. Code is available https: //g1 /gi thub. com/ http://aeaa/e////IIII. CLIPP
Introduction
Although deep learning models have achieved great success in various machine learning tasks [48, 49]. they often suffer from significant performance degradation due to distribution shifts between the training data from the source domain and the testing data from the target domain [34, 64, 15]. To address this challenge, a number of works [22, 58, 65] adopt the transductive learning principle, assuming access to both labeled source data and unlabeled target data-a scenario known as the domain adaptation setting. However, this setting contrasts with most practical scenarios, where we only have access to well-trained model and cannot re-access the source data due to privacy or data retention policies. response, researchers have proposed test-time adaptation, which leverages only the unlabeled target data stream adapt the model to duttrf-distribution domains [79, 60, 62).
Recently, large-scale vision-language models (VLMs), such as CLIP [46] and ALIGN [25], have garnered increasing attention in the research community. These models, pre-trained on massive web- scale datasets, exhibit remarkable zero-shot capabilities and open-world visual understanding [46, 70, 74, 32]. While the large-scale pre-trained (source) datasets like LAION-5B [50] are accessible, is impractical for individuals to train on them due to their immense size. Consequently, adapting VLMs to downstream tasks via efficient fine-tuning with limited annotated samples from the target domain has become a focus of recent research [85, 84, 81, 71J. However, although these methods have proven effective, they posc significant limitation: they assume the availability of annotated samples from the target domain, which is often not practical real-world scenarios. This constraint hinders the broader deployment of VLMs in diverse and dynamic environments [20, 21, 47, 75].
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A. Zero-Shot CLIP
C. TDA: Training-Free Dynamic Adapter
Hando Crafted Prongts
High Geridence Samples
(49))
Kays
hind
Test Irage
Test Inuce
Image Faasuras
Pestures
Feature
Eme-shel
B. TPT: Test-Time Prompt Tuning
D. DPE: Dual Protatype Evalving
Augreenied
Frampta
Trol Fastare
Hand Cratt-rrtted Prompla
Prototspe Baseel Inferencce
of (QLADE)
lad
Canfdraao Sarepees
Image fasturaa
Figure 1:
Comparison of our DPE with zero-shot CLIP [46], TPT [54], and TDA [27]. We denote CLIP's parallel textual and visual encoders as & and &,,, respectively. While previous methods solely adapt the CLIP model from a single modality, we design our DPE to evolve prototypes from both textual and visual modalities to progressively capture more accurate multi-modal representations for target classes during test time.
To address the label scarcity problem in practice, a number of approaches apply the test-time adaptation setting to the domain of adapting VLMs to downstream tasks, as shown in Figure 1. Specifically, Shu et al. [54] propose test-time prompt tuning to learn an adaptive prompt for each individual sample in the test data stream to enhance CLIP's zero-shot generalizability to out-of-distribution domains. Building on TPT, DifTTPT [13] incorporates diffusion-based data augmentations to facilitate more effective prompt tuning during test time. More recently, Karmanov el al. [27] propose an alternative training-free dynamic adapter approach to establish dynamic visual caches with the unlabeled test samples.
However, we recognize that existing works overlook the following inherent properties of test-time adaptation in VLMs: (I) Cumulative. We expect that with more seen samples, the performance should improve as task-specific knowledge accumulates [40, 57]. However, test-time prompt tuning methods [54, 13] treat each test instance independently. resetting to the original model for each new sample, failing to extract historical knowledge from previous test samples. (2) Multi-modal. Effective adaptation of VLMs benefits from leveraging knowledge from both textual and visual modalities [28, 35]. However, previous works only capture domain-specifi knowledge from single modality, adapting CLIP based solely on textual [54, 13] or visual [27] feature refinement.
To this end, we propose Dual Prototype Evolving (DPE), a novel test-time VLM adaptation approach that effectively accumulates task-specific knowledge from multi-modalitiis, as illustrated in Figure 1. Unlike previous methods that focus on adapting VLMs from a single modality, we create and evolve two sets of prototypes-ted textual and visual- progressivly capturing more accurate multi-modal representations for target classes during test time. To extract historical knowledge from previous test samples, we update these two sets of prototypes online using cumulative average and priority queue strategies, respectively. We further optimize these multi-modal prototypes by introducing learnable residual parameters for each individual test sample to enhance the zero-shot generalization capability of our model. Specifically, rather than solely relying on the entropy minimization objective [62, 79], our DPE also accounts for the alignment between multi-modal prototypes to ensure consistent multi-modal representations. Notably, our DPE requires only the optimization of multi-modal prototypes in the embedding space during test time, climinating the need to backpropagate gradients through the textual encoder of CLIP, required in TPT [54] and DifTTP [13].
The test-time generalization capabilities of our proposed DPE method are extensively evaluated across 15 diverse recognition datasets in two scenarios: natural distribution shifts and cross-dataset generalization. The experimental results validate the superior performance of our DPE, which achieves an average improvement of 3.55% and 4.30% over the state-of-the-art TPT [54] method in these scenarios. Moreover, our proposed DPE achieves this performance while also exhibiting 5x and over 10x test-time efficiency compared to TPT [54] and DiffTPT [13], respectively.

---

The contributions of this paper are summarized as follows:
We propose dual prototype evolving (DPE), novel test-time adaptation method for VLMs that progressively captures more accurate multi-modal representations for target classes during test time.
To promole consistent multi-modal representations, we introduce and optimize learnable residuals for each test sample to align the prototypes across modalities.
Experimental evaluations demonstrate that our DPE consistently outperforms current sfate-of-the- art methods across 15 diverse datasets while maintaining competitive computational efficiency.
2 Related Work
Vision-Language Models. Leveraging vast image-text pairs from the Internet, recent large-scale vision-language models (VLMs), such as CLIP [46] and ALIGN [25], have shown remarkable and transferable visual knowledge through natural language supervision [78, 11, 10]. These VLMSs enable a "pre-train, fine-tune" paradigm for performing downstream visual tasks, such as image recognition [46, 17, 76], pbject detection [67, 66], and depth estimation [80, 24, 73).
To effectively transfer VLMs to these downstream tasks, researchers have developed two primary methods for adapting the model with few-shot data: prompt learning methods [85, 84, 28, 51, 86, 5] and -dapter-based methods [81, 14, 77, 71, 31]. Specifically, prompt learning methods, such as CoOp [85] and CoCoOp [84], focus on learning input prompts with few-shot supervision methods, like Tip-Adapter [81] and TaskRes [71], modify the extracted visual textual representations directly to enhance model performance. However, these approaches often assume the availability of labeled samples from the target domain, which can limit their effectiveness in real-world scenarios. In this work, we address the challenge of test-time adaptation, where the model is required to adapt solely at test time without access to any training samples or ground-truth labels from the target domain. This setting is crucial for real-world deployment, as it allows for robust performance in novel and unseen environments where labeled data cannot be obtained in advance.
from downstream data. On the other hand, adapter-basee
Test-Time Adaptation. To effectively transfer a model trained on the source domain to the target domain, test-time adaptation methods [62. 79, 60, 3, 59] aim to adjust the model online using stream of unlabeled test samples. These methods enable the deployment of well-trained models in various of-distribuuionn scenarios, thereby enhancing the applicability and reliability of machine learning models in real-world applications [34, 29, 42]. Researchers have applied fest-time adaptation techniques successfully across various machine learning tasks, including semantic segmentation [23, 52, 83]. human pose estimation [33, 26]. and image super-resolution [53, 8].
Recently, increasing research efforts have focused on adapting large-scale VLMs during test time [38, 56, 1, 72, 82, 76]. As the seminal work, Shu et al. [54] firstly propose test-time prompt tuning (TPT), which enforces consistency across different augmented views of each test sample. Building on this approach, several subsequent studies have sought to further enhance TPT. For instance, DiffTPT [54] utilizes diffusion- based augmentations to increase the diversity of augmented views, while C-TPT [69] addresses the rise in calibration error during test time prompt tuning. Unlike these approaches, which treat each test sample independently, TDA [27] establishes positive and negative visual caches during test time, enhancing model performance as more samples are processed. Similarly, recent DMN [82] utilizes a dynamic memory to gather information from historical test data. However, these methods solely adapt the model from a single modality perspective, limiting their effectiveness in capturing task-specific knowledge from -uttrf-distribution domains. Given this, we design DPE to evolve two sets of prototypes from both textual and visual modalities to progressively capture more accurate multi-moda representations for target classes during test time.
3 Method
We introduce Dual Prototype Evolving (DPE) as illustrated in Figure 2, to enhance CLIP's zero-shot generalization capabilities across diverse distributions during test time. Unlike previous methods that focus solely on one modality, we design two sets of prototypes, textual and visual, which are progressively updated using the unlabeled test datasct Drest.
3

---

Descriations
Textual Protatypas
Textual Prototype Evolution
A prote of the cmas) poopptoce theccuasso
Text Encoder
Predictions All Views
Prototype Evalving
X
High Confidincc Samples
deg
Image Encoder
Indiiaiin
X
Visual Protolype Evolution
ca.t
Cantidence Setection
Visual Prototyses
Average
Aupmented Images
Image
dog
Encoder
Ca2
L..
Features
Figure 2: An overview of our DPE method. We introduce prototypes from both textual and visual modalities and enable prototype-basek inference with CLIP. For each test sample, we optimize both prototypes using learnable residual parameters with alignment loss Calign and self-entropy loss Laug. These prototypes also progressively evolved over time to capture more accurate and discriminative multi-modal representations for target classes.
3.1 Preliminaries
Zero-Shot CLIP. CLIP [46] utilizes two pre-traine pre-trained parallel encoders: a visual encoder (-)- and textual encoder E.(.), which embed images and text descriptions into shared embedding space TR.... For C-class classification task, CLIP performs zero-shot predictions by computing the similarities between the extracted image feature and the C candidate text features, written as
exp (sim (fr. ft) fo = Ex(ros))) ft. = &Too, P'cuIrly (est)) Σn exp (simn (Se, /n)/t)
(I)
where Xtest Dreat denotes the input test image, and To represents the the class-specific description input for class Me- The pairwise similarities simm::-) are calculated using cosine similarity, and t represents the temperature parameter in the softmax function.
Test-Time Prompt Tuning. To enhance the zero-shot generalizability of CLIP, TPT [54] proposes learning adaptive prompt using the test stream samples. Specifically, for each test sample Xtest, [A,,(Xees (Xtest)) W ) and averages the top p-percentile confident predictions based entropy threshold obtain the final prediction:
TPT generates N augmented views {.A,,
N
PTpr (Xees)))
Σ n=1
1 (Aarr I r] PELIP(AA (Xtest)).
PN
Here, H(p) = - ps log calculates the self-entropy of the prediction The objective of TPT is to optimize the learnable prompt to minimize the self-entropy of the final prediction, i.e., min (IPTPT (Xeos(Xeoez)).
3.2 Dual Prototype Evolving
In our DPE method, we construct and iteratively evolve sets of class-specific prototypes from both visual and textual modalities to achieve a more precise representation of each class over time.
Textual Prototype Evolution. In this work, follow CLIP [46] to multiple context prompt templates for prompt ensembling. Specifically, for each class we generate a total of text descriptions, denoted as (T. (S 1=1- The prototypes of these descriptions in the embedding space are calculated as t 1/2 E, (T ()). To further improve the quality of these prototypes over time, we design them to be updated online through cumulative average with each individual sample Xtset in

---

the test stream. The update rule is given by:
(k (k- 1)t 1 k I1(k (
t
where t = [4 tc]T Rxxd is the online updated prototype set, and t* Rx is the optimized textual prototypes for each individual sample Xtest in Equation (10). To ensure stable online updates, we set entropy threshold 7, to filter out low-confidence samples (for which H((PoLIP (Xtest)) < Tτ) from updating the online prototypes, and maintain a counter for tracking confident samples.
Visual Prototype Evolution. Inspired by TDA [27], we recognize that the historical image features of test images can also be utilized to enhance CLIP's discrimination capability. Therefore, we design a priority queue strategy to store the top-M image features for each class and symmetrically compute set of visual prototypes that evolve over time. Note that since cannot access the labels of the test samples, we assign the image features to the queue according to their predicted pscudo-labels. The priority queue for each class is initialized empty, denoted 9e 2. As test samples arrive, store the image features fe and the corresponding self-entropy he in the priority queue, represented
as qe = ((!!m)) )) The elements are sorted by self-entropy h (m) such that Ne < h (>m)
Using this priority queue, the class-specific visual prototype is obtained by: Ve - + ((m) where S. S M denotes the total number of image features stored the queue.
The priority queues are updated during testing by replacing low-confidence image features with high-confidence ones. Specifically, for each individual test sample Xtest, we first predict the pseudo-label and compute the corresponding self-entropy h as:
= arg max P CLIP (y Yel-eest), h H(FFFIP (((eoar(Xeee))).
(4)
Ve
Then, we consider the following two scenarios to iteratively update the priority queue I for class f: (1) If the priority queue is not full, we directly add the pair (Xeea))) to the queue; (2) If the priority queue is full and the entropy h of the new sample is lower than the highest entropy value (of the last element) currently in the queue, we replace the highest-entropy element with the new feature and self-entropy (E, ),h), h). If ʃ is not lower, we discard the new sample and leave the queue unchanged. After each update, we re-sort the priority queue based on the self-entropy values and re-compute the visual prototypes [v .... ..c see--oppeee Cxd for all classes.
Figure 3: t-SNE [61] visualizations of the stored image features in the priority queues. With more samples getting in, the selected image features from each class hecome more clustered, leading to more representative visual prototypes.
In Figure 3, we present the t-SNE [61] visualizations the stored image features in the priority queues (with queue size M = 6) after updating with 1500 samples (leff) and 15000 samples (right) on the Food101 [2] dataset. We highlight the stored features from 25 random classes using different colors while marking the others in gray. These visualizations illustrate that our priority queue strategy effectively accumulates high-confidence samples, progressively refining the representativeness of the visual prototypes over time.
Prototype-Base Based Inference. Based on our two sets of multi-modal prototypes (te) and {velc the final prediction for input X is given by
exp (o A(F. vo)) /t) S. = EX),, PProto vxlX)- Ee exp (IS t A(S /t))
(5)
Here, / represents the temperature parameter in the softmax function, T denotes the matrix transpose, and A(x) = exp (-3 (1 =)) is the affinity function, where is balance hyperparameter and B a sharpness ratio. Appendix A.3, we conduct a sensitivity analysis of these two hyperparameters to evaluate their impact on the overall performance of DPE.
5

---

3.3 Prototype Residual Learning
To further improve the zero-shot generalizability of our method, we introduce prototype residual learning, which optimizes tuning approaches [54, 13] that require backpropagating gradients through the text encoder to update input prompts, our method directly updates the prototype sets in the embedding space.
prototypes for each test sample. Unlike previous prompt
Specifically, after being evolved with the last test sample, the dual sets of multi-modal prototypes, denoted as t = [t L2 ... tc] E Ixd and v = [y V2 ... Vc]" E CRCd are considered as the initialization for updating with the current parameters ( i2 tc] T RCxd and v [V1 V2 vol E RCxd These parameters initialized to zero and are used to optimize the prototypes for each given test input Xteat, denoted as
sample. We further introduce learnable residual
Ve + Ve Ieve
te [te
Cell
Vc
Vell
(6)
Similar to Equation (2), we optimize these residual parameters to promote consistent predictions across a total of N different augmented views of the given test image Xeast using the unsupervised entropy minimization objective:
L Lagg H(IPDPE (((PeFE(Xteett
=
Σ
PDE (v
Ve (reat) log Pyope(y Yo (teet),
c=1
1
where POOE (X
[H (Pnroto(A((Xxeee)) S r] [ Prooo (An (XLeet)). Σ PN
(8)
However, researchers have shown that focusing solely on reducing entropy can lead the model to make overconfident predictions [69]. To address this, we apply an additional constraint to align the multi- modal prototypes during optimization, explicitly enforcing consistent mulli-modea representations between dual sets of prototypes. Specifically, we introduce a self-supervised alignment loss that utilizes the contrastive InfoNCE loss [43] to bring prototypes from the same class closer together while pushing prototypes from different classes further apart:
1 Σ
(- log exp(t, v.)
log exp(t, T Ve) )
Calign
= = c=1
(9)
ver) E. etp vo) E. exp(t, (ve)
In summary, the final objective for optimizing the multi-modal prototypes t, vs
t* v arg min (Caug + ACaliga)
(10)
where À is scale factor to balance the contribution of the alignment loss. Note that t* and v are obtained from single update step.
After optimizing the prototypes for each test sample, we evolve the online textual prototypes t as described in Equation (3), and also update the priority queues to re-compute the visual prototypes v. The evolved prototype sets then serve as the initialization for the next test sample, progressively enhancing generalization capability during test-time adaptation.
4 Experiments
this section, we evaluate our proposed method on robustness to natural distribution shifts and cross- datasets generalization across 15 various datasets. Moreover, also compare the test-time efficiency of our DPE with existing methods. Finally, we provide ablation experiments to systematically analyze the effects of different algorithm components and design choices.
4.1 Experimental Settings
Datasets. We follow previous work [54, 13] to evaluate our method on two benchmarking scenarios, namely, robustness to natural distribution shifts and cross-datasets generalization. (1) For the

---

Table I: Performance comparisons robustness to natural distribution shifts. We present top-1 accuracy (%) results for all evaluated methods employing both ResNet-50 and ViT-B/16 visual backbones of CLIP. The best results highlighted in bold.
Method
ImageNet ImageNet-A ImageNet-V V2 ImageNet-R ImageNet-S
Average OOD Average
ELIP-ResNet--3 [46]
58.16
21.83
$1.41
56.15
33.37
44.18
40.69
Ensemble
59.81
23.24
$2.91
60.72
35.48
46.43
43.09
CoOp [85]
63.33
23.06
55.40
56.60
34.67
46.61
42,43
TPT [54]
60.74
26.67
54,70
59.11
35.09
47.
43,89
DifTTP [13]
60.80
31.06
55.80
58.80
37.10
48.71
45.69
TDA [27]
61.35
30.29
55.54
62.58
38.12
49,58
46.63
TPS [56]
61.47
30.48
54.96
62.87
37.14
49.38
46.36
DMN-ZS [82]
63.87
28.57
56.12
61.44
39.84
49.97
46.49
DPE (Ours)
63.41
30.15
56.72
63.72
40.03
50.81
47.66
CLIP- ViT-B/16 [46]
66.73
47.87
60.86
73.98
46.09
59.11
57.20
Ensemble
68.34
49.89
61.88
77.65
48.24
61.20
59.42
CoOp [85]
71.51
49.71
64,20
75.21
47.99
61.72
59.28
TPT [54]
68.98
54.77
63.45
77.06
47.94
62.44
60.81
DifTTT [13]
70.30
55.68
65.10
75.00
62.28
60.52
TDA [27]
69.51
60.11
64.67
80.24
50.54
65.01
63.89
TPS [56]
70.19
60.08
64.73
80.27
49.95
65.04
63.76
DMN-ZS [82]
72.25
58.28
65.17
78.55
53.20
65.49
63.80
DPE (Ours)
71.91
59.63
65.44
80.40
52.26
65.93
64.43
evaluation of robustness to natural distribution shifts, we assess the performance of our method using the ImageNet [7] dataset alongside its variant out-of-distribution datasets, including ImageNet-A [21], ImageNet-Ve V2 [47]. ImageNet-R [19], and Touggeeet-Sketeee [63]. (2) For cross- dasssess generalization
tasks. we conduct comprehensive assessments across 10 diverse recognition datasets. including
FGVCAircraft [39], Caltech 101 [12], StandfordCars [30], DTD [6], EuroSAT [18], Flowers 1 (02 [41],
Food101 [2]. OxfordPets [44], SUN397 [68], and UCF101 [55]. These datasets offer a comprehensive benchmark for evaluating the robustness of various methods across different distributional variations.
Implementation Details. We follow previous works [54, 13] to adopt ResNet-50 [16] and ViT- B/16 [9] backbones as the visual encoder of CLIP. In Appendix C.2, we detail the specific hand-crafted prompts utilized for each dataset. Following TPT [54], generate 63 augmented views for each test image using random resized cropping to create a batch of 64 images. We learn the prototype residual parameters using AdamW [37] optimizer with a learning rate of 0.0005 for a single step. In default, the scale factor À in Equation (10) is set to 0.5, the normalized entropy threshold is set to 0.1, and the queue size M set to For the affinity function in Equation (5), we
6.0 and 3 = 5.0, respectively. All experiments are conducted on single 48GB NVIDIA RTX 6000 Ada GPU. To ensure the reliability of our results, we perform each experiment three times using different initialization seeds and report the mean accuracy achieved.
Baselines. We compare our method with established test-time adaptation approaches for CLIP: (1) TPT [54], prompt tuning method that aims to minimize self-entropy across predictions of multiple augmented views; (2) DiffTPT [13], enhanced version of TPT that utilizes diffusion-based augmentations to optimize prompts; (3) TDA [27], a training-free, adapter- based method which constructs positive and negative caches during test time. (4) TPS [56], an efficient approach that dynamically learns shift vectors for per-class prototypes based solely on the given test sample; method that utilizes a dynamic memory aggregate
(5) DMN-ZS [82], a backpropagation-Fiee information from historical test data. Additionally, we present the zero-shot performance of CLIP using the simple prompt "a photo of {CLASS}" as well the results from prompt ensembling to show the absolute performance improvements. We also report the performance of CoOp [85], a train-time adaptation method, using 16-shot annotated samples per class on ImageNet. For fair comparison, we directly report the results of these baselines from their respective original papers. Note that in the DiffTPT [13] paper, the results are based on a subset of the datasets containing 1,000 test samples. This limited sample size may introduce potential imprecision in the reported results.
7

---

Table 2: Performance comparisons cross-datasets generalization. We also present top-1 accuracy (%) for all methods on two backbones of CLIP. The best results are highlighted in bold.
Method
Aircraft
Caltech
Cars
DTD
BuroSAT
Flover
Food101
Pets
SUN397 UCFI01
Average
CLIP- NeIP-
15.66
85.88
55.70
40.37
23.69
61.75
73.97
83.57
58.80
58.84
55.82
Ensemble
16.11
87.26
55,89
40.37
25.79
62.77
74.82
82.97
60.85
59.48
56.63
CoOp [85]
15.12
86.53
55,32
37.29
26.20
61.55
75.59
87.00
58,15
59.05
56.18
TPT [54]
17.58
87.02
58.46
40.84
28.33
62.69
74.88
84,49
61.46
60.82
57.66
DifTTT [13]
17.60
86.89
60.71
40.72
41.04
63.53
79.21
83.40
62.72
62.67
59.85
TDA [27]
17.61
89.70
57.78
43.74
42.11
68.74
77.75
86,18
62.53
64.18
61.03
DPE (Ours)
19.80
90.83
59.26
50.18
41.67
67.60
77.83
85.97
64.23
61.98
61.93
CLIP-VIT-BI1
23.67
93.35
65.48
44.27
42.01
67.44
83.65
88.25
62.59
65.13
63.58
Ensemble
23.22
93.55
66.11
45.04
50.42
66.99
82.86
86.92
65.63
65.16
64.59
CoOp [85]
18.47
93.70
64.51
41.92
46.39
68.71
85.30
89.14
64.15
66.55
63.88
TPT [54]
24.78
94.16
66.87
47.75
42.44
68.98
84.67
87.79
65.50
68.04
65.10
DiffTPT [13]
25.60
92.49
67.01
47,00
43.13
70.10
87.23
88.22
65.74
62.67
65.47
TDA [27]
23.91
94.24
67.28
47.40
58.00
71.42
86.14 14
88.63
67.62
70.66
67.53
DPE (Ours)
28.95
94.81
67.31
54.20
55.79
75.07
86.17
91.14
70.07
70.44
69.40
4.2
Results and Discussions
Robustness to Natural Distribution Shifts. In Table 1, we first compare the performance of our method with other state-of-the-an methods on in-domain ImageNet and its 4 out-of-distribution variants. Due to domain shifts, zero-shot CLIP [46] underperforms in out-of-distributiio scenarios. As shown in the table, adopting prompt ensembling and prompt learning methods like CoOp [85] can enhance CLIP's generalizability. However, it is important to note that CoOp is train- time- adaptation method that requires an annotated training set, limiting its effectiveness in real-world settings. Despite this, method still exhibits significant performance gains of 4.20% and 4.21 % on average across two different backbones compared to CoOp. indicating the superiority of our DPE in enhancing generalization capability out-of-distribution domains.
Focusing on test-time adaptation methods, the experimental results demonstrate that our method datasets achieves superior zero-shot generalization performance across various state-of-the-an prompt compared to other approaches. Specifically, our method outperforms existing tuning methods, surpasses TPT [54] by 3.55% and 3.49% and DifTTPT [13] by 2.10% and 3.6 3.65% on average when using ResNet-50 and ViT-B/16 backbones, respectively. Moreover, our method also outperforms cache-based TDA [27] by margins of 1.23% and 0.92% across two different backbones, indicating the effectiveness our DPE approach. Moreover, our DPE demonstrates performance advantages over the recent TPS [56] and DMN-ZS [82] approaches. outperforming them by 1.43 % and 0.84% on average across 5 datasets using the ResNet-50 backbone, further highlighting the superiority of our method. We also demonstrate that our DPE also be effectively applied to prompts learned using CoOp [85] with a 16-shot ImageNet setup. We compare the performance with other methods on the same 5 datasets in Appendix A. where our method consistently demonstrates competitive performance. These results highlight the general effectiveness of our proposed test-time adaptation method in both in-domain and -uttof-distributton scenarios.
Generalization. In Table 2, we further assess the generalizability of our proposed method against other state-of of-the-art methods 10 fine- -ne-girined recognition datasets. Given the significant distributional differences, methods may exhibit variable performance across these datasets. Notably, our method, which is not trained on any annotated data, significantly outperforms CoOp [85] by average margins of 5.75% and 5.52% on two respective backbones. When compared to other test-time adaptation methods, DPE exhibits average performance gains of 2.08% to 0.90% compared to Di and TDA, respectively. On the more advanced ViT-B/16 backbone, DPE continues to outperform existing approaches 7 of 10 datasets, with average improvements ranging from 1.87% to 4.30%. These results demonstrate the superior robustness and adaptability of our method in transferring to diverse domains during test time, which is crucial for real-world deployment scenarios.
8

---

Formula Efficiency Comparison. Table 3 presents
Table 3: Efficiency comparison on ImageNet [7]. We report the testing time, the achieved accuracy, and the performance gains compared to zero-shot CLIP.
a comparison of our method's efficiency against other test-time adaptation ap- proaches for VLMs, evaluated on 50,000 test samples from the ImageNet [7] dataset. The comparison is conducted on a single 48GB NVIDIA RTX 6000 Ada GPU. In our DPE method, the main computational overhead arises from the visual prototype evolution and prototype residual learning components. Specifically, while zero-shot CLIP requires 10.1 1 ms to infer a single im- age, incorporating our prototype residual learning increases the inference time 64.7 ms per image. Further including the visual prototype evolution extends this to 132.1 ms per image.
Method
Testing Time
Accuracy
Gain
CLIP [46]
9 min
59.81
TPT [54]
9 h 15 min
60.74
+0.93
DifITPT [13]
> 20 h
60.80
+0,99
TDA [27]
1 h 5 min
61.35
+1.54
TPS [56]
55 min
61.47
+1.66
DPE (Ours)
1 h 50 min
63.41
+3.60
Our proposed method shows improved computational efficiency compared to other prompt tuning methods, for example, 5x faster than TPT [54] and over 10x faster than Di [13], as requires only learning the prototype residues without the need to backpropagate gradients through the textual encoder. While our method is less efficient than TDA [27] and TPS [56], as we still backpropagate gradients to update
prototypes, it offers notable performance advantages.
4.3 Ablation Studies
Different Textual Prototype Evolution Rules. In Table 4, we report the perfor- mance on ImageNet [7] using different textual prototype evolution rules. We have the following key observations: (1) Fully updating our textual prototypes t to the optimized prototypes t' for each individual test image results in collapsed perfor- mance; (2) Compared to not evolving the textual prototypes, using an exponential moving average update rule with decay rate of 0.99 leads to a slight performance improvement of 0.18%; however, setting a lower decay rate of 0.95 decreases the performance by 0.36%. (3) Our cumulative average update rule yields the highest performance, achieving a 0.48% improvement compared to no update on ImageNet [7].
Table 4: Performance comparison using different tex- tual prototype evolution rules on ImageNet [7]. For each method, present the update rule formula and report the resulting accuracy on the ImageNet dataset.
Update Rule
Accuracy
No Update
t 5-t
62.93
Full Update
t + t
21.83
Exponential Avg.
t <- 0.99t + 0.01t
63.11
Exponential Avg. t t- 0.95t + 0.05t
62.57
Cumulative Avg. t ((k: t))
63.41
Hyperparameters for Dual Prototype Evolution. We provide sensitivity analysis for the hyperpa- rameters T; and M on the Caltech 101 [12] dataset in Figure (Left). Specifically, T represents the normalized entropy threshold for evolving our textual prototypes. When T; = 0, our method does not evolve the textual prototypes, leadingg leading to a significant performance decrease, as shown in Figure (Leff). Moreover, setting т 0.1 results in the highest performance, whereas higher threshold leads to a slight decrease in performance. Additionally, the queue size M acts as a soft threshold hy perparrameeer for evolving the visual prototypes. Our setting of M = 3 consistently yields the highest performance. Lowering M causes the visual prototypes to fail in capturing the diversity of test samples from the same class, while increasing M introduces additional low-confidence noisy samples that hinder discrimination among target classes. Notably, our DPE method consistently outperforms other approaches across reasonable range of hyperparameter settings: all combinations of entropy threshold Te 2 0.1 and queue size M > 3 achieve over 90.3% accuracy on Caltech 101, whereas TPT [54] and TDA [27] only achieve 87.02% and 89.70% respectively.
Effects of Different Learnable Modules. Recall that in our DPE method, we optimize our multi- modal prototypes by introducing two sets of learnable residual parameters È and è for each individual test image. In Figure 4 (Middle), we ablate the effects of each set of learnable residual parameters and report the performance across three datasets. Specifically, on ImageNet [7], optimizing only the textual prototypes for individual samples results in 1.40% improvement, while optimizing only the visual prototypes yields non-trivial 0.36% improvement, compared to keeping both i and v fixed. Optimizing both sets of residual parameters leads a further performance increase, e.g., by 1.52% on ImageNet [7]. This indicates both learnable modules contribute to the overall effectiveness of DPE.

---

No Settnement
35
91.0
Revitual
90.5
Bath Fand
Imagetett
50.0
ImageNeri22
59.0
01.5
2011
54
50
Ste
52
Extrapy Terestoid
50
Imagelet
ImageNet-V2
59
Figure 4: Ablation studies. (Left) Sensitivity analysis of Ti and M on Caltech101 [12]; (Middle) Analysis of the performance contributions from various learnable parameter settings across three datasets; (Right) Performance on three datasets with varying scale factor in Equation (10).
Scaling the Alignment Loss. Finally, we ablate the effect of the alignment loss by varying the scale factor À in Figure 4 (Right). Compared to optimizing solely using entropy minimization loss (i.e., = 0) during test-time adaptation, applying the additional alignment loss results in performance improvement of 0.23% to 1.07% across three different datasets. However, there is trade-off between prototype alignment and self-entropy minimization: setting À too high leads to performance drop. Our experiments show that our setting of X = 0.5 yields the highest performance.
Impact of Varying Update Steps. In Equa- tion (10), we update the multi-modal prototypes with a single update step for each test instance. To evaluate the impact of different numbers of update steps on overall performance, we conduct ablation experiments by varying the number of update steps from to 5 and report the resulting performance ImageNet. As shown in Table 5, the number of update steps does not significantly influence performance (within a range of 0.2%) While increasing the update steps to 2 yields a slight performance gain of 0.04%, also leads proportional decrease in inference efficiency. Given this trade-off, we adopt the single-step update as the default for balancing efficiency and performance.
Table 5:
Ablation studies on different update steps in prototype residual learning. We vary the number of update steps from 1 to 5 and report the achieved performance on ImageNet [7].
# Steps
3
5
Accuracy
63.41 63.45
63.28
63.26
63.32
5 Conclusion
In this work, we introduce Dual Prototype Evolving (DPE), a novel and effective approach for enhancing the zero-shot generalizability of VLMs during test time. Unlike previous methods that only focus adapting the VLMs from modality, create and evolve two sets of prototypes- textual and visual- progressvoly capturing more accurate multi-modal representations for target classes during test time. Moreover, also introduce prototype residual learning optimize the dual prototype sets for each individual test sample, which further enhances the test-time generalization capabilities of VLMs. Through comprehensive experiments, demonstrate that our proposed DPE achieves state-of-tt -of-th---ttt-ttt- performance while also exhibiting competitive test-time efficiency.
Linitations. While our proposed DPE method effectively adapts CLIP to out-of-distributio domains during test time, we identify two potential limitations: (1) It still requires gradient backpropagation to optimize the multi-modal prototypes. This optimization process introduces additional computational complexity compared to zero-shot CLIP [46], which may affect its real-time performance practical deployment scenarios. (2) Since DPE needs to maintain priority queues to evolve the visual prototypes, it increases the memory cost during inference.
Broader Impacts. In this work, we aim to build more reliable machine learning systems by leveraging the extensive knowledge of current foundational models, specifically CLIP [46]. Specifically, we follow TPT [54] to apply the test-time adaptation setting to rision-languas world scenarios. By employing our DPE approach, the CLIP model can adapt itself to diverse domains during test time, which enhances its practical applicability in real-world deployment scenarios. We hope this work inspires future studies to focus on the generalization and robustness of pre-trained large-scale foundation models.
models to align with real-
10

---

Acknowledgements
This work has been funded part by the Army Research Laboratory (ARL) award --I-NF-23-2- 0007, DARPA award FA8750-23-2-1015, and ONR award N00014-23-1-2840.
References
[1] Jameel Abdul Samadh, Mohammad Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muhammad Muzammal Naseer, Fahad Shahbaz Khan, and Salman H Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. Advances in Neural Information Processing Systems, 96:80396-80413 2023.
[2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 100--011-mma 101-minin discriminative components with random forests. In European Conference on Computer Vision, pages 446-461. Springer, 2014. 5, 7,
[3] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free free online test-time adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8344-8355 8353, 2022. 3
[4] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive sanguage-image image learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2818-2829, 2023. 17,
[5] Eulrang Cho, Jooyeon Kim, and Hyunwoo J Kim. Distribution-aware prompt tuning for vision- language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22004-22013, 2023.
[6] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3606-3613, 3613, 2014. 7,20
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. 7, 9, 10, 17, 18, 20
[8] Zeshuai Deng, Zhuokun Chen, Shuaicheng Niu, Thomas Li, Bohan Zhuang, and Mingkui Tan. Efficient test-time adaptation for super-resolution with second-order degradation and reconstruction. Advances Neural Information Processing Systems, 36.74671-74701, 2023.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021. URL: https://openreviee www.eeeoerriddiiiieeooeoe
[10] Danny Driess, Fei Xia, Mehdi M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. PaLM-e: An embodied multimodal language model. International Conference on Machine Learning. pages 8469 8488. PMLR, 2023. 3
[11] Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. In Proceedings of the International Joint Conference Artificial Intelligence, pages 5436-5443, 2022.
[12] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on object categories. Computer Vision and Image Understanding, 106(1):59-70, 2007. 7, 9, 10, 20

---

[13] Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmen- tation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2704-2714, 2023. 2, 6, 7, 8, 9, 18
[14] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better dision-lanuual models with feature adapters. International Journal of Computer Vision, 132:581-595, 2024.
[15] Hao Guan and Mingxia Liu. Domain adaptation for medical image analysis: survey, IEEE Transactions on Biomedical Engineering, 69(3): 171- 1185, 2021. 1
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016. 7
[17] Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2028-2038, 2023.
[18] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
7, 20
[19] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-
generalization.
8349, 2021. 7, 20
[20] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019. URL: https:tpp// : hppp//eeeeeeeeeeeeeeee www.eeeeweiddtllaieoeee
[21] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021. 1,7, 7,7,20 20
[22] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Yan Zhu, Phillip Isola, Kate Saenko, Alexci Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning, pages 1989-1999 PMLR, 2018.
[23] Minhao Hu, Tao Song, Yujun Gu, Xiangde Luo, Jieneng Chen, Yinan Chen, Ya Zhang, and Shaoting Zhang. Fully test-time adaptation for image segmentation. In International Conference on Medical Image Computing and Computer Assisted Intervention, pages 251-260. Springer, 2021.
[24] Xueting Hu, Ce Zhang, Yi Zhang, Bowen Hai, Ke Yu, and Zhihai Hc. Learning to adapt clip for few-shot monocular depth estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5594-5603, 2024. 3
[25] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung. Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference Machine Learning, pages 4904-4916, 2021. 1, 3, 17
[26) Zhehan Kan, Shuoshuo Chen, Ce Zhang, Yushun Tang, and Zhihai He. Self-correctable and adaptable inference for generalizable human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5537-5546, 2023. 3
[27] Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2, 3, 5, 7, 9, 18, 20
12

---

[28] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-mode prompt learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19113-19122, 2023. 2, 3, 19
[29] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[30] Jonathan Krause, Michael Stark, Jia Deng. and Li Fei-Fei. 3d object representations for fine- grained categorization. Proceedings of the IEEF/CVF International Conference on Computer Vision Workshops, pages 554-561, 2013. 7, 20
[31] Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, and Xinchao Wang. Graphadapter: Tuning vision- laiiunla models with dual knowledge graph. Advances in Neural Information Processing Systems, 36: 111113348-14666 2023.
[32] Yangguang Li, Feng Liang. Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-iman image pre-traiming paradigm. In International Conference on Learning Representations, 2022. URL:
https:://peerreveez net/ wwwe/fooommmiiiie
1
[33] Yizhuo Li, Miao Hao, Zonglin Di, Nitesh Bharadwaj Gundavarapu, and Xiaolong Wang. Test-time personalization with transformer for human pose estimation. Advances in Neural Information Processing Systems, 14:2583-2599) 2021.
[34] Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. International Journal of Computer Vision, pages 1-34, 2024. 1,3
[35] Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramanan. Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models. In Proceedings of the IEEC/CVF Conference Computer Vision and Pattern Recognition, pages 19325-19337, 2023. 2
[36] Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. Remoteclip: vision language foundation model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 2024. 3
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL: id Id-Bkg6RiCqY7. 7
wttprreviviieeeefftreeeee net/forum?
[38] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision- lassunn-anguage models. Advances in Neural Information Processing Systems, 36:65252-6523 2023. 3
[39] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.515/ 2013. 7,20
[40] M Jehanzeb Mirza, Jakub Micorek, Horst Possegger, and Horst Bischof. The norm must go on: Dynamic unsupervised domain adaptation by normalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14765-14775, 2022. 2
[41] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, pages 722-729. IEEE, 2008 20
[42] Shuaicheng NIu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In International Conference on Learning Representations, 2023. URL: https: httpps://openreviee net/forunt id= ad-g2YraF751j.2
13

---

[43] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint vrXiv:1807.03744 2018.
[44] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3498-3505, 2012. 7.20
[45] Sarah Pratt, Ian Covert, Rosanne I Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15691-15701, 2023. 20
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference Machine Learning, pages 8748-8763. PMLR, 2021. 2, 4, 8, 9, 10, 18, 20
[47] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019. 7, 20
[48] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 779-788, 2016.
[49] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252, 2015. 1
[50] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25299 2022.1
[51] Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E Gonzalez, Kurt Keutzer, and Trevor Darrell. Multitask vision-language prompt tuning. In Proceedings of the IEEF/CVF Winter Conference Applications of Computer Vision, pages 5656-5667, 2024. 3
[52] Inkyu Shin, Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, In So Kweon, and Kuk-Jin Yoon. Mm-tta: Multi-modal test-time adaptation for 3d semantic segmentation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16928-16937, 2022.
[53] Assaf Shocher, Nadav Cohen, and Michal Irani. "zero-shot" super-resolution using deep internal learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 18 3126, 2018.
[54] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances Neural Information Processing Systems, 35:14274-14284 2022. 2, 3, 4, 6, 7, 8, 9, 10, 18, 20
[55] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint arXiv:1212.0400. 2012. 7, 20
[56] Elaine Sui, Xiaohan Wang, and Serena Yeung-Levy. Just shift it: Test-time prototype shifting for zero-shot generalization with vision-language models. Proceedings of the IEEF/CVF Winter Conference Applications of Computer Vision, 2025. 3, 7, 8, 19
14

---

[57] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In Interaatoonal Conference on Machine Learning, pages 9229-9248. PMLR, 2020. 2
[58] Hui Tang and Kui Jia. Discriminative adversarial domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5940-5947, 2020.
[59] Yushun Tang, Shuoshuo Chen, Zhihe Lu, Xinchao Wang, and Zhihai He. Dual-path adversarial lifting for domain shift correction in online test-time adaptation. In European Conference on Computer Vision, 2024. URL: https://wwx ecva. net net/papers/eeeee http:/pppps//ccccce0eee pdf.
sateerrrecvv_2244400 2024// 20w.2/ape22.....ppppppp
BCCC/papers////
[60] Yushun Tang, Ce Zhang, Heng Xu, Shuoshuo Chen, Jie Cheng, Luziwei Leng, Qinghai Guo, and Zhihai He. Neuro-modulated hebbian learning for fully test-time adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3728-3738, 2023. 1,3
[61] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of Machine Learning Research, 9:2579-260S. 2008. 5
[62] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021. URL: https::/t///// //tp:r/////eeevveeee/ee http:pp/wrreie.eeeeee/errrrrieie mee//foruum/dddxx3332k.rr.
1,
[63] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power In Advances in Neural Information Processing Systems, volume 32, pages 10506-10518, 2019. 7, 20
[64] Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312:135-1533 2018.
[65] Ximei Wang, Liang Li, Weirui Ye, Mingsheng Long, and Jianmin Wang. Transferable attention for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 33, pages 5345-5352, 2019. 1
[66] Yixuan Wei, Han Hu, Zhenda Xie, Ze Liu, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Improving clip fine-tuning performance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1439-54499 2023. 3
[67] Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open- pocabuacyy detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7031-7040, 2023. 3
[68] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9485-3492 2010. 7, 2,20
[69] Hee Suk Yoon, Eunscop Yoon, Joshua Tian Jin Tee, Mark A. Massegawa-Johnson Yingzhen Li, and Chang D. Yoo. C-TPT: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. In International Conference on Learning Representations, 2024. URL:
https://openrew// hwtp:eiwwwiieeeeeeeeeeeeeeeeeeee wwwpeeewwwereeeeeeeeeeeeeeereeee dow::m/mdddjje22/rFF jZZEHTBFOT. 3, 6
[70] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. URL: htttps::/oppereviieee http::///ppppeeeeeeeeeeeetren/ee wwt..//toorrrr11--E2E/PP// 1, 17
[71] Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning vision- language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10899-10909, 2023. 1,3, 19
[72] Maxime Zanella and Ismail Ben Ayed. On the test-time zero-shot generalization of vision- language models: Do we really need prompt learning? In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 23783-23793, 2024, 3
15

---

[73] Ziyao Zeng, Daniel Wang, Fengyu Yang, Hyoungseob Park, Stefano Soatto, Dong Lao, and Alex Wong. Wordepth: Variational language prior for monocular depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 9708 9718,, 2024.
[74] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-imen text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123-18133, 2022. 1, 17
[75] Ce Zhang, Simon Stepputtis, Joseph Campbell, Katia Sycara, and Yaqi Xie. HiKER-SGG: Hier- archical knowledge enhanced robust scene graph generation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 28233-28243 2024. 1
[76] Ce Zhang, Simon Stepputtis, Katia Sycara, and Yaqi Xie. Test-time prototype evolving for generalizable vision-language models. In JCML 2024 Workshop on Foundation Models in the 3
Wild, 2024. URL: https: http:///eeeeeew...eeeeeeorrrrrre
[77] Ce Zhang, Simon Stepputtis, Katia Sycara, and Yaqi Xie, Enhancing vision-language few-shot adaptation with negative learning. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, 2025. 3
[78] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024. 3
[79] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35:38629-38642, 2022.
2. 3
[80] Renrui Zhang, Ziyao Zeng, Ziyu Guo, and Yafeng Li. Can language understand depth? In Proceedings of the 30th ACM International Conference on Multimedia, pages 6868 (868-6874, 2022.
[81] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training ffree adaption of clip for few-shot classification. In European Conference on Computer Vision, pages 493-510. Springer, 2022. 1,3
[82] Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision-language models. In Proceedings the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18718-287728, 2024. 7. 19
[83] Yizhe Zhang, Shubhankar Borse, Hong Cai, and Fatih Porikli. Auxadapt: Stable and efficient test-time adaptation for temporally consistent video semantic segmentation. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 2339-223222 2348, 2022.
3
[84] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for Iision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825 2022. 1,3
[85] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. International Journal of Computer Vision, 130(9):3337-2348 2022. 1,
7, 8, 17, 18, 20
[86] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15659- 15669, 2023,
16

---

Dual Prototype Evolving for Test-Time Generalization of Vision-Languaae Models
Appendix
In this supplementary document, we provide additional details and experimental results to enhance understanding and insights into method. This supplementary document is organized follows:
Full numerical results on robustness to natural distribution shifts are detailed in Section A.I.
We present additional performance comparisons on larger-scale VLMs, specifically OpenCLIP with a ViT-L/14 backbone, in Section A.2.
Sensitivity analysis of hyperparameters and /3 is provided in Section A.3.
We provide a sensitivity analysis of queue size M on Caltech101 and ImageNet, observing performance trends based its variations in Section A.4.
We evaluate the individual impact of each component, showing the significant contribution of VPE, TPE, and PRL to overall performance in Section A.5.
We analyze the effects of the alignment and self-entropy losses, highlighting their combined performance benefits on ImageNet in Section A.6
We highlight the differences between our approach and similar methods in Section B.
Detailed statistics for all utilized datasets are provided in Section C.I.
We present the specific textual prompts we used for each dataset in Section C.2.
We list the license information for all used assets in Section D.
A
Additional Experimental Results
A.1 Full Results on Robustness Natural Distribution Shifts
In Table AI, we compare the performance of our method with other state-of-the methods on in-domain ImageNet and its out-of-distribution variants. Specifically, we demonstrate that DPE can also be applied to prompts learned using CoOp [85] with 16-shot ImageNet setup. Our methods also demonstrates competitive performance compared to other methods. also important notice that, our proposed method accumulates task-specific knowledge over time, therefore can achieve higher performance gain on larger test (e.g., ImageNet-R and ImageNet-S).
A.2 Performance Comparisons Larger-Scale VLMs
Our DPE method can thcoretically be applied to various contrastively pre-trained rision-language models, such ALIGN [25], LiT [74], and CoCa [70]. In Table A2, we use larger-scale OpenCLIP (ViT-L/14) [4] as an example and compare the performance of TDA and our method on robustness to natural distribution shifts. We observe that DPE still outperforms TDA by 1.07% average across 5 datasets, showcasing that our method generalizes well to larger-scale VLMs.
A.3 More Sensitivity Analyses of Hyper-Parameter
In our experiments on ImageNet [7].
Table A3: Sensitivity of hyper-parameters. All the results are reported on ImageNet [7] using ResNet-50 backbone.
we set the hyperparameters and 8
as defined in Eq. (5) to 6.0 and 5.0, respectively, as detailed in the im- plementation section. To thoroughly examine the impact of different hy- perparameters, we performed sen- sitivity analysis by varying each hy- perparameter individually and as- sessing the performance on ImageNet with a ResNet-50 backbone, as shown in Table A3. The results show that our selected values of 6.0 and B 5.0 provide the best performance.
2.5
4.0
5.0
6.0
7.5
10.0
62,83
63.17
63,28
63.41
63.07
62.43
2.0
3.0
4.0
5.0
6.0
7.0
62.85
63.02
63.30
63.41
63.37
63.29
17

---

Table A1: Performance comparisons robustness to natural distribution shifts. We present top-1 accuracy (%) results for all evaluated methods employing both ResNet-50 and ViT-B/16 visual backbones of CLIP. Additionally, we the performance using prompts learned by CoOp [85] with 16-shot training data per class on ImageNet [7]. The best results are highlighted in bold.
Method
ImageNet
ImageNet-A
ImageNet-V2
ImageNet-R
ImageNet-S
Average
OOD Average
LLI--ResNet-50 [46]
58.16
21.83
51.41
56.15
33.37
44.18
40.69
Ensemble
59.81
23.24
52.91
60.72
35.48
46.43
43,09
TPT [54]
60.74
26,67
54,70
59.11
35,09
47.26
43,89
DiITPTT
60.80
31.06
55.80
58.80
37.10
48.71
45.69
TDA [27]
61.35
30.29
55.54
62.58
38.12
49.58
46.63
Ours
63.41
30.15
56.72
63.72
40,03
50.81
47.66
(+: 0.23)
(#: 0.41)
(= 0.22)
(# 0.20)
(+ 0.11)
(+: 0.21)
(+ 0.22)
CoOp [85]
63.33
23.06
55.40
56.60
34.67
46.61
42.43
TPT CoDp [54]
64.73
30.32
57.83
58.99
35.86
49.55
45.75
DifTTT + CoOp [13]
64.70
32.96
61.70
58.20
36.80
50.87
47.42
Ours + CoOp
64,86
30.08
57.96
59.78
37,80
50.10
46.41
0.18)
0.27)
0.31)
0.19)
0.17)
0.22)
0.23)
CLIP-VIT--IE B/T- [46]
66.73
47,87
60.86
73.98
46.09
39.11
$7.20
Ensemble
68.34
49.89
61.88
77.65
48.24
61.20
$9.42
TPT [54]
68.98
63.45
77.06
47.94
62.44
60.81
DiFTPT [13]
70.30
55.68
65.10
75.00
46.80
62.28
60.52
TDA [27]
69.51
60.11
64.67
80.24
50.54
65.01
63.89
Ours
71.91
59.63
65.44
80.40
52.26
65.93
64.43
(#: 0.09)
0.18)
0.17)
0.24)
0.16)
0.18)
CoOp [85]
71.51
49.71
64.20
75.21
47.99
61.72
59.28
TPT + CoOp [54]
73.61
57.95
66.83
77.27
49.29
64,99
62,83
DifTTPT + CoOp [13]
75.00
58.09
66.80
73.90
49.50
64.12
61.97
Ours CoOp
73.67
59.43
66.38
78.49
50.78
65.75
63.77
(+: 0.14)
(#: 0.36)
(= 0.32)
(#: 0.06)
(::: 0.08)
(#: 0.23)
(#: 0.26)
Table A2: Performance comparisons on robustness to natural distribution shifts. We present top- 1 accuracy (%) results for all evaluated methods employing larger-scale ViT-L/14 visual backbones of OpenCLIP [4]. The best results are highlighted in bold.
Method
ImageNNt
ImageNet-A
ImageNet-V2
ImageNet- R ImageNet-S
Average
OOD Average
OpenCLIP (VIT-L/14)(6)) [46]
74.04
53.88
67.69
87.42
63.18
69.31
68.13
TDA [27]
76.28
61.27
68.4
88.41
64.67
71.81
70.69
DPE (Ours)
77.87
61.09
70.83
89.18
66.33
73.06
71.86
A.4
Ablation Study on Queue Size
In Figure 4 (Left), we provide sensitivity analysis of queue size M on the Caltech101 dataset. We further analyze the impact of hyperparameter M on larger-scale ImageNet in Table A4. Similar to the results on Caltech 101, we observe that the performance increases by 0.5% when adjusting M from I to 3 but exhibits a slight decrease of 0.2% when further increasing to 7. We speculate that initially increasing the value of M allows our priority queue to collect more diverse features and obtain representative prototypes. However, further increasing it leads to the inclusion of more low-confidence noisy samples, which has adverse effects.
A.5 Effectiveness of Each Component
We conduct additional ablation experiments to analyze the individual effect of each component in Table A5. In the table, VPE, TPE, and PRL refer to visual prototype evolution, textual prototype evolution, and prototype residual learning, respectively. Note that Experiment #3 is invalid since TPE requires optimized textual prototypes t" from PRL. As shown, VPE is the most influential component, providing ~2% improvement over zero-shot CLIP. The other two components also contribute significantly to the overall performance.
A.6 Ablation Study on Two Loss Terms
The alignment loss Latign acts from global perspective by promoting consistent multi-modal prototypes, ensuring that the representations are aligned for all subsequent test samples. The
18

---

Table A4: Ablation studies on different values of M (priority queue size).
Values of M
2
5
6
ImageNet Acc.
62.91
63.17
63.41
63.34
63.29
63.29
63.21
Table AS: Effectiveness of different algorithm components. VPE, TPE, and PRL refer to visual prototype evolution, textual prototype evolution, and prototype residual learning, respectively.
Table A6: Effects of self-entropy loss and alignment loss.. Specifically, we apply the self-entropy loss (Laug) and alignment loss (Calign) individually and in combination, and report the accuracy on ImageNet using the ResNet-50 backbone.
#
VPE
TPE
PRL
ImageNet Acc.
1
X
X
X
59.81
2
/
X
61.83
Laug
Lalign
ImageNet Acc.
3
X
4
X
X
61.59
X
X
61.90
5
J
J
X
61.90
X
63.18
6
V
X
<
62.93
X
/
62.46
7
62.48
8
63.41
/
/
63.41
self-entropy loss Laug. in contrast, greedily targets improving individual sample predictions by penalizing high-entropy predictions across augmented views. To provide a clearer understanding, analyze the effects of the two loss terms ImageNet using the ResNet-50 backbone and report the performance Table A6. We can observe that while the alignment loss alone improves the performance by 0.56%, the self-entropy loss provides greater performance gain of 1.28%. Combining both loss terms further enhances performance by an additional 0.23%.
B Further Discussions on Related Work
We acknowledge that our DPE method shares some high-level ideas with DMN-ZS, TPS, TaskRes and MaPLe. However, there are some key distinctions. Here, we discuss the differences between our method and these approaches, respectively:
DMN [82]. While DMN(-ZS) also utilizes historical test samples to enhance the test-time gen- cralizability of VLMs, it only updates the visual memory online while keeping the textual fea- tures/classifier unchanged. Therefore, we consider DMN similar to TDA, as both methods adapt CLIP only from a uni-modal (visual) perspective. In contrast, our DPE designed progressively capture more accurate multi-modal representations on the fly with test samples.
TPS [56]. Similarly, since TPS only updates the textual prototypes during testing, we categorize it with TPT and Dif which also account only for uni-modal (textual) adaptation. Moreover, TPS has similar limitations to TPT, as discussed in Lines 46-49, where it treats each test instance independently, resetting to the original model for each new sample. In contrast, our DPE can accumulate task-specific knowledge as more test samples are processed.
TaskRes [71] and MaPLe [28]. While our method shares some similarities in method details (e.g., multi-modal prototype residuals), focus
completely different test-time adaptation setting. Specifically, TaskRes and MaPLe aim to adapt CLIP using labeled few-shot samples, whereas our proposed DPE approach leverages only the unlabeled target data stream adapt the model out-of-distribution domains. Moreover, we innovatively propose textual/visual prototype evolution, which enables our method progressively capture more accurate multi-modal representations during test time. The two works mentioned above, while effective in learning from few-shot samples, do not incorporate such knowledge accumulation techniques.
C Additional Implementation Details
C.1 Dataset Details
In Table C7, we present the detailed statistics of each dataset we used in our experiments, including the number of classes, the sizes of training, validation and testing sets, and their original tasks.
19

---

Table C7: Detailed statistics of datasets used in experiments. Note that the last 4 ImageNet variant datasets are designed for evaluation and only contain the test sets.
Dataset
Classes
Training
Validation
Testing
Task
Caltech101 [12]
100
4,128
1,649
2,465
Object recognition
DID [6]
47
2,820
1,128
1,692
Texture recognition
EuroSAT [18]
13,500
5,400
8,100
Satellite image recognition
FGVCAircraft [39]
100
3,334
3,333
3,333
Fine-grained aircraft recognition
Flowers 102 [41]
102
4,093
1,633
2,463
Fine-grained flowers recognition
Food101 [2]
101
50,500
20,200
30,300
Fine-grained food recognition
ImageNct [7]
1,000
1.28M
50,000
Object recognition
OxfordPets [44]
37
2,944
736
3,669
Fine-grained pets recognition
StanfordCars [30]
196
6,509
1,635
8,041
Fine-grained car recognition
SUN397 [68]
397
15,880
3,970
19,850
Scene recognition
UCF101 [55]
101
7,639
1,898
3,783
Action recognition
ImageNct-V2 [47]
1,000
10,000
Robustness of collocation
ImageNet-Sketch [63]
1,000
50,889
Robustness of sketch domain
ImageNet-A [21]
7,500
Robustness adversarial attack
ImageNet-R [19]
200
30,000
Rohustness of multi-domains
Table C8: Textual prompts used in experiments. In addition to these prompts, we also employ
CuPL [45] prompts further enhance performance.
Dataset
Prompts
"itap of (CLASS)."
ImageNet [7]
"a bad photo of the (CLASS)."
Image/NetVV2 [47]
origami (CLASS)."
ImageNet-Sketch [63]
" photo of the large [CLASS)."
Image/Net-A [21]
(CLASS) in video game."
ImageNet-R [19]
"art of the [CLASS)."
"a photo of the small (CLASS)."
Caltech101 [12]
"a photo of (CLASS)."
DTD [6]
"ICLASS) texture.""
EuroSAT [18]
"a centered satellite photo of (CLASS)."
FGVCAircraft [39]
"a photo of a (CLASS J, type of aircraft
Flowers [41]
"a photo of a (CLASS), ), a type of flower."
Food101 [2]
"a photo of (CLASS), a type of food."
OxfordPets [44]
"a photo of a (CLASS), ), a type of pet."
StanfordCars [30]
"a photo of (CLASS)."
SUN397 [68]
"a photo of (CLASS)."
UCF101 [55]
"a photo of person doing (CLASS)."
C.2 Textual Prompts Used in Experiments
In Table C8, we detail the specific hand-crafted prompts utilized for each dataset.
D License Information
Datasets.
We list the known license information for the datasets below:
MIT License: ImageNet-A [21], ImageNet- V2 [47], ImageNet-R [19], and ImageNet-Sketch [63].
CC BY-SA 4.0 License: OxfordPets [44].
Research purposes only: ImageNet [7], StandfordCars [30], DTD [6], FGVCAircraft [39], SUN397 [68].
Code. In this work, we also use some code implementations from existing codebase: CLIP [46],
CoOp [85], TPT [54], and TDA [27]. The code used in this paper are all under the MIT License.
20

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state our contributions in Abstract and also Section 1. These contributions well validated by our experimental results in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can be expected to generalize to other settings.
It fine include aspirational goals motivation long as it is clear that these goals attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations the work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-speciffcation, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on scope the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not. penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
21

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We believe that we have clearly introduced the proposed DPE method in the main texts for reproduction. We have provided all the training details, computational resources, and hyper-parameter settings in the implementation details in Section 4.1. Also, please be assured that we will make source code publicly available upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
model (e.g., large language model), then there should either
(d) We recognize that reproducibility may be tricky in some cases, in which authors welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access data and code
Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
22

---

Justification: (1) Data: All the datasets we used in this paper are publicly available online, and all the readers are free to download them. We list the statistics and license information of all the used datasets in Appendix C.1 and D. (2) Code: Code is available at https://gg thub. com/ zhangce01/DPE-On CLIP.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https:
guides/ Sutee//iieSSubeeeeee swriiissssniiiiieee cy) for more details.
While we encourage the release of code and data, we understand that this might not possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. cttpcccpppccccccccceccccccccccll Pwrides/CodeSSubne ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have specified all the training details in Section 4.1, as well sensitivity analysis of hyperparameters in Section 4.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the standard deviation over 3 random seeds in Table Al.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap. etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar the standard deviation the standard error of the mean.
23

---

It is OK to report 1 -sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We clearly state the computational resources used for experiments in Section 4.1. We also report the total testing time on ImageNet and compare this with other state-of- state-of-the-ard test-time adaptation methods in Section 4.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /ttps://neurips
Question:
muulic/EthicsGuice
Answer: [Yes]
Justification:
We have reviewed and adhered to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of this work in Section 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will foundational research and not tied to par- ticular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for
24

---

disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
To the best of our knowledge, this paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited all the assets used in our paper. We also list the license information of the used datasets and code in Appendix D.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include a URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ hom/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged reach the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not introduce new assets.
Guidelines:
25

---

The answer NA means that the paper does not release new assets.
Rescarchers should communicate the details of the dataset/code/model part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
26


Reviews:
[{"review_id": "53OpDgS5Pe", "reviewer": "Reviewer_Sqqz", "summary": "1. This paper proposes a novel test-time adaptation method (DPE) for VLMs that captures multi-modal representations for target classes during test time.\n2. This paper introduces and optimizes learnable residuals for each test sample to align the prototypes across modalities.\n3. The results of this paper are promising while maintaining competitive computational efficiency.", "questions": "1. Many concurrent works (such as DMN-ZS, TPS) have shown similar ideas, and comparisons with them including the core idea design and performance should be provided.\n2. A detailed analysis of hyperparameter M should be provided, and how to choose the M to obtain robust performance in the test-time scenarios also should be discussed.\n3. A detailed analysis of loss functions should be provided. As shown in the paper, more alignment brings performance drop, the reason should be discussed. By the way, should the text prototype be aligned in the direction of the visual prototype? Or does the visual prototype only need to be aligned in the direction of the text? Are there any related experiments?\n4. Efficiency comparison with backpropagation-free methods (such as TDA,TPS, andDMN) should be provided.", "limitations": "1. The core idea is not novel enough, and it seems to be a combination of existing several ideas.\n2. Detailed analysis of hyperparameters and loss functions is missing.\n3. Efficiency comparison is not comprehensive.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is very clear and easy to follow. This paper proposes that former methods just focus on the single-modality, and the proposed method DPE can learn from two-modalities. Meanwhile, DPE accumulates task-specific knowledge in a residual manner, which maintains computational efficiency.\n2. The experimental results are promising. Compared with other related methods, DPE achieves improvement on 15 benchmark datasets, and DPE shows improved computational efficiency compared with TPT and DiffTPT, which is practical for the test-time scenarios.\n3. The writing and presentation of this paper is good.", "weaknesses": "1. The core idea is not novel enough. Many concurrent works with TDA which is DPE's main comparison method have shown a similar idea, such as DMN-ZS[1] actually utilizes the information of two modalities while not demanding any backpropagation, and TPS[2] which has been cited in the paper proposes that residual prototype is useful for test-time prompt learning. Though the specific designs of DPE are different from them, the core idea is somewhat similar.\n2. More analysis of hyperparameters is needed. Though the introduction of visual information is helpful intuitively, hyperparameters such as (top-)M may affect the performance obviously, which should be explored comprehensively.\n3. More analysis of loss functions is needed. Alignment loss is helpful for performance improvement, but too much alignment leads to a performance drop, the reason should be discussed.\n4. Efficiency comparison is not comprehensive. Though DPE performs faster than TPT and DiffTPT, the efficiency comparison with backpropagation-free methods (such as TDA,TPS, and DMN) should be provided.\n\n[1] Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models.\n[2] Just shift it: Test-time prototype shifting for zero-shot generalization with vision-language models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Sqqz,\n\nWe really appreciate your thorough review of our paper!\n\n---\n\n**Comment (1)**: “*Comparisons with DMN-ZS [1] and TPS [2] including the core idea design and performance should be provided*.”\n\n**Response (1)**: Thank you for pointing this out. We acknowledge that our DPE method shares some high-level ideas with DMN-ZS and TPS. However, there are some key distinctions. Here, we discuss the differences between our method and these two approaches, respectively:\n\n- **DMN-ZS**: While DMN(-ZS) also utilizes historical test samples to enhance the test-time generalizability of VLMs, it only updates the visual memory online while keeping the textual features/classifier unchanged. Therefore, we consider DMN similar to TDA, as both methods adapt CLIP only from a uni-modal (visual) perspective. In contrast, our DPE is designed to progressively capture more accurate **multi-modal representations** on the fly with test samples.\n\n- **TPS**: Similarly, since TPS only updates the textual prototypes during testing, we categorize it with TPT and DiffTPT, which also account only for uni-modal (textual) adaptation. Moreover, TPS has similar limitations to TPT, as discussed in Lines 46-49, where it treats each test instance independently, resetting to the original model for each new sample. In contrast, our DPE can **accumulate** task-specific knowledge as more test samples are processed.\n\nWe have also included a performance comparison with these methods on robustness to natural distribution shifts:\n\n\n\n| Method         | ImageNet  | ImageNet-A | ImageNet-V2 | ImageNet-R | ImageNet-S |  Average  | OOD Average |\n| -------------- | :-------: | :--------: | :---------: | :--------: | :--------: | :-------: | :---------: |\n| CLIP-RN50      |   58.16   |   21.83    |    51.41    |   56.15    |   33.37    |   44.18   |    40.69    |\n| TPS           |   61.47   | **30.48**  |    54.96    |   62.87    |   37.14    |   49.38   |    46.36    |\n| DMN-ZS         | **63.87** |   28.57    |    56.12    |   61.44    |   39.84    |   49.97   |    46.49    |\n| **DPE (Ours)** |   63.41   |   30.15    |  **56.72**  | **63.72**  | **40.03**  | **50.81** |  **47.66**  |\n\nAs shown, our proposed DPE outperforms TPS and DMN-ZS by 1.43% and 0.84% on average across 5 datasets, demonstrating the superiority of our method. We have updated the results in Table 1.\n\n---\n\n**Comment (2)**: “*A detailed analysis of hyperparameter M should be provided, and how to choose the M … should be discussed*.”\n\n**Response (2)**: Thank you for your insightful comments. In Figure 4 (Left), we provided a sensitivity analysis of $M$ on the Caltech101 dataset. Following your comments, we further analyze the impact of hyperparameter $M$ on larger-scale ImageNet:\n\n\n| Values of $M$ | 1     | 2     | 3         | 4     | 5     | 6     | 7     |\n| ------------- | ----- | ----- | --------- | ----- | ----- | ----- | ----- |\n| ImageNet Acc. | 62.91 | 63.17 | **63.41** | 63.34 | 63.29 | 63.28 | 63.21 |\n\n\nSimilar to the results on Caltech101, we observe that the performance increases by 0.5% when adjusting $M$ from 1 to 3 but exhibits a slight decrease of 0.2% when further increasing $M$ to 7. We speculate that initially increasing the value of $M$ allows our priority queue to collect more diverse features and obtain representative prototypes. However, further increasing it leads to the inclusion of more low-confidence noisy samples, which has adverse effects.\n\n---\n\n**Comment (3)**: “*... more alignment brings performance drop, the reason should be discussed. By the way, should the text prototype be aligned in the direction of the visual prototype? Are there any related experiments?*”\n\n**Response (3)**: The alignment loss $\\mathcal{L}\\_{\\mathsf{align}}$ acts from a global perspective by promoting consistent multi-modal prototypes, ensuring that the representations are aligned for all subsequent test samples. The self-entropy loss $\\mathcal{L}_{\\mathsf{aug}}$, in contrast, greedily targets on improving individual sample predictions by penalizing high-entropy predictions across augmented views. Therefore, overly emphasizing alignment may cause the method to prioritize global consistency over the refinement of individual predictions, leading to less accurate results for specific samples.\n\nBesides, rather than solely aligning the visual prototypes in the direction of the text or vice versa, our DPE mutually aligns the textual and visual prototypes, updating both prototypes simultaneously. In Figure 4 (Middle), we keep either the textual or visual prototypes fixed and only align the prototypes from the other modality towards the fixed prototypes. We demonstrated that optimizing prototypes from both modalities achieves the best performance gain of 1.52%.\n\n---\n\n**Comment (4)**: “*The efficiency comparison with backpropagation-free methods should be provided*.”\n\n**Response (4)**: Thank you for your insightful feedback. We would like to clarify a detail regarding your comment: TPS actually requires backpropagation with self-entropy loss for each sample, similar to TPT.\n\nFollowing your comment, we compare the inference time per image for each method (using a single A6000 Ada GPU): \n\n\n\n| Method              | CLIP | TPT   | DiffTPT | TPS  | TDA  | TDA* | DMN-ZS | DMN-ZS* | Ours | Ours* |\n| ------------------- | ---- | ----- | ------- | ---- | ---- | ---- | ------ | ------- | ---- | ----- |\n| Inference time (ms) | 10.1 | 668.3 | >2000   | 65.2 | 13.4 | 73.5 | 11.8   | 61.6    | 66.7 | 132.1 |\n\n\nIn the table, $^*$ indicates that pseudo-label predictions are enhanced with $N=64$ augmented views and confidence selection, which increases inference time. As shown, our DPE has a inference speed that is half that of TPS, TDA*, and DMN-ZS*. However, as we presented in our **Response (1)**, our DPE exhibits a performance gain of over 0.8% across 5 datasets compared to these methods.\n\n---\n\nIf you have additional comments or concerns, please let us know.\n\nBest,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yubSRnt3xm", "reviewer": "Reviewer_owvr", "summary": "The paper proposes a novel test-time adaptation method for CLIP models, drawing inspiration from previous works on prototype learning and CLIP-based adaptors.\n\nFor each test sample, both textual and visual prototypes are optimized using learnable residual parameters based on alignment loss and self-entropy loss. These prototypes are progressively updated to better utilize the stream of test samples.\n\nExperimental results on common benchmarks demonstrate the proposed method's effectiveness over recent baselines. Additionally, the authors provide comprehensive ablation studies to highlight the impact of hyperparameters.", "questions": "- Is the proposed method applicable to non-CLIP models?\n\n- Can a simplified version of the method be presented by removing certain components? Specifically, which components are the most influential? From the ablation studies, it appears that each component functions as an add-on.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method is a novel combination of existing methods.\n\nThe experiments and ablation studies are comprehensive.\n\nThe paper is mostly well written and the structure is clear.", "weaknesses": "- The method combines multiple existing techniques, each of which has been explored in previous works. For instance, learnable residuals have been studied in [1], where task residuals were used for CLIP-based adaptation, and learnable multi-modal representations were used in [2] with a similar motivation for CLIP-based adaptation.\n\n- The performance gain appears marginal  (Table 1), especially given the extensive hyperparameter tuning required. The results are sensitive to hyperparameters (Section 4.3). This may raise concerns about whether the method's efficacy is inherent or merely a result of meticulous hyperparameter tuning. Note that influential factors in the proposed method include (1) the entropy threshold, (2) queue size, (3) weight scale factor, and (4) prototype update rules.\n\n\n\n- There seems to be an overstatement of the method's applicability. While the title and introduction suggest broad applicability to \"Vision-Language Models,\" the method is only tested on CLIP models, with no experiments conducted beyond CLIP.\n\n[1] Yu et al., Task Residual for Tuning Vision-Language Models, CVPR 2023.\n\n[2] Khattak et al., MaPLe: Multi-modal Prompt Learning, CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer owvr,\n\nThanks for your valuable feedback!\n\n---\n\n**Comment (1)**: “*The method combines multiple existing techniques …*”\n\n**Response (1)**: While our method shares some similarities in method details (e.g., multi-modal prototype residuals), we focus on a completely different test-time adaptation setting. Specifically, TaskRes [1] and MaPLe [2] aim to adapt CLIP using **labeled** few-shot samples, whereas our proposed DPE approach leverages only the **unlabeled** target data stream to adapt the model to out-of-distribution domains.\n\nMoreover, we innovatively propose textual/visual prototype evolution, which enables our method to *progressively* capture more accurate multi-modal representations during test time. The two works mentioned above, while effective in learning from few-shot samples, do not incorporate such knowledge accumulation techniques. In our **Comment (4)**, we demonstrate that textual/visual prototype evolution contributes significantly to the overall effectiveness of our method.\n\n---\n\n**Comment (2)**: “*The performance gain appears marginal (Table 1), especially given the extensive hyperparameter tuning required*.”\n\n**Response (2)**: Thanks for your thoughtful feedback. We will respond in two aspects. \n\n- **Our Performance**: While our performance gain may seem modest at first glance, it is important to note that the test-time adaptation setting for CLIP is inherently challenging. Consequently, performance improvements across methods on this benchmark have been limited over the past few years:\n\n\n    | Method      |   CLIP   |   TPT   |  SwapPrompt   |    DiffTPT  |      TDA    |\n    | --------------- | :------: | :-----------: | :-----------: | :-----------: | :-----------: |\n    | Venue       | ICML '21 |  NeurIPS '22  |  NeurIPS '23  |   ICCV '23   |  CVPR '24  |\n    | Accuracy (Gain) |  46.61   | 47.26 (+0.65) | 47.86 (+0.50) | 48.71 (+0.85) | 49.58 (+0.87) |\n\n    Therefore, we believe our performance gain of 1.23% compared to TDA in such a challenging task is not trivial, and holds the potential to inspire future research. Moreover, we performed a t-test and found a p-value of 0.0036, demonstrating that our method's performance is significantly better than TDA.\n\n- **Hyperparameters**: We acknowledge that our method involves several hyperparameters that influence its efficacy. However, our DPE method consistently outperforms other approaches across a reasonable range of hyperparameter settings. For instance, as shown in Fig. 4 (Left), all combinations of entropy threshold $\\tau_t \\geq 0.1$ and queue size $M \\geq 3$ achieve >90.3% accuracy on Caltech101, whereas TPT and TDA only achieve 87.02% and 89.70%, respectively.\n\n---\n\n**Comment (3)**: “*Is the proposed method applicable to non-CLIP models?*”\n\n**Response (3)**: Thank you for pointing out this. While our DPE method can theoretically be applied to various contrastively pre-trained vision-language models, such as ALIGN, LiT, and CoCa, most of these methods are closed-source, preventing us from evaluating the effectiveness of our method on these models. Our method can certainly be applied to open-source CLIP-style VLMs, such as OpenCLIP [CVPR ’23], MetaCLIP [ICLR ’24], and DFN [ICLR ’24]. Here, we use OpenCLIP (ViT-L/14) as an example to compare our method with TDA:\n\n\n| Method              | ImageNet  | ImageNet-A | ImageNet-V2 | ImageNet-R | ImageNet-S | Average   | OOD Average |\n| ------------------- | --------- | ---------- | ----------- | ---------- | ---------- | --------- | ----------- |\n| OpenCLIP (ViT-L/14) | 74.04     | 53.88      | 67.69       | 87.42      | 63.18      | 69.31     | 68.13       |\n| TDA               | 76.28     | **61.27**  | 68.42       | 88.41      | 64.67      | 71.81     | 70.69       |\n| **DPE (Ours)**      | **77.87** | 61.09      | **70.83**   | **89.18**  | **66.33**  | **73.06** | **71.86**   |\n\nWe can see that our DPE still outperforms TDA by 1.25% on average across 5 datasets, showcasing that our method generalizes well to larger-scale VLMs.\n\nBesides, in this work, we followed common practices [1-3] in this field by using the CLIP model as a representative VLM due to its simplicity in design and wide applicability, without loss of generality [3]. We have specified the scope to CLIP in the abstract and introduction sections of the revised manuscript.\n\n[1] Learning to Prompt for Vision-Language Models, IJCV 2022.\n\n[2] Task Residual for Tuning Vision-Language Models, CVPR 2023.\n\n[3] Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models, NeurIPS 2022.\n\n---\n\n**Comment (4)**: “*Can a simplified version of the method be presented by removing certain components? Specifically, which components are the most influential?*”\n\n**Response (4)**: Thanks for your insightful question. Following your comments, we conducted additional ablation experiments to analyze the individual effect of each component:\n\n|  #   |   VPE    |   TPE    |   PRL    | ImageNet Acc. |\n| :--: | :------: | :------: | :------: | :-----------: |\n|  1   | &#10008; | &#10008; | &#10008; |     59.81     |\n|  2   | &#10004; | &#10008; | &#10008; |     61.83     |\n|  3   | &#10008; | &#10004; | &#10008; |       -       |\n|  4   | &#10008; | &#10008; | &#10004; |     61.59     |\n|  5   | &#10004; | &#10004; | &#10008; |     61.90     |\n|  6   | &#10004; | &#10008; | &#10004; |     62.93     |\n|  7   | &#10008; | &#10004; | &#10004; |     62.48     |\n|  8   | &#10004; | &#10004; | &#10004; |     63.41     |\n\nVPE, TPE, and PRL refer to visual prototype evolution, textual prototype evolution, and prototype residual learning, respectively. Note that Experiment #3 is invalid since TPE requires optimized textual prototypes t* from PRL. As shown, VPE is the most influential component, providing a ~2% improvement over zero-shot CLIP. The other two components also contribute significantly to the overall performance.\n\n---\n\nIf you have additional comments or concerns, please let us know.\n\nBest,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JmXFrDjwtq", "reviewer": "Reviewer_SVvN", "summary": "The paper introduces a novel test-time adaptation approach for vision-language models (VLMs) called Dual Prototype Evolving (DPE). The method effectively accumulates task-specific knowledge from multi-modalities by creating and evolving two sets of prototypes—textual and visual—during test time. This approach ensures more accurate multi-modal representations for target classes and promotes consistent representations by optimizing learnable residuals. Extensive experiments on 15 benchmark datasets demonstrate that DPE consistently outperforms previous state-of-the-art methods in both performance and computational efficiency.", "questions": "1. Can you provide more details on the computational overhead introduced by the dual prototype evolution mechanism compared to baseline methods?\n2. How does the proposed method perform when applied to vision-language models other than CLIP? Have you conducted any preliminary experiments in this direction?\n3. Could you elaborate on the potential limitations of the prototype residual learning approach and how it might affect the model's performance in different scenarios?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impacts of their work. The paper mentions two primary limitations: the additional computational complexity introduced by gradient back-propagation for optimizing multi-modal prototypes and the increased memory cost due to maintaining priority queues. The authors provide constructive suggestions for future work to address these issues, emphasizing the need for more efficient optimization techniques and memory management strategies.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+Originality: The introduction of dual prototypes (textual and visual) for evolving task-specific knowledge at test time is a novel concept that significantly improves the generalization capabilities of VLMs.\n+Quality: The experimental results are comprehensive and well-documented, covering a wide range of datasets and scenarios to validate the effectiveness of the proposed method.\n+Clarity: The paper is well-written and clearly explains the methodology, including detailed descriptions of the prototypes' evolution and optimization processes.\n+Significance: The proposed method addresses a crucial challenge in real-world applications by enabling efficient and effective test-time adaptation without the need for annotated samples from the target domain.", "weaknesses": "-Complexity: The introduction of learnable residuals and the dual prototype evolution mechanism adds complexity to the model, which might pose implementation challenges for practitioners.\n-Computational Cost: While the paper claims competitive computational efficiency, the requirement to maintain and update priority queues for visual prototypes can increase memory and computational costs, particularly for large-scale datasets.\n-Generalization to Other VLMs: The paper primarily focuses on CLIP, and it is unclear how well the proposed method generalizes to other vision-language models or tasks beyond those evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer SVvN,\n\nThank you for your insightful comments and positive recommendation of our work. We provide point-by-point responses to address your concerns below.\n\n---\n\n**Comment (1)**: “*The introduction of learnable residuals and the dual prototype evolution mechanism adds complexity to the model, which might pose implementation challenges for practitioners*.”\n\n**Response (1)**: While it is true that the introduction of learnable residuals and the dual prototype evolution mechanism adds complexity compared to zero-shot CLIP, we believe the idea of this work is straightforward and core components of our method can be implemented in under 100 lines of code. Besides, as we presented in Section 4, our DPE method significantly enhances the test-time generalization capabilities of the CLIP model. Therefore, we believe that this added complexity is worthwhile. Please also be assured that we will make the source code publicly available and provide detailed instructions upon acceptance to facilitate easier reproduction.\n\n---\n\n**Comment (2)**: “*Can you provide more details on the computational overhead introduced by the dual prototype evolution mechanism compared to baseline methods?*”\n\n**Response (2)**: Thank you for your insightful feedback! Here, we provide additional details regarding our computational overhead, including both inference time and memory usage.\n\n- **Inference time**: In our DPE method, the major computational time comes from the visual prototype evolution and prototype residual learning components. Specifically, while zero-shot CLIP requires 10.1 ms to infer one image, including our prototype residual learning increases the inference time to 64.7 ms per image. Further including the visual prototype evolution extends this to 132.1 ms per image. As a reference, TPT requires 666.3 ms per image, while TDA is more efficient, using 73.5 ms per image.\n\n- **Memory**: As you mentioned, maintaining priority queues does indeed increase memory usage. Following your comments, we compared the GPU memory usage on large-scale ImageNet with and without the priority queue mechanism. Specifically, while our DPE method without the priority queue takes 2136 MB of GPU memory, including the priority queue mechanism takes 4474 MB, which doubles the GPU consumption. However, our method still shows a memory advantage compared to TPT, which takes 18701 MB to perform inference on ImageNet.\n\n---\n\n**Comment (3)**: “*How does the proposed method perform when applied to vision-language models other than CLIP? Have you conducted any preliminary experiments in this direction?*”\n\n**Response (3)**: While our DPE method can theoretically be applied to various contrastively pre-trained vision-language models, such as ALIGN [ICML ’21], LiT [CVPR ’22], and CoCa [TMLR ’22], most of these methods are closed-source, preventing us from evaluating the effectiveness of our method on these models. Our method can certainly be applied to open-source CLIP-style VLMs, such as OpenCLIP [CVPR ’23], MetaCLIP [ICLR ’24], SigLIP [ICCV ’23], and DFN [ICLR ’24]. Here, we use larger-scale OpenCLIP (ViT-L/14) as an example and compare the performance of TDA and our method on robustness to natural distribution shifts:\n\n\n\n| Method              | ImageNet  | ImageNet-A | ImageNet-V2 | ImageNet-R | ImageNet-S | Average   | OOD Average |\n| ------------------- | --------- | ---------- | ----------- | ---------- | ---------- | --------- | ----------- |\n| OpenCLIP (ViT-L/14) | 74.04     | 53.88      | 67.69       | 87.42      | 63.18      | 69.31     | 68.13       |\n| TDA*                | 76.28     | **61.27**  | 68.42       | 88.41      | 64.67      | 71.81     | 70.69       |\n| **DPE (Ours)**      | **77.87** | 61.09      | **70.83**   | **89.18**  | **66.33**  | **73.06** | **71.86**   |\n\n$^*$ We evaluate TDA using the codes provided by the authors.\n\nWe can observe that our DPE still outperforms TDA by 1.07% on average across 5 datasets, showcasing that our method generalizes well to larger-scale VLMs.\n\n---\n\n**Comment (4)**: “*Could you elaborate on the potential limitations of the prototype residual learning approach and how it might affect the model's performance in different scenarios?*”\n\n**Response (4)**: Thank you for your insightful comments. As we discussed in our **Response (2)**, our prototype residual learning requires backpropagation, which increases the inference time from 10.1 ms per image in zero-shot CLIP to 64.7 ms per image. However, as shown in Figure 4 (Middle), our prototype residual learning also significantly enhances the zero-shot generalizability of the CLIP model. This represents a trade-off between efficiency and accuracy: in some real-time scenarios (e.g., decision-making in autonomous driving), simpler and more efficient methods like zero-shot CLIP may be a better fit. In contrast, for scenarios requiring higher precision (e.g., medical image analysis), our method can provide enhanced zero-shot robustness to the CLIP model.\n\nThat said, we believe the DPE method strikes a great balance by achieving state-of-the-art performance (+3.5% compared to TPT) with competitive computational efficiency (5x faster than TPT), which holds the potential to inspire future research.\n\n---\n\nWe hope that our responses have addressed your concerns. If you have additional comments or concerns, please let us know and we will be more than happy to answer. \n\n\nBest,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "08lZmHbnxq", "reviewer": "Reviewer_vqJd", "summary": "This paper proposes a novel method for test-time generalization of vision-language models. Specifically, the paper updates two prototypes, textual and visual prompts, online using test samples. Additionally, the authors learn task residuals by aligning the two different modality prototypes, further improving performance. Experiments are conducted on standard benchmarks.", "questions": "See weaknesses.", "limitations": "Na", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper writing is good.\n2. The motivation is clear, and the method design is reasonable.\n3. Experiments validate the effectiveness of the method.", "weaknesses": "1. More ablation studies are expected: What are the different impacts of the two loss terms in Eq 10?\n2. Clarification is needed: Are t and v in Eq 10 achieved after loss convergence, or are they obtained from a single update step? Is there a significant difference between the two?\n3. Discussion and comparison with a closely related method (DMN [a]) are expected. Both methods maintain the online queue/memory and weight cached features to obtain new prototypes. The difference is that this paper averages cached features to obtain prototypes, while DMN uses an attention mechanism. The authors should discuss both methods and the differences in prototype generation.\n\n[a] Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models, CVPR2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer vqJd,\n\nWe greatly appreciate your valuable feedback on our paper. We address the raised concerns and questions below.\n\n---\n\n**Comment (1)**: “*More ablation studies are expected: What are the different impacts of the two loss terms in Eq 10?*”\n\n**Response (1)**: Thank you for your valuable feedback. The alignment loss $\\mathcal{L}\\_{\\mathsf{align}}$ acts from a global perspective by promoting consistent multi-modal prototypes, ensuring that the representations are aligned for all subsequent test samples. The self-entropy loss $\\mathcal{L}_{\\mathsf{aug}}$, in contrast, greedily targets on improving individual sample predictions by penalizing high-entropy predictions across augmented views.\n\nTo provide a clearer understanding, we analyze the effects of the two loss terms on ImageNet using the ResNet-50 backbone and report the performance in the following table:\n\n\n|  #   | $\\mathcal{L}_{\\mathsf{aug}}$ | $\\mathcal{L}_{\\mathsf{align}}$ | ImageNet Acc. |\n| :--: | :--------------------------: | :----------------------------: | :-----------: |\n|  1   |           &#10008;           |            &#10008;            |     61.90     |\n|  2   |           &#10004;           |            &#10008;            |     63.18     |\n|  3   |           &#10008;           |            &#10004;            |     62.46     |\n|  4   |           &#10004;           |            &#10004;            |     63.41     |\n\nWe can observe that while the alignment loss alone improves the performance by 0.56%, the self-entropy loss provides a greater performance gain of 1.28%. Combining both loss terms further enhances performance by an additional 0.23%.\n\n---\n\n**Comment (2)**: “*Clarification is needed: Are t and v in Eq 10 achieved after loss convergence, or are they obtained from a single update step? Is there a significant difference between the two?*”\n\n**Response (2)**: Thank you for your careful review of our paper and sorry for any confusion caused. In Eq. 10, t* and v* are obtained from a single update step. Following your comments, we have conducted a new ablation experiment on ImageNet using the ResNet-50 visual backbone, varying the number of update steps from 1 to 5. The results are as follows:\n\n\n| Number of Update Steps | 1     | 2     | 3     | 4     | 5     |\n| ---------------------- | ----- | ----- | ----- | ----- | ----- |\n| ImageNet Acc.          | 63.41 | **63.45** | 63.28 | 63.26 | 63.32 |\n\n\nAs shown, the number of update steps does not significantly influence performance (within a range of 0.2%). Although setting the update steps to 2 slightly increases performance, it also linearly decreases inference efficiency. Therefore, we use a single-step update by default.\n\nWe have specified this setting in revised Section 3.3, and will include this ablation experiment in the revised manuscript.\n\n---\n\n**Comment (3)**: “*Discussion and comparison with a closely related method (DMN [a]) are expected. The authors should discuss both methods and the differences in prototype generation*.”\n\n**Response (3)**: Thank you for pointing out this. We will respond from three perspectives:\n\n- **Motivation**: While DMN(-ZS) also utilizes historical test samples to enhance the test-time generalizability of VLMs, it is important to note that DMN only updates the visual memory online while keeping the textual features/classifier unchanged. Therefore, we consider DMN similar to TDA, as both methods adapt CLIP only from a uni-modal (visual) perspective. In contrast, as we motivated in Section 1, our DPE is designed to progressively capture more accurate multi-modal representations on the fly with test samples.\n\n- **Technical Details**: Focusing solely on the visual modality, the method formulation also differs. DMN-ZS constructs a large dynamic memory (e.g., 50 image features per class) and, during testing, computes the similarity with all features in the memory for each test sample to obtain attention weights. In contrast, our DPE evolves a single representation (i.e., prototype) for each class by maintaining a relatively very small priority queue (e.g., M=3 features per class). Our prototype-based inference requires evaluating only the similarity between test features and prototype features. This technical difference results in our DPE requiring 3x less GPU memory on ImageNet compared to DMN-ZS, as shown in the following table.\n\n\n    | Method     | DMN-ZS   | Ours    | Ours (w/o priority queue) |\n    | ---------- | -------- | ------- | ------------------------- |\n    | GPU Memory | 14110 MB | 4474 MB | 2138 MB                   |\n\n- **Performance Comparison**: We have also included a performance comparison between DPE and DMN-ZS on robustness to natural distribution shifts using the ResNet-50 backbone of CLIP. The results are as follows:\n\n\n    | Method         | ImageNet  | ImageNet-A | ImageNet-V2 | ImageNet-R | ImageNet-S |  Average  | OOD Average |\n    | -------------- | :-------: | :--------: | :---------: | :--------: | :--------: | :-------: | :---------: |\n    | DMN-ZS         | **63.87** |   28.57    |    56.12    |   61.44    |   39.84    |   49.97   |    46.49    |\n    | **DPE (Ours)** |   63.41   | **30.15**  |  **56.72**  | **63.72**  | **40.03**  | **50.81** |  **47.66**  |\n\n\n    As shown in the table, our proposed DPE outperforms DMN-ZS by 0.84% on average across 5 datasets, demonstrating the superiority of our method. Additionally, our method also outperforms DMN-ZS on the ViT-B/16 backbone. We have included the full results in Table 1 and added the corresponding discussions about DMN in the revised paper.\n\n---\n\nWe hope that our responses have addressed your concerns. If you have additional comments or concerns, please let us know and we will be more than happy to answer. \n\n\nBest,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear AC and Reviewers,

We are sincerely grateful to you all for dedicating time and efforts in providing these detailed and thoughtful reviews, which helped us to improve the quality of our paper. We also want to thank all the reviewers for your **unanimous recognition and positive recommendations** of this work.

Here, apart from the point-by-point responses to each reviewer, we would like to summarize the contributions of this work and highlight our new results added during the rebuttal phase.

---

We are delighted that the reviewers appreciate and recognize the following strengths and contributions of this work:
- The proposed method addresses a crucial challenge in real-world applications by enabling efficient and effective test-time adaptation without the need for annotated samples from the target domain. **[SVvN]**
- The motivation and idea are very clear and easy to follow, and the method design is reasonable. **[Sqqz, vqJd]**
- The introduction of dual prototypes (textual and visual) for evolving task-specific knowledge at test time is a novel concept. **[SVvN]**
- Comprehensive experiments on 15 various datasets verify that our proposed DPE significantly improves the generalization capabilities of VLMs. DPE also shows improved computational efficiency compared with TPT and DiffTPT. **[Sqqz, SVvN, vqJd]**
- The paper is well-written, the structure is clear and the methodology is well explained. **[All Reviewers]**

---

In this rebuttal, we have included the following discussions and experiments to address reviewers’ comments:
- We discuss the unique contributions of our work compared to DMN-ZS, TaskRes. **[Sqqz, owvr, vqJd]**
- **[Table A]** We provide further performance comparisons with DMN-ZS and TPS to verify the effectiveness of our method. **[Sqqz, vqJd]**
- **[Table B]** We test our DPE method on the larger-scale OpenCLIP model with the ViT-L/14 backbone to verify that our method generalizes well to other VLMs. **[owvr, SVvN]**
- **[Table C-F]** We conduct more detailed ablation studies analyzing different algorithm components, two loss terms, update steps, and hyperparameter $M$, explaining their different impacts in greater detail. **[Sqqz, owvr, vqJd]**
- **[Table G-H]** We offer a more detailed analysis of our computational overhead in terms of both inference time and GPU memory. **[Sqqz, SVvN, vqJd]**

---

Again, thank you for your time in reviewing our work!

Best,

Authors

Meta-review: The paper introduces a novel method for test-time generalization in vision-language models (VLMs), specifically applied to CLIP. The method, DPE, evolves both textual and visual prototypes during test time to capture accurate multi-modal representations. The method aims to improve performance on out-of-distribution tasks by progressively accumulating knowledge from unlabeled test samples. The approach is evaluated on 15 benchmark datasets, demonstrating superior results compared to existing methods.

While the performance improvement is marginal and the complexity increased, the method introduces a novel and technically solid approach to improving VLMs in test-time adaptation settings. The comprehensive experimental evaluation and potential to inspire future research in test-time generalization make the paper a worthwhile contribution to the field, justifying acceptance to NeurIPS.
Decision: Accept (poster)