Historical Test-time Prompt Tuning for Vision Foundation Models
Authors: Jingyi Zhang
Jiaxing Huang
Xiaoqin Zhang
Ling Shao
Shijian Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: test-time prompt tuning
vision foudation model
prompt learning
Primary area: machine_vision

Abstract: Test-time prompt tuning, which learns prompts online with unlabelled test samples during the inference stage, has demonstrated great potential by learning effective prompts on-the-fly without requiring any task-specific annotations. However, its performance often degrades clearly along the tuning process when the prompts are continuously updated with the test data flow, and the degradation becomes more severe when the domain of test samples changes continuously. We propose HisTPT, a Historical Test-time Prompt Tuning technique that memorizes the useful knowledge of the learnt test samples and enables robust test-time prompt tuning with the memorized knowledge. HisTPT introduces three types of knowledge banks, namely, local knowledge bank, hard-sample knowledge bank, and global knowledge bank, each of which works with different mechanisms for effective knowledge memorization and test-time prompt optimization. In addition, HisTPT features an adaptive knowledge retrieval mechanism that regularizes the prediction of each test sample by adaptively retrieving the memorized knowledge. Extensive experiments show that HisTPT achieves superior prompt tuning performance consistently while handling different visual recognition tasks (e.g., image classification, semantic segmentation, and object detection) and test samples from continuously changing domains.

Historical Test-time Prompt Tuning for Vision Foundation Models
Jingyi Zhang', Jiaxing Huang', Xiaoqin Zhang" Ling Shao', Shao',Shijian Lu
College of Computing and Data Science, Nanyang Technological University, Singapore College of Computer Science and Technology. Zhejiang University of Technology, China UCAS-Terminit AI Lab, University of Chinese Academy of Sciences, China
Abstract
Test-time prompt tuning, which learns prompts online with unlabelled test samples during the inference stage, has demonstrated great potential by learning effective prompts on-the-fly without requiring any task-specifie annotations. However, its performance often degrades clearly along the tuning process when the prompts continuously updated with the test data flow, and the degradation becomes more severe when the domain of test samples changes continuously. We propose HisTPT, a Historical Test-time Prompt Tuning technique that memorizes the useful knowledge of the learnt test samples and enables robust test-time prompt tuning with the memorized knowledge. HisTPT introduces three types of knowledge banks, namely, local knowledge bank, sardle knowledge bank, and global knowledge bank, each of which works with different mechanisms for effective knowledge memorization and test-time prompt optimization. In addition, HisTPP features an adaptive knowledge retrieval mechanism that regularizes the prediction of each test sample by adaptively retrieving the memorized knowledge. Extensive experiments show that HisTPT achieves superior prompt tuning performance con- sistently while handling different visual recognition tasks (e.g., image classification, semantic segmentation, and object detection) and test samples from continuously changing domains.
Introduction
Vision Foundation Models (VFMs) D 2 3 have demonstrated impressive zero-shot generalization capabilities over various downstream tasks at the cost domain expertise for crafting appropriate To cireumvent this limitation, prompt learning [4], which aims to adapt VFMs to fit downstream tasks by optimizing prompts as learnable vectors with few-shot task training samples, has been extensively explored recently. However, existing prompt tuning methods generally suffer from two constraints: 1) they require labelled training data for each downstream task which be tedious and laborious collect α and 2) the learnt prompts tend to overfit the few-shot training samples, leading to degraded generalization toward downstream tasks [9] 10 [1]. Test-time prompt tuning [7] instead learns prompts with online flow of unlabelled test samples during the inference stage. It has attracted increasing attention recently as it allows learning effective prompts on-the-fly without requiring any task-specific annotations as illustrated in Fig. (a).
task-specific prompts 4 5 o
Existing test-time prompt tuning methods usually start with an initial template prompt like "a photo of [class]" and optimize it with a self-supervised objective over test images together with their model predictions [7 8). However, these methods often experience a clear performance degradation along the tuning process when the prompts continuously updated with the test data flow, largely due to the lack of test-sample annotations as illustrated Fig. T (b). Specifically, these methods
"oorresooniigg author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1FT CUIITT
Leumabic Pempll
Prediction
Traised Lage
Superieed Objesivv
Bloc
Viea Mudak
Updan
Data Flow
Inconun Sanglee
Camrer Sempll
Leand Sauppes
Der Floe Text Domiin 61
Domaii
Derrain
(a)
(b)
Figure 1: (a) Test-time Prompt Tuning learns and optimizes prompts from continuous flow of unlabelled test samples during the inference stage. (b) Most existing test-time prompt tuning methods such as TPT [ and Dif [8] tend to 'forget' historical knowledge learnt from previous test samples when the prompts are continuously updated with the test data flow. They learn effective prompts at early tuning stage. but the learnt prompts degrade gradually along the tuning process. This phenomenon becomes more apparent when the domain of test samples changes continuously. The curves are derived from 100 runs over 3 different domains [16 17.. In each run, the order of the 3 domains as well as the samples within each domain is randomly shuffled simulate continuously changing test domains.
learn prompts well at the early test-time tuning stage, and the learnt prompt outperforms the initial template prompts clearly. However, while the tuning continues, the learnt prompts deteriorate and gradually perform even worse than the initial template prompt especially when the test domain changes continuously. These results show that existing methods 7 g learn effective prompts via self-supervised objectives at the early training stage, but tend to forget the useful knowledge learnt from previous test samples, and the forgetting largely due to the accumulation of prediction errors over the unlabelled test samples along the tuning process [2]
Inspired by prior studies [4 15 in memory- based learning, we propose Historical Test-time Prompt Tuning (HisTPT) that introduces three types knowledge banks to help memorize the previously learnt useful knowledge mitigate the knowledge 'forgetting' problem. The three types of knowledge banks are local knowledge bank, hard-sample knowledge bank and global knowledge bank, each of which stores complementary historical information and works with different mechanisms. Specifically, local knowledge bank buffers fresh information from the recent batches of test images, capturing up-to-date distribution changes. Hard-samin sard-s knowledge bank identifies and stores the features of hard samples from local knowledge bank, capturing difficult and rare corner cases along the tuning process. Global knowledge bank stores global information by accumulating the features from the local knowledge bank and hard hard-sample knowledge bank, leading to comprehensive memorization that captures representative features. In addition, HisTPT introduces an adaptive knowledge retrieval mechanism which retrieves memorized knowledge adaptively for each test image for prediction regularization and prompt optimization. To this end, HisTPT builds up comprehensive memorization that preserves useful knowledge from previous test samples, mitigating the knowledge forgetting and enabling robust test-time prompt tuning as illustrated in Fig. (b).
The contributions of this work can be summarized in three aspects. First, we design HisTPT, a general test-time prompt tuning framework that explores memory learning to learn effective prompts on-the-fly. To the best of our knowledge, this is the first work that explores memory learning for test-time prompt tuning. Second, HisTPT constructs three types of knowledge banks that store complementary historical information and introduces an adaptive knowledge retrieval mechanism that retrieves memorized knowledge adaptively for each test image, mitigating the 'forgetting' of learnt useful knowledge along the prompt tuning process and ultimately leading to robust prompt learning with unlabelled test samples. Third, extensive experiments over multiple benchmarks show that HiT achieves superior performance consistently across different visual recognition tasks such as image classification, semantic segmentation, and object detection, especially when the domain of test images continuously changes.

---

2 Related Work
Test-time Adaptation, which is a type of domain adaptation technique 18
19 20 21 aims for 23
24 Early studies such as test-time training (TTT) and its variants [22 23 introduce auxiliary tasks (e.g., rotation prediction task (25)) into the supervised training objective to improve the model generalization at the training stage, and then adapt the pre-trained model to test samples via self-supervised objectives at the inference stage. Differently, recent studies [24] 20 26 27 28 [9] 30 31 generally focuses on fully test-time adaptation, where the model is adapted to test samples only during the inference stage, without introducing any auxiliary task into the training phase. For example, TENT (24) minimizes the batch-wise prediction entropy for test images while MEMO [27] enforces the prediction consistency between different augmentations of each test sample. With the advent of vision foundation models (VFMs), test-time prompt tuning [7,] has recently been explored for adapting pre-trained VFMs toward downstream tasks via prompt tuning at the inference stage.
designing the technique improve model generalization over test samples [22
Prompt Learning of Vision Foundation Models (VFMs) [2, has been studied extensively as VFMs despite their impressive zero-shot generalization capabilities over various downstream tasks often require to design appropriate task-specific prompts for optimal adaptation. Inspired by the "prompt learning" in NLP 32 one typical prompt learning approach for VFMs 4 p
33
34
35 36
38 39. 40 [1] Iearns to optimize prompts as learnable vectors with few-shot labelled samples of downstream tasks, Despite its effectiveness, requires to label task-specific training data which is often laborious with poor scalability [7]. In addition, the learnt prompts tend to overfit to few-shot task samples, and this often degrades the generalization of VFMs while adapting toward various downstream tasks [7]. Different from prompt learning, test-time prompt tuning 7 g explores a new prompt learning setup that learns prompts on-the-fly with an online flow of unlabelled test images during the inference stage.
Test-time Prompt Tuning (TPT) aims to learn prompts on-the-fly using the test samples inference. It has attracted increasing attention recently 7 8
as it can learn effective prompts online with unlabelled test samples flow continuously. Most existing test-time prompt tuning studies focus on image classification tasks 77 g
45
For example, TPT (7] optimizes prompts by minimizing the prediction entropy between each test sample and its augmented views. DifTTT (8] improves the TPT by introducing the pre-trained diffusion model 46 to produce multiple diverse and informative augmented views. Different from these studies g
45
45).
HisTPT aims to mitigate the knowledge 'forgetting' problem in test-time prompt tuning when the text tokens are continuously updated with the test data flow. HisTPT achieves it by constructing comprehensive memorization capturing useful historical knowledge. In addition, HisTPP achieves superior performance across various visual recognition tasks consistently, and can effectively handle the challenging scenario where the domain of test samples changes continuously.
Memory-based Learning has been studied extensively in computer vision 112
49
50
53 54, 55, (57,, such as semi-supervised learning [58)) long-term video understanding
15
59)
and domain adaptation 660 61 14] For the adaptation of vision foundation models (VFMs), several
studies employ memory for improving the performance on downstream tasks 62 63
64
65 66)
For instance, (66) tackles image captioning challenge by memorizing visual-related sentences which helps VFMs to generate high-qualit captions with fewer hallucinations. [65] replaces text features by identity-specific sequence features extracted by CLIP, which effectively facilitates video-based person re- re-identification. [64] and (62) enable efficient training-free VFMs adaptation by caching category- specific data features. Different from these studies, HisTPT designs three types of knowledge banks for memorizing useful knowledge learnt from previously test samples and introduces an adaptive knowledge retrieval mechanism that retrieves memorized knowledge for each test sample adaptively, aiming for mitigating the knowledge 'forgetting' problem in test-time prompt tuning.
Method
3.1 Preliminaries and Task Definition
Preliminaries of Vision Foundation Models (VFMs). We denote a pre-trained VFM by F {Fl, FFF), where Fl and FT are image encoder and text encoder respectively, Given test image Xrest and the names of its possible belonged classes yf Viest { the VFM image
3

---

encoder and text encoder can produce image features and category-wi -wigee text features, respectively ie., v F' (a:) and ur F (v"). The predictions can be obtained by calculating the similarity
between the image features and the category-wise text features:
exp (cos(uE, v))/T t
è = arg maxp",.', p- = Σj= exp (cos(uy, u))/ T²
(I)
exp
where cos(.) denotes the cosine similarity, and is temperature hyper-paramett that controls the density of the encoded feature.
Instead of directly obtaining text features using the raw class names, certain hand-crafted template prompts, e.g.. "a photo of a [class]". are often adopted for generating task-related textual descriptions. However, designing appropriate prompts for each downstream task is non-trivial task which often requires domain expertise. To this end, prompt learning 4 9 has been extensively studied, aiming to adapt VFMs to fit downstream tasks by optimizing prompts learnable text tokens with few-shot task samples. Specifically, M learnable text tokens are adopted to append the raw class names, i.e., t {41, .2,..., each being vector of dimension D (e.g.. D 512). Thus, the textual description for class e becomes (t; ))) The learnable text tokens t are optimized with a task-related loss (e.g.. cross-entropy loss) over the few-shot labelled training samples.
Task Definition. Different from conventional prompt learning, this work focuses on continual test- time prompt tuning that adapts VFMs via prompt tuning with unlabelled test images. The objective of test-time prompt tuning is to optimize the text tokens t for test image x with certain self-supervised training losses fself that can be formulated by:
t* arg min Ceeis(F, (F, x).
(2)
Note that the test data is presented in continuous flow, where the text tokens are continuously updated with the test data flow.
3.2 Historical Test-time Prompt Tuning
We design three types of knowledge banks to help memorize the useful knowledge learnt from the previous test samples and adaptively exploit the memorized knowledge for regularizing the prediction of the current test samples. As illustrated in Fig. 2 local knowledge bank buffers features of the recent test images, capturing up-to-date distribution changes along the tuning process. Hard- sample knowledge bank actively identifies and stores hard samples from the local knowledge bank, which helps to capture difficult and corner features. Global knowledge bank maintains global and representative information along the whole prompt tuning process by accumulating all the features from the local knowledge bank and hard-sampe sample knowledge bank. In addition, HisTPT introduces an adaptive knowledge retrieval mechanism that adaptively retrieves relevant memorized knowledge for prediction regularization and prompt optimization for each test image.
Given a continuous flow of N test samples Xeest = { we take the time step n as an example to describe the knowledge bank construction with the previous test sample .n-1 and the prompt optimization of the current sample 2n, with the memorized knowledge.
Knowledge Bank Construction. HisTPT comes with three types of knowledge banks for capturing fresh and representative knowledge during the test-time prompt tuning with previous test samples.
Local Knowledge Bank captures and stores fresh and up-to-date knowledge by buffering the features of the recent test samples. It works as a FIFO queue with a fixed size of L, where the features of the oldest test sample will be dequeued and the features of the most recent test sample will be enqueued to update the local knowledge bank, i.e, Miocal = (w/noett 1 )/ on the flow. Specifically, for the latest test sample 1 and its learnt text tokens t, local knowledge bank enqueues its text feature un and prediction probability Pn-1, i.c., un = [u - where - = F ((tn--ii:))) and Pa-1 = [pr | where pf. is calculated via Eq. Note that the size of local knowledge bank L much smaller than the total number of test samples N since local knowledge bank aims to capture fresh information and up-to-date distribution changes of test samples along the test-time prompt tuning process.
CC
Hard-sample Knowledge Bank identifies hard samples from local knowledge bank for capturing difficult and corner information. We identify hard samples by those having high classification

---

Adaptive Category-wise Prototypes
Rogularized Prediction
Local Knowledge
Hard- Sample Knowledge Bank
Global Knowledge
Adaptive Knowledge
Bank
Bank
Retrioval
Store
Online Update
Vision Foundation
Models
Xp-1
Prediction
Previous Test Images with the learnt text tokems
Current Test Image
Figure 2: Overview of the proposed HiSTPT. HisTPT features three types of knowledge banks, namely, local knowledge bank, hard-sample knowledge bank, and global knowledge bank, which learn and memorize up-to-date, difficult and representative knowledge, respectively, from previous test samples (e.g., In-2 and 1n-1) ) and their learnt text tokens (e.g.. tn-2 and fn-1) along the test-time prompt tuning process. For the current sample Vn. HisTPT regularizes its prediction by retrieving the memorized knowledge via an adaptive knowledge retrieval mechanism, enabling prompt optimization for Zy with the self-supervised loss Leels-
uncertainty, where the uncertainty is measured by their prediction entropy which can be computed from their prediction probability as stored in the local knowledge bank:
E Elulocat) = --==--iie
(l.c) (1.c) Plocat log Piocal
where the first K samples with the highest entropy are selected and stored in the hard-sample knowledge bank. To enable robust memorization, we first compact the features of K selected samples via category- -iiee average and store the compacted feature in the hard-sample knowledge bank. Similar to the local knowledge bank, hard-sample knowledge bank also works as a FIFO queue with fixed size of H. i.e.e,,,M..
= (whard) M=
Global Knowledge Bank stores global and representative knowledge the whole prompt tuning process by accumulating all the features from the local knowledge and hard-sample knowledge banks. Specifically, we compact the features Ugtobal and Uhard dequeued from the local and hard-sample knowledge banks to generate category-wise feature prototype 6ytobai (ottoatt/ where d global = 1/2 (ffoaa + + ( To facilitate stable and sustainable global memorization along the tuning process, update the global knowledge bank with compacted feature prototype in momentum way:
8global +- (1 7) Oytobal + ò globol,
where ögtobal denotes the old global feature prototype and y is coefficient for smooth feature update in the global knowledge bank.
Prompt Optimization with the Constructed Knowledge Banks. With the built comprehensive memorization, HisTPT introduces an Adaptive Knowledge Retrieval Mechanism that enables adaptive retrieval of memorized knowledge for prediction regularization and prompt optimization of each test sample.
Given the test sample zn and the text tokens learnt at time step - 1, i.e.. tn-1, the category- prediction probability P., 1 1C can be obtained by measuring the similarity between the image feature vn Fl (xx) and category-wise text feature uf. = FT ((t via Eq. The prediction Pr. can be enhanced via regularization with the three types of knowledge banks. For temporary knowledge in the local and hard-sample knowledge banks, we first compact the stored features into category-wise feature prototypes, i.c., òtocal and ōhard, via an average operation:
ò ( Snard = (Shard) where &iccal
Σ
(3.c) Wlccal Shard
Σ
(h.c) Whrdd
(5)
5

---

The new prediction for Iı can thus be obtained based on the derived prototypes òiocal. óhard, and gglobut- Take the local prototype dieeat as an example. The prediction regularization of zn can be obtained with the local knowledge bank Plocat by
exp (cos(6incal) Un))/
Placal = {Piocul) Piocal
(6)
Σj-1 exp (cos(8) (Roeat: Un))/>
The prediction regularization by the hard-sample and global knowledge banks can be obtained in a similar way. Generally, the prediction with higher confidence (i.e., lower entropy) means that the corresponding fcature prototype is better aligned with the current test sample in feature space, and should contribute more to the final prediction Pr. that can be obtained as follows:
Pr Σ W. Psy W/ Softmax > log p!!)),
where E {local, hard, global). The softmax operation is performed across the entropy of different predictions.
With the regularized prediction probability Pu. the text tokens - can be optimized for the current test sample Fn with the self- sulf-supe loss defined as follows:
Leelf U(nn Pn))
(8)
where ((-) denotes a task-related loss, e.g., the standard cross-entropy loss for image classification.
Experiments
This section presents experiments including datasets, implementation details, benchmarking with the state-of-the-art, as well as discussion of our designs.
4.1 Datasets
We evaluate HisTPT over multiple datasets across three widely studied visual recognition tasks:
Semantic Segmentation:
We benchmark HisTPT over 6 image segmentation datasets with pixel-
wise annotations, including Cityscapes [16), BDD100K 667 Mapillary (68), ADE20K (69,, Pascal Content [70] and ACDC [17].
Image Classification: We benchmark HisTPT over 10 classification datasets, including Flow-
ers102 71.. DTD (72), Oxford-Pets 773. StanfordCars 174). UCF101
775 Caltech101 [76].
Food101 777. SUN397 78,, Aircraft (79] and EuroSAT [0).
Object Detection: We benchmark HisTPT over 4 object detection datasets, including Cityscapes 16
BDD100K 677, ADE20K 669 and ACDC 11
4.2 Implementation Details
Semantic Segmentation:
Following we adopt SEEM B with two vision backbones including Focal-Tiny 82 and Davit-Large 83 as the segmentation foundation models. In training, we employ
AdamW optimizer
84 with weight decay of 0.05, and set the initial learning rate as 0.0001.
Image Classification:
Following
g we use CLIP [ with two backbones, i.e., ResNet-50 g5 and
ViT-B/16 [86) the classification foundation models. In training, we adopt AdamW optimizer with a weight decay of 0.01, and set the initial learning rate as 0.005.
S4
Object Detection: For object detection task, we adopt SEEM 3 with two vision backbones including Focal-Tiny 82) and Lavit-Large [83] as the detection foundation models. In training, we employ 84 with weight decay of 0.05, and set the initial learning rate as 0.0001.
AdamW optimizer
For all experiments, the prompt is initialized as "a photo of a" and the corresponding 4 tokens (i.e.. M = 4) of dimension D = 512 are optimized as in [7 8). Unless otherwise specified, we set the size of the local knowledge bank and hard-sample knowledge bank at H 32 and the number the selected hard-sample features K at 16. We set the update coefficient 7 of the global knowledge bank at 0.99. Following 77, we set the optimization step in test-time prompt tuning at by default. All the experiments are conducted on one NVIDIA Tesla V100 GPU with batch size 1.

---

Table I: Test-time prompt tuning semantic segmentation over 6 widely adopted datasets. mloU is reported.
Method
Citysapes
BDD
Mapillary
ADE
Pascal
ACDUPss
ACDC
ACDC
ACIIC
Mcan
SEEM-Tiny
39.2
37.4
14.7
14.6
45.1
34.6
20.7
33.1
35.8
30.5
IPT
42.3
38.9
15.4
16.1
46.8
35.2
21.4
34,9
36.5
31,9
TPT
HisTPT
45.1
41.8
17.5
17.6
49.4
37.2
22.9
37.2
37.8
34.0
42.9
35.3
32.3
DITIPT HiSTPT
45.4
42.1
16.7
17.9
49.2
47.6
22.7
37.7
38.1
35.2
НKTPT
44.7
41,2
17.2
17.3
48.7
36.8
22.1
36.7
37.1
33.5
SEUM-Large
37.1
37.4
TPT
2
50.1
45.2
19.1
13.7
40.2
48.7
32.4
47.9
45.7
38.3
TPT
HiTTPT
52.1
47.4
21.3
17.1
45.8
52.1
33.4
49.4
48.8
40.5
DiTTIT
50.4
45.7
19.3
16.1
41.2
49.1
32.2
48.2
46.3
38.7
DiSTPT
HisTPT
52.4
47.8
21.1
17.4
46.3
52.4
33.6
49.7
49.1
41.0
Table 2: Test-time prompt tuning image classification over 10 widely adopted datasets. Top-1 classification accuracy is reported.
Method
Flower
DTD
Pets
Cars
UCF101
Caltech101
Food101
SUN397
Aircraft
EuroSAT
Mean
CLIP-RNNO
61.7
40.3
83.5
55.7
58.8
85.8
73.9
58.8
15.6
23.6
55.8
40.1
83.9
58.3
60.3
86.3
74.4
16.7
27.4
DiATPT
63.1
39.7
82.9
60.1
62.1
86.4
78.3
62.4
17.3
39.3
59.2
TIS
67.6
41.3
84,9
61.3
64.1
87.2
81.3
63.5
18.1
42.5
61.2
CLIP-WI-RR16
67.4
44.2
88.2
65.4
65.1
93.3
83.6
62.5
23.6
42.0
63.5
IPT [7]
68.2
47.3
87.1
66.5
67.7
93.7
84.2
65.1
24.3
42.1
64.6
DITTIT
69.4
46.3
87.9
66.4
68.1
92.3
86.5
65.3
25.1
42.8
HIsTPT
48.9
89.1
69.2
70.1
94.5
89.3
67.2
26.9
4.3
Comparisons with State of the Arts
Semantic Segmentation. We evaluate and benchmark HisTPT over 6 semantic segmentation datasets. Since there little prior study on test-time prompt tuning on semantic segmentation, we benchmark HisTPT by reproducing methods 87. which designed for image classification task, semantic segmentation task. Table T shows experimental results. We can observe that HisTPT achieves superior segmentation performance, largely due its comprehensive memorization that helps regularize the predictions of test samples and mitigates the knowledge forgetting problem in test-time prompt existing methods and produces clear and consistent
tuning. In addition, HisTPT is complementary performance boosts. This is attributed to the proposed HisTPT which can effectively mitigate the knowledge forgetting existing methods.
Image Classification. Following 7 []. we evaluate HisTPT over 10 image classification tasks. To suit the setup in this work, reproduce methods [7 by keeping their prompts continuously updated during the test-time adaptation. As shown Table HisTPT outperforms state-of-the- methods consistently over different classification tasks such classic classification
Flowers 102 [71],
texture classification on DTD 722 and human action recognition on UCF101 (7). This demonstrates the superior generalization ability while HisTPT faces diverse downstream data.
Object Detection. We evaluate and benchmark HisTPT over 4 object detection datasets. Similar to semantic segmentation benchmarking, we benchmark HisTPT by reproducing methods 7 g (designed for image classification task) on the object detection task. As shown in Table 3 HisTPT achieves superior detection performance and can well handle wide range of detection tasks including detection under various weather conditions [17] across different scenes [16, 69). The superior detection performance is largely attributed to the knowledge banks in HisTPT which effectively help generate more accurate predictions and learn better prompts for test samples.
4.4 Ablation Studies
We examine the proposed HisTPT by performing ablation study over Cityscapes semantic segmenta- tion task. As shown in Table 4 the three types of knowledge banks can work well alone and improve
7

---

Table 3: Test-time prompt tuning object detection over 4 widely adopted datasets. mAP30 is reported.
Method
Citsscapes
BDD
ADE
ACDCFog
ACDCNight
ACDCRain
ACDCS
Mean
SEEM-Timy
30.5
26.1
15.7
44.2
22.3
25.9
33.9
28.3
TPT 7
30.9
27.0
16.2
44.8
23.1
26.3
34.4
28.9
DiATTP
X
31.2
27.4
16.8
45.1
23.3
26.7
34.6
29.3
HiSTPT
31.9
28.3
17.5
46.2
24.7
27.2
35.6
30.2
SEEM-Large
31.4
31.8
18.3
55.2
31.4
34.8
43.7
35.2
TPT 7
31.8
32.2
18.5
55.6
31.9
35.1
44.2
35.6
DiATPT 8
32.5
32.3
18.9
56.1
32.3
35.4
44.8
36.0
HiSTPT
33.2
33.4
19.4
33.1
36.4
36.8
Table 4: Ablation study of the proposed HisTPT over Cityscapes semantic segmentation task.
Histrocial Knowledge Banks
Method
Adaptive knowledge retrioval
local knowledge bank
hard hard-samp sample knowledge bank global knowledge bank
SEEM-Tioy
39.2
41.1
40.9
41.7
42.2
42.8
42.5
HisTPT
44.7
the performance consistently, indicating that all the stored historical knowledge is helpful in prompt tuning. In addition, the three types of knowledge banks are complementary to each other, largely because the three knowledge banks store different types of knowledge, i.e., local knowledge bank stores fresh information, hard-sample knowledge bank stores difficult corner case information, and global knowledge bank stores the global and representative features. On top of the three types of knowledge, including the proposed adaptive knowledge retrieval improves the performance further. This shows that adaptively retrieving different types of memorized information for each test image could generate more accurate prediction and ultimately lead to better test-time prompt tuning.
4.5 Discussion
Complementarity Prompt I Learning Methods. a test-time tuning technique, the proposed HisTPT is complementary to prompt learning methods that learn prompts at the training stage. We examine this feature by setting the learnt prompts by prompt learning [ as the initial prompts of HisTPT. As Table shows, equipping TisTPT with the learnt prompts improves the performance clearly, indicating that HisTPT as a plug-in can greatly enhance existing prompt learning methods.
Optimization Steps. We examined how the optimization step affects HisTPT by increasing it from 10. Figure shows the mean mloU over 6 semantic segmentation datasets with SEEM- Tiny. We can observe that increasing the optimization step im- proves segmentation consistently. Nevertheless, the performance gain becomes marginal after 6-8 optimization steps. The actual optimization step can be set by balancing the inference efficiency and the inference accuracy.
2
Figure HisTPT with multiple optimization steps.
Continuously Changing Test Domains. As discussed in Sec-
tion []] HisTPT can handle challenging scenarios when the domain of test samples changes continu- ously. We examine this feature over semantic segmentation data that were collected under normal
weather [16] and various adverse weathers 17 87 (fog, night, rain and snow). As Table 6(a) shows, the performance of existing test-time prompt tuning methods TPT 7 and DifFTPT [8] degrades gradually along the tuning process when the weather changes from normal to adverse, largely due to increasing error accumulation and 'foggetting" while the test domain changes continuously, As

---

Table 5: Complementarity to state-of- of-the---te prompt learning methods CoOp 4 and CoCoOp 9 The mean top-1 accuracy across 10 image classification datasets is reported, and CoOp and CoCoOp supervised with 16-shot labelled training data per category.
Method
CLIP-RN50
CoOp
CoCoOp
HisTPT
HisTPT + CoOp
HisTPT + CoCoOp
Mean Accuracy
55.8
56.1
57.2
61.2
62.4
63.1
Table 6: Test-time prompt tuning on semantic segmentation across continuously changing test domains. mloU is reported.
Norual
Fox
Rai
Ouder
Nigh:
Nornal
SEEM Tiy
39.2
346
20.7
33.1
35.8
SEEM Tay
35.8
53.1
20.7
34.6
39.2
TPT
13
00
31.2 1-4)
3004 5.2)
TPT
76
34.00000
20.10-000
32 22.9
35.86
DIITPPT
429/43.7)
35.3.40.6)
20 30364444
32.0-1.)
31.4-4.)
DMITIPT
36.64088)
34.6+
20.56-0.3)
32.9/-1.7)
3..1-3.01
H&TPT
369-223)
23.6-2.0
37
H:TPT
374(-13)
36
22.1(-1144)
32.0-2-4)
74.94577)
(a)
(b)
comparison, HisTPT improves the performance consistently across different weathers, and this is largely due to two factors: 1) HisTPT effectively preserves representative and up-to-date knowledge from past test samples along the tuning process; 2) HisTPT retrieves relevant memorized hoowledge for each test sample, mitigating the 'forgetting" and leading more robust test-time prompt tuning. Similar results are obtained when the test domain changes from adverse weather to normal weather as shown in Table b(b), further verifying HisTPT's effectiveness and robustness while facing changing test domains.
Comparisons to Existing Memory-based Learning Methods. We examine how the proposed HisTPT performs as compared with existing memory-based learning techniques. We benchmark with two categories of memory- -based learning techniques: 1) memory-based learning in traditional network training [60] 61 [4] and 2) memory-based learning with vision foundation models [66] [65]
621. Table shows experimental results on the task of semantic segmentation on Cityscapes with
SEEM- Tiny, It can be seen that HisTPT outperforms all existing memory learning techniques (60] 61
14
62 with clear margins. The superior performance largely attributed two factors: His TPT memorizes comprchensive knowledge of previous test samples on the fly along the prompt tuning process and 2) HisTPT features a retrieval mechanism that adaptively retrieves the memorized knowledge to learn specific prompts for each current test sample.
66
65
Table 7: Comparison with existing memory-based learning methods over Cityscapes semantic segmentation task on SEEM-Tiny, mloU is reported.
Method
HCL 60
MeGA 61
BiMem [14
MeaCap [66]
TF-Clip 65
TDA (62
HisTPT
mloU
40.3
40.7
41.2
41.9
41.4
42.6
44.7
5 Conclusion
This paper introduces Historical Test-time Prompt Tuning (HisTPT), general test-time prompt tuning framework that aims to mitigate the 'knowledge forgetting' problem across various visual recognition tasks. HisTPT introduces three types of knowledge banks, including local knowledge bank, hard- sample knowledge bank and global knowledge bank, each of which works with different mechanisms for memorizing useful knowledge. With the three knowledge banks, TT builds up comprehensive memorization that preserves useful knowledge from previous test samples, mitigating the knowledge forgetting and enabling robust test-time prompt tuning. In addition, HisTPT comes with adaptive knowledge retrieval mechanism that regularizes the prediction of the current test sample by adaptively retrieving the memorized knowledge. Extensive experiments show that HisTPT achieves superior performance consistently across various vision tasks. In addition, HisTPT can effectively handle the challenging scenario where the domain of test samples changes continuously. Moving forwards, will further investigate memory-based learning for adaptation of vision foundation models.
A Acknowledgement. This study was funded by the MOE Tier-1 project RG18/22.

---

References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In JCML, pages 8748-8763, PMLR, 2021.
[2] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[3] Xueyan Zou, Jianwei Yang. Hao Zhang. Feng Li, Linjie Li. Jianfeng Wang. Lijuan Wang. Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances Neural Information Processing Systems, 36, 2024.
[4] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. I/CV, 130(9)23337-2348 2022.
[5] Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge-guided context optimization. In CVPR, pages 6757-6767, 2023.
[6] Tz- Ying Wu, Chih-Hui Ho. and Nuno Vasconcelos. Protect: Prompt tuning for hierarchical consistency. arXiv preprint arXiv:2306.0 23l.:3220022200 2023.
[7] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35:14274-14289, 2022.
[8] Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2704-2714, 14, 2023.
[9] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, pages 16816-16825, 2022.
[10] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu.. Fsdr: Frequency space domain randomization for domain generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6891-69022 2021.
[11] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[12] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analyxis and machine intelligence, 40(12):2935-2 2947, 2017.
[13] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7201-7211, 2022.
[14] Jingyi Zhang, Jiaxing Huang, Xueying Jiang, and Shijian Lu, Black-box unsupervised domain adaptation with bi-directional atkinson-shiffrin memory. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 11771-11782, 2023.
[15] Ho Kei Cheng and Alexander Schwing. Xmem: Long-tern video object segmentation with an atkinson- shiffrin memory model. In European Conference on Computer Vision, pages 640 658. Springer, 2022.
[16] Marius Cordis, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213 0223,,22,2222 2016.
[17] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Acde: The adverse conditions dataset with corre- spondences for semantic driving scene understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10765-10775, 2021.
[18] Jingyi Zhang, Jiaxing Huang, Zichen Tian, and Shijian Lu. Spectral unsupervised domain adaptation for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9829-9840, 2022.
[19] Jiaxing Huang. Dayan Guan, Aoran Xiao, Shijian Lu, and Ling Shao, Category contrast for unsupervised domain adaptation in visual tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1203 1214-121,440 2022.
10

---

[20] Jian Liang. Dapeng Hu. and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning, pages 6028-6039. PMLR, 2020.
[21] Jingyi Zhang, Jiaxing Huang, Zhipeng Luo, Gongjie Zhang, Xiaoqin Zhang, and Shijian Lu. Da-detr: Do- main adaptive detection transformer with information fusion. In Proceedings of the JEEEJCVE Conference on Computer Vision and Pattern Recognition, pages 217998,,798, 2023.
[22] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexci Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference machine learning, pages 9229-9248. PMLR, 2020.
[23] Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Tu++: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems,
820, 2021.
[24] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
[25] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.00 1803007788 2018.
[26] Chaithanya Kumar Mummadi, Robin Hutmacher, Kilian Rambach, Evgeny Levinkov, Thomas Brox, and Jan Hendrik Metzen. Test-time adaptation to distribution shift by confidence maximization and input transformation. arXiv preprint arXiv:2106. 14999, 2021.
[27] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances neural information processing systems, 35:38629 -36622, 2022.
[28] Shuai Wang. Daoan Zhang, Zipei Yan, Jianguo Zhang, and Rui Li. Feature alignment and uniformity for test time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10050-20060, 2023.
[29] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pages 16888-16905. PMLR, 2022.
[30] Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Fcotta: Memory-effiient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19900-11929, 2023.
[31] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for generalization. Advances in Neural Information Processing Systems, 34:2427-2440, 2021.
domain
[32] Pengfei Liu, Weizhe Yuan, Jinlan Fu. Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9): 1-35, 2023.
[33] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In CVPR, pages 520655206220 5215, 2022.
[34] Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor Guilherme Turrisi da Costa. Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Variational prompt tuning improves generalization of lasion-language models. arXiv preprint arXiv:2210.02390, 2022.
[35] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. arXiv preprint ar0iv: 14055 2022.
[36] Xuchai He, Diji Yang, Weixi Feng. Tuu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, and Xin Eric Wang. Cpl: Counterfactual prompt learning for vision and language models. arXiv preprint arXiv:2210.1083 10362, 2022.
[37] Guangyi Chen, Weiran Yao, Xiangehen Song. Xinyue Li, Yongming Rao, and Kun Zhang. Prompt learning with optimal transport for vasion-language models. arXiv preprint arXiv:2210001253 2022.
[38] Tony Huang. Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-languan models. arXiv preprint 2204 03649, 2022.
11

---

[39] Sheng Shen, Shijia Yang. Tianjun Zhang, Bohan Zhai, Joseph E Gonzalez, Kurt Keutzer, and Trevor Darrell. Multitask vision-language prompt tuning. arXiv preprini arXiv:2211.11220, 2022.
[40]
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. arXiv preprint arXiv:2210003117. 2022.
[41] Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, and Yanning Zhang. Class-aware visual prompt tuning for vision-language pre-trained model. arXiv preprint arXiv:2208. 08340, 2022.
[42] Jameel Hassan, Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muzammal Naseer, Fahad Shahbaz Khan, and Salman Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. arXiv preprint arXiv:2311 1.t.:2391..114, 2023.
[43] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. Advances in Neural Information P Processing Systems, 36, 2024.
[44] Hee Suk Yoon. Eunseop Yoon. Joshua Tian Jin Tee. Mark A Maasegawa-Johnson Yingzhen Li, and Chang D Yoo. C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. In The Twelfth International Conference on Learning Representations, 2023.
[45] Shuai Zhao, Xiaohan Wang, Linchao Zhu, and Yi Yang. Test-time adaptation with clip reward for zero-shot generalization in vision-language models. arXiv preprint arXiv:2.305. 18010, 2023.
[46] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resohution image synthesis with latent diffuusin models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[47] Yaoyao Liu, Bernt Schiele, and Qianru Sun. Rmm: Reinforeed memory management for class-incremente learning. Advances in Neural Information Processing Systems, 34:3478 3490, 2021.
[48] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):351--3526. 2017.
[49] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint arXiv:1410.3916 2014.
[50] Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9729-9738, 2020.
[51] Samuli Laine and Timo Aila. Temporal ensembling for semi-super learning. arXiv preprint arXiv:161..02242.220 2016.
[52] Antti Tarvainen and Harri Valpola. Mcan teachers are better role models: Weight-averaged consistency targets improve supessupe deep learning results. Advances in neural information processing systems, 30, 2017.
[53] Xun Wang, Haozhi Zhang, Weilin Huang, and Matthew R Scott. Cross-batch memory for embedding learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6388-6397, 2020.
[54] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with meeoory-aggmente neural networks. In International conference on machine learning. pages 1852-1850. PMLR, 2016.
[55] Linchao Zhu and Yi Yang. Compound memory networks for few-shot video classification. In Proceedings of the European Conference Computer Vision (ECCV), pages 751-766, 2018
[56] Inigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano, and Ana C Murillo. Semi Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise wise memory bank. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8219-8228, 2021.
[57] Guanxiong Sun. Yang Hua, Guosheng Hu, and Neil Robertson. Mamba: Multi-level aggregation via memory bank for video object detection. In Proceedings of the AAAI Conference on Artificia Intelligence, volume 35, pages 2620-2623 2021.
[58] Yanbei Chen, Xiatian Zhu, and Shaogang Gong. Semi Semi-supervised deep learning with memory. In Proceedings of the European conference on computer vision (ECCV), pages 268 283,, 2018.
12

---

[59] Pavel Tokmakov, Karteek Alahari, and Cordelia Schmid. Learning video object segmentation with visual memory. In Proceedings of the IEEE international conference on computer vision, pages 4481-4490, 2017.
[60] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu, Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data, Advances Neural Information Processing Systems, 34, 2021.
[61] Vibashan VS, Vikram Gupta, Poojan Oza, Vishwanath A Sindagi, and Vishal M Patel. Mega-cda: Memory guided attention for category-aware unsupervised domain adaptive object detection. In Proceedings of the [EEE/CVF Conference Computer Vision and Pattern Recognition, pages 4516 4526, 2021.
[62] Adilhek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb EI Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. arXiv preprint arXiv:2403. /8293, 2024.
[63] Xinyao Yu, Hao Sun, Ziwei Niu, Rui Qin, Zhenjia Bai, Yen-Wei Chen, and Lanfen Lin. Memory-inspire temporal prompt interaction for lext-image classification. arXiv preprint arXiv:2401. 14856, 2024.
[64] Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision- lannulla models, arXiv preprint ar arXiv:2403. 17023.177789) 17589, 2024.
[65] Chenyang Yu, Xuehu Liu, Yingquan Wang, Pingping Zhang. and Huchuan Lu. Tf-clip: Learning text- free clip for video-based person re-identification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6764-6772, 2024.
[66] Zequn Zeng. Yan Xie, Hao Zhang. Chiyu Chen, Zhengjue Wang, and Bo Chen. Meacap: Memory- augmented zero-shot image captioning. arXiv preprint arXiv:2403.0377 2024.
[67] Fisher Yu, Haofeng Chen, Xin Wang. Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 2636-2645, 2020.
[68] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pages 4990 4999, 2017.
[69] Bolei Zhou, Hang 7hao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pages 633-641, 2017.
[70] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, pages 891 898, 2014.
[71] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, pages 722-729. IEEE, 2008
[72] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, pages 3606-3613, 2014.
[73] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, pages 3498-3505. IEEE, 2012.
[74] Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting a large-scale dataset of fine-grain fine-grained cars. 2013.
[75] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 121.0:22 2012.
[76] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. CVPR workshop, pages 178 178. IEFF, 2004.
[77] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food- Bood-101-an 101-mining discriminative components with random forests. In ECCV, pages 446-461. Springer, 2014.
[78] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485 3492, IEEE, 2010.
13

---

[79] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Fine-grained visual classification of aircraft. arXiv preprint 1rX1v.1300.5151, 2013.
[80] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land and land cover classification. JSTARS, 12(7):2217-2222 2019.
[81] Jiaxing Huang, Kai Jiang, Jingyi Zhang, Han Qiu, Lewei Lu, Shijian Lu, and Eric Xing. Learning to prompt segment anything models. arXiv preprint arXiv:2401 240.:22000..4651,, 2024.
[82] Jianwei Yang, Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks. Advances in Neural Information Processing Systems, 15:4203-42111 2022.
[83] Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. In European Conference Computer Vision, pages 74-92. Springer, 2022.
[84] Nlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint urXiv:1711.05001,222 2017.
[85] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[86] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth Transformers image recognition scale, arXiv preprint 4cXiv:2010.11929 2020.
16x16 words:
[87] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Rda: Robust domain adaptation via fourier adversarial attacking. Proceedings of IEEE/CVF international conference on computer vision, pages 8988-8999, 2021.
[88] Minguk Jang, Sae-Your Young Chung, and Hye Won Chung. Test-time adaptation via self-training with nearest neighbor information. arXiv preprint 1rXiv:2207.107933 2022.
[89] Longhui Yuan, Binhui Xie, and Shuang Robust test-time adaptation in dynamic scenarios. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15922- 15932, 2023.
14

---

Appendix
A Datasets Details
We benchmark our HisTPT extensively over different visual recognition tasks with multiple datasets, including 10 image classification datasets, 6 semantic segmentation datasets and 4 object detection datasets. These datasets have rich diversity as shown in table[8] Specifically, the 10 image classification datasets involves wide range of visual recognition tasks from line-grained classification, to human action recognition and texture classification. Similarly, the images of the semantic segmentation and object detection datasets are also in rich diversity, spinning from street scene images collected from various cities with different weather conditions, to images collected under indoor scenes such as office and kitchen.
Table 8: Details of the datasets used for benchmarking HisTPT.
Datasets
Test Images
Classes
Description
Image Clussification
Flower F11
2,463
102
Flower images with various sizes and illumination environments.
DIT 72
1,692
dataset texural images for image recognition.
Oxford-IITT PETS 78
3,669
37
A dataset for pet recognition with cat and dog images of 37 breeds.
Stanford Cars
74
8,041
196
Car images for fine grained recognition.
UCFI01
75
3,783
101
A video dataset for human action recognition.
Caltech101
2,465
101
A dataset for common object recognition.
Food-1011 77
30,300
101
Food images for fine-graines grainnaindd recognition.
SUN397 78
19,850
397
Indoor and outdoor scene images for fine-grained recognition.
Aircraft 79
3.333
100
A dataset of 100 aircraft model variants for aircrutt model recognition.
AT 80
8,100
10
A dataset of satellite images for land use and land cover recognition.
Semantic Segmentation
Cityscupes 16
500
19
Scene images collected in different cities for street scene understanding. Street scene images collected at different times of the day. A dataset of street-level images with high resolution.
BDD1OKK 67
1,000
19
Mapillary 68
2,000
65
ADE20K
69
2,000
150
A large-scule dataset of images collected from outdoor and indoor scenes. An extension of PASCAL VOC 2010 dataset with pixel-wise annotations.
Pascal Content
70
5101
59
ACDC 07
406
19
Scene images with adverse weather conditions, i.c., fog, night. rain, snow.
Object Detection
Citssaapes 16
500
Scene images collected in different cities for street scene understanding. Street scene images collected at diffcrent times of the day. A largo-scale dataset of images collected from outdoor and indoor scenes. Scene images with adverse weather conditions, i.e. log, night, rain, snow.
BOD110KK 67
1,000
ADE20K 69
2,000
100
ACDC 17
406
8
B Parameter Analysis
We study the size of the local knowledge bank and hard sample knowledge bank (L and H), the parameter K used in hard-sample sample knowledge bank update, and the update coefficient y used in Eq. 4 for global knowledge bank, over the semantic segmentation task Cityscapes with SEEM-Tiny.
Size of the local knowledge bank T.. As discussed in the main text, the size of local knowledge bank T. is much smaller than the total number of test samples, since local knowledge bank aims buffer fresh information from recent previous test samples. Here we study how it affects the test-time prompt tuning. As shown Table 2 (a). HisTPT yields robust performance when L is relatively small (from 8 to 64). while the performance drops slightly when it becomes too large. This show that the local knowledge bank with relatively small size could effectively capture fresh information and up-to-date distribution changes along the tuning process.
Size of the hard-sample knowledge bank H. Hard-sample knowledge bank stores the features of hard-samples, capturing different and rare corner cases during the test-time prompt tuning process. Table g (b) show that HisTPT is quite robust when H is between 8 to 128, Hence, we simply set it as the same as the size of the local knowledge bank, i.c., H = L = 32.
The number of selected hard-sample features K. As discussed in the main text, hard-sample identifies and stores K hard-sample features from the local knowledge bank. Here we study the sensitivity of K by increasing it from to 24 with a step of As shown in Table Ic). the performance quite tolerant to the parameter N and the best performance obtained when K- 16.
Update coefficient y. The update coefficient 7 Eq. determines the update speed global knowledge bank. where the larger update coefficient results in the slower update of global knowledge bank. From Table (d). we can observe that HisTPT is robust when is large enough (i.e., from 0.9 to 0.999) while the performance of HisTPT drops slightly when 7 becomes too small. This demonstrates that a large update coefficient, ensuring
15

---

smooth and gradual updates, facilitates stable global memorization. Conversely, a too small update coefficient leads rapid updates of the global knowledge bank, resulting unstable memorization and less effective test-time prompt tuning.
Table 9: Parameter analysis of HisTPT over semantic segmentation task Cityscapes with SEEM-Ting -Tny.
L
8
16
32
64
128
512
H
16
32
64
128
512
HisTPT
44.5
44.7
44.7
44.6
44.2
43.9
HISTPT
44.7
44.6
44.7
44.5
44.6
43,5
(a) The size of local knowledge bank L.
(b) The size of hard-sample knowledge bank H.
12
16
20
24
0.1
0.5
0.9
0.99
0.999
HiSTPT
44.6
44.5
44.7
44.6
44.6
HisTPT
43.1
43.9
44.5
44.7
44.6
(c) The number of hard-sample features K.
(d) The update coefficient ?.
C More Discussion about the Design of Historical Knowledge Banks
Update of the hard-sample knowledge bank. As discussed in the main text, hard-sample knowledge bank works FIFO queue with fixed size, and is updated using the hard-sample features selected from local knowledge bank with an average compaction operation. Here we provide more discussion about the different update ways of hard-sample knowledge bank, including 1) directly update using the selected fcatures and 2) update using the compacted features with an average operation. From Table [T] we can observe that updating hard-sample knowledge bank using the selected features with average compaction operation performs better, which is largely due that the compacted features enabling to filter out some noises and results more robust memorization of difficult and corner-case information.
Table 10: Comparison of different update ways of hard-sample knowledge bank over semantic segmentation task Cityscapes with SEEM-Tiny.
Method
Directly update
Update with average operation
mloU
43.9
44.7
Update the global knowledge bank. As descrihed the main text, we update the global knowledge hank using the features dequeued from both the local knowledge bank and hard-sample knowledge bank. Here we study its effectiveness with different update ways of global knowledge bank. including 1) update global knowledge bank with only the features dequeued from local knowledge bank: 2) update global knowledge bank with only the features dequeued from harl-sample knowledge bank and 3) update global knowledge bank with the features dequeued from the local knowledge bank and hard-samplc knowledge bank. Table Isshwws shows the experimental results. It can be observed that updating global knowledge bank with the features dequcued from both the local knowledge bank and hard-sample knowledge bank performs the best, which indicates that the features stored in local knowledge bank and hard-sample knowledge bank are complementary to each other, working together to help build a more comprehensive and representative global memorization.
Table 11: Comparison of different update ways of global knowledge bank over semantic segmentation task Cityscapes with SEEM-Tiny.
Method
local knowledge bank hard-sample knowledgge bank global& hard-sample knowledge banks
mloU
44.2
43.8
44.7
D
More Comparisons with Memory-based Learning Methods
We provide more comparisons with existing memory-based learning methods differs in two major aspects: Memory Types - HisTPT designs three types of knowledge banks for capturing and storing both fresh and representative features; Memory Retrieval HisTPT designs an Adaptive Knowledge Retrieval Mechanism for retrieving the memorized information adaptively for each test image. Due to the very clearly shown in Table 12
g9
Our HisTPT
different designs, HisTPT outperforms 31 88 89 28
16

---

Table 12: Comparison with existing memory- based learning methods Cityscapes semantic segmentation task on SEEM-Tiny mloU is reported.
Method
T3A
31
TAST
[8
RoTTA
89
FAU
28
HisTPT
mloU
41.8
42.0
41.9
42.2
44.7
E
Pseudo Codes of HisTPT
We provide the pseudo codes of the proposed historical test-time prompt tuning (HisTPT), shown in Algo- rithm We initialize the three knowledge banks with the features of the first test sample and then gradually update them as in Lines 3-7 along the test-time prompt tuning process. Note that, for the first sample, skip the prediction regularization in Line 10 and optimize the tokens for it with the vanilla self since the knowledge banks have not been constructed at that time.
self-trainin training objective
Algorithm Historical Test- Time Prompt Tuning.
Require: Online optimized text tokens pre-trained vision foundation model F {F', FT), a continuous flow of test samples Kest = {I,] I= and their possible belonged class names Drest {//
1: Initialization: Initialize t. as to
2: for 1 to N do
3: Knowledge bank construction with Tn-1 and tn-1:
4: Encode In-1: un-1 (tnnii
5: Update local knowledge bank: dequeue old feature Nsocal and enqueue Un-1
Update hard-sample knowledge bank: dequeue old feature Thard and enqueue new feature selected by Eq. 3
7: Update global knowledge bank: generate new category-wise feature prototype using Ntocal and Uhard, and update the global knowledge bank by Eq.
8: Prompt optimization for x,, with the constructed knowledge banks:
9: Generate prediction Pa for Fn with fn-1 via Eq.
Generate the regularized prediction Pm. by adaptively retrieving the memorized knowledge as in Eqs. 7
11:
Optimize the text token for In. i.c. tu + tx by Eq.
12: end for
F
Quantification of the Forgetting Mitigation Ability of HisTPT
Following prior study (29) we measure the forgetting by randomly selecting one of the five datasets in Table 6 the reference domain and perform continual adaptation toward the other four datasets. During the continuous adaptation process, we evaluate HisTPT's ability preserving the knowledge vision foundation models by measuring its performance on the reference domain. As shown in the Figure HisTPT shows less performance degradation on the reference domain consistently, demonstrating its effectiveness in preserving the knowledge of vision foundation models and mitigating forgetting during the adaptation process.
G Further Analysis of the Three Knowledge Banks
We analyse the three knowledge banks by visualizing their stored features along the test-time adaptation process. Three points can be drawn as illustrated in Figure SS: 1) the global prototypes exhibit slow and gradual shift from the initial feature prototypes, preserving the knowledge of pre-trained vision foundation models and facilitating stable test-time adaptation; 2) the features in the local knowledge bank change rapidly, validating their effectiveness in capturing fresh and up-to-date distribution changes along the test-time adaptation process; 3) most features in the hard sard-sample knowledge bank lies around inter-category boundary, indicating their effectiveness in capturing difficult and rare corner cases along the tuning process. With the three types of complementary knowledge. HisTPT enables adaptive regularization for the prediction of current test samples.
17

---

37
Horra
Ferforanne Weater (MC3C
Terformace Ba Westher
21-
Figure 4: Comparison of preventing forgetting continual test-time adaptation task with SEEM Tiny. For each experiment, one dataset is selected as the reference domain, and then we perform the continual adaptation the other datasets. We record the performance change on the reference domain for measuring the forgetting during the continual adaptation process, Our HisTPT shows clearly less performance degradation the reference domain, demonstrating the effectiveness of HisTP1 in mitigating forgetting during the adaptation process.
Car
Isaal
Clobal
Locall
Samples of local kaon
Hardoample
Sumptio of otlhrrd--nnpie bnowloige
Cauggey Trede
Inaut proweyps
Global proveype
proteyye
? Samplax of of browtage bast
Itars-ampp porcapp
V Sanglae ortuadd--gge bnowdedge hast
100-th sample
(a) 200-th sample
400 test sample
Figure T-SNE visualization of the features stored in each knowledge bank with Cityscapes semantic segmentation task on SEEM-Tiny, For clear illustration, we select two categories (i.c., car and truck) for visualization. T-SNE visualization shows that 1) global prototype shifts slowly from the initial prototype, preserving the original knowledge of pre-trained vision foundation models; 2) local knowledge bank updates rapidly, capturing fresh information and reflecting real-time distribution changes and 3) hard-sample knowledge bank captures challenging and rare cases situated near decision boundaries.
H Analysis with Error Bars
In experiments, we observe negligible variance on the results between multiple random runs. Nevertheless, we provide the error har with 5 random runs to analyze the proposed HisTPT on semantic segmentation task with SEEM-Ting, -iny, image classification task with CLIP- and object detection with SEEM-Ting -iny, respectively. From Table [3] we can observe that our proposed HisTPT performs well consistently over multiple random runs.
Table 13: Analysis of our proposed HisTPT with error bars.
Method
Semantic segmentation task (Mean) Image classification task (Mean)
Object detection task (Mean)
HisTPT
33,5 +0.2
61.2 +0.1
30,2 +0.2
18

---

I
Qualitative Results
We present qualitative illustrations and comparisons over semantic segmentation task on Cityscapes. As shown Fig. HisTPT yields the best segmentation consistently which is well aligned with the quantitative results.
Original Image
SEEM-Tiny
3
TPT 7
HisTPT(OOrs)
Ground Truth
Figure 6: Qualitative comparison of Hi with the baseline model (SEEM-Tiny -iny) over semantic segmentation task on Cityscapes.
B and TPT
J Broader Impacts and Limitations
Broader Impacts. This work explores novel pipcline for transfer learning with vision foundation models, namely, test-time prompt tuning. Our proposed method offers great advantages by eliminating the need for labelled task-specific data and allowing learning prompts from test samples on-the-fly. thus makes very valuable contribution to the computer vision research community by providing a novel and efficient transfer learning pipeline. The feature of requiring no labelled task-specific training data enables efficient adoption of vision foundation models in various downstream tasks, broadening the applicability of vision foundation models significantly.
Limitations. As discussed in Section 4.2 main text, HisTPT offers general framework that can perform well across different computer vision tasks. enables effective test-time prompt tuning with the generic text prompt that is universally applicable across all vision foundation models (VFMs), thus avoiding the complexity of task-spe designs in VFM adaptation. At the other end, task-specific designs allow incorporating task-relevant knowledge which often helps improve performance. For instance, the incorporation of specific visual prompts, such as points bounding boxes, segmentation detection foundation models often lead more precise segmentation masks and bounding boxes. We will investigate how to incorporate task-specific prompt tuning in our future work.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately describe the paper's contributions and scope.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of the work in Section of the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well rell-specifications asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
20

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provided detailed instructions for reproducing the main experimental results in Section 3 Method and Section 4 Experiment including the details of the proposed framework, and the datasets, base models and the parameters used for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access hosted model (e.g., in the of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
example, if the contribution
While NeurIPS does require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should deseribe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results way to reproduce the model (e.g.. with
datasct instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed source models, it may be that access to the model is limited in some way (c.g.. registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions faithfully reproduce the main experimental results, described supplemental material?
Answer: [No]
Justification: Code will be released after being accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines hhttpp::/nips.cc//// cc/public/
Sriidea/CodeSule ses.uuaaiiiioooooooooooo for more details.
While we encourage release of code and data, we understand that this might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : //tn://nipp cc/public/ oul.dess/Cooeeuutmiiesssoooooo.o for more details.
21

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provided the detailed implementation details Scction 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provided the analysis with error bar in Section H of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be elearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap. etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should he careful symmetric error hars would yield results that are out of range (c.g. negative error rates).
show in tables figures
error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: provided sufficient information on computation resources required for reproduce the experiments in Section 4.1 Implementation Details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
22

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the broader impacts of the work in Section of the Appendix.
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system Icarns from feodback over time, improving the efficiency accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datascts)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model implementing safety filters.
23

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly eredited and are the license and terms of usc explicitly mentioned and properly respected?
Answer: [Yes]
Justification: properly credited the original owners of assets used in the paper and properly respect their license and terms of use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperewi thcode. cwm/ca////aa//oc daaa/dattstt has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage in country of data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
25


Reviews:
[{"review_id": "ZCvzmaxX4S", "reviewer": "Reviewer_43p7", "summary": "This paper proposes Historical Test-time Prompt Tuning (HisTPT), aimed at addressing the performance degradation issue of test-time prompt tuning methods in scenarios where test samples continuously change. HisTPT establishes a local knowledge bank, hard-sample knowledge bank, and global knowledge bank, each storing the recent, hard, and global features of test samples, respectively. Furthermore, it employs an adaptive knowledge retrieval mechanism to compute pseudo-labels for individual test samples and carry out prompt optimization. Overall, HisTPT is not entirely new, and has achieved performance surpassing previous methods in classification, segmentation, and detection tasks. Comprehensive ablation experiments have also verified the wide applicability of HisTPT.", "questions": "See weakness.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- HisTPT is simple yet effective, demonstrating scalability in classification, segmentation, and detection tasks.\n- Sufficient experiments and ablation studies demonstrate the effectiveness of the proposed method.\n- The paper is well-organized and easy to follow.", "weaknesses": "- Entropy is not a good metric for calibrating confidence. However, HisTPT extensively uses entropy as a tool for weight calculation. It's advised to test different confidence metrics to verify the robustness of HisTPT.\n- It is unclear how the ablation experiments are designed without Adaptive Knowledge Retrieval In Table 4.\n- How does HisTPT regularize the bounding box for object detection?\n- What are the computational and storage overhead compared to TPT?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Response 1] Robustness of HisTPT to different confidence metrics:**\n\nThank you for your suggestion! We conduct the suggested studies by adopting different confidence metrics, i.e., Softmax probability [a], MC dropout [b], and Mahalanobis distance [c], over Cityscapes semantic segmentation task with SEEM-Tiny. As shown in the table below, HisTPT can work with different confidence metrics. We chose entropy as it is simple and widely adopted.\n\n||Softmax probability[a] | MC dropout[b] | Mahalanobis distance[c] | **Entropy (Default)**|\n|:----:|:----:|:----:|:----:|:---:|\nHisTPT| 44.6 | 44.7|44.5|**44.7**|\n\n[a] A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks, ICLR 2017.\n\n[b] Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, ICML 2016.\n\n[c] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, NeurIPS 2018.\n\n**[Response 2] Clarification of the ablation study design:**\n\nThank you for pointing out this issue. For the ablation experiments without Adaptive Knowledge Retrieval, we compute the self-supervised loss $L_{self}$ with the regularized prediction $\\hat p_n$ which is obtained without the adaptive weighting as in Eq.7. \n\nSpecifically, for experiments using a single knowledge bank in Table 4, we compute $L_{self}$ by directly adopting the prediction of respective knowledge bank by Eq.6 as the final regularized prediction. For the rest experiments using more than a single knowledge bank, we compute $L_{self}$ by averaging the predictions of respective knowledge banks as the final regularized prediction. \n\nWe will clarify this point in Section 4.4 in the updated manuscript.\n\n**[Response 3] How does HisTPT regularize boxes for object detection task:**\n\nWe would clarify that, as discussed in Section Method in the main manuscript and Section Limitations in the appendix, HisTPT achieves effective text prompt tuning by regularizing the category prediction only, which is general and applicable to various vision foundation models on image classification, semantic segmentation and object detection. \n\nOn the other hand, the bounding box prediction is the task-specific design for object detection task. We did not consider it during method design as we aim to build a general test-time prompt tuning framework that can work for various vision tasks.\nNevertheless, we believe that introducing additional task-specific designs like regularizing bounding boxes for object detection tasks may improve the performance. We will investigate how to incorporate task-specific designs (e.g., regularizing bounding boxes) in our future work.\n\n**[Response 4] Computational and storage overhead compared to TPT:**\n\nThank you for your suggestion! We conduct the suggested efficiency benchmarking with TPT in terms of run time and GPU memory usage, under the same setting with CLIP ResNet-50 on dataset Flowers-102 with a single GPU. As the table below shows, TPT has much longer inference time and higher GPU usage because 1) TPT involves heavy augmentations of test images (e.g., 64 augmentations for each test image) while HisTPT does not and 2) HisTPT efficiently memorizes the historical information by storing compacted data within fixed-size memory banks as described in Lines 155-157, 169-172 and 541-550.\n\n\n|Methods|Run time per image (s) | GPU memory usage (MB)|\n|:---:|:---:|:---:|\n|TPT | 0.66|4,210|\n|HisTPT| 0.12|3,888|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XHb9Vvenpb", "reviewer": "Reviewer_hT3z", "summary": "The paper makes a comprehensive investigation of the memory bank in test-time prompt tuning for CLIP. To address the forgetting issue, the author proposes HisTPT. HisTPT aims to address this by memorizing useful knowledge from learned test samples, using three types of memory banks: local, hard-sample, and global. Extensive experiments on segmentation and classification demonstrate the superiority of HisTPT.", "questions": "1. The authors emphasize mitigating knowledge forgetting. However, they do not clearly define how forgetting is quantified. In TTA, performance on the original domain is typically used to measure forgetting. I recommend the authors modify their motivation or conduct experiments to demonstrate the superiority of HisTPT in mitigating the forgetting issue.\n\n[1] Niu S, Wu J, Zhang Y, et al. Efficient test-time model adaptation without forgetting. In ICML. 2022", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. HisTPT is easy to follow. The proposed framework is well-introduced and the construction of memory banks is intuitive。\n2. The evaluation is comprehensive. HisTPT is evaluated across multiple visual recognition tasks including image classification, semantic segmentation, and object detection, demonstrating consistent performance improvements over state-of-the-art test-time prompt tuning methods.", "weaknesses": "1. The main experiment does not match the motivation. The motivation emphasizes solving continuously changing test domains [1]. However, the main experiments only focus on single datasets, lacking evaluation of the continuous test-time adaptation. Alternatively, the author should make a formal claim indicating whether HisTPT addresses covariate shift or other issues.\n2. The experiment is insufficient. Memory bank is a common technique in TTA [2-5]. However, the author does not discuss it in the related work, or make comparison in the experiment. \n\n[1] Wang Q, Fink O, Van Gool L, et al. Continual test-time domain adaptation. In CVPR. 2022.\n\n[2] Iwasawa Y, Matsuo Y. Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS. 2021\n\n[3] Jang M, Chung S Y, Chung H W. Test-time adaptation via self-training with nearest neighbor information. In ICLR. 2023.\n\n[4] Yuan L, Xie B, Li S. Robust test-time adaptation in dynamic scenarios. In CVPR. 2023.\n\n[5] Wang S, Zhang D, Yan Z, et al. Feature alignment and uniformity for test time adaptation. IN CVPR. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Response 1] Evaluation of HisTPT over the continuous test-time adaptation task:**\n\nWe would clarify that we evaluated HisTPT over continuously changing test domains in Table 6 of the main manuscript. As discussed in Lines 280-293, HisTPT can tackle challenging scenarios when the domain of test samples changes continuously.\n\n**[Response 2] Comparisons with other memory-based TTA methods:**\n\nThank you for sharing the prior studies [2-5] and we will review and compare with them in the revised paper. Our HisTPT differs in two major aspects: \n1) Memory Types - HisTPT designs three types of knowledge banks for capturing and storing both fresh and representative features; \n2) Memory Retrieval - HisTPT designs an Adaptive Knowledge Retrieval Mechanism for retrieving the memorized information adaptively for each test image. \n\nAs a comparison, the shared studies generally employ a single type of vanilla memory that stores either samples or class centroids. Due to the very different designs, HisTPT outperforms the shared work clearly as shown in the table below (on Cityscapes semantic segmentation task with SEEM-Tiny).\n\n|Cityscapes|T3A [2] | TAST [3] | RoTTA [4] | FAU [5] | **HisTPT**|\n|:---:|:---:|:---:|:---:|:---:| :---:|\n|mIoU|41.8 | 42.0 | 41.9 | 42.2 | **44.7**|\n\n**[Response 3] Quantification of the forgetting mitigation ability of HisTPT:**\n\nThank you for your suggestion! Since the source domain of vision foundation models (pre-training) is generally huge with data from multiple sources, it is very challenging to measure the forgetting by directly testing the performance over such source domain. Instead, we randomly select one of the five datasets in Table 6 as the reference domain and perform continual adaptation toward the other four datasets. During the continuous adaptation process, we evaluate HisTPT's ability of preserving the knowledge of vision foundation models by measuring its performance on the reference domain. As shown in the **Figure 1** in the attached PDF, HisTPT shows less performance degradation on the reference domain consistently, demonstrating its effectiveness in preserving the knowledge of vision foundation models and mitigating forgetting during the adaptation process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eWUGHYn7IO", "reviewer": "Reviewer_ejR5", "summary": "This paper takes a deep investigation into the memory bank and proposes a framework with three different memory banks for storing local knowledge, hard-sample knowledge, and global knowledge. The experimental results show that the proposed method delivers superior performance on many tasks.", "questions": "Please refer to the `Weaknesses` section.", "limitations": "The authors have discussed the limitations in the appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper explore an interesting about the memory bank for test-time prompt tuning. This is direction is meaningful for making the test-time prompt tuning more robust in practice. \n2. The experiments in this paper is comprehensive and solid. The results on semantic segmentation, object detection, and image classification tasks reveal the superiority of the proposed method.", "weaknesses": "1. Further analysis of the three memory banks should be presented to show which samples and representations are stored in the memory during the testing. This would be helpful to evaluate how each memory can help the final performance in various aspects.\n2. The comparison methods are limited in this paper, e.g., [1, 2]. Please discuss them in the related work.\n3. The running time of the proposed method should be reported.\n\n[1] SwapPrompt: Test-Time Prompt Adaptation for Vision-Language Models. NeurIPS 2023\n\n[2] Efficient Test-Time Adaptation of Vision-Language Models. CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Response 1] Further analysis of the three knowledge banks:**\n\nThank you for your suggestion! We analyze the three knowledge banks by visualizing their stored features along the test-time adaptation process. Three points can be drawn as illustrated in **Figure 2** in the attached PDF: \n1) The global prototypes exhibit slow and gradual shift from the initial feature prototypes, preserving the knowledge of pre-trained vision foundation models and facilitating stable test-time adaptation. \n2) The features in the local knowledge bank change rapidly, validating their effectiveness in capturing fresh and up-to-date distribution changes along the test-time adaptation process.\n3) Most features in the hard-sample knowledge bank lies around inter-category boundary, indicating their effectiveness in capturing difficult and corner cases along the tuning process. \n\nWith the three types of complementary knowledge, HisTPT enables adaptive regularization for the prediction of current test samples. We will include the above analysis and the attached visualization in the updated manuscript.\n\n**[Response 2] Comparisons with more related works:**\n\nThank you for sharing the two prior studies! We will discuss and compare with them in the updated manuscript. Specifically, [1] leverages self-supervised contrastive learning to facilitate the test-time prompt adaptation, while [2] introduces a training-free dynamic adapter that caches category-specific feature for efficient test-time adaptation. Differently, HisTPT focuses on mitigating the knowledge 'forgetting' problem in test-time prompt tuning, and it achieves it by constructing comprehensive memorization that captures useful historical knowledge. As shown in Table below, HisTPT performs clearly better than [1,2], demonstrating its effectiveness in tackling test-time prompt tuning challenge. The experiments are conducted on Cityscapes semantic segmentation task with SEEM-Tiny.\n\n|Cityscapes|SwapPrompt[1] | TDA[2] | **HisTPT**|\n|:---:|:---:|:---:|:---:|\n|mIoU|43.4 | 43.7  | **44.7**|\n\n\n**[Response 3] Running time of HisTPT:**\n\nThank you for your suggestion! The table below shows the run time of the proposed HisTPT with CLIP ResNet-50 on a single GPU over dataset Flowers-102. We can see that HisTPT is clearly more efficient than TPT. The better efficiency is largely attributed to two factors: 1) TPT involves heavy augmentations of test images (e.g., 64 augmentations for each test image), while HisTPT does not; 2) HisTPT introduces memory to store historical information efficiently as described in Lines 155-157, 169-172 and 541-550.\n\n|Methods|Run time per image (s) | \n|:---:|:---:|\n|TPT | 0.66|\n|HisTPT| 0.12|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7soCfqccXj", "reviewer": "Reviewer_CdiG", "summary": "This paper introduces a framework designed to mitigate the knowledge forgetting problem in test-time prompt tuning. The proposed method, HisTPT, employs three types of knowledge banks-local, hard-sample, and global-to memorize useful knowledge from previous test samples, thereby enhancing prompt optimization during inference. An adaptive knowledge retrieval mechanism is integrated to regularize predictions and optimize prompts based on memorized knowledge. Extensive experiments across various visual recognition tasks, such as image classification, semantic segmentation, and object detection, demonstrate HisTPT's superior performance, particularly in scenarios where test domains change continuously.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The introduction of knowledge banks and an adaptive knowledge retrieval mechanism provides a clear and effective solution to address the issue of knowledge forgetting during test-time prompt tuning.\n2. The method is validated through extensive experiments on multiple benchmarks, showing its effectiveness across different visual recognition tasks and varying test domains.", "weaknesses": "1. While the three memory banks are highlighted as a solution to the knowledge forgetting issue, the paper lacks an explanation of why and how these banks effectively address this problem, relying mostly on empirical results without sufficient analysis.\n2. The paper suffers from repetitive text, which affects readability and clarity. The authors often reiterate the same points, and the figures do not effectively illustrate the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Response 1] Further explanation of how and why the proposed knowledge memory banks effectively address the knowledge forgetting issue:**\n\nThank you for pointing out this issue.\n\nAs discussed in Lines 37-45, HisTPT introduces three types of knowledge banks to memorize the previously learnt knowledge which helps mitigate the 'forgetting' issue. Specifically, the forgetting is largely caused by the accumulation of prediction errors over unlabelled test samples along the tuning process. HisTPT exploits the three types of complementary knowledge that collaborate to denoise the prediction of test samples along the tuning process, alleviating error accumulation and ultimately mitigating the forgetting issue.\n\nWe analyze how the three memory banks help mitigate the forgetting by visualizing the change of their stored features along the test-time adaptation process. As shown in **Figure 2** of the attached PDF, the three types of knowledge banks store complementary historical features: \n1) The global prototypes exhibit slow and gradual shift from the initial feature prototypes, preserving the knowledge of pre-trained vision foundation models and facilitating stable test-time adaptation. \n2) The features in the local knowledge bank change rapidly, validating their effectiveness in capturing fresh and up-to-date distribution changes along the test-time adaptation process. \n3) Most features in the hard-sample knowledge bank lies around inter-category boundary, indicating their effectiveness in capturing difficult and rare corner cases along the tuning process. \n\nIn this way, HisTPT can leverage the comprehensive and complementary knowledge stored in the three memory banks to denoise predictions of test samples along the tuning process, reducing error accumulation and effectively mitigating the issue of forgetting.\n\nIn addition, we follow prior test-time adaptation study [a] and conduct experiments to analyze the forgetting mitigation ability of HisTPT. Specifically, we randomly select one of the five datasets in Table 6 as the reference domain and perform continual adaptation toward the other four datasets. During the continuous adaptation process, we evaluate HisTPT's ability of preserving the knowledge of vision foundation models by measuring its performance on the reference domain. As shown in **Figure 1** of the attached PDF, HisTPT shows clearly less performance degradation on the reference domain consistently, demonstrating its effectiveness in mitigating forgetting during the adaptation process.\n\nWe will include above analysis in our revised manuscript.\n\n[a] Efficient test-time model adaptation without forgetting, ICML 2022.\n\n**[Response 2] The texts and figures need to be improved:**\n\nThank you for your comments! We will check through the manuscript carefully to remove repetitive text as suggested. In addition, we will improve Figure 2 of the main manuscript to illustrate the framework of the proposed method more clearly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for your insightful feedback and constructive comments. We are highly encouraged by the reviewers' acknowledgement that our proposed method has good scalability in various vision tasks [43p7,CdiG] and effectively explores memory in test-time prompt tuning [ejR5], the conducted experiments are comprehensive and solid [43p7,hT3z,ejR5,CdiG], and the paper is well-organized and easy to follow [43p7,hT3z].

We address the questions and concerns raised by each reviewer point-by-point in the respective threads below. In addition, attached below is the PDF containing the figures required for some responses.

Meta-review: This paper proposes a method to improve test-time prompt tuning using historical information from memory bank in an online manner. In particular, the method introduces three types of knowledge banks: local, hard-sample, and global, aimed at memorizing learned knowledge to prevent performance degradation. The resulting method benefits from the design with more stable test-time-training performance over time. The method is applied to various tasks, including image classification, semantic segmentation, and object detection.

The reviews of the paper are borderline and slightly diverging. While the use of memory bank and the overall design does not stand out from the literature as ground breaking, the work does tackle an important problem in TPT and presents a very sound solution. The AC is also impressed that the work considered various visual recognition applications instead of focusing on just the image classification ones as other TPT works usually do. The AC thus considers this work with sufficient contributions and quality for publication.

That being said, some concerns from reviewers such as the discussion/comparison with other online/temporal/memory methods. It is also a pity that the authors did not highlight this method in video applications, even though there is one video dataset (UCF-101). It is not entirely clear to the readers how this work could benefit video applications.
Decision: Accept (poster)