Historical Test-time Prompt Tuning for Vision Foundation Models

Authors: jingyi  zhang
jiaxing  huang
xiaoqin  zhang
ling  shao
shijian  lu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 07:59:32

Abstract: Test-time prompt tuning, which learns prompts online with unlabelled test samples during the inference stage, has demonstrated great potential by learning effective prompts on-the-fly without requiring any task-specific annotations. However, its performance often degrades clearly along the tuning process when the prompts are continuously updated with the test data flow, and the degradation becomes more severe when the domain of test samples changes continuously. We propose HisTPT, a Historical Test-time Prompt Tuning technique that memorizes the useful knowledge of the learnt test samples and enables robust test-time prompt tuning with the memorized knowledge. HisTPT introduces three types of knowledge banks, namely, local knowledge bank, hard-sample knowledge bank, and global knowledge bank, each of which works with different mechanisms for effective knowledge memorization and test-time prompt optimization. In addition, HisTPT features an adaptive knowledge retrieval mechanism that regularizes the prediction of each test sample by adaptively retrieving the memorized knowledge. Extensive experiments show that HisTPT achieves superior prompt tuning performance consistently while handling different visual recognition tasks (e.g., image classification, semantic segmentation, and object detection) and test samples from continuously changing domains.

Main paper: 
 introduction: 
 Vision Foundation Models (VFMs) [1, 2, 3] have demonstrated impressive zero-shot generalization capabilities over various downstream tasks at the cost of domain expertise for crafting appropriate task-specific prompts [4, 5, 6] . To circumvent this limitation, prompt learning [4] , which aims to adapt VFMs to fit downstream tasks by optimizing prompts as learnable vectors with few-shot task training samples, has been extensively explored recently. However, existing prompt tuning methods generally suffer from two constraints: 1) they require labelled training data for each downstream task which can be tedious and laborious to collect [7, 8] , and 2) the learnt prompts tend to overfit to the few-shot training samples, leading to degraded generalization toward downstream tasks [9, 10, 11] . Test-time prompt tuning [7] instead learns prompts with a online flow of unlabelled test samples during the inference stage. It has attracted increasing attention recently as it allows learning effective prompts on-the-fly without requiring any task-specific annotations as illustrated in Fig. 1 (a) .Existing test-time prompt tuning methods usually start with an initial template prompt like "a photo of a [class]" and optimize it with a self-supervised objective over test images together with their model predictions [7, 8] . However, these methods often experience a clear performance degradation along the tuning process when the prompts are continuously updated with the test data flow, largely due to the lack of test-sample annotations as illustrated in Fig. 1 Pre-Trained Large Vision Models … … [class] 🔥 ❄Online UpdateCurrent Sample Learnt Samples Incoming Samples [7] and DiffTPT [8] tend to 'forget' historical knowledge learnt from previous test samples when the prompts are continuously updated with the test data flow. They learn effective prompts at early tuning stage, but the learnt prompts degrade gradually along the tuning process. This phenomenon becomes more apparent when the domain of test samples changes continuously. The curves are derived from 100 runs over 3 different domains [16, 17] . In each run, the order of the 3 domains as well as the samples within each domain is randomly shuffled to simulate continuously changing test domains.learn prompts well at the early test-time tuning stage, and the learnt prompt outperforms the initial template prompts clearly. However, while the tuning continues, the learnt prompts deteriorate and gradually perform even worse than the initial template prompt especially when the test domain changes continuously. These results show that existing methods [7, 8] learn effective prompts via self-supervised objectives at the early training stage, but tend to forget the useful knowledge learnt from previous test samples, and the forgetting is largely due to the accumulation of prediction errors over the unlabelled test samples along the tuning process [12, 13] .Inspired by prior studies [14, 15] in memory-based learning, we propose Historical Test-time Prompt Tuning (HisTPT) that introduces three types of knowledge banks to help memorize the previously learnt useful knowledge to mitigate the knowledge 'forgetting' problem. The three types of knowledge banks are local knowledge bank, hard-sample knowledge bank and global knowledge bank, each of which stores complementary historical information and works with different mechanisms. Specifically, local knowledge bank buffers fresh information from the recent batches of test images, capturing up-to-date distribution changes. Hard-sample knowledge bank identifies and stores the features of hard samples from local knowledge bank, capturing difficult and rare corner cases along the tuning process. Global knowledge bank stores global information by accumulating the features from the local knowledge bank and hard-sample knowledge bank, leading to comprehensive memorization that captures representative features. In addition, HisTPT introduces an adaptive knowledge retrieval mechanism which retrieves memorized knowledge adaptively for each test image for prediction regularization and prompt optimization. To this end, HisTPT builds up comprehensive memorization that preserves useful knowledge from previous test samples, mitigating the knowledge forgetting and enabling robust test-time prompt tuning as illustrated in Fig. 1 (b) .The contributions of this work can be summarized in three aspects. First, we design HisTPT, a general test-time prompt tuning framework that explores memory learning to learn effective prompts on-the-fly. To the best of our knowledge, this is the first work that explores memory learning for test-time prompt tuning. Second, HisTPT constructs three types of knowledge banks that store complementary historical information and introduces an adaptive knowledge retrieval mechanism that retrieves memorized knowledge adaptively for each test image, mitigating the 'forgetting' of learnt useful knowledge along the prompt tuning process and ultimately leading to robust prompt learning with unlabelled test samples. Third, extensive experiments over multiple benchmarks show that HisTPT achieves superior performance consistently across different visual recognition tasks such as image classification, semantic segmentation, and object detection, especially when the domain of test images continuously changes. 
related work: 
 Test-time Adaptation, which is a type of domain adaptation technique [18, 19, 20, 21] , aims for designing the technique to improve model generalization over test samples [22, 23, 24] . Early studies such as test-time training (TTT) and its variants [22, 23] , introduce auxiliary tasks (e.g., rotation prediction task [25] ) into the supervised training objective to improve the model generalization at the training stage, and then adapt the pre-trained model to test samples via self-supervised objectives at the inference stage. Differently, recent studies [24, 20, 26, 27, 28, 29, 30, 31] generally focuses on fully test-time adaptation, where the model is adapted to test samples only during the inference stage, without introducing any auxiliary task into the training phase. For example, TENT [24] minimizes the batch-wise prediction entropy for test images while MEMO [27] enforces the prediction consistency between different augmentations of each test sample. With the advent of vision foundation models (VFMs), test-time prompt tuning [7, 8] has recently been explored for adapting pre-trained VFMs toward downstream tasks via prompt tuning at the inference stage.Prompt Learning of Vision Foundation Models (VFMs) [1, 2, 3] has been studied extensively as VFMs despite their impressive zero-shot generalization capabilities over various downstream tasks often require to design appropriate task-specific prompts for optimal adaptation. Inspired by the "prompt learning" in NLP [32] , one typical prompt learning approach for VFMs [4, 9, 33, 34, 35, 36, 37, 38, 39, 40, 41] learns to optimize prompts as learnable vectors with few-shot labelled samples of downstream tasks. Despite its effectiveness, it requires to label task-specific training data which is often laborious with poor scalability [7] . In addition, the learnt prompts tend to overfit to few-shot task samples, and this often degrades the generalization of VFMs while adapting toward various downstream tasks [7] . Different from prompt learning, test-time prompt tuning [7, 8] explores a new prompt learning setup that learns prompts on-the-fly with an online flow of unlabelled test images during the inference stage.Test-time Prompt Tuning (TPT) aims to learn prompts on-the-fly using the test samples at inference. It has attracted increasing attention recently [7, 8, 42, 43, 44, 45] as it can learn effective prompts online with unlabelled test samples flow continuously. Most existing test-time prompt tuning studies focus on image classification tasks [7, 8, 42, 43, 44, 45] . For example, TPT [7] optimizes prompts by minimizing the prediction entropy between each test sample and its augmented views.DiffTPT [8] improves the TPT by introducing the pre-trained diffusion model [46] to produce multiple diverse and informative augmented views. Different from these studies [7, 8, 42, 43, 44, 45] , HisTPT aims to mitigate the knowledge 'forgetting' problem in test-time prompt tuning when the text tokens are continuously updated with the test data flow. HisTPT achieves it by constructing comprehensive memorization capturing useful historical knowledge. In addition, HisTPT achieves superior performance across various visual recognition tasks consistently, and it can effectively handle the challenging scenario where the domain of test samples changes continuously.Memory-based Learning has been studied extensively in computer vision [12, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57] , such as semi-supervised learning [51, 58] , long-term video understanding [15, 59] and domain adaptation [60, 61, 14] . For the adaptation of vision foundation models (VFMs), several studies employ memory for improving the performance on downstream tasks [62, 63, 64, 65, 66] . For instance, [66] tackles image captioning challenge by memorizing visual-related sentences which helps VFMs to generate high-quality captions with fewer hallucinations. [65] replaces text features by identity-specific sequence features extracted by CLIP, which effectively facilitates video-based person re-identification. [64] and [62] enable efficient training-free VFMs adaptation by caching categoryspecific data features. Different from these studies, HisTPT designs three types of knowledge banks for memorizing useful knowledge learnt from previously test samples and introduces an adaptive knowledge retrieval mechanism that retrieves memorized knowledge for each test sample adaptively, aiming for mitigating the knowledge 'forgetting' problem in test-time prompt tuning. 
preliminaries and task definition: 
 Preliminaries of Vision Foundation Models (VFMs). We denote a pre-trained VFM by F = {F I , F T }, where F I and F T are image encoder and text encoder respectively. Given a test image x ∈ X test and the names of its possible belonged classes y c ∈ Y test = {y c } C c=1 , the VFM image encoder and text encoder can produce image features and category-wise text features, respectively , i.e., v = F I (x) and u c = F T (y c ). The predictions can be obtained by calculating the similarity between the image features and the category-wise text features:EQUATIONwhere cos(•) denotes the cosine similarity, and τ is a temperature hyper-parameter that controls the density of the encoded feature.Instead of directly obtaining text features using the raw class names, certain hand-crafted template prompts, e.g., "a photo of a [class]", are often adopted for generating task-related textual descriptions. However, designing appropriate prompts for each downstream task is a non-trivial task which often requires domain expertise. To this end, prompt learning [4, 9] Note that the test data is presented in a continuous flow, where the text tokens are continuously updated with the test data flow.We design three types of knowledge banks to help memorize the useful knowledge learnt from the previous test samples and adaptively exploit the memorized knowledge for regularizing the prediction of the current test samples. As illustrated in Fig. 2 , local knowledge bank buffers features of the recent test images, capturing up-to-date distribution changes along the tuning process. Hardsample knowledge bank actively identifies and stores hard samples from the local knowledge bank, which helps to capture difficult and corner features. Global knowledge bank maintains global and representative information along the whole prompt tuning process by accumulating all the features from the local knowledge bank and hard-sample knowledge bank. In addition, HisTPT introduces an adaptive knowledge retrieval mechanism that adaptively retrieves relevant memorized knowledge for prediction regularization and prompt optimization for each test image.Given a continuous flow of N test samples X test = {x n } N n=1 , we take the time step n as an example to describe the knowledge bank construction with the previous test sample x n-1 and the prompt optimization of the current sample x n with the memorized knowledge. Knowledge Bank Construction. HisTPT comes with three types of knowledge banks for capturing fresh and representative knowledge during the test-time prompt tuning with previous test samples.Local Knowledge Bank captures and stores fresh and up-to-date knowledge by buffering the features of the recent test samples. It works as a FIFO queue with a fixed size of L, where the features of the oldest test sample will be dequeued and the features of the most recent test sample will be enqueued to update the local knowledge bank, i.e, M local = {u l local , p l local } L l=1 on the flow. Specifically, for the latest test sample x n-1 and its learnt text tokens t n-1 , local knowledge bank enqueues its text feature u n-1 and prediction probability p n-1 , i.e., u n-1 = {u c n-1 } C c=1 where u c n-1 = F T ((t n-1 ; y c )), and p n-1 = {p c n-1 } C c=1 where p c n-1 is calculated via Eq. 1. Note that the size of local knowledge bank L is much smaller than the total number of test samples N since local knowledge bank aims to capture fresh information and up-to-date distribution changes of test samples along the test-time prompt tuning process.Hard-sample Knowledge Bank identifies hard samples from local knowledge bank for capturing difficult and corner information. We identify hard samples by those having high classification ) and their learnt text tokens (e.g., t n-2 and t n-1 ) along the test-time prompt tuning process. For the current test sample x n , HisTPT regularizes its prediction by retrieving the memorized knowledge via an adaptive knowledge retrieval mechanism, enabling prompt optimization for x n with the self-supervised loss L self .𝑥 ! 𝑥 !"# 𝑥 !"$ … 𝐭 !"#uncertainty, where the uncertainty is measured by their prediction entropy which can be computed from their prediction probability as stored in the local knowledge bank:EQUATIONwhere the first K samples with the highest entropy are selected and stored in the hard-sample knowledge bank. To enable robust memorization, we first compact the features of K selected samples via category-wise average and store the compacted feature in the hard-sample knowledge bank. Similar to the local knowledge bank, hard-sample knowledge bank also works as a FIFO queue with a fixed size of H, i.e., M hard = {u h hard } H h=1 . Global Knowledge Bank stores global and representative knowledge the whole prompt tuning process by accumulating all the features from the local knowledge and hard-sample knowledge banks. Specifically, we compact the features u global and u hard dequeued from the local and hard-sample knowledge banks to generate category-wise feature prototypeδ global = {δ c global } C c=1 , where δ c global = 1/2 (u c local +u c hard ).To facilitate stable and sustainable global memorization along the tuning process, we update the global knowledge bank with compacted feature prototype in a momentum way:EQUATIONwhere δ global denotes the old global feature prototype and γ is a coefficient for smooth feature update in the global knowledge bank.Prompt Optimization with the Constructed Knowledge Banks. With the built comprehensive memorization, HisTPT introduces an Adaptive Knowledge Retrieval Mechanism that enables adaptive retrieval of memorized knowledge for prediction regularization and prompt optimization of each test sample.Given the test sample x n and the text tokens learnt at time step n -1, i.e., t n-1 , the category-wise prediction probability p n = {p c n } C c=1 can be obtained by measuring the similarity between the image feature v n = F I (x n ) and category-wise text feature u c n = F T ((t n-1 ; y c )) via Eq.1. The prediction p n can be enhanced via regularization with the three types of knowledge banks. For temporary knowledge in the local and hard-sample knowledge banks, we first compact the stored features into category-wise feature prototypes, i.e., δ local and δ hard , via an average operation:EQUATIONThe new prediction for x t can thus be obtained based on the derived prototypes δ local , δ hard , and δ global . Take the local prototype δ local as an example. The prediction regularization of x n can be obtained with the local knowledge bank p local byEQUATIONThe prediction regularization by the hard-sample and global knowledge banks can be obtained in a similar way. Generally, the prediction with higher confidence (i.e., lower entropy) means that the corresponding feature prototype is better aligned with the current test sample in feature space, and it should contribute more to the final prediction pn that can be obtained as follows:EQUATIONwhere i ∈ {local, hard, global}. The softmax operation is performed across the entropy of different predictions.With the regularized prediction probability pn , the text tokens t n-1 can be optimized for the current test sample x n with the self-supervised loss defined as follows:EQUATIONwhere l(•) denotes a task-related loss, e.g., the standard cross-entropy loss for image classification. 
experiments: 
 This section presents experiments including datasets, implementation details, benchmarking with the state-of-the-art, as well as discussion of our designs.We evaluate HisTPT over multiple datasets across three widely studied visual recognition tasks: Semantic Segmentation: We benchmark HisTPT over 6 image segmentation datasets with pixelwise annotations, including Cityscapes [16] , BDD100K [67] , Mapillary [68] , ADE20K [69] , Pascal Content [70] and ACDC [17] .Image Classification: We benchmark HisTPT over 10 classification datasets, including Flow-ers102 [71] , DTD [72] , Oxford-Pets [73] , StanfordCars [74] , UCF101 [75] , Caltech101 [76] , Food101 [77] , SUN397 [78] , Aircraft [79] and EuroSAT [80] .Object Detection: We benchmark HisTPT over 4 object detection datasets, including Cityscapes [16] , BDD100K [67] , ADE20K [69] and ACDC [17] .Semantic Segmentation: Following [81] , we adopt SEEM [3] with two vision backbones including Focal-Tiny [82] and Davit-Large [83] as the segmentation foundation models. In training, we employ AdamW optimizer [84] with a weight decay of 0.05, and set the initial learning rate as 0.0001.Image Classification: Following [7, 8] , we use CLIP [1] with two backbones, i.e., ResNet-50 [85] and ViT-B/16 [86] , as the classification foundation models. In training, we adopt AdamW optimizer [84] with a weight decay of 0.01, and set the initial learning rate as 0.005.Object Detection: For object detection task, we adopt SEEM [3] with two vision backbones including Focal-Tiny [82] and Davit-Large [83] as the detection foundation models. In training, we employ AdamW optimizer [84] with a weight decay of 0.05, and set the initial learning rate as 0.0001.For all experiments, the prompt is initialized as "a photo of a" and the corresponding 4 tokens (i.e., M = 4) of dimension D = 512 are optimized as in [7, 8] . Unless otherwise specified, we set the size of the local knowledge bank and hard-sample knowledge bank at L = H = 32 and the number of the selected hard-sample features K at 16. We set the update coefficient γ of the global knowledge bank at 0.99. Following [7] , we set the optimization step in test-time prompt tuning at 1 by default. All the experiments are conducted on one NVIDIA Tesla V100 GPU with batch size 1. Since there is little prior study on test-time prompt tuning on semantic segmentation, we benchmark HisTPT by reproducing methods [7, 8] , which are designed for image classification task, on semantic segmentation task. Table 1 shows experimental results. We can observe that HisTPT achieves superior segmentation performance, largely due to its comprehensive memorization that helps to regularize the predictions of test samples and mitigates the knowledge forgetting problem in test-time prompt tuning. In addition, HisTPT is complementary to existing methods and produces clear and consistent performance boosts. This is attributed to the proposed HisTPT which can effectively mitigate the knowledge forgetting existing methods.Image Classification. Following [7, 8] , we evaluate HisTPT over 10 image classification tasks. To suit the setup in this work, we reproduce methods [7, 8] by keeping their prompts continuously updated during the test-time adaptation. As shown in Table 2 , HisTPT outperforms state-of-the-art methods consistently over different classification tasks such as classic classification on Flowers102 [71] , texture classification on DTD [72] and human action recognition on UCF101 [75] . This demonstrates the superior generalization ability while HisTPT faces diverse downstream data.Object Detection. We evaluate and benchmark HisTPT over 4 object detection datasets. Similar to semantic segmentation benchmarking, we benchmark HisTPT by reproducing methods [7, 8] (designed for image classification task) on the object detection task. As shown in Table 3 , HisTPT achieves superior detection performance and can well handle a wide range of detection tasks including detection under various weather conditions [17] across different scenes [16, 69] . The superior detection performance is largely attributed to the knowledge banks in HisTPT which effectively help generate more accurate predictions and learn better prompts for test samples.We examine the proposed HisTPT by performing ablation study over Cityscapes semantic segmentation task. As shown in Table 4 , the three types of knowledge banks can work well alone and improve the performance consistently, indicating that all the stored historical knowledge is helpful in prompt tuning. In addition, the three types of knowledge banks are complementary to each other, largely because the three knowledge banks store different types of knowledge, i.e., local knowledge bank stores fresh information, hard-sample knowledge bank stores difficult corner case information, and global knowledge bank stores the global and representative features. On top of the three types of knowledge, including the proposed adaptive knowledge retrieval improves the performance further. This shows that adaptively retrieving different types of memorized information for each test image could generate more accurate prediction and ultimately lead to better test-time prompt tuning.Complementarity to Prompt Learning Methods. As a test-time tuning technique, the proposed HisTPT is complementary to prompt learning methods that learn prompts at the training stage. We examine this feature by setting the learnt prompts by prompt learning [4, 9] as the initial prompts of HisTPT. As Table 5 shows, equipping HisTPT with the learnt prompts improves the performance clearly, indicating that HisTPT as a plug-in can greatly enhance existing prompt learning methods. Optimization Steps. We examined how the optimization step affects HisTPT by increasing it from 1 to 10. Figure 3 shows the mean mIoU over 6 semantic segmentation datasets with SEEM-Tiny. We can observe that increasing the optimization step improves segmentation consistently. Nevertheless, the performance gain becomes marginal after 6-8 optimization steps. The actual optimization step can be set by balancing the inference efficiency and the inference accuracy.Continuously Changing Test Domains. As discussed in Section 1, HisTPT can handle challenging scenarios when the domain of test samples changes continuously. We examine this feature over semantic segmentation data that were collected under normal weather [16] and various adverse weathers [17, 87] (fog, night, rain and snow). As Table 6 (a) shows, the performance of existing test-time prompt tuning methods TPT [7] and DiffTPT [8] degrades gradually along the tuning process when the weather changes from normal to adverse, largely due to increasing error accumulation and 'forgetting' while the test domain changes continuously. As a HisTPT memorizes comprehensive knowledge of previous test samples on the fly along the prompt tuning process and 2) HisTPT features a retrieval mechanism that adaptively retrieves the memorized knowledge to learn specific prompts for each current test sample. Acknowledgement. This study was funded by the MOE Tier-1 project RG18/22.We benchmark our HisTPT extensively over different visual recognition tasks with multiple datasets, including 10 image classification datasets, 6 semantic segmentation datasets and 4 object detection datasets. These datasets have rich diversity as shown in table 8 . Specifically, the 10 image classification datasets involves a wide range of visual recognition tasks from fine-grained classification, to human action recognition and texture classification.Similarly, the images of the semantic segmentation and object detection datasets are also in rich diversity, spinning from street scene images collected from various cities with different weather conditions, to images collected under indoor scenes such as office and kitchen. 2,000 65 A dataset of street-level images with high resolution. ADE20K [69] 2,000 150 A large-scale dataset of images collected from outdoor and indoor scenes. Pascal Content [70] 5101 59 An extension of PASCAL VOC 2010 dataset with pixel-wise annotations. ACDC [17] 406 19 Scene images with adverse weather conditions, i.e., fog, night, rain, snow.Cityscapes [16] 500 8 Scene images collected in different cities for street scene understanding. BDD100K [67] 1,000 8 Street scene images collected at different times of the day. ADE20K [69] 2,000 100 A large-scale dataset of images collected from outdoor and indoor scenes. ACDC [17] 406 8 Scene images with adverse weather conditions, i.e., fog, night, rain, snow.We study the size of the local knowledge bank and hard-sample knowledge bank (L and H), the parameter K used in hard-sample knowledge bank update, and the update coefficient γ used in Eq. 4 for global knowledge bank, over the semantic segmentation task Cityscapes with SEEM-Tiny.Size of the local knowledge bank L. As discussed in the main text, the size of local knowledge bank L is much smaller than the total number of test samples, since local knowledge bank aims to buffer fresh information from recent previous test samples. Here we study how it affects the test-time prompt tuning. As shown in Table 9 (a), HisTPT yields robust performance when L is relatively small (from 8 to 64), while the performance drops slightly when it becomes too large. This show that the local knowledge bank with relatively small size could effectively capture fresh information and up-to-date distribution changes along the tuning process.Size of the hard-sample knowledge bank H. Hard-sample knowledge bank stores the features of hard-samples, capturing different and rare corner cases during the test-time prompt tuning process. Table 9 (b) show that HisTPT is quite robust when H is between 8 to 128. Hence, we simply set it as the same as the size of the local knowledge bank, i.e., H = L = 32.The number of selected hard-sample features K. As discussed in the main text, hard-sample identifies and stores K hard-sample features from the local knowledge bank. Here we study the sensitivity of K by increasing it from 8 to 24 with a step of 4. As shown in Table 9 (c), the performance is quite tolerant to the parameter N and the best performance is obtained when K = 16.Update coefficient γ. The update coefficient γ in Eq. 4 determines the update speed of global knowledge bank, where the larger update coefficient results in the slower update of global knowledge bank. From Table 9 (d), we can observe that HisTPT is robust when γ is large enough (i.e., from 0.9 to 0.999) while the performance of HisTPT drops slightly when γ becomes too small. This demonstrates that a large update coefficient, ensuring smooth and gradual updates, facilitates stable global memorization. Conversely, a too small update coefficient leads to rapid updates of the global knowledge bank, resulting in unstable memorization and less effective test-time prompt tuning. C More Discussion about the Design of Historical Knowledge Banks Update of the hard-sample knowledge bank. As discussed in the main text, hard-sample knowledge bank works as an FIFO queue with a fixed size, and it is updated using the hard-sample features selected from local knowledge bank with an average compaction operation. Here we provide more discussion about the different update ways of hard-sample knowledge bank, including 1) directly update using the selected features and 2) update using the compacted features with an average operation. From Table 10 , we can observe that updating hard-sample knowledge bank using the selected features with average compaction operation performs better, which is largely due to that the compacted features enabling to filter out some noises and results in more robust memorization of difficult and corner-case information. We provide more comparisons with existing memory-based learning methods [31, 88, 89, 28] . Our HisTPT differs in two major aspects: Memory Types -HisTPT designs three types of knowledge banks for capturing and storing both fresh and representative features; Memory Retrieval -HisTPT designs an Adaptive Knowledge Retrieval Mechanism for retrieving the memorized information adaptively for each test image. Due to the very different designs, HisTPT outperforms [31, 88, 89, 28] clearly as shown in Table 12 . We provide the pseudo codes of the proposed historical test-time prompt tuning (HisTPT), as shown in Algorithm 1. We initialize the three knowledge banks with the features of the first test sample and then gradually update them as in Lines 3-7 along the test-time prompt tuning process. Note that, for the first test sample, we skip the prediction regularization in Line 10 and optimize the tokens for it with the vanilla self-training objective since the knowledge banks have not been constructed at that time.Algorithm 1 Historical Test-Time Prompt Tuning.Require: Online optimized text tokens t, a pre-trained vision foundation model F = {F I , F T }, a continuous flow of test samples X test = {x n } N n=1 and their possible belonged class namesY test = {y c } C c=1 1: Initialization: Initialize t as t 0 2: for n = 1 to N do 3:Knowledge bank construction with x n-1 and t n-1 :4: Encode x n-1 : u n-1 = F T (t n-1 ; Y test ) 5:Update local knowledge bank: dequeue old feature u local and enqueue u n-1Update hard-sample knowledge bank: dequeue old feature u hard and enqueue new feature selected by Eq. 3Update global knowledge bank: generate new category-wise feature prototype using u local and u hard , and update the global knowledge bank by Eq. 4 8:Prompt optimization for x n with the constructed knowledge banks:9:Generate prediction p n for x n with t n-1 via Eq. 1 10:Generate the regularized prediction pn by adaptively retrieving the memorized knowledge as in Eqs. 5-7 11:Optimize the text token for x n , i.e., t n ← t n-1 , by Eq. 8 12: end for F Quantification of the Forgetting Mitigation Ability of HisTPT Following prior study [29] , we measure the forgetting by randomly selecting one of the five datasets in Table 6 as the reference domain and perform continual adaptation toward the other four datasets. During the continuous adaptation process, we evaluate HisTPT's ability of preserving the knowledge of vision foundation models by measuring its performance on the reference domain. As shown in the Figure 4 , HisTPT shows less performance degradation on the reference domain consistently, demonstrating its effectiveness in preserving the knowledge of vision foundation models and mitigating forgetting during the adaptation process.We analyse the three knowledge banks by visualizing their stored features along the test-time adaptation process. Three points can be drawn as illustrated in Figure 5 : 1) the global prototypes exhibit slow and gradual shift from the initial feature prototypes, preserving the knowledge of pre-trained vision foundation models and facilitating stable test-time adaptation; 2) the features in the local knowledge bank change rapidly, validating their effectiveness in capturing fresh and up-to-date distribution changes along the test-time adaptation process; 3) most features in the hard-sample knowledge bank lies around inter-category boundary, indicating their effectiveness in capturing difficult and rare corner cases along the tuning process. With the three types of complementary knowledge, HisTPT enables adaptive regularization for the prediction of current test samples. For each experiment, one dataset is selected as the reference domain, and then we perform the continual adaptation on the other datasets. We record the performance change on the reference domain for measuring the forgetting during the continual adaptation process. Our HisTPT shows clearly less performance degradation on the reference domain, demonstrating the effectiveness of HisTPT in mitigating forgetting during the adaptation process. For clear illustration, we select two categories (i.e., car and truck) for visualization. T-SNE visualization shows that 1) global prototype shifts slowly from the initial prototype, preserving the original knowledge of pre-trained vision foundation models; 2) local knowledge bank updates rapidly, capturing fresh information and reflecting real-time distribution changes and 3) hard-sample knowledge bank captures challenging and rare cases situated near decision boundaries.In experiments, we observe negligible variance on the results between multiple random runs. Nevertheless, we provide the error bar with 5 random runs to analyze the proposed HisTPT on semantic segmentation task with SEEM-Tiny, image classification task with CLIP-RN50 and object detection with SEEM-Tiny, respectively. From Table 13 , we can observe that our proposed HisTPT performs well consistently over multiple random runs. We present qualitative illustrations and comparisons over semantic segmentation task on Cityscapes. As shown in Fig. 6 , HisTPT yields the best segmentation consistently which is well aligned with the quantitative results.Original Image SEEM-Tiny [3] TPT [7] HisTPT(Ours) Ground Truth Figure 6 : Qualitative comparison of HisTPT with the baseline model (SEEM-Tiny) [3] and TPT [7] over semantic segmentation task on Cityscapes.Broader Impacts. This work explores a novel pipeline for transfer learning with vision foundation models, namely, test-time prompt tuning. Our proposed method offers great advantages by eliminating the need for labelled task-specific data and allowing learning prompts from test samples on-the-fly. It thus makes a very valuable contribution to the computer vision research community by providing a novel and efficient transfer learning pipeline. The feature of requiring no labelled task-specific training data enables efficient adoption of vision foundation models in various downstream tasks, broadening the applicability of vision foundation models significantly.Limitations. As discussed in Section 4.2 of the main text, HisTPT offers a general framework that can perform well across different computer vision tasks. It enables effective test-time prompt tuning with the generic text prompt that is universally applicable across all vision foundation models (VFMs), thus avoiding the complexity of task-specific designs in VFM adaptation. At the other end, task-specific designs allow incorporating task-relevant knowledge which often helps improve performance. For instance, the incorporation of specific visual prompts, such as points and bounding boxes, in segmentation or detection foundation models often lead to more precise segmentation masks and bounding boxes. We will investigate how to incorporate task-specific prompt tuning in our future work.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provided detailed instructions for reproducing the main experimental results in Section 3 Method and Section 4 Experiment including the details of the proposed framework, and the datasets, base models and the parameters used for experiments.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: Code will be released after being accepted.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provided the detailed implementation details in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We provided the analysis with error bar in Section H of the appendix.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provided sufficient information on the computation resources required for reproduce the experiments in Section 4.1 Implementation Details.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly credited the original owners of assets used in the paper and properly respect their license and terms of use.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This paper proposes a method to improve test-time prompt tuning using historical information from memory bank in an online manner. In particular, the method introduces three types of knowledge banks: local, hard-sample, and global, aimed at memorizing learned knowledge to prevent performance degradation. The resulting method benefits from the design with more stable test-time-training performance over time. The method is applied to various tasks, including image classification, semantic segmentation, and object detection.

The reviews of the paper are borderline and slightly diverging. While the use of memory bank and the overall design does not stand out from the literature as ground breaking, the work does tackle an important problem in TPT and presents a very sound solution. The AC is also impressed that the work considered various visual recognition applications instead of focusing on just the image classification ones as other TPT works usually do. The AC thus considers this work with sufficient contributions and quality for publication.

That being said, some concerns from reviewers such as the discussion/comparison with other online/temporal/memory methods. It is also a pity that the authors did not highlight this method in video applications, even though there is one video dataset (UCF-101). It is not entirely clear to the readers how this work could benefit video applications.

Reviews: review: paper_summary: paper_summary: This paper proposes Historical Test-time Prompt Tuning (HisTPT), aimed at addressing the performance degradation issue of test-time prompt tuning methods in scenarios where test samples continuously change. HisTPT establishes a local knowledge bank, hard-sample knowledge bank, and global knowledge bank, each storing the recent, hard, and global features of test samples, respectively. Furthermore, it employs an adaptive knowledge retrieval mechanism to compute pseudo-labels for individual test samples and carry out prompt optimization. Overall, HisTPT is not entirely new, and has achieved performance surpassing previous methods in classification, segmentation, and detection tasks. Comprehensive ablation experiments have also verified the wide applicability of HisTPT.
strength_weakness: strength_weakness: - Entropy is not a good metric for calibrating confidence. However, HisTPT extensively uses entropy as a tool for weight calculation. It's advised to test different confidence metrics to verify the robustness of HisTPT.
- It is unclear how the ablation experiments are designed without Adaptive Knowledge Retrieval In Table 4.
- How does HisTPT regularize the bounding box for object detection?
- What are the computational and storage overhead compared to TPT?
questions: questions: See weakness.
limitations: limitations: Yes
paper_summary: paper_summary: The paper makes a comprehensive investigation of the memory bank in test-time prompt tuning for CLIP. To address the forgetting issue, the author proposes HisTPT. HisTPT aims to address this by memorizing useful knowledge from learned test samples, using three types of memory banks: local, hard-sample, and global. Extensive experiments on segmentation and classification demonstrate the superiority of HisTPT.
strength_weakness: strength_weakness: 1. The main experiment does not match the motivation. The motivation emphasizes solving continuously changing test domains [1]. However, the main experiments only focus on single datasets, lacking evaluation of the continuous test-time adaptation. Alternatively, the author should make a formal claim indicating whether HisTPT addresses covariate shift or other issues.
2. The experiment is insufficient. Memory bank is a common technique in TTA [2-5]. However, the author does not discuss it in the related work, or make comparison in the experiment. 

[1] Wang Q, Fink O, Van Gool L, et al. Continual test-time domain adaptation. In CVPR. 2022.

[2] Iwasawa Y, Matsuo Y. Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS. 2021

[3] Jang M, Chung S Y, Chung H W. Test-time adaptation via self-training with nearest neighbor information. In ICLR. 2023.

[4] Yuan L, Xie B, Li S. Robust test-time adaptation in dynamic scenarios. In CVPR. 2023.

[5] Wang S, Zhang D, Yan Z, et al. Feature alignment and uniformity for test time adaptation. IN CVPR. 2023.
questions: questions: 1. The authors emphasize mitigating knowledge forgetting. However, they do not clearly define how forgetting is quantified. In TTA, performance on the original domain is typically used to measure forgetting. I recommend the authors modify their motivation or conduct experiments to demonstrate the superiority of HisTPT in mitigating the forgetting issue.

[1] Niu S, Wu J, Zhang Y, et al. Efficient test-time model adaptation without forgetting. In ICML. 2022
limitations: limitations: N/A
paper_summary: paper_summary: This paper takes a deep investigation into the memory bank and proposes a framework with three different memory banks for storing local knowledge, hard-sample knowledge, and global knowledge. The experimental results show that the proposed method delivers superior performance on many tasks.
strength_weakness: strength_weakness: 1. Further analysis of the three memory banks should be presented to show which samples and representations are stored in the memory during the testing. This would be helpful to evaluate how each memory can help the final performance in various aspects.
2. The comparison methods are limited in this paper, e.g., [1, 2]. Please discuss them in the related work.
3. The running time of the proposed method should be reported.

[1] SwapPrompt: Test-Time Prompt Adaptation for Vision-Language Models. NeurIPS 2023

[2] Efficient Test-Time Adaptation of Vision-Language Models. CVPR 2024
questions: questions: Please refer to the `Weaknesses` section.
limitations: limitations: The authors have discussed the limitations in the appendix.
paper_summary: paper_summary: This paper introduces a framework designed to mitigate the knowledge forgetting problem in test-time prompt tuning. The proposed method, HisTPT, employs three types of knowledge banks-local, hard-sample, and global-to memorize useful knowledge from previous test samples, thereby enhancing prompt optimization during inference. An adaptive knowledge retrieval mechanism is integrated to regularize predictions and optimize prompts based on memorized knowledge. Extensive experiments across various visual recognition tasks, such as image classification, semantic segmentation, and object detection, demonstrate HisTPT's superior performance, particularly in scenarios where test domains change continuously.
strength_weakness: strength_weakness: 1. While the three memory banks are highlighted as a solution to the knowledge forgetting issue, the paper lacks an explanation of why and how these banks effectively address this problem, relying mostly on empirical results without sufficient analysis.
2. The paper suffers from repetitive text, which affects readability and clarity. The authors often reiterate the same points, and the figures do not effectively illustrate the proposed method.
questions: questions: N/A
limitations: limitations: N/A

Comments: comment: Thank you for your response. I would like to keep my positive score for this paper.
Dear Reviewer CdiG,

Thank you for your insightful feedback. We have carefully considered your questions and suggestions and have addressed them accordingly. We sincerely appreciate your constructive comments, which have helped strengthen our paper. As the discussion phase is nearing its conclusion, we would appreciate it if you could let us know if there are any additional questions or suggestions.

Best regards,

Authors
Dear Reviewer ejR5,

Thank you for your insightful feedback. We have carefully considered your questions and suggestions and have addressed them accordingly. We sincerely appreciate your constructive comments, which have helped strengthen our paper. As the discussion phase is nearing its conclusion, we would appreciate it if you could let us know if there are any additional questions or suggestions.

Best regards,

Authors
Dear Reviewer 43p7,

Thank you for your insightful feedback. We have carefully considered your questions and suggestions and have addressed them accordingly. We sincerely appreciate your constructive comments, which have helped strengthen our paper. As the discussion phase is nearing its conclusion, we would appreciate it if you could let us know if there are any additional questions or suggestions.

Best regards,

Authors
Dear Reviewer hT3z,

Thank you for your positive evaluation of our work. We sincerely appreciate your feedback and suggestions.

Best regards,

Authors
Dear Authors,

Thank you for your response. I maintain my positive rating.

Best,

Reviewer hT3z