BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping
Authors: Taolin Zhang
Jinpeng Wang
Hang Guo
Tao Dai
Bin Chen
Shu-Tao Xia
Venue: NeurIPS 2024 poster (2024)
Keywords: Test-time adaptation
Vision-Language models
Primary area: machine_vision

Abstract: Adaptation of 
pretrained vision-language models such as CLIP to various downstream tasks have raised great interest in recent researches. 
Previous works have proposed a variety of test-time adaptation (TTA) methods to achieve strong generalization without any knowledge of the target domain. 
However, existing training-required TTA approaches like TPT necessitate entropy minimization that involves large computational overhead, while training-free methods like TDA overlook the potential for information mining from the test samples themselves.
In this paper, we break down the design of existing popular training-required and training-free TTA methods and bridge the gap between them within our framework.
Specifically, we maintain a light-weight key-value memory for feature retrieval from  instance-agnostic historical samples and instance-aware boosting samples. 
The historical samples are filtered from the testing data stream and serve to extract useful information from the target distribution, while the boosting samples are drawn from regional bootstrapping and capture the knowledge of the test sample itself.
We theoretically justify the rationality behind our method and empirically verify its effectiveness on both the out-of-distribution and the cross-domain datasets, showcasing its applicability in  real-world situations.

BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping
Taolin Zhang'
Jinpeng Wang" Hang Guo
Tao Dai*?
Bin Chen Shu-tao Xia 1,4
Tsinghua University 2 Shenzhen University
3 Harbin Institute of Technology PengCCeng Laboratory
tttps://giiie ggt.tb..tuuut../ggg cwm/ lanchangg/oostdaattta...
Abstract
Adaptation of pretrained vision-language models such as CLIP to various down- stream tasks have raised great interest recent researches. Previous works have proposed variety of test-time adaptation (TTA) methods to achieve strong gener- alization without any knowledge of the target domain. However, existing training- required TTA approaches like TPT necessitate entropy minimization that involves large computational overhead, while training-free methods like TDA overlook the potential for information mining from the test samples themselves. In this paper, we break down the design of existing popular training- required and training- free TTA methods and bridge the gap between them within our framework. Specif- ically, we maintain a light-weight key-value memory for feature retrieval from ingtance-agnostic historical samples and instance-ave aware boosting samples. The historical samples are filtered from the testing data stream and serve to extract use- ful information from the target distribution, while the boosting samples are drawn from regional bootstrapping and capture the knowledge of the test sample itself. We theoretically justify the rationality behind our method and empirically verify its effectiveness on both the out-of-distribution and the cross-domain datasets, showcasing its applicability in real-world situations.
1 Introduction
Vision Language models [49, 16, 23-25, 7] have shown incredible performance in downstream vision tasks [1], such classification [29, 55, 54, 8]. generation [20, 38, 9] and recognition [46, 47]. Among these models, CLIP [36] has been trained with large-scale noisy image-text pairs and can generalize well in zero-shot recognition tasks. The key idea behind CLIP modality alignment during training and similarity comparison during testing for classification. However, CLIP suffers from domain shift problems during test-time inference. In the presence of out-of-distribution issues [27, 43, 12] that commonly appear in real-world scenarios, CLIP may fail to effectively align the feature across modalities, leading to performance degradation.
Test-tin bist-time adaptation (TTA) has been widely explored in recent approaches [43, 15, 41, 17] to mitigate misalignment issues and improve performance in downstream tasks. Current mainstream TTA meth- ods can be divided into training- trauning-required required methods and training-frre methods, as depicted in Figure. la and Figure. lb. Training requiree approaches [43, 41, 39] adjust model parameters or learnable prompts based on self-supervised objectives like entropy and increase the prediction confidence of model for distribution adaptation. TPT [41] applies entropy minimization the laniun-language model first. Furthermore, inspired by consistency regularization, TPT performs information mining
*Corrspongding author: Tao Dai (daitao.edu@e aaaaoeelll@gama.ommmmmmm
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

luerige
ImAge Encoder
Imagereic
Entropy
Augmented
Bir
Iragenel
Leamable Promptt
Dog
(a) Training roquired Test daptation
Historical Cache
Fautte)
Calech
Feacure Rebreez!
Images
Biid
Hand crafted Prompt
Cat
Encoder
SUNO07
DTD
Julin
CUP
(b) Training- fro: Test time A Adaptation
(c) Perforancc comparison on two benchmarks
Figure I: (a) Existing training-required TTA methods utilize self-supervised objective like entropy minimization for better generalization. (b) Existing training-free TTA methods perform feature retrieval the historical samples adjust the model prediction. (c) Performance comparison on the Out-of-Distribund benchmark and Cross-Datasets benchmark.
from the test sample itself by random regional cropping in saff-bootstrapping style. However, training-required methods require gradient descent that is monssconsuinng with large training over- head, which prevents them from being applied in computationally limited situations. Training-free approaches [15, 52, 17] utilize memory networks, cache, or prototypes to store information regarding target samples and distributions, which is then used to adaptively modify the model's prediction, For example, TDA [17] leverages historical samples from the test data steam to build a dynamic key-value cache. updates the prior knowledge encoded in CLIP through feature retrieval and output prediction based on the similarity between the test sample and the high-quality data stored in the memory bank. However, existing training- approaches only consider interaction with other historical samples in the cache and do not effectively exploit the information within the test sample itself. This limitation prevents them from performing well especially in tasks that require fine-grained information.
Both of these approaches demonstrate excellent performance in enhancing the robustness of vision- language models to unknown distributions. However, the connection between them remains unclear. In this paper, we aim to answer three questions: (1) How are training-requing training-fred free methods like TDA connected? (2) How can we combine these two methods based on their shared nature? (3) Does vision- languale models benefit from the combination of these methods?
methods like TPT and
In order answer these questions, first consider that the augmented images of test samples form regional bootstrapping distribution of the original data. By filtering out the noisy augmentations based on mutual information with the predefined CLIP text embedding clusters, we obtain boosting distribution from which high-quality samples close to the target clusters can be drawn. Based on this, we delve into the connection between the target operations over the boosting distribution, i.e., cross- entropy optimizations and cache classifier, which reveals the shared nature between entropy-based and cache-based methods. Specifically, pinpoint that with the samples derived from the bootstrapping distribution, entropy minimization over them performs equivalently to feature retrieval from the cache consisting of them. Motivated by this analysis, we propose a brand-new adaptation strategy, dubbed BoostAdapter to improve training-free free adapters by incorporating the samples derived from boosting distribution to the memory bank. Particularly, the cache in BoostAdapter consists of cestance-agnostic historical samples filtered from the test data stream, along with instance-aware boosting samples generated through regional bootstrapping from the sample itself. The interactions between intrae and cross-sample operations make BoostAdapter effective and efficient by

---

incorporating the idea of information mining from Beiiing-requiiee methods while maintaining the efficiency of training-fre free methods. Theoretical analyses and empirical results are also provided to validate the effectiveness of BoostAdapter.
To summarize, we make the following contributions in this paper.
We first discuss the relationship between training-requieee and training-free methods in test-time adaptation and establish connections between them.
We propose BoostAdapter, a brand new adaptation strategy in test-time adaptation of vision- language models, which improves training-free free adapters by introducing high-quality samples from regional bootstrapping into the memory.
We theoretically derive target domain error bound of BoostAdapter and shows that BoostAdapter benefit from incorporating seff-bootstrappinn data.
Extensive experiments conducted over two benchmark demonstrate the superior performance of BoostAdapter under test-time adaptation settings.
2
Related Works
Vision-Languaae Models have shown remarkable potential in generalization by contrastive pre- training over amounts of text-image pairs [16, 36, 24, 25] One typical work is CLIP [36], which benefits from the alignment of 400 million curated image-tex( pairs and predicts the most relevant text description for given image based on cosine similarity. Adapting CLIP the downstream applications has attracted much attention and has been widely explored in recent approaches [55, 54, 52, 26, 56, 30]. CoOp [55] introduces learnable prompts [22, 51, 50, 28] and CoCoOp [54] conditions the text prompts on image embedding for better generalization. Maple [18] performs prompting for both vision and language branches and improves the alignment of the embedding between modalities. These approaches have demonstrated significant performance enhancements, but they still require few training data from the target domain. In contrast, we focus on test-time adaptation where there is no information about the target distribution and aim to generalize the model to any unknown scenarios.
meiiiiggrequireed Test-time Adaptation updates partial wights of the model like prompts [41, 39] or BN layer [43] with self-supervised objectives that benefit the downstream tasks without requiring additional training data. Tent [43] reduces generalization error on shifted data by test- time entropy minimization. For vision- language models, Test-time prompt tuning (TPT) [41] is method that dynamically optimizes prompts during the testing phase, enhancing the model's zero-shot generalization ability. Specifically, TPT generates multiple augmented views of the test sample and then minimizes the entropy of the model's output logits across them to ensure consistent prediction. Recently, many works built upon TPT have been proposed to further enhance the performance of vision-language models. Particularly. DiffTPT [6] leverages the power of diffusion models to generate semantically consistent augmented images for entropy minimization. PromptAlign [39] bridges the gap between the test sample and source distribution by aligning token statisties, including mean and variance. Nevertheless, these approaches require gradient descent over the augmented images, which computationally expensive and time-consuming.
Training-free Test-time Adaptation applies cache model or prototypes to make prediction of test samples non-parametrie manner [15, 17, 53]. T3A [15] utilizes prototypes as downstream classifiers and dynamically adjusts the weights. AdaNPC [53] leverages the data from the source domain to address the issues of computation overhead and domain forgetting. For vision- language models, TDA [17] introduces both positive cache and negative cache to obtain high-quality test samples from the target domain. However, these methods only consider inter- sample interactions and may fail to generalize well when the downstream tasks require fine-gu fnneggrined knowledge or there is insufficient similarity across samples.
3 Methodology
3.1 Preliminary
Problem setting. We begin by introducing the basic notations in test-time adaptation. We consider binary classification for simplicity and the theory can be easily extended to multi-classifications settings. Let (z, y) denotes the joint distribution of image and labels in the target distribution, and
3

---

Classiter Weights
Teat Semple
Chesster Distance
Sampo pie with Different Classes
Optimizatoon Direction
Fealure Center of the Samples
(a) Cross Entropy Optimization
(b) Cache Classifier
Figure 2: Connection between cross ents-pyy optimization and cache classifier well-clune well-clustered samples with a frozen feature encoder. With optimization of cross- entropy, samples will pull the classifier weights closer of the same class while pushing them away from different class weights. Since the feature space well-clustered, the classifier weights will ultimately converge near the feature center of the samples. Finally, the optimal classifier achieved through cross-entropy minimization will exhibit similar hehavior with the cache classifier.
simply assume that samples {(1,, B1)) are drawn ii.d. from the distribution with y, represents the one-hot label.
Definition 1. (Classification error.) Given f as binary classification function. The error incurred by hypothesis I E H X - 1) under the distribution pe pr y) can be defined as
<(f) = Ep + v] Bp. (f) yy]
(1)
the last equality holds in binary classification setting. Definition 2. (Excess error.) Given the Bayes classifier under distribution Po(x): (())= 1 1/2} and the optimal classfier f", the excess error of f is defined as
[1
1 [
E()) =()==(J)- - =(s") 2B 2.x
CLIP classifier Let g be the image encoder of CLIP, C' be the feature dimension, N denotes the number of categories, w; E RC represents the ith text embedding cluster. Considering normalized embedding and g(z), we can derive a simplified version of the output of CLIP for class i:
Z = : g(z).
(3)
And we denote the output logits as p(r) = [21, Zz, ZN] E R..
Cache classifier Given an unseen sample encoder with dimensional C, cache size K and number categories N, the cache classfier conduct feature retrieval based on the similarity with the data {(T, y in the cache. The predictions based on Tip-Adapter [52] are follows:
Peache (x) = (g(x)G".aehe) Y,
where A(z) scalar 3, Geache E RKxx represents the feature of K samples } in the cache and Y E R the corresponding labels {yr { Considering the number of samples class yis We can also derive simplified version of (4) follows, by ignoring the scaling function and adopting an instance-wing computation style:
denotes scaling function with weighting factor and smoothing
Peache (I)
Σ ai [9(x;) g(x))) Vh. 1=1
(5)
where a;
for class balance or O; Σラ [a(x)) for normalization across all the samples.
3.2 A Closer Look at Entropy-based and Cache- Methods
We start with analyzing the filtering operation of augmentated images in TPT. Pseudo-labels tends to be noisy in the test time, and entropy can serve confidence metric to identify trustworthy samples

---

Text Embedding
Image Embedding
CLIP logits
Final logits
Cache logils
Update
Boceting
Historical
Image Encoder
Hietorical Samples
Augment
Boosting Samples
Filter by Entropy
Boosting Distribution
Prompt
Encoder
Dog
Text Embedding
Figure 3: Overall architecture of BoostAdapter. BoostAdapter leverages knowledge from the target domain and employs self-hootstrapping with historical and boosting samples boosting cache, respectively.
among augmented views [43, 41, 33]. These high-quality samples can be considered drawn i.i.d. from the so-called boosting distribution as defined below.
Definition 3. (Boosting Distribution.) Given test sample from target distribution x ~ pe(a),, let H()) be the entropy measuring function and Aug(.) )) be the regional augmentation, By filtering noisy samples based on thresthould we have the following property of boosting distribution ps(r)):
~ (x() + {
Aug(z) A H (p(z)) S <T}
(6)
We also terms the samples from the boosting distribution boosting samples. Then we can connect entropy-basee -based methods and cache classfier by the following proposition:
Proposition 1. (Informal) Given n samples {(x; b with freeze encoder that effectively performing feature clustering with respect labels, the gradient descent optimization direction of the classifier's 's weights based on eross-entroyy generally tends towards making predictions using the cache classifier with class balance weights defined on these samples.
An intuitive illustration of Proposition depicted in Figure 2, where the weights of optimal classfier behave like the feature centers across different classes with of the well-clusterd samples. Revisiting the catropy-based method TPT, when provided with high-quality boosting samples with low entropy drawn from the boosting distribution, the objective function of entropy minimization optimizes in a manner similar to conducting cross-entropy optimization over the pseudo-labels. According to Proposition 1, TPT performs similarly to the mache-based methods with cache comprising the same boosting samples from the boosting distribution.
3.3 Boosting your Training-free Adapters
Existing cache-based methods store historical test samples only as useful information for prediction. In light of the analysis above, we can integrate the idea behind TPT into these training- -fee adapters by incorporating boosting samples into the memory bank. In particular, each sample can participate in both satele and intra-sample interactions with the aggnstcc historical samples and the instance-aware boosting samples in the cache, respectively.
Specifically, with k selected historical samples and kn selected boosting samples comprise the cache, we extend the classifier defined in (4) and formulate our BoostAdapter as follows:
Pboost (x) = (o(z)) Y.
where A is the same scaling function defined in (4), Geache E + denotes the features of the combination of both the historical and boosting samples, and Y Rak is the label.
5

---

Since we do not have access to the labels of the test samples, we generate one-hot pseudo-labels for them using argmax operations. However, these pseudo-labels tend be noisy in the target domain. Therefore, we apply filtering based on entropy thresholds on the test data stream following [41] to obtain trustworthy historical samples. We employ similar operation to select boosting samples from multiple augmented views of the current sample. In practice, we dynamically adapt the entropy thresholds 7 for each test sample, with a fixed percentile p. The cache continuously updates with lower entropy historical samples from the test data stream, while the current test sample augments the cache with self-boosting samples and forms an independent cache that only affects its own prediction. Additionally, to maintain diversity while considering the relevance each test sample, set maximum shot capacity for each class in the cache. This means that samples in the cache will be replaced by lower-entrops historical sample boosting sample when necessary.
An important issue whether introducing boosting samples brings improvements the training- free adapters. We will first make some necessary assumptions and then theoretically verify the effectiveness in reducing target error by incorporating samples from the boosting distribution.
Assumption 1. (Strong Density Condition) For any test sample zq in the target distribution xi ~ Pe(a)) and the boosting distribution Pa(oo), given positive lower bound m and upper bound M, positive scaling constant et and Co, the radius bound R > O, and B(x,r) = {x' :ll x' x || r} is the ball centered on with radius We assume pt (x) and pa pa())) are absolutely continuous with respect to the Lebesgue measure in R.. For r E (0, R), we assume
Alp((z) n (B r)] cx\iB(oo r)] (pp) n (B r)] 2 co (B r)] m
(8)
dp((x) < M;m dd
dpm(z) < M,
dd
where À is the Lebesgue measure in Euclidean space. Assumption 2. (L-Lipschitz Condition) Let f be the classification function and L be a positive
constant. For all feasible I, we have [f(x)) [(x'] < L 1---- x' ||.
Assumption 3. (Low Noise Condition). Let C3 be positive constants and we assume pt p((x) satisfies Pz Pz-pe(a) ([f(x) - 21 < t) C2t for all 0.
Remark Assumption intuitively ensures that for any test sample, there is surrounding neighbor- hood with significant presence samples from the target domain and the boosting distribution. More importantly, for specific sample Zo. boosting samples other samples x ~ pt (x) from the target domain, i.e., generally, we have ot S Cb- Assumption 2 and 3 describe the smoothness of functions and imply high level of confidence in predictions around the threshold, respectively.
po(xo) should be closer to than
Proposition 2. (Historical Cache reduce Emperical Risk) Given f as the training- free classfier consisting of historical samples only defined by (4). Let ne to be the number of confudent previously predicted samples the target domain and kt as the number of historical samples in the cache, with assumptions 1-3, the following results hold with high-probability for large enough ke and ni-
((
1/4
K2
1/d
1+3
E() 50
+
CETRE
Proposition 3. (Historical Cache benefits from Boosting Samples) Let The to be all confident previously predicted samples in the target domain and na be the number of boosting samples that are drawn from the boosting distribution. Given kt and Kū, to be the number of historical samples and the number of boosting samples be selected as the nearest neighbors stored in the cache, respectively. Let was and Whi be the weights defined in Eq. (5) of the historical samples and boosting samples. We have the following bound for the empirical risk of the cache classfier defined in 7.
((x+tt))
1/4
kt
1/d
R6.
kib
1/d
1+8
E()) 0
kbb
+
Wli
+
Wba
(10)
cem
1=1
1=1

---

Table Full results on the OOD benchmark with ViT-B/16 backbone. We report top-1 accuracy and "Average" is calculated by taking the mean accuracy across all four OOD datasets.
Imagenet-V2
Imagenet-Sketch
Imagenet-A
Imagenet-R
Average
CLIP [36]
60.86
46.09
47.87
73.98
$7.20
CLIP+TPT [41]
64.35
47.94
54.77
77.06
60.81
CoOp [55]
47.99
49.71
75.21
59.28
CoOp+TPT [41]
66.83
49.29
57.95
77.27
62.84
Co-CoOp [54]
64.07
48.75
50.63
76.18
59.91
Co-CoOp+TPP [41]
64.85
48.27
58.47
78.65
62.61
Maple [18]
64.07
49.15
50,90
76.98
60.28
Maple TPT [41]
64.87
48.16
58.08
78.12
62.31
PlomptAAign [39]
65.29
50.23
59.37
79.33
63.55
DiffIPT [6]
65.10
46,80
55.68
75.00
60.52
TDA (17)
64.67
50.54
60.11
80.24
63.89
BoostAdapter
65.51
51.28
64.53
80.95
65.57
Table 2: Full results on the Cross-Domain Benchmark with ViT-B/16 backbone. We report top-1 accuracy and "Average" is calculated by taking the mean accuracy across all ten datasets. The error bound is +0.17.
Caltech
Flowers
Food101
Aireraft
SUN397
EuroSAT
UCF101
Avergee
Pets
Cars
DTD
CL.P [36]
93.35
88.25
65.48
67.44
83.65
23.67
62.59
44.27
42.01
65.13
63.58
CLIP+IPT [41]
94.16
87.79
66.87
68.98
84.67
24.78
65.50
47.75
42.44
68.04
65.10
CoOp [55]
93.70
89.14
64.51
68.71
85.30
64.15
41.92
46.39
66.55
63.88
CoCoOp [54]
93.79
90.46
64.90
70.85
83.97
22.29
66.89
45.45
39.23
68.44
64.63
MaPLe [18]
93.53
90.49
65.57
72.23
86.20
24.74
67.01
46.49
48.06
68.69
66.30
MaPLe+TPT [41]
93.59
90.72
66.50
72.37
86.64
24.70
67.54
45.87
47.80
69.19
66.50
DiffTPT [6]
92.49
88.22
67.01
70.10
87.23
25.60
65.74
47.00
43.13
62.67
65.47
PromptAlign [39]
94.01
90.76
68.50
72.39
86.65
24.80
67.54
47.24
47.86
69.47
66.92
TDA [17]
94.24
88.63
67.28
71.42
86.14
23.91
67.62
47.40
58.00
70.66
67.53
BoostAdapter
94.77
89.51
69.30
71.66
87.17
27.45
68.09
45.69
61.22
71.93
68.68
Remark Proposition 2 provides a guarantee of the effectiveness of selecting k, out of n, historical samples comprise the cache. The empirical risk quite small when n, - 30 since the cache captures the full information of the target domain. Proposition 3 demonstrates that the historical cache can further reduce empirical risk by incorporating Kú boosting samples.
Experiments
4.1 Experimental Setup
Datasets Following the setting TPT [41], we conduct experiments on both Out-of-Distribution (OOD) benchmark and Cross-Domain benchmark. The OOD benchmark evaluates the model's
robustness natural distribution shifts on 4 ImageNet [4] Variants, including ImageNetV2 [37],
Imagehet-Sketch [44], ImageNet-A [14] and ImageNet-R [13]. We evaluate the transferring per-
formance on 11 datasets in the Cross-Domain benchmark: Aircraft [31], Caltech101 [5]. Cars [19],
DTD [3], Au [1]], Flower 102 [32], Food [2], Pets [34], SUN397 [4d] UCF101 [42].
follow the split in [55] and report the top-1 accuracy. The error bound are also provided.
Implementation details We utilize pre-trained VIT-B/16 of CLIP as the foundation model. In test-time adaptation, the batch size is
be We search for the optimal shot capacity balance diversity and relevance of samples. For boosting samples, we utilize random crop and then random horizontal flip as augmentations. Moreover, we empirically set the entropy threshold percentile to p 0.1 and filter 64 augmented views based on random cropping to obtain the boosting samples. and filter 64 augmented views obtain the boosting samples. The top-1 accuracy and the error bound reported on the test sets. All our experiments are conducted with Nvidia 3090 24GB GPU.
7

---

27.5
Entropy Retrieval
Top-1 accuracy
70
25.0
Tee-1 accuract
22.5 CUP
20.0
17.5
66
CLIP
BoostAdapter
15.0
BoostAdapter
Number of augmented views
Baouting Sarppes
Total shot capacity
(a) Number of augmented views
(b) Adaptation methods
Total shot capacity
Figure 4: Ablation studies of (a) number of augmented views to generate boosting samples (b) different adaptation methods and (c) total shot capacity of the cache.
Table 3: Ablation study on historical samples and boosting samples on the OOD benchmark with ViT-B/16 backbone. We report top-1 accu- racy and the error bound is +0.12.
Table 4: Full results on the OOD benchmark with RN-50 backbone. We report top-1 accu- racy and the error bound is +0.06.
-V2
-Sketch
-R
Average
-Sketch
-A
Average
CLIP [36]
TVT
$4.70
51.41
33.37
21.83
56.15
40.69
CLIP
60.86
46.09
47.87
73.98
$7.20
CoOp [55]
CALIP (10)
55.40
53.70
35.61
34.67
23.06
23.96
56.60
42.43
Historical Samples
64.93
50.23
63.80
80.43
64.85
CoCaOp [54]
58.72
34.48
23.32
57.74
42.82
Boosting Samples
65.40
50.59
64.40
65.34
DarIFT
58.80
37.10
38.12
BoostAdapter
65.51
51.28
64.53
80.95
65.57
BoostAdaateer
56.14
38.87
35.12
62.66
48.20
4.2 Out-of-Distribution Generalization
To verify the robustness of BoostAdapter, evaluate method on the OOD benchmark, in comparison with existing taiming-requins methods including CoOp [55], CoCoOp [54], TPT [41], DiffTPT [6], Maple [18] and PromptAlign [39], as well training-free method TDA [17]. As be seen from Table 8, the most striking observation emerging from the comparison that BoostAdapter significantly outperforms other baselines on average and improves the generalization ability of the model. For training-free methods such as TPT, DiffTPT and PromptAlign, BoostAdapter achieves superior performance while saving on optimization computation overhead. For training-frres free methods like TDA, BoostAdapter gains consistent performance improvements with the introduction of the boosting samples. Notably, BoostAdapter surpasses TDA by 4.42% on ImageNet-A and 0.84% on ImageNet-V22 V2, respectively. This enhancement indicates the effectiveness of ceff-bootstrapping when historical samples may not provide sufficient useful information.
4.3 Cross-Domain Transter
We further highlight our improvements in the transfer ability of CLIP on the Cross-Domain benchmark and present the results in Table 2. Compared with existing training- reahinreerrrredddd and training-free methods, Boost/daapee achieves state-of- titthhe-rt performance on 7 out of 10 tasks, surpassing the strongest baselines by an average of 1.15%. With diverse classes at test time, regional boosting enables BoostAdapter to adaptively extract knowledge that makes classes distinct from each other in multi-scale manner. Notably, for datasets requiring fine-grained information for classification such as Aircraft, the improvement of BoostAdapter is most significant.
4.4 Ablation Study
Historical Samples and Boosting Samples. To demonstrate the effect historical and boosting samples, we introduce two variants of BoostAdapter that utilize only historical samples or only boosting samples, respectively. Additionally, we provide the zero-shot results CLIP for comparison. As shown in Table 3, CLIP significantly benefits from both historical samples and boosting samples, resulting in notable improvements performance. The consistent improvement of Adaster compared to the variant that utilizes only historical samples further confirms the effectiveness of

---

Table 5: Full results on the Cross-Domain Benchmark with RN-50 backbone. We report top-1
accuracy and "Average" is calculated by taking the mean accuracy across all ten datasets. The error bound +.005.
Caltech
Flowers
Food101
Aireraft
SUN397
EuroSAT
UCFI0
Average
Pots
Cars
DTD
CLIP [36]
85.88
83.57
55.70
61.75
73.97
15.66
58.8
40.37
23.69
58.84
55.82
CLIP + TPT [41]
87.02
84.49
58.46
62.69
74.88
17.58
61.46
40.84
28.33
60.82
57.66
CALIP [10]
87.71
86.21
56.27
66.38
77.42
17.76
58.59
42.39
38.90
61.72
59.34
DiffTPT [6]
86.89
83.40
60.71
63.53
79.21
17.60
62.72
40.72
41.04
62.67
59.85
CuPL [35]
89.29
84.84
57.28
65.44
76.94
19,59
62.55
48.64
38.38
58.97
60.19
TDA [17]
89.70
86.18
57.78
68.74
77.75
17.61
62.53
43.74
42.11
64.18
61.03
BoostAdapter
88.48
85.75
59.67
68.25
78.78
18.93
62.83
43.85
44.40
64.42
61.54
Table 6: Comparisons with baselines ImageNet-C at severity level 5 regarding accuracy (%).
Noise
Blur
Weather
Digital
Gauss.
Shot
Impul.
Defoc:
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
Avg.
CLIP-VIT-BI6
15.15
16.28
15.26
25.83
16.87
26.34
24.43
34,56
33.01
39.10
57.78
18.45
14.71
35.62
35.81
27.28
TDA
17.50
18.59
18.12
59.12
19.02
28.25
26.24
37.30
35,30
41.57 59.04
21.06
17.61
31.58 39.25
37.78
37,26
BoostAdapter
17.53
18.89
18.39
59.70
19.07
28.62
27.33
38.21
36.13
42331
59.63
21.22
18.23
38.07
32.17
incorporating boosting samples into the training-free free adapters. See Section E in the Appendix for more results.
Number of Augmented Views for Boosting Samples. We augment the testing samples and filter them by mutual information with the CLIP text embedding to obtain the boosting samples. We vary the number of augmented views and investigate the performance of BoostAdapter on UCF101 in Figure 4a. With a larger number of augmented views, the performance improves due to more bootstrapping information of the test sample, which is consistent with the conclusions of TPT [41] and PromptAlign [39]. However, the computational overhead also increases with more augmented views, and selecting 64 augmented views a fair trade-off between boosting performance and efficiency.
Adaptation Methods. Training-required methods use entropy as a self-supervised objective, whereas training-free frre methods classify samples based on feature retrieval. We compare the performance of these two adaptation methods under the constraints of historical samples only, boosting samples only, or both, and present the results on Flower10) 102 in Fig. 4b. Entropy minimization requires gradient descent and model optimization, resulting in high training costs and relatively lower performance across all three settings. contrast, the training-free methods based on feature retrieval offer significant performance improvements with lower computational overhead. Additionally, both adaptation methods benefit from combining historical samples and boosting samples, consistent with the conclusions in Table 3.
Total shot capacity. BoostAdapter maintains low-entropy samples per class in the cache, and Figure 4c studies the influence of different total shot capacities containing historical samples and boosting samples of each class on Aircraft. As can be observed from the results, when the cache capacity small, the low-entropy samples maintained by BoostAdapter do not necessarily provide benefit for classification compared to CLIP. As the shot capacity increases, BoostAdapter will achieve the best balance of diversity and relevance, and a larger capacity does not guarantee better performance.
Versatility. To demonstrate the versatility of BoostAdapter, we apply it the RN-50 backbone and present the results in Tables 4 and 5. The improvement is consistent and on average, BoostAdapter outperforms TDA by 1.57% on the OOD benchmark and 0.49% on the Cross- Doss--Dominn benchmark.
4.5 Discussions
Generalization on Corruption Datasets To further evaluate the generalization ability of Boost- Adapter in new test-time scenarios, we compare BoostAdapter with baseline methods on the Imagenet- C dataset at the highest severity level 5. The key observation from Table 6 is that BoostAdapter

---

Table 7: Efficiency analysis. We evaluate different methods single NVIDIA 3090 24GB GPU and report the frames per second (fps) and memory cost (GB).
Augmentation
Views
Inference Speed (Eps)
Mernry (GB)
OOD Results
Cross: Domain Results
CLIP DITTPT TDA
82.3
D.7
57.20
65.58
Augmix
65.10
Dillusion
0.10
14.4
60.52
66.92
Augmix
64
11.89
1.2
63.8
67.3 33
BoostAdapter
Rand. Crop
Rand, Horiz. Flip
11.23
1,2
65.57
68.68
Sample
Boosiing Sample
Semple
Boosting Sample
Table & Unification more training-required methods. BoostAdapter benefits from different
trainieg-required methods.
Average
Metal
Snall
Crutch
Microphone
CLIP-VIT-8/1
60.86
47.87
TDA
64.67
50.54
60.11
80.24
63. 89
Figure 5: Qualitative results. The model predictions provided below the images. Boosting samples with low entropy improves information extraction from the
BoostAdapeer
63.31
31.25
64.33
80.95
65:57
Rooat Achetteee TSD
68.49
51.50
64.37
81.15
65.61
BoostAdapir- DEYO
65.71
51.52
64.85
$1.43
65.83
test sample and helps model to distinguish better.
consistently outperforms TDA across all 15 corruption types, highlighting its practical applicability in real-world situations. The superior performance of BoostAdapter stems from its capability to capture the knowledge of the test sample even under severe corruption. This achieved with the help of the boosting samples, which effectively filter out noisy parts while retaining useful information.
Efficiency Analysis BoostAdapter requires augmentation over the test samples, which may slightly affect the inference speed during testing. We conduct an efficiency analysis of BoostAdapter in comparison with existing Test Time Augmentation (TTA) methods and provide the results in Table BoostAdapter is slightly slower than the cache-based method TDA, yet still significantly faster than trquuie--required methods. The memory cost of BoostAdapter is also comparable to other baselines.
Unification of Trainieg-required and Training-free Methods. From the unified perspective, we also enhance training-free free adapters with additional training- methods. Here take TSD [45] and DEYO [21] as the showcase. Specifically, in the BoostAdapter+DEEY variant, we filter augmented views with PLPD lower than 0.2. For the BoostAdapter TSD variant, we discard augmented views that have different cache predictions and CLIP predictions to ensure consistency of the boosting samples. When equipping BoostAdapter with the technique of TSD and DEYO, observe further improvement and find that training-free free adapters can benefit from various boosting techniques of caiiing-requiree methods.
Qualitative Results The qualitative results are provided in Figure. 5. By incorporating samples with low entropy from regional bootstrapping, the model is enhanced to more effectively capture the fine-grained information of the test samples, thereby improving the overall performance.
5 Conclusions
In this work, we present insightful analysis of existing training-required and training-free TTA methods to bridge the gap between them. In particular, we improve training-free adapters by incorpo- rating self-boosting samples into the memory bank inspired by the idea of regional bootstrapping from entropy-bass based methods. The cache in our method, containing anstance-agnostic historical samples and instance-aware boosting samples, is capable of performing knowledge mining both the target domain and the testing sample itself. We also derive error bounds in the test-time adaptation setting and show that this cache benefits from both historical samples and boosting samples. Extensive experiments on the two benchmarks demonstrate the effecttivneess of our method.
Despite the promising performance of our method, it also has some limitations. It requires slightly more computation overhead than existing training-free adapters due to the multiple augmentation of the test samples, as discussed in Appendix. One future direction is to develop more efficient augmentation method to obtain boosting samples, rather than merely randomly cropping and then filtering over the test samples.
10

---

Acknowledgements
This work supported in part by the National Natural Science Foundation of China, un- der Grants (624B2088, 62302309, 62171248), Shenzhen Science and Technology Program and PCNL KEY project (PCL20234S6-1)
References
[1] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0722 2021.
[2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mood-100-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
[3] Mircea Cimpoi, Subhransu Maji, Jasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[4] Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. IEEE, 2009.
[5] Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference on Computer Vision and Pattern Recognition Workshop, pages 178-178. IEEE, 2004.
[6] Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmen- tation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2704-2742000 2144-214,, 2023.
[7] Kuofeng Gao, Jindong Gu, Yang Bai, Shu-Tao Xia, Philip Torr, Wei Liu, and Zhifeng Li. Energy-latency manipulation of multi-modal large language models via verbose samples. arXiv preprint arXiv:2404. 16557, 2024.
[8] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter Better vision-languag models with feature adapters. International Journal of Computer Vision, 132(():581 1-555 2024.
[9] Hang Guo, Tao Dai, Zhihao Ouyang, Taolin Zhang, Yaohua Zha, Bin Chen, and Shu-tao Xia. Refir: Grounding large restoration models with retrieval augmentation. arXiv preprint 2rX:v:2410.05001, 2024.
[10] Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. Calip: Zero-shot enhancement of clip with frremeter-free attention. In Proceedingg of the AAAI Conference on Artificial Intelligence, volume 37, pages 746-754, 2023.
[11] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2117-2266, 2019.
[12] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 1903. /2261, 2019.
[13] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of
generalization. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages
2021.

---

[14] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song, Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15262-15271, 2021.
[15] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34:2427-2440, 2021.
[16] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and -lanuale representation learning with noisy text supervision. In ICML, pages 4904-4916. PMLR, 2021.
[17] Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. arXiv preprint arXiv:2403. (80.::2303..8293,, 2024.
[18] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[19] Jonathan Krause, Michael Stark, Jia Deng. and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
[20] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
[21] Jonghyun Lee, Dahuin Jung, Saehyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang, and Sungroh Yoon. Entropy is not enough for teme-time adaptation: From the perspective of disentangled factors. arXiv preprint arXiv:2403.07336 2024.
[22] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffiiient prompt tuning. arXiv preprint (xXiv:211440889// 2021.
[23] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
[24] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified dision-language understanding and generation. In International conference on machine learning, pages 12888-12900 PMLR, 2022.
[25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19744 PMLR, 2023.
[26] Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, and Xinchao Wang. Graphadapter: Tuning linion-language models with dual knowledge graph. Advances in Neural Information Processing Systems, 36, 2024.
[27] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection in neural networks. arXiv preprint arXiv: 170.002690,0 2017.
[28] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable fine-tuning universally across scales and tasks. arXiv preprint 1rXiv:2110.076600 2021.
[29] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 5206-5215, 2022.
[30] Zhihe Lu, Jiawang Bai, Xin Li, Zeyu Xiao, and Xinchao Wang. Beyond sole strength: Cus- tomized ensembles for generalized vision-language models, arXiv preprint arXiv:231 2.1.. 17091 2023.
12

---

[31] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification aircraft. arXiv preprint arXiv:1306.513: 2013.
[32] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
[33] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. arXiv preprint arXiv:2302. 12400, 2023.
[34] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3498 -300. IEEE, 2012.
[35] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15691-15701, 2023.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 01448-876... PMLR, 2021.
[37] Benjamin Recht. Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[39] Jameel Hassan Abdul Samadh, Hanan Gani, Noor Hazim Hussein, Muhammad Uzair Khattak, Muzammal Naseer, Fahad Khan, and Salman Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. In Thirty-seventth Conference on Neural Information Processing Systems, 2023.
[40] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
[41] Manli Shu, Weili Nie. De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in -sioun-anguuage models. Advances in Neural Information Processing Systems, 35:14274-14289, 2022.
[42] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(11), 2012.
[43] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
[44] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[45] Shuai Wang, Daoan Zhang, Zipei Yan, Jianguo Zhang, and Rui Li. Feature alignment and uniformity for test time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 20050-20060, 2023.
[46] Xiang Wang, Shiwei Zhang, Jun Cen, Changxin Gao, Yingya Zhang, Deli Zhao, and Nong Sang. Clip-guided prototype modulating for few-shot action recognition. International Journal of Computer Vision, pages 1-14, 2023.
13

---

[47] Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Vita-clip: Video and text adaptive clip via multimodal prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23034-23044 2023.
[48] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485-3492. IEEE, 2010.
[49] Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15671-15680, 2022.
[50] Sheng Yang, Jiawang Bai, Kuofeng Gao, Yong Yang, Yiming Li, and Shu-Tao Xia. Not all prompts are secure: A switchable backdoor attack against pre-trained vision transfomers. In CVPR, 2024.
[51] Yaohua Zha, Jinpeng Wang, Tao Dai, Bin Chen, Zhi Wang, and Shu-Tao Xia. Instance-aware dynamic prompt tuning for pre-trained point cloud models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14161-14170, 2023.
[52] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, pages 493-510. Springer, 2022.
[53] Yifan Zhang, Xue Wang, Kexin Jin, Kun Yuan, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Adanpe: Exploring non-parametric classifier for test-time adaptation. In International Conference on Machine Learning, pages 41647-41676. PMLR, 2023.
[54] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learn- ing for vision-language models. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[55] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV), 2022.
[56] Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2605-2615, 2023.
14

---

Appendix
A
Dataset and Licenses
Table presents the statistics and details of datasets used in the paper. We also provide the corre- sponding license information of the datasets and source code.
Datasets. Below are the datasets used in this paper that have known license information: The
following datasets used in this paper are under the MIT License: ImageNet-A [14], ImageNet-V2 [37],
ImageNet-R [13], ImageNet-Skette [44], EuroSAT AT [11], Food101 [2]. The following datasets used in this paper are under the CC BY-SA 4.0 License: Oxford-Pets [34],
Caltech101 [5].
The following datasets used in this paper are for research purposes only: DTD [3], StanfordCars [19], SUN397 [48], FGVC-Aircraft [31]. Flower102 [34], UCF101 [42].
Source code. We use the implementation of existing baseline methods for reporting their results in this paper. Below are their license information: Source code used in this paper that under the MIT
License: CLIP [36], PromptAlign [39] and TDA [17].
Dataset
Description
Classes
Test Size
Out-of-Distribution Benchmark
ImageNet-V. V2
New Validation Sets of ImageNet
1,000
10,000
ImageNet-S
Sketch Images
1,000
50,000
ImageNet-A
Natural Adversarial Examples
200
7,500
ImageNet-R
Rendition Extension of ImageNet
200
30,000
Cross-Domain Benchmark
Aireraft
Aircraft Model Classification
100
3,333
Caltech101
Natural Image Classification
100
2,465
Cars
Cars Classification
196
8,041
DTD
Describable Textures Dataset
1,692
EuroSAT
Satellite Images
10
8,100
Flowers102
Flowers Classification
102
2,463
Food101
Food Classification
101
30,300
Pets
Pets Classification
3,669
SUN397
Scene Categorization Benchmark
397
19,850
UCF101
Action Recognition Dataset
3,783
Table 9: Datasets statistics.
B Broader Impacts
In this paper, we focus on bridging the gap between training-required and training-free methods to improve the generalization ability of lisiuanlanguaee models. We also theoretically derive the error bound of incorporating boosting samples into the historical cache. We hope that our work will inspire the community to explore test-time adaptation in effective and efficient way.
C Theoretical Proof
C.1 Cross-entropy Optimization behaves like Cache Classifier over well-clustered Samples (Proof of Proposition 1)
Given well-clustered samples in the feature space and the classifier defined in Eq.(3), we first derive the distance between the weights of the classifier and the optimal weights and then establish the connection between the optimal weights with the features center of the samples.
15

---

Suppose the classifier function f over samples is convex and differentiable, and also L-smooth. Let the distance between imitial weight w and optimal weight w" to be D = || w I|. GD updates
by ((++1) = "² - fi V (w=") with step size fi = t. and then GD enjoys the following convergence guarantee:
to w01l 2L 10000) w 11
LD T
O
(11)
T
We then showcase the relationship between w. and the features center μ4 of class i, = 2, ..... N. Since optimize on well-clustered samples, we consider the scenarios of perfect clusters, where samples in the class will be encoded into the same point μe, by the encoder g. and these points should be farthest enough between each other. Given samples { yk )= with the number of samples in class to be n4, the cross-entropy loss function L can be written as:
L=
Σ
log Pvy =yxlaa)
(12)
Substitute the sample g(xx)) n4 from class i, we derive the probability P(y i|0x)) using the softmax function from (3) is:
explw? pii P(y = il2xa) = E/=1 exp(us H&i)
(13)
Thus, the cross-entropy loss for sample (Ir i) is:
explu? Hi)
LA = log
(14)
E/= 1= exp(w? pi)
For all samples, the total loss is:
explwtt pss)
I.
Σ n: log
(15)
Σラ=1 exp(w] ps)
N
The gradient of the loss with respect to wi can be simplified as:
0L
cxp(w? 14k)
uim, + 14
72k
(16)
ow;
Ej=1 exp(u? p4)
When converges the optimal weight, we have the condition of fixed point dwy OL
And we have
exp((wt)) (ux) = 0. j=1 exp((w3) ((u))
14774 + 14
(17)
Thus, have
N
cxp((wF))
(4x)
T4.
(18)
Σn exp((w7) (Tz)TTL)
(upt))))))))))) Given well-clustered samples, could have ezp((w)) )' YTuu mma) > exp((w;)) uu) for specific i when wf is near μgg Then since the equality in (18) will hold for each class and for class i i= 1,2, N we have
+ 14.
(19)
Combining (11) and Eq.(18), with iteration steps T, show that the weight of classfier will finally converge to the feature center of each class:
LD2
w - μ11l S w|w
w01l
|| + 1| - 50
(20)
T
16

---

And we have the output logits of the optimal weights with the encoder g:
Perozs (x) = ((())
(21)
Next we discess the behavior of the cache classifier over these samples, Given the number of well- clustered samples class to be n,, the output logits the cache classifier defined Eq.(5) using samples (( Uk ) can be described as follows:
n
Peache (z)
[o(an)) g(x))u
ni |u? g(x))v
=
| g
(22)
Combining (q1( and Eq.(22), we draw the conclusion that cross-entropy optimization behaves like cache classifier over well-clustered samples.
C.2 Historical Cache reduce Empirical Risk (Proof of Proposition 2)
follow the proofs [53] and extend conclusion to boosting samples.
C.2.1 Additional Definitions and Assumptions
Definition 4. (Wasserstein-diittane and the dual form). Wasserstein distance measures the distance between two probability distributions on a given metric space. It is defined using the concept of optimal transport. For two distributions P. Q The p-th Wasserstein distance is defined as
1/8
f da, (o.yy)dya
(T, Q)
inf (E
(23)
Here, [IP Q) denotes the set of all couplings (or transport plans) y of and Q, i.e., joint distributions on X x X with marginals F and Q The idea is to find the optimal way to transport the mass from one distribution to the other with the minimal cost, where the cost is given by the p-th power of the distance.
The first Wasserstein distance, W (TP Q) ,often referred to as the Earth- Mover Distancel EMD), has particularly elegant dual representation. The dual form of W1 leverages the Kantorovich Rubiistteeee duality and can be expressed as:
sup
(24)
Here, the supremum is taken over all -Lipschitz,uunctinss functions f whhich are functions satisfying (f f(y)l I d(æ, y) for all x, y E X This representation shows that W1 can be seen the maximum difference in expected values of 1-Lipschitz function over the two distributions. the following part, funccioot,,unction
Wasserstein distance represents the first Wasserstein distance for simplicity and we utilize W(.-) instead of W.(s.).
Given the definition of the Wasserstein distance, we have the following proposition that derive the empirical risk the target domain according to Theorem from [40].
Proposition 4. Given two distributions PP, Q, denote f arg (E EQ(s)) and & Ep(f") + +f())) Assume all hypotheses are L-Lipschitz continuous, the risk of hypothesis is then bounded by
colf) sE(c(ff) +2WW Q)
(25)
17

---

C.2.2 Distance between the Ball Distribution with the Target Distribution
When using the cache classifier with historical samples, a large number of samples that are not similar enough from the target domain will be filtered and the selected samples with high weight are all close to the target data. Thus extend the conclusion in [53] to the distance between the ball distribution with the target distribution. Considering a test sample from the target distribution di E Pe (a) and distribution consisting of ball center of all the test samples ? == Un (p())(())) B( r), informally, according to (2.(,3), we have the distance between the ball distribution with the target distribution as follows:
// 24 Iball || da( Tball),
inf W((2,e(()) (E.
(26)
where for each Tball E Ω, find at least overall distance will then be bounded by r. Specifically, we can choose a density function y where " (Iball, 21) > 0 only if Tball Iz I otherwise 0, then have
Pr (z(x) such that Zball It || r. the
semin // Tball - Dt dr xt)
((S2,pr()))
inf (EM(R..
$ <Iʃ
|| Dhaul zz ll 11 (Iraall Te drraaus S r.
(27)
However, there no guarantee that each data Dt E pr(x)) can find neighbor B(x,, r) with [B(x,,r) with all the small We then provide the probability that the of neighbors B(x,,r) r) of each v, E pr())) is not measuring zero with respect to the radius r.
As defined in the cache classfier Eq,(5), we denote ke is the number of historical samples select in the cache and n, the total number of data from the historical stream. With the strong density assumption, given the coefficient bound m and M, for any Ie E pc(x), r < R, according to Assumption 1, we have
= f ( dd
dpe(x) (Er)d
|dfe E pr(x) A B(x,r))
2 m)(B(ar))) (p(()))
> mo
(28)
where X(B(O, 1)) is the volume of the d dimension unit ball and À is the Lebesgue measure of 24 set in a Euclidean space. Set To memaan van, )//d, with additional assumption that we utilize a small
we have ru < R. Then for any Ia E (x(),, according to k compared to n, so that ke m (28), we have
2
2k
f E (x) A E B(x,, ro)l mamar
(29)
TT,
Since o (2) are independently drawn from the target distribution, let II()) be the Indicator funciton and Sn, (x,) ) = Σx= I( E B(x,, ro)) denote the number of data of E pt (x() that fall into B(I ro). then Sn, (Ix) follows the Binomial distribution. Let W ~ Binomial 2x., according to the Chernoff inequality, we have
P(Sn, (xte < hia) P(W kt)
= P(W E(W1 < -kz)
< exp( -k, /2/2]][]]
exp(-kk 44).
(30)
where the second inequality holds since Sa(x) has larger mean than W. With a large ke, the probability that (x(x) < ky is small for any 3, E Pt (a). Denoting 24 (6) the in nearest sample to Iz among B(I,, ro) in the cache, we have for any 2, (x)
18

---

P(I 2, (h) 2z 1o) P(S 2 exp(-k,/4)
(31)
Combine (31) with the assumption that the distribution pz(x)) is finite with cardinality Np. and the desired probability part is shown by union bound.
n
P(I de (Ke)
(kx)
||5
n P(Sn(z) 2 ks)
ihe Epee
are (p)
U P(Sn(z) < k()
ZtEpe(x)
Np. exp
(-4)
exp (-4
kt
+ log Pip
(32)
And then we have the following proposition.
Proposition 5. Given the target domain distributions pt (3) that finite with cardinality Spp and = Upepi(z) B(xr, r), where B(x,r) = {x' :ll x' - x || r denotes a ball centered on x with radius r. Denote f = arg min min +en(()) and Ç =
Assume all hypotheses are L-Lipschitz continuous, the risk of hypothesis f on the unseen target domain is then bounded by
1/d
2k.
a()) R 15n++enl + 2T.
(33)
with probability exp(- + log Np.)
C.2.3 Excess Error Bound of Cache Classifier
Let s; to be the softmax probability softmar(Peackee for class i in the the cache classifier from Eq,(5), we can simplify the classifier as feache 1[ }}} on the binary classification setting.
Then feache (x+)) # f" (x+) implies that [cache( f (x+)) 2 (f 111. We then bridge the gap between the excess error and the classify error follows:
[F(ee - 1 (xraoee(x)) -
-(a))111 11
Ee( = ~p(a)))
(-xx))
(34)
We want to bound sup,, feache f (xx) S combining with the marginal assumption in Assumption and the fact that
E 1 112 < 2() SIP(Z
(35)
where Z = (x)) 1 so we have E,()) < Cgt To bound [faach((1)) - f (2x)) we denote
(oi, (9) S1. (8) as the ith nearest data and the corresponding labels v. in B(x,, ro). The result of the cache classfier with normalized weight will be
de
( (2))0) ((x)) 10
Scache(51))
(36)
9 (1) g(x)
j=1
wayh (5)
(37)
8(2,, (9) x(x)
where Σー (x)))))) o(())
is the normalized weight and Eh w, = 1. Based on the assump-
tions and notions above, have for any xe (x()
19

---

(eaccho(1e)- - (2x))
Σ Wiite
(1)
(3re)
Ne
Rx
With
(8)
Σ
waf
(2s (0)
+ Σ
wif (s, -f((z)
i=1
(38)
With
(1)
Σ
f
(0)
Σ Wi f (2) (2)()) - fr (x()
2m
1
where 2 is easy to bound. According to the assumption that f* is C-Smoothness, we have
Σ wi lr E ())))- (
Σ w.C. || E (5) 2z || C. ||
Lt (k))
(39)
1/d
According Eq.(31), with probability at least 1 - exp(-ky/
C
2h
Note
mc
that We store the target sample into the cache only when its prediction confidence is large enough. Therefore, is natural to assume that:
Eyix
0 ( f (x.").
(40)
Then we use the Hoeffding inequality to obtain the upper bound of !
lr
be
PX.
Σ
weth
(2f
re]
Ex
Prix
Σ
wiyi (1)
Σ (
i=1
2ca
exp(
Σ
m
(41)
We simplify the bound by assuming that the weights in the target domain are evenly distributed in the subset of all samples with respect specific class controlled by coefficient y, according to Assumption and Proposition 4. That is, we have E w
En
nk, ( nh
Set = (1/kg) ((1c==1/)))111
we have, with probability, least -3exp( -3cxp((21//kk)) ① S (1/k:)
2 <
2h: 1/d and then fcache (xt) (xx) $ (1/kg)1/44 C
1/d
C
2h men
According to
(31) and (35), the excess error is bounded by
1
1/4
2kt
1/d 1+8
Ex( <
2Cs
+C
1/1
hr
1/d
1+9
+ Ci
(42)
kg
c
with constant C1. When appropriately choosing k; = (log n4), we have
min(1 - 2exp( - / kc), exp( k//4))
2 1 - 2 expp -2/ 2// Kt) - -kpp())//4)
(43)
> 1 3 exxp( 1osp(22nV kz)
1 - 3 exxp( (exp(-O(11) -0(1) llggn
20

---

where the third line is because //4> kg for large enough kg. Namely, with probability at least 1 - 3exp(-v/lo 01)) the following bound holds true.
E, E so ((10gm)
1/4
log n4
1/d
1+0
(44)
C
C.3 Historical Cache benefits from Boosting Samples (Proof of Proposition 3)
To study the effect of the boosting samples, we consider the cache classfier containing both ky historical samples { (9) Dr ( and ky boosting samples {2ib" (4) (4)
Jimi
9/b
the nearest data to Fiml
Ra
( a(w)
Ie in B(xr, ro). With the normalized weights wai
and
()) g(x)
+ (2) a(a))
(0 ( o(w)
Wbi LI [ (1) a(x) +- 9 (2) a(a)) the prediction result of the cache classifier will be fetc E/e WeiVie
(8)
i=1 Wbil/b
hs
(4)
Then have:
(xe) -
by Σ
Wrihe
Weif (1t))(())) +
While
Whi of (x1)
1=1
ht
Weelit
(1)
wei (2fe
+
Whis S (fi (0)
Σ
I (1x)
1=1
1=1
Ho
Wbiya
woif (x((1))
+
Σ
Whif
x
worf (x1)
As
le
hb
Wea gh
+
Wiil f (2e (9)
Σ
Whif
(2((8)
kbb
Witt
(2t (1))
f fat))
Whi
fr
(1(
f (xx)
Similar to E40) we have the following assumption the boosting distribution:
Exix [16 (0) (x)
(45)
According to (41),,, we have
k Weil (Et (2,
hu
Px,
WHithe (6)
Whilbb
whis x
(i))
1=1
1=1
ls
woef (z) |)]
(46)
=Ex
Prix
Weiti
+
Wbrlbb
(4)
wii 8 (2, (6))
Σ
< 2eexpp221(6 + kob)
21

---

c (1/(k, + k4)) we have, with probability, at least 1 3exp(-2nV / + ko)), ① -
(L/(kr kb)) Then, according to (39), we have
Av
h+
we
f 11 ()
(
C.C. || [e zz 11 S.C- B
(1 (11) 24
(47)
and
kg
Σ Whi fr (1b (1)(0) - (2(e)) S 2=1
Whe C- 1b (0)
Iz || S.C- ||
dy (ks) It |I
(48)
where St = Ex Wts. S. = Σe=
== Whi are the sum of weights of historical samples and boosting samples, respectively, and have St + Sb 1.
Then we have the following results similar:
2k momaan
1/d
2kb
1/d
S.S.C
3 <
&SSC
(49)
Finally, the excess error under the covariate shift setting can be bounded by
114 (1a.tr,))
2k
1/d
2kb
1/d
1+.8
&(f) 5 2Cnp 2Cp
S.C
+ +SC
+ kin
mC
1/4
ht
1/d
kb
1/d
1+8
+ CiSe
CiSb
(50)
hi + kbb
cm
CbTlb
1+9
1/4
hi
hrs
1/d
kb
1/a
+ C1
Wti
C1
Whi
kit + kbb
CeTle
Compared (50) (42) and St + S 1, it is easy verify that
2(kk + kb)
1/d
2k,
1/d
2hb
1/d
(S. C
S.C
S.C
mc;
mcG
(51)
2kg
1/d
2kg
1/d
2 S
S.C
mC
mO
In general, the boosting distribution is more close to the test sample than the target distribution and we have C > 6s. Thus the difference (51) then larger than 0, namely incorporating boosting samples into the memory bank, the excess enror can be further reduced.
D More Experiments
Independent Cache for Boosting Samples. In BoostAdater, due to the cost of augmentation, the number of boosting samples is relatively smaller than the number of historical samples. Therefore, we use a joint cache for storing both historical and boosting samples to facilitate intra-sample and inter-sample interactions. Table 10 and Table 11 study the influence of using an independent cache for the boosting samples. As can be observed from the results, BoostAdaptes suffers from slight performance degradation due to the independent cache.
Table 10: Independent cache for boosting samples on the 0OD benchmark.
Imagenet-V2
Imagenet-Sketch
Imagenct-A
Imagenet-R R
Average
Independent Cache
65.37
50.62
64.56
80.96
65.38
Joint Cache
65.51
51.28
64.53
80.95
65.57
22

---

Table 11: Independent cache for boosting sample on the Cross-Domain Benchmark.
Caltech
Flowers
Food101
Aircraft
SUN397
EuroSAT
UCFION
Average
Pets
Cars
D
Independent Cache
94.69
88.88
69.19
71.94
86.99
26.76
67.64
44.21
61.20
69.63
68.11
Joint Cache
94.77
89.51
69.30
71.66
87.17
27.45
68.09
45.69
61.22
71.93
68.68
Different Augmentation for Boosting Samples. We make use of random crop followed by random horizontal flip as augmentations for generating boosting samples. Additionally, we further explore the influences different augmentations applied the randomly cropped images. The comparison methods include: (i) Random Brighness: Randomly set the brighness of image from 50% to 150%. (ii) Random Auto Contrast: Apply auto contrast over image with probability p = 0.5. (iii) Random Rotate: Randomly rotate the image from -45 degree to 45 degree. (iv) Random Vertical Flip: Apply vertical flip over image with probability p = 0.5. (v) Random Horizontal Flip Adapttr). Apply horizontal flip over image with probability p 0.5. The results presented in Table 12 and Table 13, The results indicate that random horizontal flipping outperforms other augmentation methods, primarily because the images generated from horizontal flips are closer to the original distribution when training CLIP.
Table 12: Comparison of different augmentations on the OOD benchmark. Default settings are marked in gray.
Imagenet-V.
Imagenet-Sketel
Imagenet-A
Imagenet-R
Average
Random Brightness
65.10
51.24
62.10
81.03
64.87
Random Auto Contrast
65.50
50.79
64.33
80.57
65.30
Random Rotate
61.14
47.67
60.83
78.15
61.95
Random Vertical Flip
63.39
49.67
60.77
78.55
63.10
Random Horizontal Flip
65.51
51.28
64.53
80.95
65.57
Table 13: Comparison of different augmentations on the Cross-Doman Benchmark, Default settings are marked in gray
Caltech
Flowers
Food101
Aireraft
SUN397
EuroSAT
UCF101
Average
Pets
Cars
DTD
Random Brightness
94.60
89.70
69.28
71.70
86.88
26.67
68.24
45.57
61.63
71.45
68.57
Random Auto Contrast
94.48
89.67
69.33
71.90
87.24
27.39
68.16
45.51
61.67
71.77
68.71
Random Rotate
94.52
89.59
67.74
71.30
85.91
24.27
67.56
45.45
60.72
70.66
67.77
Random Vertical Flip
94.89
89.53
68.75
72.19
86.78
24.99
67.72
45.27
61.56
70.82
68.25
Random Horizontal Flip
94.77
89.51
69.30
71.66
87.17
27.45
68.09
45.69
61.22
71.93
68.68
E
Additional Ablation Results
Historical Samples and Boosting Samples. We provide more ablation results the historical and boosting samples the Cross-Datases benchmark in Table 14. The observation is consistent with the results in Table 3, showing that CLIP gains improvements from both historical and boosting samples. Furthermore, when applied to various downstream tasks, the importance of regional bootstrapping becomes more significant, as indicated by the gap between BoostAdapter and the variant that uses boosting samples only.
Number Augmented Views for Boosting Samples. The complete results on the number of augmented views are presented in Table 15 and Table 16. With more augmented views, BoostAdapter able better extract the ine-grained information from the original test sample, achieving improved performance.
23

---

Table 14: Ablation study on historical samples and boosting sample on the Cross-Domain Benchmark.
Caltech
Flowers
Food101
Aireraft
SUN397
EuroSAT
UCF101
Averuge
Pets
Cars
DTD
CLIP
93.35
88.25
65.48
67.44
83.65
23.67
62.59
44.27
42.01
65.13
63.58
Historical Samples
94.16
89.42
66.87
72.11
85.93
24.69
67.24
44.80
61.85
69.81
67.69
Boosting Samples
94.32
88.64
68.38
71.54
87.12
27.30
67.42
44.68
45.93
69.34
66.47
BoostAdapter
94.77
89.51
69.30
71.66
87.17
27.45
68.09
45.69
61.22
71.93
68.68
Table 15: Results of different views on the OOD benchmark. Default settings are marked in gray
Imagenet-V2
Imagenet- -keechh
Imagenet-A
Imagenet-R
Average
16 Views
79.41
49.01
62.08
63.68
63.54
32 Views
80.32
50.73
63.22
64.91
64.80
Views
80.95
51.28
64.53
65.51
65.57
128 Views
80.95
51.91
64.06
65.27
65.55
Table 16: Results of different views on the Cross-Domant Benchmark. Default settings marked gray.
Caltech
Flowers
Food101
Aireraft
SUN397
EuroSAT
UCF101
Average
Pets
Cars
DTD
16 Views
93.95
89.62
68.06
71.62
86.76
25.71
67.33
45.39
62.07
70.97
68.15
32 Views
94.48
89.59
69.07
71.54
87.01
27.18
67.97
45.45
61.22
71.56
68.51
64 Views
94.77
89.51
69.3
71.66
87.17
27.45
68.09
45.69
61.22
71.93
68.68
128 Views
94.77
89.62
69.15
71.34
87.28
27.15
68.15
45.86
61.19
71.87
68.64
Fixed shot capacity. We search the optimal total shot capicity in BoostAdapter. We also find that fixing the cache size to be can generalize well in different task settings, as shown in Table 17 and Table 18.
F More Qualitative Results
More qualitative results are provided in Fig. 6.
24

---

Table 17: Results of fixed shot capacity on the OOD benchmark.
Imagenet- V2
Bingenet-Sketche
Imagenet-A
Imagenet-R
Average
CL.IP
60.86
46.09
47.87
73.98
57.20
CLIP+TPT
64.35
47.94
54.77
77.06
60,81
PromptAlign
65,29
50.23
59.37
79.33
63.55
TDA
64.67
50.54
60.11
80.24
63.89
Fixe
65.13
50.66
63.96
80.44
65.05
BoostAhapter-Seurch
65.03
50.66
64.27
80.64
65.15
Table 18: Results of fixed shot capacity on the Cross-Domain Benchmark.
Caltech
Flowers
Food101
Aireraft
SUN397
EuroSAr
UCF101
Average
Pets
Cars
DTD
CLIP
93.35
88.25
65.48
67.44
83.65
23.67
62.59
44.27
42.01
65.13
63.58
CLIP+TPT
94.16
87.79
66.87
68.98
84.67
24.78
65.50
47.75
42.44
68.04
65.10
PromptAlign
94.01
90.76
68.50
72.39
86.65
24.80
67.54
47.24
47.86
69.47
66.92
TDA
94.24
88.63
67.28
71.42
86.14
23.91
67.62
47.40
58.00
70.66
67.53
BoostAdaptee-Fiinn Fixed
94.77
88,85
69.30
71.66
87.17
27.00
67.64
44.33
61.22
69.73
68.17
BoostAdapter- Search
94.77
89.51
69.30
71.66
87.17
27.45
68.09
45.69
61.22
71.93
68.68
Test Sample
Boosting Sample
Test Sample
Boosting Sample
Test Sample
Boosting Sample
mitten
cottontail rabbit
A380
777-200
Industrial Buildings
Highway or Road
ImagNet-A
Aircraft
EuroSAT
Figure 6: More qualitative results ImagNet-A, Aircraft and EuroSAT.
25

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope?
Question:
Answer: [Yes]
Justification: The main claims made in the abstract and introduction reflect the main idea described in Section 3.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: The limitation and the discussion about computational overhead can be found in the Section 5. These assumptions are reasonable in domain adaptation and parameter analysis is conduct in the ablation studies in Section 4.4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The theoretical proof of the proposition used in the paper can be found in Section C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
26

---

All the theorems, formulas, and proofs in the paper should be numbered and cross
referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the implementation details in Section 4.1 for reproduction. Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will open source the code once accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips
mublic/guided CoodeeSbmissiioooo..
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
27

---

reproducible, they including code, unless this is central to the contribution (e.g., for new open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. cctpublic/ c/guid/gssssseee Suse/euedmubiiic ississsinn pessoooniccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the implementation details in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide the error bound along with the main results. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We provide the information about compute resources in the implementation details in Section 4.1.
28

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ett:///nn.ippp/e cc cc/public/ Ceubllic/Etti wttpettihiiiieeeeeeeeeeeeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurlPS Code of Ethics. Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid-
eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We provide discussions of broader impacts in Section B Appendix. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring
29

---

that users adhere to usage guidelines or restrictions to access the model or implementing
safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors
should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We mention the licenses of existing assets in the Section A in Appendix. Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
30

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "jk7MexfT2H", "reviewer": "Reviewer_YJpm", "summary": "The paper focuses on gradient-free test time adaptation of CLIP model with ViT-B/16 and ResNet-50 backbones on out-of-distribution datasets. The authors take inspiration of augmentations from gradient-based test time methods and incorporate this concept of augmentations in gradient-free and memory (cache) based test time methods. Previous work considers only historic samples to be in cache, whereas this work also includes the augmentations of test samples with lower entropy into the cache. Results on OOD benchmark and cross-domain benchmark show improved average performance compared to prior works.", "questions": "Please refer to Weakness above.", "limitations": "Limitations are discussed in the work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tProposed a simple but effective approach to include low entropy augmentations into the memory.\n\n2.\tEstablished theoretical bounds to justify the inclusion of augmentations into memory and its relation to minimization of empirical risk.\n\n3.\tProposed approach brings significant improvements on ImageNet-A, Aircraft and EuroSAT datasets.", "weaknesses": "I don’t have major concerns on the proposed approach, as the method is simple and straightforward. My concern lies in increase of computation overhead and extended run time due to running CLIP model on multiple augmentations during test time (as acknowledged by the authors). In addition, it can be noticed that results are comparable on almost all datasets except ImageNet-A, Aircraft and EuroSAT datasets. It would be interesting to provide rationale for the proposed approach to work much better on these datasets. Results are shown on single ViT architecture, however providing results on multiple CLIP based ViT backbones would be interesting as the method targets for test time performance. It would be helpful to evaluate the method on more OOD distributions like corruptions (ImageNet-C) to understand it better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Computation overhead and efficiency.** \n\nPlease refer to Q2 in Global Response. \n\n**Q2. Specific Datasets.**\n\nAs shown in Figure 5 in the appendix, BoostAdapter benefits from boosting samples to capture fine-grained knowledge of the test samples. We further provide more qualitative results on the ImageNet-A, Aircraft, and EuroSAT datasets in Figure 4 of the rebuttal PDF to investigate how BoostAdapter performs on these datasets.\n\nImageNet-A consists of real-world examples that are misclassified by ResNet models, while Aircraft is a benchmark dataset for the fine-grained visual categorization of aircraft, and EuroSAT consists of satellite images for land use and land cover classification.\n\nThese datasets require fine-grained information for classification, and the boosting samples filter out noisy parts of the test samples while retaining useful information, contributing to significant performance enhancements.\n\n**Q3. More backbones.**\n\nIn order to further validate the robustness of our method, we compare BoostAdapter with baseline models across various backbones including both RN and ViT checkpoints. \n\nThe results in Table 2 and Table 3 of the rebuttal PDF indicate that BoostAdapter shows strong compatibility and consistently outperforms TDA in most of the cases over different backbones. For instance, BoostAdapter shows superior performance to TDA over 7, 8, and 8 out of 10 tasks with RN-101, ViT-B/32, and ViT-L/14 checkpoints on the Cross-Domain Benchmark, respectively.  \n\n**Q4. ImageNet-C.**\n\nPlease refer to Q3 in Global Response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a46j5bW6Da", "reviewer": "Reviewer_a236", "summary": "This paper studies the problem of test-time vision-language model adaptation. The authors devise training-free method by maintaining a key-value memory for feature retrieval from both historical and boosting samples. The boosting samples are drawn from regional bootstrapping and capture the knowledge of the test sample it self. Experiments demonstrate the effectiveness of the proposed method.", "questions": "Could the authors provide some computational complexity analyses and comparisons for the proposed method, including wall-clock time and GPU memory consumption?\n\nIt would also be much better to conduct the analysis in Figure 4(c) on more datasets.\n\nIs the proposed method applicable to, or have the authors tested it on, pure CNN/ViT models? If not, I recommend including ‘vision-language model adaptation’ in the title.\n\nThe authors claim that *“prior methods like TDA only consider inter-sample interactions and may fail to generalize well when the downstream tasks require fine-grained knowledge or there is 109 insufficient similarity across samples.”* Although I acknowledge the technological soundness of the proposed method can perform better on more fine-grained classification, are there any empirical evidences to further justify this?\n\nIn the Boosting Cache, do the authors store the original samples or their corresponding features? Storing images may pose privacy and additional computation issues. It would be helpful to indicate this in Figure 3.\n\n[minor] How about the performance of the proposed on corruption datasets such as ImageNet-C?", "limitations": "the potential limitation is the computational efficiency of the proposed method, which can be alleviated if the method works well on multiple datasets with a small number of augmentations", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The studied training-free adaptation setting is practical, broadening the application scope of test-time adaptation in real-world scenarios.\n\nThe authors propose incorporating augmentations of each sample into the cache to enable the cache-based classifier to perform better on more fine-grained classifications. This approach is both interesting and technologically sound.\n\nThe authors also provide theoretical analyses to establish connections between training-required and training-free TTA methods.", "weaknesses": "It would be beneficial for the authors to discuss the detailed technical differences from TDA more thoroughly. From my perspective, the key difference appears to be the introduction of additional augmented views of the same sample into the Boosting Cache for intra-sample iteractions. If this is the case, the technical contribution of this work might be a bit limited.\n\nThe proposed method relies on multiple augmentations, requiring multiple forward passes to achieve better performance than TDA, which sacrifices efficiency. For Figure 4(a), could the authors provide results on more datasets (both OOD domains and cross domains) to demonstrate the sensitivity of the proposed method to this hyper-parameter? This would help verify whether the proposed method can achieve good performance with fewer augmentations across multiple datasets.\n\nThe improvement on Cross-Domain Benchmarks with RN-50 backbone is a bit marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Technical insights and differences with existing baselines.** \n\nPlease refer to Q1 in Global Response.\n\n**Q2, Q5. Computation overhead and efficiency.**\n\nPlease refer to Q2 in Global Response. \n\n**Q3, Q11. Number of augmented views.**\n\nThe analysis of the augmented views can be found in Table 12 of the Technical Appendix. It can be observed that BoostAdapter shows superior performance on average compared to TDA with 32 views on the OOD benchmark and only 16 views on the Cross-Domain Benchmark.\n\nWe would like to point out that TDA also utilizes augmentation to obtain high-quality embeddings of test samples to store in the cache, as depicted in Table 1 of the rebuttal PDF. Furthermore, most of the existing TTA methods use 64 views of augmentation by default. Therefore, using 64 views is not considered particularly large and is acceptable from a computational cost perspective.\n\n**Q4. Cross-Domain Benchmarks results with RN-50 backbone.**\n\nThe marginal improvement is mainly due to the limited performance of the RN-50 backbone. \nTo verify the robustness of BoostAdapter, we further provide results with more checkpoints in Tables 2 and 3 of the rebuttal PDF. BoostAdapter shows promising improvements and consistently outperforms TDA in 7, 8, and 7 out of 10 tasks on the Cross-Domain Benchmarks with RN101, ViT-B/32, and ViT-L/14 backbones, respectively.\n\n**Q6. Shot capacity.**\n\nWe provide ablation results of the shot capacity over all four datasets on the OOD benchmarks in Figure 4 of the rebuttal PDF. The results are consistent with the conclusion in the manuscript that the boosting cache will achieve a balance of diversity and relevance as the shot capacity increases.\n\n**Q7. Applicability on Vision-Only Backbones.**\n\nYes. We believe it is possible to apply BoostAdapter to vision-only TTA, but transferring might be non-trivial since vision-language TTA and vision-only TTA are generally two distinct research sub-fields with different settings. Our paper focuses on the adaptation of the vision-language model, while addressing vision-only models could be considered for future work. \n\nWe appreciate your suggestion on refining the title of this paper and will incorporate it in the revision.\n\n**Q8. Fine-grained knowledge.**\n\nWe provide qualitative results in Figure 5 of the appendix and Figure 4 of the rebuttal PDF to investigate how BoostAdapter leverages boosting samples to extract fine-grained knowledge. \nAs depicted in the figures, boosting samples with low entropy incorporate the prior of label systems to filter out the noisy parts of the test images and guide the model on where to focus. \nMost importantly, we only need to perform random cropping and random horizontal flipping to achieve this, making BoostAdapter more applicable in real-world scenarios.\n\n**Q9. Storage in Boosting Cache.**\n\nIn practice, we store the features of both the historical and boosting samples to construct the key-value cache, which is privacy-preserving and time-efficient. Thanks for your advice, and we will modify the figure for better clarity in the revision.\n\n**Q10. ImageNet-C.**\n\nPlease refer to Q3 in Global Response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tixD6PIPdm", "reviewer": "Reviewer_HZ6e", "summary": "This paper proposes a training-free test-time adaptation approach for vision-language models. It combines the idea of entropy minimization with a training-free adaptor to enhance adaptation performance. The experimental results generally demonstrate the effectiveness of the proposed method.", "questions": "1. What unique insights or techniques does your approach introduce that differentiate it from existing methods?\n2. Why the proposed BoostAdaptor cannot perform as well as clip when the total shot capacity is less than 3? Is this a general phenomenon for all datasets?\n3. Could you explain why the independent cache is good for some datasets as shown in Table 7?", "limitations": "1. While the method is effective, the paper lacks significant novel insights or groundbreaking ideas. The combination of entropy minimization and a training-free adaptor, though practical, may be viewed as an incremental improvement rather than a major innovation.\n2. Refer to the weakness and question part.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper effectively combines entropy minimization for storing sample information and a training-free adaptor for test-time adaptation on VLMs. This approach proves to be effective in most cases.\n2. The figures presented in the paper are clear and easy to follow. The performance figure, in particular, is very intuitive.\n3. The method is theoretically supported, which adds credibility to the proposed approach.\n4. The experiments are well-conducted and cover a broad range of considerations.", "weaknesses": "1. While the proposed method is effective, it lacks significant novel ideas or insights. The contributions could be seen as incremental rather than groundbreaking.\n2. **Figure 2 (b):** The double-arrow directions in Figure 2 (b) should be colored in green and orange for clarity. This part is confusing.\n3. **Figure 3:** The term \"Boosting Cache\" in Figure 3, which appears to store both boosting samples and historical samples, is not explained or used elsewhere in the manuscript, leading to confusion.\n4. Several typographical errors need to be corrected.\n5. The paper misses parameter study such as the threshold $\\tau$.\n6. In Figure 4b, the setup for how the model performs entropy minimization, updates learnable prompts or updates the LN/FULL model is missing. This information is crucial for understanding the ablation study.\n7. The \"Hand-crafted Prompt\" is used in this paper, but it is unclear what it entails. Additionally, it is important to investigate if the method maintains stable performance across different prompts.\n8. I am also wondering if the method could be categorized into TTA, as it maintains low-entropy samples per class in the cache. Even with such extra information for one or two, it cannot surpass the zero-shot clip. (Figure 4.c)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1, Q9, Q12. Technical insights and difference with existing baselines.** \n\nPlease refer to Q1 in Global Response.\n\n**Q2, Q3, Q4. Figure 2 (b), Figure 3 and typo.**\n\nThanks for your advice. We will revise the figures, rewrite the corresponding part, and fix the typo in the revision.\n\n**Q5. Ablation study on threshold $\\tau$.**\n\nWe follow the setting in TPT [1] and empirically set the threshold $\\tau=0.1$. To have a better understanding of the threshold, we provide the ablation results on the Aircraft dataset in Figure 1 of the rebuttal PDF.\n\nThe results are consistent with the conclusion from Figure 4(b) in the original TPT paper, showing that a threshold near $0.1 \\sim 0.2$ contributes to the best results, while higher thresholds lead to noisy samples that may result in misleading predictions. \n\nWith the default setting that utilizes 64 augmented views, we can find that $6$ high-quality boosting samples are sufficient to bring significant improvements to the cache models.\n\n\n**Q6. More details about entropy minimization.**\n\nWhen implementing entropy minimization of the training-required methods, we follow the pipeline of TPT [1] and update the learnable prompts in the input while freezing other weights of the model. Taking entropy as a self-supervised objective, we perform gradient descent over both historical and boosting samples. Prompt tuning is more lightweight than full model tuning but still requires a large computational cost during model optimization.\n\nIt is observed from Figure 4 (b) of the manuscript that either training-required or training-free methods benefit from both historical and boosting samples, in line with the theoretical analysis provided in Propositions 2 and 3. This further highlights our contribution to bridging the gap between training-required and training-free methods.\n\nThanks for your advice and we will modify these parts to provide a clearer description. \n\n\n**Q7. Hand-crafted Prompt.**\n\nWe follow the pipeline of TDA and adopt hand-crafted prompts in the training-free adapters. Take the Action Recognition dataset UCF101 for instance, TDA utilizes the prompt \"a photo of a person doing \\{\\}.\" to better incorporate the prior knowledge of the dataset into the model.\n\nTo study the influence of these hand-crafted prompts, we further equip CLIP, TDA, and BoostAdapter with different prompts (the standard prompt \"a photo of \\{\\}.\" and the hand-crafted prompt \"a photo of a person doing \\{\\}.\") and compare their performance on the UCF101 dataset.\n\nAs can be seen from the results in Figure 3 of the rebuttal PDF, BoostAdapter surpasses TDA and CLIP with both the standard prompt and the hand-crafted prompt. Additionally, the hand-crafted prompt brings improvements in performance to all the methods to some extent.\n\n\n**Q8, Q10. Comparison with zero-shot clip when the shot capacity is less than 3.**\n\nOur methods could be categorized into TTA since it adaptively makes predictions for streaming data during test time based on feature retrieval of historical and boosting samples.\n\nBesides, the reviewer points out that BoostAdapter cannot perform as well as CLIP when the shot capacity is less than 3. However, this is not always the case. As shown in Figure 2 of the rebuttal PDF, BoostAdapter outperforms CLIP across all 4 tasks on the OOD benchmarks. On the Aircraft dataset, the low-entropy samples stored in the boosting cache with a small shot capacity may be biased and not diverse enough, thereby leading to the performance drop.\n\n\n**Q11. Independent cache v.s. Joint cache.**\n\nThe independent cache will retain all the knowledge from both historical and boosting samples, whereas in the joint cache, we update the cache of historical samples with boosting samples. Due to the limited cache size, the historical samples in the joint cache will be replaced by lower-entropy boosting samples when necessary.\n\nIn most cases, BoostAdapter performs better with the joint cache compared to the independent cache. However, in some cases, when the test sample benefits from a sufficient amount of diverse cross-sample interactions, the replacement of boosting samples in the joint cache may lead to a slight performance drop. \n\nGenerally, it is preferred to utilize the joint cache rather than the independent cache due to lower storage cost and better performance on average.\n\n**Reference**\n\n[1] Shu, Manli, et al. \"Test-time prompt tuning for zero-shot generalization in vision-language models.\" Advances in Neural Information Processing Systems 35 (2022): 14274-14289.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable feedback and are encouraged by the positive comments on our contributions, including 

1. Soundness and Novelty:
    - Training-free adaptation broadens real-world applicability (Reviewer a236).
    - Innovative use of sample augmentations in the cache for fine-grained classifications (Reviewer a236 & YJpm).

2. Theoretical Contribution:
    - Establishes connections and bounds between training-required and training-free TTA methods (Reviewer a236 & YJpm).
    - Adds credibility and justification to the approach (Reviewer HZ6e).

3. Solid Experiments:
    - Significant improvements on ImageNet-A, Aircraft, and EuroSAT datasets (Reviewer YJpm).
    - Well-conducted experiments cover a broad range of considerations (Reviewer HZ6e).

4. Presentation:
    - Clear and intuitive figures, especially the performance figure (Reviewer HZ6e).

In the following parts, we will first respond to the common questions raised by reviewers and then respond to the rest of the concerns of each reviewer from point to point. We believe the comments & revisions have made the paper stronger and thank all the reviewers for their help. ***Please let us know if these address your concerns and if there are any further questions that need clarification.***

**Q1. Technical insights and difference with TDA.**

The mainstream of training-required TTA methods is entropy minimization, while TDA serves as a training-free baseline that maintains a key-value cache of historical samples. We argue that our work is far more than merely introducing augmentation views into training-free adapters. **Our main contribution lies in the theoretical and experimental connection between training-required and training-free methods.**

Specifically, we provide a bound for empirical risk as a theoretical guarantee in unifying the two streams. We also show that both methods can benefit from each other, enhancing performance through either entropy minimization or feature retrieval of both historical and boosting samples, as illustrated in Figure 4(b) of the manuscript. 
In practice, we focus on boosting training-free methods rather training-required ones due to their lower computational cost and better generalization performance.

**Additional evidence on this point:** From the unified perspective, we can also enhance training-free adapters with additional training-required methods. Here we take EATA [2], which introduces momentum statistics to combat forgetting in the test-time adaptation, as the showcase. 
When equipping BoostAdapter with the technique of EATA, we observe further improvement and find that training-free adapters can benefit from various boosting techniques of training-required methods. These results can be found in Table 2 of our rebuttal PDF.


**Q2. Computation overhead and efficiency.**

We agree that efficiency is an important metric, and we have already provided the computational efficiency analysis in Table 16 in the Technical Appendix. More information is available in Table 1 of our rebuttal PDF.

We would like to point out that TDA follows TPT [1] and utilizes 64 views with AugMix to obtain high-quality embeddings of test samples, which leads to a similar computation cost with the boosting augmentation in BoostAdapter. 
As can be seen from the results, the inference time for BoostAdapter is slightly longer than TDA but remains significantly faster than training-required methods such as TPT and DiffTPT.
Therefore, considering the performance enhancement it provides, the additional inference cost and comparable memory consumption of BoostAdapter are acceptable.


**Q3. More results on ImageNet-C.**

To further evaluate the generalization ability of BoostAdapter in new test-time scenarios, we compare BoostAdapter with baseline methods on the Imagenet-C dataset at the highest severity level 5. These results are presented in Table 5 of the rebuttal PDF.

The key observation from these results is that BoostAdapter consistently outperforms TDA across all 15 corruption types, highlighting its practical applicability in real-world situations.

BoostAdapter's superior performance stems from its capability to capture the knowledge of the test sample even under severe corruption. This is achieved with the help of the boosting samples, which effectively filter out noisy parts while retaining useful information in the images. 

**Reference**

[1] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models." Advances in Neural Information Processing Systems 35 (2022): 14274-14289.

[2] Niu, Shuaicheng, et al. "Efficient test-time model adaptation without forgetting." International conference on machine learning. PMLR, 2022.

Meta-review: This paper introduces a training-free test-time adaptation method for vision-language models that proposes to include augmented samples that are predicted with low entropy into the memory bank. Theoretical justification is provided to show that these new samples reduce empirical risk. Reviewers thought the method was simple but well-justified theoretically, and that there were comprehensive experiments showing improvements. However, reviewers also had concerns about novelty, marginal performance improvements and increased computational overhead due to the use of multiple augmentations. There was a good discussion between reviewers and authors that resolved concerns regarding computational overhead though concerns about novelty remained. 

The AC's view is that while in practice the differences with the closely related TDA method appear small, the conceptual contribution of linking gradient-based TTA and memory-based TTA methods, along with theoretical justification for the apparently simple addition are useful technical contributions. Thus, on balance, even though the improvements are somewhat small, the AC recommends acceptance.
Decision: Accept (poster)