BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping

Authors: taolin  zhang
jinpeng  wang
hang  guo
tao  dai
bin  chen
shu-tao  xia
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 08:24:22

Abstract: Adaptation of 
pretrained vision-language models such as CLIP to various downstream tasks have raised great interest in recent researches. 
Previous works have proposed a variety of test-time adaptation (TTA) methods to achieve strong generalization without any knowledge of the target domain. 
However, existing training-required TTA approaches like TPT necessitate entropy minimization that involves large computational overhead, while training-free methods like TDA overlook the potential for information mining from the test samples themselves.
In this paper, we break down the design of existing popular training-required and training-free TTA methods and bridge the gap between them within our framework.
Specifically, we maintain a light-weight key-value memory for feature retrieval from  instance-agnostic historical samples and instance-aware boosting samples. 
The historical samples are filtered from the testing data stream and serve to extract useful information from the target distribution, while the boosting samples are drawn from regional bootstrapping and capture the knowledge of the test sample itself.
We theoretically justify the rationality behind our method and empirically verify its effectiveness on both the out-of-distribution and the cross-domain datasets, showcasing its applicability in  real-world situations.

Main paper: 
 introduction: 
 Vision Language models [49, 16, [23] [24] [25] 7] have shown incredible performance in downstream vision tasks [1] , such as classification [29, 55, 54, 8] , generation [20, 38, 9] and recognition [46, 47] . Among these models, CLIP [36] has been trained with large-scale noisy image-text pairs and can generalize well in zero-shot recognition tasks. The key idea behind CLIP is modality alignment during training and similarity comparison during testing for classification. However, CLIP suffers from domain shift problems during test-time inference. In the presence of out-of-distribution issues [27, 43, 12] that commonly appear in real-world scenarios, CLIP may fail to effectively align the feature across modalities, leading to performance degradation.Test-time adaptation (TTA) has been widely explored in recent approaches [43, 15, 41, 17] to mitigate misalignment issues and improve performance in downstream tasks. Current mainstream TTA methods can be divided into training-required methods and training-free methods, as depicted in Figure . 1a and Figure . 1b. Training-required approaches [43, 41, 39] adjust model parameters or learnable prompts based on self-supervised objectives like entropy and increase the prediction confidence of model for distribution adaptation. TPT [41] applies entropy minimization to the vision-language model first. Furthermore, inspired by consistency regularization, TPT performs information mining from the test sample itself by random regional cropping in a self-bootstrapping style. However, training-required methods require gradient descent that is time-consuming with large training overhead, which prevents them from being applied in computationally limited situations. Training-free approaches [15, 52, 17] utilize memory networks, cache, or prototypes to store information regarding target samples and distributions, which is then used to adaptively modify the model's prediction. For example, TDA [17] leverages historical samples from the test data steam to build a dynamic key-value cache. It updates the prior knowledge encoded in CLIP through feature retrieval and output prediction based on the similarity between the test sample and the high-quality data stored in the memory bank. However, existing training-free approaches only consider interaction with other historical samples in the cache and do not effectively exploit the information within the test sample itself. This limitation prevents them from performing well especially in tasks that require fine-grained information.Both of these approaches demonstrate excellent performance in enhancing the robustness of visionlanguage models to unknown distributions. However, the connection between them remains unclear.In this paper, we aim to answer three questions: (1) How are training-required methods like TPT and training-free methods like TDA connected? (2) How can we combine these two methods based on their shared nature? (3) Does vision-language models benefit from the combination of these methods?In order to answer these questions, we first consider that the augmented images of test samples form a regional bootstrapping distribution of the original data. By filtering out the noisy augmentations based on mutual information with the predefined CLIP text embedding clusters, we can obtain a boosting distribution from which high-quality samples close to the target clusters can be drawn. Based on this, we delve into the connection between the target operations over the boosting distribution, i.e., crossentropy optimizations and cache classifier, which reveals the shared nature between entropy-based and cache-based methods. Specifically, we pinpoint that with the samples derived from the bootstrapping distribution, entropy minimization over them performs equivalently to feature retrieval from the cache consisting of them. Motivated by this analysis, we propose a brand-new adaptation strategy, dubbed BoostAdapter, to improve training-free adapters by incorporating the samples derived from the boosting distribution to the memory bank. Particularly, the cache in BoostAdapter consists of instance-agnostic historical samples filtered from the test data stream, along with instance-aware boosting samples generated through regional bootstrapping from the sample itself. The interactions between intra-sample and cross-sample operations make BoostAdapter effective and efficient by Since the feature space is well-clustered, the classifier weights will ultimately converge near the feature center of the samples. Finally, the optimal classifier achieved through cross-entropy minimization will exhibit similar behavior with the cache classifier.we simply assume that samples {(x i , y i )} n i=1 are drawn i.i.d. from the distribution with y i represents the one-hot label. Definition 1. (Classification error.) Given f as a binary classification function. The error incurred by hypothesis f ∈ H : X → {0, 1} under the distribution p t (x, y) can be defined asEQUATIONthe last equality holds in a binary classification setting. Definition 2. (Excess error.) Given the Bayes classifier under distribution p t (x): f * (x) = I{f (x) ≥ 1/2} and the optimal classfier f * , the excess error of f is defined asEQUATIONCLIP classifier Let g be the image encoder of CLIP, C be the feature dimension, N denotes the number of categories, w i ∈ R C represents the i th text embedding cluster. Considering normalized embedding w and g(x), we can derive a simplified version of the output of CLIP for class i:EQUATIONAnd we denote the output logits as p(x) = [Z 1 , Z 2 , ..., Z N ] ∈ R N .Cache classifier Given an unseen sample x, encoder g with dimensional C, cache size K and number of categories N , the cache classfier conduct feature retrieval based on the similarity with the data {(x i , y i )} k i=1 in the cache. The predictions based on Tip-Adapter [52] are as follows:EQUATIONwhere A(z) = αexp(-β(1-z)) denotes a scaling function with a weighting factor α and a smoothing scalar β, G cache ∈ R K×C represents the feature of K samples {x i } K i=1 in the cache and Y ∈ R K×N is the corresponding labels {y i } K i=1 . Considering the number of samples in class y i , We can also derive a simplified version of Eq.( 4) as follows, by ignoring the scaling function and adopting an instance-wise computation style:EQUATIONwhereα i = 1 ny i for class balance or α i = 1 k j=1 [g(xj ) T g(x)] for normalization across all the samples. 
a closer look at entropy-based and cache-based methods: 
 We start with analyzing the filtering operation of augmentated images in TPT. Pseudo-labels tends to be noisy in the test time, and entropy can serve as a confidence metric to identify trustworthy samples among augmented views [43, 41, 33] . These high-quality samples can be considered drawn i.i.d. from the so-called boosting distribution as defined below. Definition 3. (Boosting Distribution.) Given a test sample from target distribution x ∼ p t (x), let H(•) be the entropy measuring function and Aug(•) be the regional augmentation. By filtering noisy samples based on thresthould τ , we have the following property of boosting distribution p b (x):EQUATIONWe also terms the samples from the boosting distribution as boosting samples. Then we can connect entropy-based methods and cache classfier by the following proposition: Proposition 1. (Informal) Given n samples {(x i , y i )} n i=1 with a freeze encoder g that effectively performing feature clustering with respect to labels, the gradient descent optimization direction of the classifier's weights based on cross-entropy generally tends towards making predictions using the cache classifier with class balance weights defined in 5 on these samples.An intuitive illustration of Proposition 1 is depicted in Figure 2 , where the weights of optimal classfier behave like the feature centers across different classes with of the well-clusterd samples. Revisiting the entropy-based method TPT, when provided with high-quality boosting samples with low entropy drawn from the boosting distribution, the objective function of entropy minimization optimizes in a manner similar to conducting cross-entropy optimization over the pseudo-labels. According to Proposition 1, TPT performs similarly to the cache-based methods with a cache comprising the same boosting samples from the boosting distribution.Existing cache-based methods store historical test samples only as useful information for prediction. In light of the analysis above, we can integrate the idea behind TPT into these training-free adapters by incorporating boosting samples into the memory bank. In particular, each sample can participate in both inter-sample and intra-sample interactions with the instance-agnostic historical samples and the instance-aware boosting samples in the cache, respectively. Specifically, with k t selected historical samples and k b selected boosting samples to comprise the cache, we extend the classifier defined in Eq.( 4) and formulate our BoostAdapter as follows:EQUATION)where A is the same scaling function defined in Eq.( 4), Gcache ∈ R (kt+k b )×C denotes the features of the combination of both the historical and boosting samples, and Ỹ ∈ R (kt+k b )×N is the label.Since we do not have access to the labels of the test samples, we generate one-hot pseudo-labels for them using argmax operations. However, these pseudo-labels tend to be noisy in the target domain. Therefore, we apply filtering based on entropy thresholds on the test data stream following [41] to obtain trustworthy historical samples. We employ a similar operation to select boosting samples from multiple augmented views of the current sample. In practice, we dynamically adapt the entropy thresholds τ for each test sample, with a fixed percentile p. The cache continuously updates with lower entropy historical samples from the test data stream, while the current test sample augments the cache with self-boosting samples and forms an independent cache that only affects its own prediction. Additionally, to maintain diversity while considering the relevance to each test sample, we set a maximum shot capacity for each class k in the cache. This means that samples in the cache will be replaced by a lower-entropy historical sample or boosting sample when necessary.An important issue is whether introducing boosting samples brings improvements to the trainingfree adapters. We will first make some necessary assumptions and then theoretically verify the effectiveness in reducing target error by incorporating samples from the boosting distribution.Assumption 1. (Strong Density Condition) For any test sample x 0 in the target distribution x 0 ∼ p t (x) and the boosting distribution p b (x 0 ), given positive lower bound m and upper bound M , positive scaling constant c t and c b , the radius bound R > 0, and B(x, r) = {x ′ :∥ x ′ -x ∥≤ r} is the ball centered on x with radius r. We assume p t (x) and p b (x 0 ) are absolutely continuous with respect to the Lebesgue measure in R d . For r ∈ (0, R], we assumeEQUATION)where λ is the Lebesgue measure in Euclidean space.Assumption 2. (L-Lipschitz Condition) Let f be the classification function and L be a positive constant. For all feasible x, x ′ we have |f(x) -f (x ′ )| ≤ L ∥ x -x ′ ∥.Assumption 3. (Low Noise Condition). Let β, C β be positive constants and we assume p t (x) satisfiesP x∼pt(x) f (x) -1 2 < t ≤ C β t β for all t > 0.Remark Assumption 1 intuitively ensures that for any test sample, there is a surrounding neighborhood with a significant presence of samples from the target domain and the boosting distribution. More importantly, for a specific sample x 0 , boosting samples x ∼ p b (x 0 ) should be closer to x 0 than other samples x ∼ p t (x) from the target domain, i.e., generally, we have c t ≤ c b . Assumption 2 and 3 describe the smoothness of functions and imply a high level of confidence in predictions around the threshold, respectively.Proposition 2. (Historical Cache reduce Emperical Risk) Given f as the training-free classfier consisting of historical samples only defined by Eq.( 4). Let n t to be the number of confident previously predicted samples in the target domain and k t as the number of historical samples in the cache, with assumptions 1-3, the following results hold with high-probability for large enough k t and n t .EQUATIONProposition 3. (Historical Cache benefits from Boosting Samples) Let n t to be all confident previously predicted samples in the target domain and n b be the number of boosting samples that are drawn from the boosting distribution. Given k t and k b to be the number of historical samples and the number of boosting samples to be selected as the nearest neighbors stored in the cache, respectively. Let w ti and w bi be the weights defined in Eq.( 5) of the historical samples and boosting samples. We have the following bound for the empirical risk of the cache classfier defined in 7. Remark Proposition 2 provides a guarantee of the effectiveness of selecting k t out of n t historical samples to comprise the cache. The empirical risk is quite small when n t → ∞ since the cache captures the full information of the target domain. Proposition 3 demonstrates that the historical cache can further reduce empirical risk by incorporating k b boosting samples.EQUATION)Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: The theoretical proof of the proposition used in the paper can be found in Section C. Guidelines:• The answer NA means that the paper does not include theoretical results. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experimental setup: 
 Datasets Following the setting in TPT [41] , we conduct experiments on both Out-of-Distribution (OOD) benchmark and Cross-Domain benchmark. The OOD benchmark evaluates the model's robustness to natural distribution shifts on 4 ImageNet [4] Variants, including ImageNetV2 [37] , ImageNet-Sketch [44] , ImageNet-A [14] and ImageNet-R [13] . We evaluate the transferring performance on 11 datasets in the Cross-Domain benchmark: Aircraft [31] , Caltech101 [5] , Cars [19] , DTD [3] , EuroSAT [11] , Flower102 [32] , Food101 [2] , Pets [34] , SUN397 [48] ,and UCF101 [42] .We follow the split in [55] and report the top-1 accuracy. The error bound are also provided.We utilize a pre-trained ViT-B/16 of CLIP as the foundation model. In test-time adaptation, the batch size is set to be 1. We search for the optimal shot capacity to balance diversity and relevance of samples. For boosting samples, we utilize random crop and then random horizontal flip as augmentations. Moreover, we empirically set the entropy threshold percentile to p = 0.1 and filter 64 augmented views based on random cropping to obtain the boosting samples. and filter 64 augmented views to obtain the boosting samples. The top-1 accuracy and the error bound is reported on the test sets. All our experiments are conducted with a Nvidia 3090 24GB GPU. To verify the robustness of BoostAdapter, we evaluate our method on the OOD benchmark, in comparison with existing training-require methods including CoOp [55] , CoCoOp [54] , TPT [41] , DiffTPT [6] , Maple [18] and PromptAlign [39] , as well as training-free method TDA [17] . As can be seen from Table 8 , the most striking observation emerging from the comparison is that BoostAdapter significantly outperforms other baselines on average and improves the generalization ability of the model. For training-free methods such as TPT, DiffTPT and PromptAlign, BoostAdapter achieves superior performance while saving on optimization computation overhead. For training-free methods like TDA, BoostAdapter gains consistent performance improvements with the introduction of the boosting samples. Notably, BoostAdapter surpasses TDA by 4.42% on ImageNet-A and 0.84% on ImageNet-V2, respectively. This enhancement indicates the effectiveness of self-bootstrapping when historical samples may not provide sufficient useful information.We further highlight our improvements in the transfer ability of CLIP on the Cross-Domain benchmark and present the results in Table 2 . Compared with existing training-required and training-free methods, BoostAdapter achieves state-of-the-art performance on 7 out of 10 tasks, surpassing the strongest baselines by an average of 1.15%. With diverse classes at test time, regional boosting enables BoostAdapter to adaptively extract knowledge that makes classes distinct from each other in a multi-scale manner. Notably, for datasets requiring fine-grained information for classification such as Aircraft, the improvement of BoostAdapter is most significant.Historical Samples and Boosting Samples. To demonstrate the effect of historical and boosting samples, we introduce two variants of BoostAdapter that utilize only historical samples or only boosting samples, respectively. Additionally, we provide the zero-shot results of CLIP for comparison. As shown in Table 3 , CLIP significantly benefits from both historical samples and boosting samples, resulting in notable improvements in performance. The consistent improvement of BoostAdapter compared to the variant that utilizes only historical samples further confirms the effectiveness of Number of Augmented Views for Boosting Samples. We augment the testing samples and filter them by mutual information with the CLIP text embedding to obtain the boosting samples. We vary the number of augmented views and investigate the performance of BoostAdapter on UCF101 in Figure 4a . With a larger number of augmented views, the performance improves due to more bootstrapping information of the test sample, which is consistent with the conclusions of TPT [41] and PromptAlign [39] . However, the computational overhead also increases with more augmented views, and selecting 64 augmented views is a fair trade-off between boosting performance and efficiency.Adaptation Methods. Training-required methods use entropy as a self-supervised objective, whereas training-free methods classify samples based on feature retrieval. We compare the performance of these two adaptation methods under the constraints of historical samples only, boosting samples only, or both, and present the results on Flower102 in Fig. 4b . Entropy minimization requires gradient descent and model optimization, resulting in high training costs and relatively lower performance across all three settings. In contrast, the training-free methods based on feature retrieval offer significant performance improvements with lower computational overhead. Additionally, both adaptation methods benefit from combining historical samples and boosting samples, consistent with the conclusions in Table 3 .Total shot capacity. BoostAdapter maintains low-entropy samples per class in the cache, and Figure 4c studies the influence of different total shot capacities containing historical samples and boosting samples of each class on Aircraft. As can be observed from the results, when the cache capacity is small, the low-entropy samples maintained by BoostAdapter do not necessarily provide a benefit for classification compared to CLIP. As the shot capacity increases, BoostAdapter will achieve the best balance of diversity and relevance, and a larger capacity does not guarantee better performance.Versatility. To demonstrate the versatility of BoostAdapter, we apply it to the RN-50 backbone and present the results in Tables 4 and 5 . The improvement is consistent and on average, BoostAdapter outperforms TDA by 1.57% on the OOD benchmark and 0.49% on the Cross-Domain benchmark. consistently outperforms TDA across all 15 corruption types, highlighting its practical applicability in real-world situations. The superior performance of BoostAdapter stems from its capability to capture the knowledge of the test sample even under severe corruption. This is achieved with the help of the boosting samples, which effectively filter out noisy parts while retaining useful information.Efficiency Analysis BoostAdapter requires augmentation over the test samples, which may slightly affect the inference speed during testing. We conduct an efficiency analysis of BoostAdapter in comparison with existing Test Time Augmentation (TTA) methods and provide the results in Table 7 .BoostAdapter is slightly slower than the cache-based method TDA, yet still significantly faster than training-required methods. The memory cost of BoostAdapter is also comparable to other baselines.Training-free Methods. From the unified perspective, we can also enhance training-free adapters with additional training-required methods. Here we take TSD [45] and DEYO [21] as the showcase. Specifically, in the BoostAdapter+DEYO variant, we filter out augmented views with a PLPD lower than 0.2. For the BoostAdapter TSD variant, we discard augmented views that have different cache predictions and CLIP predictions to ensure consistency of the boosting samples. When equipping BoostAdapter with the technique of TSD and DEYO, we observe further improvement and find that training-free adapters can benefit from various boosting techniques of training-required methods.The qualitative results are provided in Figure . 5. By incorporating samples with low entropy from regional bootstrapping, the model is enhanced to more effectively capture the fine-grained information of the test samples, thereby improving the overall performance. 
conclusions: 
 In this work, we present an insightful analysis of existing training-required and training-free TTA methods to bridge the gap between them. In particular, we improve training-free adapters by incorporating self-boosting samples into the memory bank inspired by the idea of regional bootstrapping from entropy-based methods. The cache in our method, containing instance-agnostic historical samples and instance-aware boosting samples, is capable of performing knowledge mining on both the target domain and the testing sample itself. We also derive error bounds in the test-time adaptation setting and show that this cache benefits from both historical samples and boosting samples. Extensive experiments on the two benchmarks demonstrate the effectiveness of our method.Despite the promising performance of our method, it also has some limitations. It requires slightly more computation overhead than existing training-free adapters due to the multiple augmentation of the test samples, as discussed in Appendix. One future direction is to develop a more efficient augmentation method to obtain boosting samples, rather than merely randomly cropping and then filtering over the test samples. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification:We provide the error bound along with the main results. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Table 9 presents the statistics and details of datasets used in the paper. We also provide the corresponding license information of the datasets and source code.Datasets. Below are the datasets used in this paper that have known license information: The following datasets used in this paper are under the MIT License: ImageNet-A [14] , ImageNet-V2 [37] , ImageNet-R [13] , ImageNet-Sketch [44] , EuroSAT [11] , Food101 [2] .The following datasets used in this paper are under the CC BY-SA 4.0 License: Oxford-Pets [34] ,Caltech101 [5] .The following datasets used in this paper are for research purposes only: DTD [3] , StanfordCars [19] , SUN397 [48] , FGVC-Aircraft [31] , Flower102 [34] , UCF101 [42] .Source code. We use the implementation of existing baseline methods for reporting their results in this paper. Below are their license information: Source code used in this paper that are under the MIT License: CLIP [36] , PromptAlign [39] and TDA [17] . DatasetIn this paper, we focus on bridging the gap between training-required and training-free methods to improve the generalization ability of vision-language models. We also theoretically derive the error bound of incorporating boosting samples into the historical cache. We hope that our work will inspire the community to explore test-time adaptation in an effective and efficient way.C.1 Cross-entropy Optimization behaves like Cache Classifier over well-clustered Samples (Proof of Proposition 1)Given well-clustered samples in the feature space and the classifier defined in Eq.( 3), we first derive the distance between the weights of the classifier and the optimal weights and then establish the connection between the optimal weights with the features center of the samples.Suppose the classifier function f over samples is convex and differentiable, and also L-smooth. Let the distance between initial weight w (0) and optimal weight w * to be D = ||w (0) -w * ||. GD updates by w (t+1) = w t -f * t ∇f (w t ) with step size f * t = 1 L , and then GD enjoys the following convergence guarantee:EQUATION)We then showcase the relationship between w * and the features center µ i of class i, i = 1, 2, ..., N . Since we optimize on well-clustered samples, we consider the scenarios of perfect clusters, where samples in the class i will be encoded into the same point µ i by the encoder g, and these points should be farthest enough between each other. Given n samples {(x k , y k )} n k=1 , with the number of samples in class i to be n i , the cross-entropy loss function L can be written as:EQUATIONSubstitute the sample g(x k ) = µ i from class i, we derive the probability P (y = i|x k ) using the softmax function from Eq.( 3) is:EQUATION)Thus, the cross-entropy loss for a sample (x k , y k = i) is:EQUATIONFor all samples, the total loss is:EQUATION)The gradient of the loss with respect to w i can be simplified as:EQUATION)When converges to the optimal weight, we have the condition of fixed point ∂L ∂w * i = 0. And we haveEQUATIONThus, we haveEQUATION)Given a well-clustered samples, we could have exp((w * i ) T µ k ) ≫ exp((w * j ) T µ k ) for a specific i when w * i is near µ k . Then since the equality in Eq.( 18) will hold for each class and for class i = 1, 2, ..., N we haveEQUATIONCombining Eq.( 11) and Eq.( 18), with iteration steps T , we show that the weight of classfier will finally converge to the feature center of each class:EQUATIONAnd we have the output logits of the optimal weights with the encoder g:EQUATIONNext we discess the behavior of the cache classifier over these samples. Given the number of wellclustered samples in class i to be n i , the output logits of the cache classifier defined in Eq.( 5) using samples {(x k , y k )} n k=1 can be described as follows:EQUATIONCombining Eq.( 21) and Eq.( 22), we draw the conclusion that cross-entropy optimization behaves like cache classifier over well-clustered samples.We follow the proofs in [53] and extend the conclusion to boosting samples.Definition 4. (Wasserstein-distance and the dual form). Wasserstein distance measures the distance between two probability distributions on a given metric space. It is defined using the concept of optimal transport. For two distributions P, Q, The ρ-th Wasserstein distance is defined asEQUATIONHere, Π(P, Q) denotes the set of all couplings (or transport plans) γ of P and Q, i.e., joint distributions on X × X with marginals P and Q.The idea is to find the optimal way to transport the mass from one distribution to the other with the minimal cost, where the cost is given by the p-th power of the distance.The first Wasserstein distance, W 1 (P, Q) ,often referred to as the Earth-Mover Distance(EMD), has a particularly elegant dual representation. The dual form of W 1 leverages the Kantorovich-Rubinstein duality and can be expressed as:W 1 (P, Q) = sup ∥f ∥Lip≤1 X f dP - X f dQ (24)Here, the supremum is taken over all 1-Lipschitz functions f ,which are functions satisfying |f (x)f (y)| ≤ d(x, y) for all x, y ∈ X .This representation shows that W 1 can be seen as the maximum difference in expected values of a 1-Lipschitz function over the two distributions. In the following part, Wasserstein distance represents the first Wasserstein distance for simplicity and we utilizeW (•, •) instead of W 1 (•, •).Given the definition of the Wasserstein distance, we have the following proposition that derive the empirical risk on the target domain according to Theorem 1 from [40] .Proposition 4. Given two distributions P, Q, denote f * = arg min f ∈H (ϵ P (f ) + ϵ Q (f )) and ξ = ϵ P (f * ) + ϵ Q (f * ). Assume all hypotheses h are L-Lipschitz continuous, the risk of hypothesis f is then bounded byEQUATIONWhen using the cache classifier with historical samples, a large number of samples that are not similar enough from the target domain will be filtered and the selected samples with high weight are all close to the target data. Thus we extend the conclusion in [53] to the distance between the ball distribution with the target distribution. Considering a test sample from the target distribution x t ∈ p t (x) and a distribution consisting of ball center of all the test samples Ω := xt∈pt(x) B(x t , r), informally, according to Eq.( 23), we have the distance between the ball distribution with the target distribution as follows:EQUATIONwhere for each x ball ∈ Ω, we can find at least one x t ∈ p t (x) such that ∥ x ball -x t ∥≤ r, the overall distance will then be bounded by r. Specifically, we can choose a density function γ * where γ * (x ball , x t ) > 0 only if ∥ x ball -x t ∥≤ r otherwise 0, then we haveEQUATION)However, there is no guarantee that each data x t ∈ p t (x) can find a neighbor B(x t , r) with |B(x t , r)| > 0 with all the small r. We then provide the probability that the set of neighbors B(x t , r) of each x t ∈ p t (x) is not measuring zero with respect to the radius r.As defined in the cache classfier Eq.( 5), we denote k t is the number of historical samples we select in the cache and n t is the total number of data from the historical stream. With the strong density assumption, given the coefficient bound m and M , for any x t ∈ p t (x), r < R, according to Assumption 1, we haveEQUATIONwhere π d = λ(B(0, 1)) is the volume of the d dimension unit ball and λ is the Lebesgue measure of a set in a Euclidean space. Set r 0 = ( 2k mctπ d nt ) 1/d , with a additional assumption that we utilize a small k t compared to n t so that kt nt <ctmπ d r d µ 2, we have r 0 < R. Then for any x t ∈ p t (x), according to Eq.( 28), we haveEQUATION)Since xt ∈ p t (x) are independently drawn from the target distribution, let I(•) to be the Indicator funciton andS nt (x t ) = nt i=1 I( xt ∈ B(x t , r 0 )) denote the number of data xt ∈ p t (x) that fall into B(x t , r 0 ), then S nt (x t ) follows the Binomial distribution. Let W ∼ Binomial(n t , 2knt ), according to the Chernoff inequality, we haveEQUATIONwhere the second inequality holds since S n (x) has a larger mean than W . With a large k t , the probability that S n (x) < k t is small for any x t ∈ p t (x). Denoting xt (i) as the i th nearest sample to x t among B(x t , r 0 ) in the cache, we have for any x t ∈ p t (x)EQUATION) Combine Eq.( 31) with the assumption that the distribution p t (x) is finite with cardinality ℵ pt and the desired probability part is shown by union bound.EQUATION)And then we have the following proposition. Proposition 5. Given the target domain distributions p t (x) that is finite with cardinality ℵ pt , and Ω := x∈pt(x) B(x, r), where B(x, r) = {x ′ :∥ x ′ -x ∥≤ r} denotes a ball centered on x with radius r. Denote f * = arg min f ∈H (ϵ t (f )+ϵ Ω (f )) and ξ = ϵ t (f * )+ϵ Ω (f * ). Assume all hypotheses h are L-Lipschitz continuous, the risk of hypothesis f on the unseen target domain is then bounded byEQUATION)with probability 1 -exp(-kt 4 + log ℵ pt )Let s i to be the softmax probability sof tmax(p cache ) for class i in the the cache classifier from Eq.( 5), we can simplify the classifier as fcache = I{s 1 ≥ 1 2 } on the binary classification setting. Then fcache (x t ) ̸ = f * (x t ) implies that fcache (x t ) -f * (x t ) ≥ f * (x t ) -1 2 . We then bridge the gap between the excess error and the classify error as follows:EQUATIONWe want to bound sup xt fcache (x t ) -f * (x t ) ≤ t, combining with the marginal assumption in Assumption 3 and the fact thatEQUATIONwhereEQUATION) as the i th nearest data and the corresponding labels to x t in B(x t , r 0 ). The result of the cache classfier with normalized weight will beEQUATIONEQUATIONwherew i = g( xt (i) ) T g(x) k t j=1 g( xt (j) ) T g(x)is the normalized weight and kt i=1 w i = 1. Based on the assumptions and notions above, we have for anyEQUATIONwhere 2 is easy to bound. According to the assumption that f * is C-Smoothness, we haveEQUATIONAccording to Eq.( 31), with probability at least1 -exp(-k t /4), 2 ≤ C 2kt mctπ d nt 1/d. Note that We store the target sample into the cache only when its prediction confidence is large enough. Therefore, it is natural to assume that:EQUATIONThen we use the Hoeffding inequality to obtain the upper bound of 1EQUATIONWe simplify the bound by assuming that the weights in the target domain are evenly distributed in the subset of all samples with respect to a specific class controlled by coefficient η, according to Assumption 1 and Proposition 4. That is, we havekt i=1 w 2 i ≈ ηkt i=1 ( 1 ηkt ) 2 = 1 ηkt . Set ϵ = (1/k t ) 1/4, we have, with probability, at least 1 -3 exp(-2η√ k t ), 1 ≤ (1/k t ) 1/4 , 2 ≤ C 2kt mctπ d nt 1/d, and then fcache (x t ) -f * (x t ) ≤ (1/k t ) 1/4 + C 2kt mctπ d nt 1/d. According to Eq.( 31) and Eq.( 35), the excess error is bounded byEQUATIONwith constant C 1 . When appropriately choosing k t = O(log n t ), we haveEQUATIONwhere the third line is because k t /4 > 2η √ k t for large enough k t . Namely, with probability at least 1 -3 exp(-√ log n t ) O (1) , the following bound holds true.EQUATIONC.3 Historical Cache benefits from Boosting Samples (Proof of Proposition 3)To study the effect of the boosting samples, we consider the cache classfier containing bothk t historical samples { xt (i) , ŷt (i) } kt i=1 and k b boosting samples { xb (i) , ŷb (i) } k b i=1 as the nearest data to x t in B(x t , r 0 ). With the normalized weights w ti = g( xt (i) ) T g(x) k t j=1 g( xt (j) ) T g(x) + k b j=1 g( xb (j) ) T g(x)andw bi = g( xb (i) ) T g(x) k t j=1 g( xt (j) ) T g(x) + k b j=1 g( xb (j) ) T g(x), the prediction result of the cache classifier will befcache (x t ) = kt i=1 w ti ŷt (i) + k b i=1 w bi y (i)b . Then we have:fcache (x t ) -f * (x t ) = kt i=1 w ti ŷt (i) - kt i=1 w ti f * (x t ) + k b i=1 w bi y (i) u - k b i=1 w bi f * (x t ) ≤ kt i=1 w ti ŷt (i) - kt i=1 w ti f * ( xt (i) ) + kt i=1 w ti f * ( xt (i) ) - kt i=1 w ti f * (x t ) + k b i=1 w bi y (i) u - k b i=1 w bi f * x (i) u + k b i=1 w bi f * x (i) u - k b i=1 w bi f * (x t ) ≤ kt i=1 w ti ŷt (i) + k b i=1 w bi y (i) u - kt i=1 w ti f * ( xt (i) ) - k b i=1 w bi f * x (i) u 1 + kt i=1 w ti f * ( xt (i) ) -f * (x t ) 2 + k b i=1 w bi f * x (i) u -f * (x t )3 Similar to Eq.( 40), we have the following assumption on the boosting distribution:EQUATIONAccording to Eq.( 41), we haveEQUATIONSet ϵ = (1/(k t + k b )) 1/4 , we have, with probability, at least 1 -3 exp(-2η(k t + k b )), 1 ≤ (1/(k t + k b )) 1/4. Then, according to Eq.( 39), we haveEQUATIONandEQUATION)whereS t = kt i=1 w ti , S b = k bi=1 w bi are the sum of weights of historical samples and boosting samples, respectively, and we have S t + S b = 1.Then we have the following results in similar:2 ≤ S t C 2k t mc t π d n t 1/d ; 3 ≤ S b C 2k b mc b π d n b 1/d (49)Finally, the excess error under the covariate shift setting can be bounded byEQUATIONCompared Eq.( 50) to Eq.( 42) and S t + S b = 1, it is easy to verify that(S t + S b )C 2(k t + k b ) mc t π d n t 1/d -S t C 2k t mc t π d n t 1/d -S b C 2k b mc b π d n b 1/d ≥ S b C 2k t mc t π d n t 1/d -S b C 2k b mc b π d n b 1/d (51)In general, the boosting distribution is more close to the test sample than the target distribution and we have c b > c t . Thus the difference in Eq.( 51) is then larger than 0, namely incorporating boosting samples into the memory bank, the excess error can be further reduced. Historical Samples and Boosting Samples. We provide more ablation results of the historical and boosting samples on the Cross-Dataset benchmark in Table 14 . The observation is consistent with the results in Table 3 , showing that CLIP gains improvements from both historical and boosting samples. Furthermore, when applied to various downstream tasks, the importance of regional bootstrapping becomes more significant, as indicated by the gap between BoostAdapter and the variant that uses boosting samples only.Views for Boosting Samples. The complete results on the number of augmented views are presented in Table 15 and Table 16 . With more augmented views, BoostAdapter is able to better extract the fine-grained information from the original test sample, achieving improved performance. Fixed shot capacity. We search the optimal total shot capicity in BoostAdapter. We also find that fixing the cache size to be 3 can generalize well in different task settings, as shown in Table 17 and Table 18 .More qualitative results are provided in Fig. 6 . Justification: The limitation and the discussion about computational overhead can be found in the Section 5. These assumptions are reasonable in domain adaptation and parameter analysis is conduct in the ablation studies in Section 4.4. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: This paper introduces a training-free test-time adaptation method for vision-language models that proposes to include augmented samples that are predicted with low entropy into the memory bank. Theoretical justification is provided to show that these new samples reduce empirical risk. Reviewers thought the method was simple but well-justified theoretically, and that there were comprehensive experiments showing improvements. However, reviewers also had concerns about novelty, marginal performance improvements and increased computational overhead due to the use of multiple augmentations. There was a good discussion between reviewers and authors that resolved concerns regarding computational overhead though concerns about novelty remained. 

The AC's view is that while in practice the differences with the closely related TDA method appear small, the conceptual contribution of linking gradient-based TTA and memory-based TTA methods, along with theoretical justification for the apparently simple addition are useful technical contributions. Thus, on balance, even though the improvements are somewhat small, the AC recommends acceptance.

Reviews: review: paper_summary: paper_summary: The paper focuses on gradient-free test time adaptation of CLIP model with ViT-B/16 and ResNet-50 backbones on out-of-distribution datasets. The authors take inspiration of augmentations from gradient-based test time methods and incorporate this concept of augmentations in gradient-free and memory (cache) based test time methods. Previous work considers only historic samples to be in cache, whereas this work also includes the augmentations of test samples with lower entropy into the cache. Results on OOD benchmark and cross-domain benchmark show improved average performance compared to prior works.
strength_weakness: strength_weakness: I don’t have major concerns on the proposed approach, as the method is simple and straightforward. My concern lies in increase of computation overhead and extended run time due to running CLIP model on multiple augmentations during test time (as acknowledged by the authors). In addition, it can be noticed that results are comparable on almost all datasets except ImageNet-A, Aircraft and EuroSAT datasets. It would be interesting to provide rationale for the proposed approach to work much better on these datasets. Results are shown on single ViT architecture, however providing results on multiple CLIP based ViT backbones would be interesting as the method targets for test time performance. It would be helpful to evaluate the method on more OOD distributions like corruptions (ImageNet-C) to understand it better.
questions: questions: Please refer to Weakness above.
limitations: limitations: Limitations are discussed in the work.
paper_summary: paper_summary: This paper studies the problem of test-time vision-language model adaptation. The authors devise training-free method by maintaining a key-value memory for feature retrieval from both historical and boosting samples. The boosting samples are drawn from regional bootstrapping and capture the knowledge of the test sample it self. Experiments demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: It would be beneficial for the authors to discuss the detailed technical differences from TDA more thoroughly. From my perspective, the key difference appears to be the introduction of additional augmented views of the same sample into the Boosting Cache for intra-sample iteractions. If this is the case, the technical contribution of this work might be a bit limited.

The proposed method relies on multiple augmentations, requiring multiple forward passes to achieve better performance than TDA, which sacrifices efficiency. For Figure 4(a), could the authors provide results on more datasets (both OOD domains and cross domains) to demonstrate the sensitivity of the proposed method to this hyper-parameter? This would help verify whether the proposed method can achieve good performance with fewer augmentations across multiple datasets.

The improvement on Cross-Domain Benchmarks with RN-50 backbone is a bit marginal.
questions: questions: Could the authors provide some computational complexity analyses and comparisons for the proposed method, including wall-clock time and GPU memory consumption?

It would also be much better to conduct the analysis in Figure 4(c) on more datasets.

Is the proposed method applicable to, or have the authors tested it on, pure CNN/ViT models? If not, I recommend including ‘vision-language model adaptation’ in the title.

The authors claim that *“prior methods like TDA only consider inter-sample interactions and may fail to generalize well when the downstream tasks require fine-grained knowledge or there is 109 insufficient similarity across samples.”* Although I acknowledge the technological soundness of the proposed method can perform better on more fine-grained classification, are there any empirical evidences to further justify this?

In the Boosting Cache, do the authors store the original samples or their corresponding features? Storing images may pose privacy and additional computation issues. It would be helpful to indicate this in Figure 3.

[minor] How about the performance of the proposed on corruption datasets such as ImageNet-C?
limitations: limitations: the potential limitation is the computational efficiency of the proposed method, which can be alleviated if the method works well on multiple datasets with a small number of augmentations
paper_summary: paper_summary: This paper proposes a training-free test-time adaptation approach for vision-language models. It combines the idea of entropy minimization with a training-free adaptor to enhance adaptation performance. The experimental results generally demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. While the proposed method is effective, it lacks significant novel ideas or insights. The contributions could be seen as incremental rather than groundbreaking.
2. **Figure 2 (b):** The double-arrow directions in Figure 2 (b) should be colored in green and orange for clarity. This part is confusing.
3. **Figure 3:** The term "Boosting Cache" in Figure 3, which appears to store both boosting samples and historical samples, is not explained or used elsewhere in the manuscript, leading to confusion.
4. Several typographical errors need to be corrected.
5. The paper misses parameter study such as the threshold $\tau$.
6. In Figure 4b, the setup for how the model performs entropy minimization, updates learnable prompts or updates the LN/FULL model is missing. This information is crucial for understanding the ablation study.
7. The "Hand-crafted Prompt" is used in this paper, but it is unclear what it entails. Additionally, it is important to investigate if the method maintains stable performance across different prompts.
8. I am also wondering if the method could be categorized into TTA, as it maintains low-entropy samples per class in the cache. Even with such extra information for one or two, it cannot surpass the zero-shot clip. (Figure 4.c)
questions: questions: 1. What unique insights or techniques does your approach introduce that differentiate it from existing methods?
2. Why the proposed BoostAdaptor cannot perform as well as clip when the total shot capacity is less than 3? Is this a general phenomenon for all datasets?
3. Could you explain why the independent cache is good for some datasets as shown in Table 7?
limitations: limitations: 1. While the method is effective, the paper lacks significant novel insights or groundbreaking ideas. The combination of entropy minimization and a training-free adaptor, though practical, may be viewed as an incremental improvement rather than a major innovation.
2. Refer to the weakness and question part.

Comments: title: Follow up from reviewer
Please engage in discussion
comment: Dear authors,

I thank you for the responses, and additional experiments provided in the rebuttal. I have read fellow reviewers comments and authors responses. I find that the method is slightly better than TDA consistently across all benchmarks, particularly helpful for ImageNet-A. However, my major concern on limited novelty still exists and hence I tend to keep a borderline rating. I will discuss with my fellow reviewers for a final decision.
Sorry for the confusion about the memory usage of CLIP.  It is reported by mistake when we compare the memory consumption of BoostAdapter with TDA. Actually, the memory consumption of CLIP is just 0.7 GB, which is lower than the 1.2 GB of BoostAdapter and TDA. We hope that your questions are addressed.
Thanks for the clarifications. It gets clearer now.

> "We only perform parallel forward propagation for different views of the same test sample."

In this implementation, a further question concerning memory usage is: why do BoostAdapter and CLIP have the same reported memory consumption? Shouldn’t the parallel forward passes of 64 views in BoostAdapter require more memory than a single view forward propagation in CLIP?
**Q1. More details**

We follow the setting in TDA and deal with the test samples from the data stream one by one. Therefore, we cannot increase the batch size to handle multiple different test samples simultaneously, but only for different views of the same test sample. Thus, we augment 64 views of the test image as TDA does and use 64 as the batch size to obtain the corresponding features.

In BoostAdapter, we utilize a simpler augmentation (random crop and random horizontal flip) than AugMix in TDA to save time. Additionally, we set num_workers=8 in the dataloader to leverage multiprocessing for acceleration. Furthermore, we perform feature retrieval over the stored features instead of images. The additional retrieval time compared to TDA comes from the operation of updating the cache with boosted samples.

All our experiments are conducted with a 64-core Nvidia 3090 24GB GPU.



**Q2. FPS and memory.** 

For a better view of the time consumption, we provide the average wall-clock time consumption of each component for 1000 samples in TableA. The total time can be mainly divided into three parts: data augmentation, model forwarding, and feature retrieval.

- Regarding augmentation time, note that we set num_workers = 8 for the dataloader, so the augmentation takes up a small percentage of the total time. TDA and BoostAdapter take a similar amount of time since they both utilize 64 views of augmentations.
- The model forwarding of BoostAdapter utilizes approximately 8 times more time than CLIP. This is reasonable since we use parallel forward propagation for the 64 views instead of sequential forward propagation, so the difference in comsumption overhead will not be as large as 64 times. The only difference is the batch size of the model input (1 for CLIP and 64 for BoostAdapter).

- We build a new cache from the historical cache and update it with boosting samples so the feature retrieval time of BoostAdapter takes slightly longer than TDA. Overall, the time consumption in feature retrieval is much smaller than model forwarding. 

**TableA. Computation cost of each component.**

|              | Data Augmentation      | Model Forwarding        | Feature Retrieval      | Total                  |
| ------------ | ---------------------- | ----------------------- | ---------------------- | ---------------------- |
| CLIP         | -                      | 0.01208 seconds (100%)  | -                      | 0.01208 seconds (100%) |
| TDA          | 0.00116 seconds (1.4%) | 0.08065 seconds (96.6%) | 0.00167 seconds (2.0%) | 0.08348 seconds (100%) |
| BoostAdapter | 0.00098 seconds (1.1%) | 0.07989 seconds (91.5%) | 0.00644 seconds (7.4%) | 0.08731 seconds (100%) |

The memory consumption of BoostAdapter is similar to TDA because both utilize 64 views of augmentation, and the model takes 64 views of the image (batch size 64) as input. This model forwarding part accounts for most of the memory usage. During feature retrieval, we store the features in the cache without significant memory overhead, even with the integration of boosting samples' features.

In the online test-time adaptation setting, we deal with test samples from the data stream one by one, so we cannot increase the batch size by parallel modeling of different test samples. We only perform parallel forward propagation for different views of the same test sample. Therefore, changing the batch size here is not appropriate, as it corresponds to the number of augmented views for the current test sample. 



**Q3. Efficiency analysis table.**

Thanks for your insightful advice, and we add the performance into the efficiency analysis in the TableB.

  **TableB. Efficiency analysis with performance results.**

|              | Augmentation                   | Views | Inference Speed (fps) | Memory (GB) | OOD Benchmarks Results | Cross-Domain Benchmarks Results |
| ------------ | ------------------------------ | ----- | --------------------- | ----------- | ---------------------- | ------------------------------- |
| CLIP         | -                              | -     | 82.3                  | 1.2         | 57.20                  | 63.58                           |
| TPT          | Augmix                         | 64    | 0.29                  | 4.5         | 60.81                  | 65.10                           |
| DiffTPT      | Diffusion                      | 64    | 0.10                  | 14.4        | 60.52                  | 66.92                           |
| TDA          | Augmix                         | 64    | 11.89                 | 1.2         | 63.89                  | 67.53                           |
| BoostAdapter | Rand. Crop & Rand. Horiz. Flip | 64    | 11.23                 | 1.2         | 65.15                  | 68.52                           |
Thanks for the authors’ response. Regarding the computational efficiency, could the authors provide more details of the experimental setup, including but not limited to the GPU, and batch size?

For FPS, I am confused about why BoostAdapter (64 forward passes of image encoder) achieves 11.23 fps and CLIP only achieves 82.3 fps which only needs 1 forward propagation. Meanwhile, the memory consumption of BoostAdapter and CLIP is the same. Do you test Inference Speed (fps) using batch size 64 (64 views also equal to batch size 64), and test Memory using a different batch size? 

Moreover, the performance could also be directly reported in Table 1 of the PDF.
Thanks a lot for your efforts and discussion!
Dear reviewer, 

We would like to thank you for your insightful feedback. We hope that your concerns are addressed with our rebuttal. As we are getting really close to the deadline of the discussion phase, please let us know if there are any further questions that need clarification.

Many thanks,

Authors
Dear reviewer, 

We would like to thank you for your insightful feedback. We hope that your concerns are addressed with our rebuttal. As we are getting really close to the deadline of the discussion phase, please let us know if there are any further questions that need clarification.

Many thanks,

Authors
Thank you for your reply and your additional experiments. Although I feel like the novelty of this work is still kind of limited, I would like to increase my score due to your prompt and detialed reply.
We thank the reviewer for the insightful comments, and we are happy to discuss some implementation details. 

**Q1. The implementation steps of BoostAdapter.** 

Your description of the implementation steps is correct. We would like to add some noteworthy points. 

- The boosting cache (memory bank) in step 2 is instance-adaptive. We create a copy of the historical cache (referred to as the boosting cache) and construct boosting samples for current test image to update the this cache for feature retrieval.
- These boosting samples will be discarded and will not be used by other images after step 3. 

It is rational because the boosting samples are close to the current test image rather than others, ensuring the bound of empirical risks in Proposition 3. 



**Q2. The relationship between historical and boosting samples**

For the current test image, the boosting samples will be derived only from itself rather than from historical samples.

We check the survey and find that all the relevent methods perform techniques like augmentation and clustering over **only historical samples** in the memory bank, without considering any information of the current test image.

We would like to point out that these methods may show poor generalization performance especially in downstream tasks that require fine-grained knowledge or when historical samples share insufficient similarity.

So we construct the instance-aware boosting cache to perform information mining over the current test sample and incorporate this knowledge with historical samples during feature retrieval.

The survey provides a clear description of memory-bank-based methods, so we will mention it and rewrite the corresponding section for a detailed discussion with these methods in the revision.



**Q3. Using EATA as additional training-required methods.**

The online test time adaptation setting discussed in BoostAdapter can be seen as a special case of continual TTA since we deal with samples from the test data stream. Therefore, we previously use EATA [1] simply due to its importance in test time adaptation and its applicability to our tasks. In practice, we follow the idea of a diversity-based selective strategy proposed by EATA to construct boosting samples in the cache.

We further incorporate techniques from more training-required methods, including the Pseudo-Label Probability Difference metric (PLPD) from DEYO [2] and the consistency filter from TSD [3]. Specifically, in the BoostAdapter+DEYO variant, we filter out augmented views with a PLPD lower than 0.2. For the BoostAdapter TSD variant, we discard augmented views that have different cache predictions and CLIP predictions to ensure consistency of the boosting samples. The results are provided in Table A and we can observe performance improvements with the help of different training-required methods, demonstrating the versatility of BoostAdapter.



**TableA: Unification of more training-required methods.**

|                       | -V    | -S    | -A    | -R    | Average |
| --------------------- | ----- | ----- | ----- | ----- | ------- |
| CLIP-ViT-B/16         | 60.86 | 46.09 | 47.87 | 73.98 | 57.20   |
| TDA                   | 64.67 | 50.54 | 60.11 | 80.24 | 63.89   |
| BoostAdapter          | 65.03 | 50.66 | 64.27 | 80.64 | 65.15   |
| BoostAdapter+EATA [1] | 65.27 | 50.82 | 64.83 | 81.15 | 65.52   |
| BoostAdapter+DEYO [2] | 65.51 | 51.01 | 64.57 | 81.11 | 65.55   |
| BoostAdapter+TSD [3]  | 65.49 | 51.50 | 64.37 | 81.15 | 65.63   |



**Reference**

[1] Niu, Shuaicheng, et al. "Efficient test-time model adaptation without forgetting." International conference on machine learning. PMLR, 2022.

[2] Lee, Jonghyun, et al. "Entropy is not enough for test-time adaptation: From the perspective of disentangled factors." The International Conference on Learning Representations. ICLR (2024).

[3] Wang, Shuai, et al. "Feature alignment and uniformity for test time adaptation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
Dear Reviewers,

The authors have provided responses to the reviews. Please take a look at the responses (and other reviews) and engage in a discussion.

Thanks for your service to NeurIPS 2024.

Best, 
AC
Thank you for your reply. I appreciate your additional experiments and explanation.

Actually, after check out your rebuttal, I still feel confused about the Q1 and answer in your global reply. I've checked TDA and the idea of this paper is quite aligned with them. While discussing entropy minimization in TTA (and as you claimed ``Our main contribution lies in the theoretical and experimental connection between training-required and training-free methods.''), for me, it seems like combining effective training-free and training-required techniques.

In this case, I would like to explain this paper in the way:
1. Using a Tip-adapter-like (also TDA-like) training-free adapter.
2. Using a memory bank to save reliable (historical and ) and diverse (augmented) test samples, using clip prediction as evidence. 
3. Apply this stored information to ``correct'' the clip logits for final prediction.
Please correct me if I have any misunderstanding.

I am also confused about the relationship between historical and boosting samples. Will the boosting samples constructed by augmented filtered historical test samples? If this is the case, I saw so many similar ideas in the online test-time adaptation on pages 12-13 [R1]. Could you explain further about the insights of your strategy and why it is better than others? How to differentiate your contribution among them?

It will be better to make the whole sample saving or caching process a bit clearer. In the current version, it is hidden among the equations. 

Another question is, could you specify why using EATA as the TTA baseline? EATA has carefully designed the FIM module and used the source sample information. It is also better for continual TTA. Why don't use other TTA methods?

Thank you.

Reference:

[R1]. Wang, Z., Luo, Y., Zheng, L., Chen, Z., Wang, S., & Huang, Z. (2023). In search of lost online test-time adaptation: A survey. arXiv preprint arXiv:2310.20199.