DOTA: Distributional Test-time Adaptation of Vision-Language Models
Authors: Zongbo Han
Jialong Yang
Guangyu Wang
Junfan Li
Qianli Xu
Mike Zheng Shou
Changqing Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Test-Time Adaptation
Primary area: applications
TL;DR: We proposes a new Distributional Test-time Adaptation (DOTA) method, which continuously estimates the distribution of test samples.

Abstract: Vision-language foundation models (VLMs), such as CLIP, exhibit remarkable performance across a wide range of tasks. However, deploying these models can be unreliable when significant distribution gaps exist between training and test data, while fine-tuning for diverse scenarios is often costly. Cache-based test-time adapters offer an efficient alternative by storing representative test samples to guide subsequent classifications. Yet, these methods typically employ naive cache management with limited capacity, leading to severe catastrophic forgetting when samples are inevitably dropped during updates. In this paper, we propose DOTA (DistributiOnal Test-time Adaptation), a simple yet effective method addressing this limitation. Crucially, instead of merely memorizing individual test samples, DOTA continuously estimates the underlying distribution of the test data stream. Test-time posterior probabilities are then computed using these dynamically estimated distributions via Bayes' theorem for adaptation. This distribution-centric approach enables the model to continually learn and adapt to the deployment environment. Extensive experiments validate that DOTA significantly mitigates forgetting and achieves state-of-the-art performance compared to existing methods.

DOTA: DistributiOnal Test-time Adaptation of Vision-Language Models
Zongbo Han Jialong Yang", Guangyu Wang Junfan Li',Qianli Xu²,
Mike Zheng Shou Changqing Zhang
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications
College of Intelligence and Computing, Tianjin University
School of Computer Science and Technology, Harbin Institute of Technology Shenzhen Institute for Infocomm Research (I²R), A STAR Research Entities (ARES) Show Lab, National University of Singapore
Abstract
Vision-language foundation models (VLMs), such as CLIP, exhibit remarkable performance across wide range of tasks. However, deploying these models be unreliable when significant distribution gaps exist between training and test data, while fine-tuning for diverse scenarios is often costly. This creates a need for methods that can efficiently adapt to new data at test time without expensive retraining. Cache-based test-time adapters serve this purpose by storing repre- sentative test samples to guide subsequent classifications. Yet, these methods typically employ naive cache management with limited capacity, leading to severe catastrophic forgetting when samples are inevitably dropped during updates. In this paper, we propose Dota (DistributiOnal Test-time Adaptation), simple yet fective method addressing this limitation. Crucially, instead of merely memorizing individual test samples, Dota continuously estimates the underlying distribution of the test data stream. Test-time posterior probabilities are then computed using these dynamically estimated distributions via Bayes' theorem for adaptation. This distribution-cente approach enables the model to continually learn and adapt the deployment environment. Extensive experiments validate that Dota signifi- cantly mitigates forgetting and achieves state-of-the-art performance compared to existing methods. Code is available at https:/gittbbboo///yyiccceee///d
Introduction
Recent advances in vision-language foundation models have shown remarkable vision understanding capabilities across broad range of tasks by training on web-scale image-text pairs [39 30 51].
Taking CLIP an example, it can conduct zero-shot classification without the need for additional training data using predefined prompts [39] However, CLIP may still face challenges when handling various specific applications during test time, especially when there is significant distribution gap between the training and test data 41 25 [3]. To adapt a foundational model to diverse deployment environments and personalized application requirements, fine-tuning is often necessary, which can be resource-intensive in terms of time, computational effort, and training data requirements.
Test-time adaptation (TTA) methods provide an efficient solution for addressing distributional shifts between training and testing domains 5 6 45 TTA allows for the dynamic adjustment of a pre-
"Corresponding author: wuw.eedu.ubredcccccc mike.zheng.sheee shou @ gmail.com. zhangchangging @tju.edu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

trained model during the inference phase by leveraging incoming test data to refine the model's s parameters. This adaptation process optimizes the model's performance for the specific test distribu- tion, eliminating the need for resource-intensive retraining. As such, TTA aligns seamlessly with real-world applications, where models must rapidly adapt to diverse and changing environments. There are two primary lines to achieve TTA on VLMs. Early works advocate learning prompts during test time with the test data [3] However, these methods require significant computational resources to optimize the learnable prompts via backpropagation and gradient descent, rendering them unsuitable for applications demanding fast test-time inference. Therefore, more efficient methods, cache- based methods, have been proposed recently 25 551. Typically, to avoid the need for training with backpropagation, Training-free free Dynamic Adapter (TDA) maintains a lightweight cache during testing to store representative test samples, which helps guide the classification of subsequent test samples.
While cache-based classifiers like TDA offer significant efficiency, they are fundamentally constrained by their finite cache capacity. These methods typically store limited set of 'typi- cal' test samples, updating the cache by replac- ing older entries with newer, cigh-confidence ones. However, this reliance instance-level memorization and forced sample discarding in- evitably Icads catastrophic forgetting. As the model adapts to new data patterns, it loses information about previously seen variations, hindering the formation of a stable and com- prehensive understanding of the evolving test distribution. Consequently, classifiers depend- ing solely on cached samples can be suboptimal, and ultimately performance-limite size [55]. As illustrated Fig. [1 to overcome these limitations of instance-level caching, we introduce DistributiOnal Test-time Adaptation (Dota). This represents a paradigm shift in test- time adaptation. Instead of passively memoriz- ing discrete samples in the cache with limited size, the core motivation Dota lies actively and continuously estimating the underlying sta- tistical distribution of the incoming test data, thereby enabling the full utilization of all avail- able test samples.
stream of test samples
Cache
Test-time distribution estimation
(Limited Size)
Classifier (limited data support)
Classifier
by the cache
(built from all samples)
Cache-based TTA (Previous Methods)
Distribution-based TTA (Ours)
Figure I:
Cache-based TTA methods store indi- vidual test samples within a limited cache, which often Ieads to underutilization of the available test data. In contrast, the proposed method continu- ously estimates the underlying distribution of the test data, enabling the full exploitation of all avail- able test samples.
Specifically, Dota continually estimates the distribution of test samples to adapt the test environment. Under the mild assumption that the embedding distribution of each class follows Gaussian dis- tribution [19], we propose an efficient method to continually estimate the distribution of different classes. Once the distributions of different classes are estimated, we can easily calculate the posterior probabilities of subsequent test samples based on Bayes' theorem and obtain test-time classifier for test-time adaptation. Similar to cache-based methods, this process does not require gradient back- propagation, avoiding the complex computational overhead during testing, leading to more than 20 times faster inference speed. Moreover, unlike cache-basee methods memorizing only representative test samples, Dota can continually adapt to the test environment by estimating the distribution of test samples. The contributions this paper are:
We propose novel continual test-time learning framework which improves the performance of pretrained foundation models in downstream tasks by learning the statistical distribution of test data, rather than caching a limited set of individual samples.
Within this framework, we propose a simple and effective method to enhance foundation models by estimating distribution of different categories test time and using Bayes' theorem to create an adaptive classifier.
Extensive experiments on diverse datasets validate the effectiveness of the proposed method, demonstrating a significant improvement. The code has been released at here

---

2
Related work
Test-time adaptation (TTA) for classical classification neural networks focuses on addressing the distribution shift between training and test data by learning from the test data. Early efforts to improve TTA performance primarily involve adjusting batch normalization layers and designing
unsupervised objective functions [35 44 28 32 For example, TENT [44] optimizes the affine parameters in batch normalization layers by minimizing the entropy of the prediction probability. MEMO [5 applies variant augmentation methods to a single test sample and optimizes model parameters by minimizing the entropy of the prediction probability. A recent advancement in test-time adaptation with distribution shift, which introduces the concept of universal TTA to address domain non-stationarity and temporal correlation, ensuring robust model performance across diverse scenarios 34. The most relevant of these traditional TTA methods to is T3A 24, which achieves test-time adaptation by adjusting the trained linear classifier using prototypes. Compared to T3A, which naively stores typical test samples, we achieve continuous adaptation by estimating the distribution of test samples.
TTA for VLMs. To enhance the performance of VLMs during testing. TPT 41
introduces adaptive text prompts and optimizes the prompts through entropy minimization. Building on this, DPf [3 leverages pre-trained stable diffusion models to generate diverse augmented data for use in test-time prompt tuning. However, TPT and DifTTPT rely heavily on gradient backpropagation to optimize the prompts, making them computationally expensive and resource-intensive. TDA (5] proposes a lightweight test-time adaption method by storing representative test samples. Building upon TDA, Boostadapter (55] enhances the cache sample selection strategy through Regional Bootstrapping. Compared to TDA and Boostadapter, which naively stores typical test samples, we achieve continuous adaptation by estimating the distribution of test samples, leading a more adaptive solution.
Distribution estimation for recognition. Distribution estimation plays crucial role in adapting recognition models by leveraging data's statistical properties for dynamic updates. This approach is
particularly effective when encountering new classes or shifting data distributions [99 48 For exam-
ple, 3 developed an open-world recognition system that continuously learns new object categories by evolving the nearest class mean algorithm into a nearest non-outlier variant. Similarly, Prototypical Networks 2 utilize distribution estimation by defining class prototypes as the mean of embedded support examples; classification then relies on metric distances, yielding strong performance in few-shot and zero-shot settings. Further addressing dynamic data, previous method [9] proposed a system for continual prototype evolution, facilitating online learning from non- non-stationary streams via efficient memory management and a novel objective function. Recently, a training-free CLIP adaptation method has been proposed by introducing Gaussian Discriminant Analysis [48] which estimates class means and shared covariance from data to build an ensemble classifier integrating zero-shot CLIP predictions. Building upon these principles, particularly from the continual learning literature, this paper introduces Dota to enhance the test-time performance of vision-language models.
Vision-language models have demonstrated strong vision understanding capabilities benefiting from training on large-scale datasets 33 51 30.. Among them, CLIP 39 the most representative method by maximizing the similarity between image and their corresponding text embeddings. To further enhance performance of CLIP on downstream tasks, prompt learning-basee methods have been proposed by optimizing the prompts of the text encoder [57 58 27
Moreover, to reduce prompt learning, efficient CL.IP adaptation methods have been introduced [16] 54] 48 31 (5). These methods enable downstream task adaptation using only a small number of training samples in the embedding space. Orthogonal to above methods, this paper focuses continuously adapting to environments during testing by leveraging test samples.
the computational cost associated with gradient calculations
3 Method
3.1 Zero-Shot Classification with Prompt
Zero-shot classification. During the pre-training stage, ELIIPtrainn its image and text encoders using large-scale image-text pairs. This is achieved by maximizing the cosine similarity between the image
Dota is also applicable other similar models.
3

---

and text embeddings through contrastive loss. Unlike traditional classifiers trained on closed-set labels, CLIP leverages open-set semantic information the image-text pairs learn broader range of visual concepts. Consequently, during the test stage, CLIP can perform zero-shot classification without additional training. Specifically, given a test sample æ for K-class classification, where represents the image embedding obtained from the image encoder, the corresponding zero-shot prediction probability 1% for class is calculated as:
exp(cos(x, Wx)/T)
P (y kt(z) - exp(cos(11, wx)/T)
(I)
where zs refers to zero-shot. we is the classification weight for class ks obtained by encoding the corresponding prompt, e.g., "a photo of [class]", , with the class token replaced by the specific category name. T is the learned temperature parameter in CLIP, and cos(..) denotes the cosine similarity. The above classification process can be understood as comparing the obtained image embedding with the text prompt and selecting the most similar category as the final decision.
3.2
Distributional Test-time Adaptation
Key motivation. When CLIP is deployed across different environments, its performance often deteriorates due changes in the data distribution, especially when the test data significantly deviates from the training data. TTA can effectively enable the foundational model to adapt quickly to new environments during the test phase. Current state-of-the-art methods typically maintain cache of representative samples from various classes, which guide the classification of subsequent test samples 25 55). However, due to the limited cache size, eache-based TTA methods face several challenges:
Limited information storage: These methods store embeddings for only a limited number of samples, which restricts the breadth and depth of information captured. As result, they may fail to adequately represent the full complexity and nuances of the test data distribution.
Lack of deep association learning: These methods primarily rely on cached samples for similarity matching or guidance, rather than learning or updating the model's understanding the deeper, intrinsic relationships between sample features and their corresponding semantic labels in the new test environment.
Significant risk of test-time forgetting: Due to the above limitations, when the cache must be updated (e.g., by replacing older samples), the model struggles to retain the adaptive knowledge gained previously. This leads to high risk of catastrophic forgetting, compromising both performance and stability as new data arrives.
To address these challenges, as shown in Fig. [ we propose distributional test-time adaptation (Dota), which continuously estimates the evolving test sample distribution during testing. By leveraging Bayes' theorem to infer the posterior distribution of different classes, Dota allows for dynamic adaptation based on an accurate understanding of the current data distribution. This distributional estimation ensures richer, more reliable information for classification, overcoming the limitations of cache-based methods and reducing the risk of forgetting, thus maintaining stable performance.
Classification with classical Gaussian discriminant analysis. Formally, inspired by classical Gaussian discriminant analysis (1). we assume that the embedding distribution of each class k follows a Gaussian distribution, i.c., P(æly=k) = N(ug, Es), where #4 and Σ; are the mean vector and covariance matrix of class k. respectively. Using Bayes' theorem, the posterior probability P(y - k/z) of class can be given by
Ply=k(æ) P(zly= (ooly=k)a (()
P(x)
where P(x) = EK_ P(zly=k)PPyybk and P(y=k) k) is the prior probability. In practice, we set P(y=k) to 1/ for simplicity. Then P(y= k((zz) can be obtained with
ex)(f((x))
P(y 4 (x)
ΣK exp(fx(x))
where [x() = -a1 1b) 2 log Ex]. The discriminant function (x) measures
how well sample fits the distribution of class k. The detail can be found in the Appendix A.6 For

---

stream test samples
Test-time distribution estimation
Zero-shot classifier
Updiae form of zest-time dstribution estimaton
MM E. Pasfy male,
4 "
E. Prato - aule -85 Ale A y
Party (t))
S- Ee Ps m.)
Boyes moben
Zero-shot classification probability
Test-time classifier
Figure 2: Pipeline of Dota. During test time, a stream test samples is evaluated with original zero-shot classifier, and we estimate the distributions for the test samples during testing, enabling the model to continually learn from the test samples and the zero-shot classification probabilities. As the number of lest samples increases, the estimated test sample data distribution will become more accurate. Finally the test-time classifier can be obtained with the estimated distributions according to Bayes' theorem for test-time adaptation.
the k-th class with N& training samples, the mean and covariance matrix of different classes can be estimated as follows:
Nv N₂
No Σg 2 (I,
(xx) )4x)
14
where fág and E. represent the estimated mean and covariance matrix the k-th class, respectively, an denotes the input embeddings. However, there are two problems with the estimation of the mean and covariance in Eq. 3 that make it unsuitable for test-time distribution estimation. First, during testing, class labels not available to compute the mean and covariance for class k. Second, during testing, it is impossible to access all test samples at once for distribution estimation, as the test samples are provided in the form of data stream. We will address these two issues by one below.
Parameter estimation with zero-shot predictive probability. When conducting test-time distribu- tion estimation, one main challenge is that we cannot access to the ground-trum labels for the N test samples. Therefore, we try to use the zero-shot predictive probability to estimate the distribution [19]. To this end we introduce the Prop. 3.1 The proof can be found in the appendix A.I
Proposition 3.1 (Parameter estimation with shro predictive probability using the Expectation- Maximization (EM) algorithm). Let PP k an)) li=1 be the zero-shot prediction probabilities { and covariances {E&]k= of the distribution can be estimated as a single iteration of the EM algorithm, where the expectation step computes the zero-shot prediction and the maximization step updates the parameters by maximizing the likelihood. The estimates are:
for n-th test sample. The means {jx k=
Pas y
2n Σg
ΣN m- PRO = 1: xr (x,, (a - P
f1
pas (20 (y= K xn.
Σx (y= (F,))
Prop.
3.1 shows that the estimation process with zero-shot predictive probability. The estimation can also be intuitively understood as reweighting, where the zero-shot probabilities are used as weights to adjust the contributions of different samples, thereby mitigating the impact of the potential inaccuracies in the zero-shot prediction.
Online test-time distribution estimation. When estimating data distribution at test time, one another challenge that we evaluate the test samples sequentially a streaming manner instead of accessing all samples simultaneously. This necessitates a strategy to appropriately adjust the estimation method in Eq. throuugh effective initialization, and then allowing the parameters to be updated quickly as new test samples arrive. To achieve this goal, Dota maintains the distribution information of different classes (i.e., mean and covariance matrix) during testing, and updates distribution information based on its representation information after obtaining new samples. Initialization {يg Σn K We can initialize the estimated mean of different classes in the following way:
0 3 w1 and
Σg = 1
(5)
5

---

where w and o² is hyperparameter that determines the initial mean and variance, I is the identity matrix. Update of (pg. ) We employ the update form described in [8], which is capable of
estimating Gaussian distribution parameters in an online setting. Theoretically, for any sequence, the average regret of the update form converges to zero in the limit. Specifically, given a batch of test
S samples at step i, the updated juke Σ can be computed based on the p%
2-1
Σg as follows:
k
14 +
+
P (y = K kn E
R
+
K 2n S
(6)
c + pzs (y nn
Σ
(y
n
where
) - p c represents the effective sample size, defined by the cumulative confidences of the observed samples of class it at step t 1. Specifically, we set of = 0 and dk is updated as e = - + Σ p (u=k(ar,,) When we obtain the estimated jt, E, we can use Eq. 2 to calculate the test-time adapted posterior probability.
= (In = p
In practice, Eq. is a generalized vector update version that works effectively with different test batch sizes. For consistency with comparison methods, we set the batch size to in our experiments. To reduce computational complexity when inverting the covariance matrix Σg. similar to the approach in [114, we approximate the covariance by averaging across all classes, reducing the number of matrix inversions from K to 1, thereby improving efficiency and reducing the number of parameter estimates. Additionally, we apply shrinkage regularization to the precision matrix to enhance the stability of the inversion process as follows: A - [[1 -e) + = where 10-4 is the shrinkage parameter. The term e/ ensures that the eigenvalues of the covariance matrix are well-conditioned, maintaining the desired properties such as positive definiteness and rank stability.
3.3 Adaptive fusion of zero-shot and test-time classifier
As the number of test samples increases, the reliability of the estimated test sample distribution improves [8]. However, when the number of test samples is insufficient, the estimated distribution may be unreliable. To address this, we introduce a dynamic zero-shot classification and test-time result fusion approach, allowing the model to rely more on zero-shot classification during stages where the sample size for distribution estimation is insufficient. Formally, the final fusion probability is defined as follows:
Pa k/z) us)) ((a(a))
[exp(cos(x; wx)/r + Afe(æ)))
where À = min(pe, 17). Here, e represents the number of test samples, and p and are hyperpa- rameters that control the weight of the test-time classifier logits. The value of À increases with the number of test samples when this number is insufficient, gradually approaching the maximum value This approach encourages the model rely on the zero-shot classifier results when the test samples are insufficient to estimate the distribution, mitigating the potential negative impact of the test-time classifier. The whole pseudo code is shown in Alg.
The additive fusion in Eq. 7 is a heuristic yet effective strategy designed to balance the robust prior knowledge from the pre-trained model with dynamic adaptation test-time shifts. This logit-level combination is computationally efficient and has been widely adopted as a common and effective practice in the TTA literature for leveraging pre-trained knowledge while incorporating new test-time information 225 (2).. While this approach is largely heuristic, it dynamically ensures minimal interference early stages while effectively refining predictions as Dota's per-class modeling becomes more accurate, proving highly effective in practice.
4 Experiments
Benchmarks. Consistent with prior works 41
we conduct our main experiments on cross-domain generalization and natural distribution shifts (NDS) scenarios. In the cross-domes generalization scenario, we evaluate the performance of the model across 10 diverse image classifica- tion datasets, each representing a distinct domain with different classes: Aireraft [3). Caltech 101
13 25
12.. Cars [29], DTD 71, FuroSAT (20). Flower102 (36,, Food101 Pets (37. SUN397 [49]. and UCF101 43 This benchmark provides a comprehensive evaluation of the adaptability of the

---

Algorithm 1:
The pseudocode of Dota. The embedding of test samples {.,, ) in an streaming way, zero-shot classification
Input:
weights [w,,... w Initializing the distribution of different class; for each test sample æ; do
Obtain the zero-shot probability with Eq. [
Update the distribution of different class with Eq. Obtain the test-time classification probability with Eq
Obtain the final classification result with Eq.
end for
Table 1: Top-1 accuracy (%) under the cross-domain generalization scenario. "PE" represents the use of prompt enhancement
Caltech
Flower102
Aireral
EuroSAT
Food101
SUN397
UCFIOI
Average
Cars
DTDD
Method
Zero-Shot
23.22
93.55
66.11
45.04
50.42
66.99
82.86
86.92
65.63
65.16
64.59
TPT 41
24.78
94.16
66.87
47.75
42.44
68.98
84.67
87.79
65.50
68.04
65.10
DiffIPT 13
25.60
92.49
67.01
47.00
43.13
70.10
87.23
88,22
65.74
62.67
65.47
TDA
23.91
94.24
67.28
47,40
58.00
71.42
86.14
88.63
67.62
70.66
67.53
BoostAdapter
55
27.45
94.77
69.30
45.69
61.22
71.66
87.17
89.51
68.09
71.93
68.68
HisTPT
26.90
94.50
69.20
48.90
49.70
71.20
89.30
89.10
67.20
70.10
67.60
ZERO
25.21
93.66
68 04
46.12
34,33
67.68
86.53
87.75
65.03
67.77
64.21
Dota
26.25
94.16
69.56
47.64
62.78
75.23
87.08
92.01
69.80
72.54
69.71
DMN w/PE [56
30.03
95.38
67.96
55.85
59.43
74.49
85.08
92.04
70.18
72.51
70.30
Dota w/ PE
29.82
94.85
69.06
55.97
58.35
77.06
87.07
92.40
70.97
74.86
71.04
model during test time across various class spaces. For the natural distribution shifts scenario, we utilize multiple datasets including ImageNet (10), ImageNet-A [22], ImageNet-R 21, ImageNet-S [6) and ImageNet- V2 [40], which serve measures of the robustness of our approach. We also evaluate Dota using medical image datasets and a caaholoogy-retrainedd foundation model [23), where the labels were converted into descriptive sentences for evaluation (e.g., transforming "tumor" into "H&E image tumor"). The evaluation includes three datasets: the Kather Colon dataset (26), comprising nine different tissue types; the PanNuke dataset [15), focusing benign versus malignant classifications; and the WSSS4LUAD dataset [18), which distinguishes between tumor and normal samples.
Comparison Method. compare the proposed method with the following approaches: (1) CLIP's zero-shot method, which utilizes an ensemble of 80 prompts [39]: (2) Prompt-based training methods, including TPT [1], DiffTPT [13,, and Historical Prompt Tuning [52]. These methods focus on optimizing input prompts rather than modifying the model's parameters. TPT and DiffTPT adapt the prompt at test time, with DifITPT also introducing more diverse test sample augmentation using diffusion model. Historical Prompt Tuning uses prior task knowledge to refine prompts for better task adaptation. (3) Efficient test-time adaptation methods, which do not require backpropagation and rely on cache of representative samples for adaptation, demonstrated in 25 56 55 The results for the above methods are obtained from the original papers.
Table 2: Performance under the NDS scenario.
Table 3: Performance comparison across datasets PLIP model.
Method
ImageNet
ImageNet-A
ImageNet-R R
ImageNet-S
Average
Shot
68.34
77.65
Dataset Kather
Zero-Shot
TDA
CTA
TPT [1]
68.98
54.77
77.06
47.94
62.19
45.60
49.35
61.92
DiTTPT [B]
70.30
55.68
75.00
46.80)
61.95
PanNuke
69.49
69.70
74.68
TDA(25
69.51
60.11
80.24
50.54
65.10
WSSS4LUAD
70.31
71.96
75.07
ZERO [
69.31
59.61
77.22
48.40
63.64
average
61.80
63.67
70.56

---

4.1 Comparison with state-of-the-ard methods
Results under the cross-domain generalization scenario. We first compare Dota with state-of-the- art methods under the cross-doman generalization scenario across 10 diverse image classification datasets, each from distinct domain with different classes. Tab. [ presents the experimental results. The proposed method achieved the best performance most datasets and the top two performance on all datasets. For example, when using the ViT-B/16 backbone network, the average performance was improved by 1. 1.03%.
Results under the natural distribution shifts scenario. We then compare Dota with state-of-the-a methods in the context of natural distribution shifts. Treeetess the experimental results, revealing the following key observation. Leveraging distribution modeling of the representation of test data, Dota achieves superior performance without requiring gradient backpropagation. For instance, using the ELLI--VTTB/// backbone network, Dota outperforms the second-best method by an average of 1.21 %, achieving gfate-of-the-an results across all datasets.
Performance validation on pathological image classification with PLIP. To validate the per- formance of the proposed method across other pretrained models and application scenarios, we conducted experiments using the PLIP model in the context of pathological image classification. The results are presented in Tab. 3 As shown in the table, the proposed method achieved superior performance, with an average improvement of 6.89% over TDA.
Table 4: Comparisons of our Dota with other methods in terms of efficiency (Test- ing Time) and effectiveness (Accuracy).
Table 5: Comparisons of our Dota with other methods on the ImageNerV2 dataset, where each class contains only 10 samples.
Method
Testing Time
Accuracy
Gain
Method
VIT-B/16
ResNet-50
Zero-Shot
11.82min
68,34
0
Zero-Shot
61.88
52.91
TPT
447min
68.98
+0.64
DinTPT
1346min
70.30
+1.96
TDA
64.67
55.54
TDA
22min
69.51
+1.17
Dota (All test samples)
64.50
55.19
Dota (last 50% test samples
65.20
55.72
Dota (Ours)
22min
70.69
+2.35
Inference time comparison. To illustrate the efficiency of the proposed method, we conduct evaluation about the inference time using the ViT-B/16 backbone on the ImageNet [I0] dataset. The experimental results are shown in Tab [ From the table, we can that the proposed method is faster than the methods that require gradient backpropagation. For example, Dota is 24 times faster than TPT, and 61 times faster than Dif Therefore, test-time adaptation methods that require gradient backpropagation may not be applicable during deployment due to the performance limitations of the inference device. At the same time, compared with TDA, the speed of the proposed method is comparable, but the performance is higher.
Failure case study. While our approach highlights the advantages of continuously estimating the distribution of test data and adapting to it, it does not consistently outperform TDA across all datasets, particularly those with a limited number of samples. For instance, as shown in Tab. 5 on the Imagenet 40), which contains only 10 samples per class, the performance of Dota is slightly lower than TDA. This is likely because the limited number of samples per class is insufficient accurately estimate the data distribution online. However, its performance shows a marked improvement on the latter 50% of the test samples. This observation suggests that the proposed model has the potential for further enhancement as the number of available test samples increases.
Table 6: Performance of Dota and TDA, comparing overall accuracy and the last 50% of test samples to show continuous adaptability. The results show that the performance our method continual improving. However, TDA is different, and its performance has declined on several datasets.
Method
Aircraft
Cakech101
Cars
DTD
Flower102
Food101
Pets
SUN397
UCF101
TDA (all test samples)
23.91
94.24
67.28
47.40
71.42
86.14
88.63
67.62
70.66
TDA (last 50% test samples)
26.57
93.59
66.95
46.22
71.75
86.02
89.26
67.86
72.20
Dota (All samples)
26.25
94.16
69.56
47.64
75.23
87.08
92.01
72.54
Dota (last 50% test samples)
27.65
94.65
69.98
50.47
76.46
87.12
93.30
70.69
73.73
8

---

4.2 Ablation studies and further analysis
Analysis of continuous learning ability and test- time forgetting of TDA. We conducted this experi- ment to validate our research motivation and demon- strate experimentally that Dota possesses the capabil- ity for continuous learning. During testing on the Im-
DOTA
2.5
TDA
Accuracy Improvement
ageNet dataset, we recorded the performance of the
most recent 5,000 test samples and compared it with
the original zero-shot classifier's performance. We analyzed the relationship between the improvement in model performance and the number of test sam- ples processed. The results are illustrated in Fig. 3 From the experimental results, it can be shown that the proposed method progressively enhances model performance the number of test samples increases. In contrast, TDA shows an initial improvement that subsequently declines, indicating its inability to con- tinuously learn from the test data stream. Further analysis on additional datasets, shown in Tab. 6 highlights the performance on the last 50% test samples as well as all samples. The results clearly demonstrate that the performance of the last 50% of test samples for Dota is significantly higher than the overall performance. This improvement can be attributed to the increasing reliability of the estimated distribution as more test samples are observed. However, TDA exhibits a different pattern, with its performance declining on several datasets.
1.0
0.0
10
Test Samples Seen
X5e3
Figure 3: Improvement of different methods in model performance as the number of en- countered test samples increases.
Table Ablation study compare the performance Dota with two variants: (1) rely solely on nigh-confidence samples and their predictive labels for estimating the distribution; (2) using only the mean, without updating the covariance matrix the estimation of the Gaussian distribution.
Method
Aireraft
Caltech101
Cars
DTD
EunSAT
Flower102
Food101
Pets
SUN997
UCF101
Average
Dota
26.25
94.16
47.64
62.78
75.23
87.08
92.01
72.54
69.71
high confidence samples only
-1.62
.0.04
-2.45
-1.30
53.28
-9.50
72.07
-3.16
.0.61
-1.33
-1.56
69:28
-3.26
-2.49
wro covariance
25. 29
94.16
67,47
45.62
55.06
-7.72
71.34
86.44
40.64
90.57
67.88
-1,92
69.34
-3.20
67.32
The Importance of distribution estimation using predictive probabilities of all test samples in an EM framework. We compared the performance of the Dota with a simplified version that only uses high-confidence samples and their corresponding predictive labels for estimating the distribution. The experimental results are shown in Tab. 7 The experimental results demonstrate that updating the data distribution using the proposed zero-shot predictive probabilities achieves better performance compared to using pseudo-labels from only high-confidence samples. For instance, overall performance sees a decline of approximately 2.49% when relying solely on high-confidenee pseudo-label.
The necessity of distribution estimation. We com- pared the performance of the Dota with a simplified version that uses only the mean, excluding the es- timation of the Gaussian distribution by removing the updates to the covariance matrices. This experi- ment aimed to understand the necessity continual distribution estimation in enhancing model accuracy. The experimental results are shown in Tab. 7 The fifth row in the table presents the accuracy reduc- tions across different datasets when the covariance matrix is not updated. The results indicate consis- tent decrease in accuracy across all datasets, with a particularly notable drop of 7.72% on the EuroSAT dataset. These findings highlight the importance of continual distribution estimation.
Table 8: Hyperparameters analysis on the o² and (p. combinations.
o²
0.0001
0.001
0.002
0.004
0.008
0.02
Acc
70.72
70.72
70.69
70.70
70.60
70.42
v/
0.005
0.01
0.02
0.03
0.2
70.69
70.66
70.59
70.54
0.3
70.64
70.55
70.36
70.28
0.4
70.64
70.51
70.24
70.13
0.5
70.64
70.48
70.15
70.00
Hyperparameters analysis. To validate the sensitivity of our model to hyperparameters. we conduct systematic experiments and analyses. First, we evaluate the hyperparameter o² while keeping other fixed. The results showed minimal impact accuracy, with performance ranging from 70.42 to 70.72.

---

Next, we test different p and 7) combinations, observing stable performance across combinations. For instance, accuracy ranged from 70.69 to 70.00 as p and 7) varied. Another hyperparameter we consistently set to 0.001. Notably, all hyperparameter combinations show that the proposed method outperforms the original zero-shot classifier, indicating that TTA can usually significantly enhance performance even without validation set for hyperparameter tuning.
The necessity of adaptive fusion of zero-shot and test-time classifier. We conduct ablation study to show that adaptive fusion of zero-shot and test-time classifier is necessary. The specific experimental results are shown on the Tab. It can be observed that as p increases (indicating the diminishing effect of dynamic fusion), the performance of Dota consistently decreases.
5 Conclusion and Limitations
In this paper, we proposed DistributiOnal Test-time Adaptation (Dota), method that overcomes the limitations of cache-based test-time adaptation by continuously estimating the underlying distribution of test data. Unlike traditional methods, which are constrained by fixed cache sizes and suffer from catastrophic forgetting. Dota dynamically updates class distributions, leading to more efficient and adaptive test-time inference. Our experiments show that Dota significantly reduces forgetting. improves performance, and offers over 20 times faster inference compared to test-time prompt training methods. This approach provides effective solution for deploying vision-language models in dynamic environments, offering a scalable and computationally efficient alternative to traditional adaptation techniques.
This method has three main limitations that suggest directions for future work. First, its single Gaussian distribution assumption restricts its ability to model complex data, which could be addressed by exploring more flexible models. Second, the assumption of uniform class priors is inconsistent with real-world class imbalances and could be improved with an adaptive prior estimation mechanism. Finally, the heuristic fusion strategy could be replaced by a more principled approach that dynamically adjusts weights based on real-time sample uncertainty.
Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China (No. 624B2100, No. 62376193, No. 61925602, No. T2522008, and No. 62272055), New Comnesttnee Science Foundation through the XPLORER PRIZE. We thank Prof. Qinghua Hu for his help with the manuscript. Mike Shou does not receive any funding for this work. The authors appreciate the valuable feedback from anonymous reviewers.
References
[1] Theodore Wilbur Anderson, Theodore Wilbur Anderson, Theodore Wilbur Anderson, and Theodore Wilbur Anderson. An introduction to multivariate statistical analysis, volume 2. Wiley New York, 1958.
[2] Yichen Bai, Zongbo Han, Bing Cao, Xiaoheng Jiang, Qinghua Hu, and Changqing Zhang. Id- like prompt learning for few-shot cut-of-distributioo detection. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17480- 17489, 2024.
[3] Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1893-1902, 2015.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 100d-100- discriminative components with random forests. In Computer Aition-ECCV 2014: 13th European conference, zurich, Switzerland, September 6- 12, 2014, proceedings, part VI 13, pages 446-461 Springer, 2014.
[5] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 8344-8353, 2022.
10

---

[6] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test- eisttimme adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 295-305, 2022.
[7] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[8] Sanjoy Dasgupta and Daniel Hsu. On-line estimation with the multivariate gaussian distribution. In International Conference on Computational Learning Theory, pages 278-292. Springer, 2007.
[9] Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non-stationary data streams. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8250-8259, 2021.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. leee, 2009.
[11] Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, and Elisa Ricci. Frus- tratingly easy test-time adaptation of vision-language models. Advances in Neural Information Processing Systems, 37:1 37:129002-129993, 2024.
[12] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference on Computer Vision and Pattern Recognition workshop, pages 178-178. IEEE, 2004.
[13] Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmen- tation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2704 1100-2714, 2714, 2023.
[14] Jerome H Friedman. Regularized discriminant analysis. Journal of the American statistical association, 84(405):165-11 175, 1989.
[15] Jevgenij Gamper, Navid Alemi Koohbanani, Ksenija Benet, Ali Khuram, and Nasir Rajpoot. Pannuke: an open pan-cancer histology dataset for nuclei instance segmentation and classifica- tion. In Digital Pathology: 15th European Congress, ECDP 2019, Warwick, UK, April 10-13, 2019, Proceedings 15, pages 1 11-19. Springer, 2019.
[16] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision- langua-a models with feature adapters. International Journal of Computer Vision, (32(2):581-555 2024.
[17] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
[18) Chu Han, Xipeng Pan, Lixu Yan, Huan Lin, Binghing Li, Su Yao, Shanshan Lv, Zhenwei Shi, Jinhai Mai, Jiatai Lin, et al. Wsss4luad: Grand challenge on weakly-supervised tissue semantic segmentation for lung adenocarcinoma. arXiv preprint arXiv:2204. 06455, 2022.
[19] Trevor Hastie and Robert Tibshirani. Discriminant analysis by gaussian mixtures. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):155-176, 1996.
[20] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
[21] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of out-of distribbitiiittonn generalization. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 8340-83000000 8349, 2021.
11

---

[22] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song, Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15262-15271, 2021.
[23] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A -angul- foundation model for pathology image analysis using medical twitter. Nature Medicine, pages 1-10, 2023.
[24] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances Neural Information Processing Systems, 34:2427-2440, 2021.
[25] Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14162-14171, 2024.
[26] Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios Valous, Dyke Ferber, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS medicine, 16(1)x 2019.
[27] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[28] Ansh Khurana, Sujoy Paul, Piyush Rai, Soma Biswas, and Gaurav Aggarwal. Sita: Single image test-time adaptation. arXiv preprint arXiv:2112.0233: 2021.
[29] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition workshops, pages 554-561, 2013.
[30] Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mido Assran, Andrew Gordon Wilson, Aaron C. Courville, and Nicolas Ballas. Modeling caption diversity in contrastive vision- language pretraining. In Forty-first International Conference on Machine Leurning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[31] Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, and Xinchao Wang. Graphadapter: Tuning lision-language models with dual knowledge graph. Advances in Neural Information Processing Systems, 36, 2024.
[32] Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. TTN: A domain-shift aware batch normalization in test-time adaptation. In The Eleventh International Conference Learning Representations, 2023.
[33] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification aircraft. arXiv preprint arXiv: 13065555 2013.
[34] Robert A Marsden, Mario Döbler, and Bin Yang. Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction. Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 2555-2561 2024.
[35] Zachary Nado, Shreyas Padhy, Sculley, Alexander D' Amour, Balaji Lakshminarayanan, and Jasper Snoek. Exaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.10963 2020.
[36] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008.
[37] Omkar Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3498-3505, IEEE, 2012.
12

---

[38] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for shoo image classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15691-15701, 2023.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[40] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
[41] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35:14274-14289 2022.
[42] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 30, 2017.
[43] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(11):1-7, 2012.
[44] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[45] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In 9th International Conference on Learning Representations, 2021.
[46] Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[47] Shuoyuan Wang, Jindong Wang, Guoqing Wang, Bob Zhang, Kaiyang Zhou, and Hongxin Wei. Open-vocabulary calibration for fine-tuned clip. In International Conference on Machine Learning, pages -17554. PMLR, 2024.
[48] Zhengbo Wang, Jian Liang. Lijun Sheng, Ran Hc, Zilei Wang, and Tieniu Tan. A hard-to-beat baseline for training-free frree clip-based adaptation. In The Twelfth International Conference Learning Representations, 2024.
[49] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. 2010 IEEE computer society Conference on Computer Vision and Pattern Recognition, pages 1485-3492, IEEE, 2010.
[50] Tao Yu, Zhihe Lu,, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning vision- language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10899-10909, 2023.
[51] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceddings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11975-11986. 2023.
[52] Jingyi Zhang, Jiaxing Huang, Xiaoqin Zhang, Ling Shao, and Shijian Lu. Historical test-time prompt tuning for vision foundation models. In Advances in Neural Information Processing Systems, volume 37, pages 12872-12896. 2024.
[53] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35:38629-3888 -36422, 2022.
13

---

[54] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training- free adaption of clip for few-shot classification. In European conference on computer vision, pages 493-510. Springer, 2022.
[55] Taolin Zhang, Jinpeng Wang, Hang Guo, Tao Dai, Bin Chen, and Shu-Tao Xia. Boostadapter: Improving vision-language test-time adaptation via regional bootstrapping. In Advances in Neural Information Processing Systems, volume 37, pages 67795-66669 67825, 2024.
[56] Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 28718-28777 2024.
[57] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu, Conditional prompt learn- ing for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[58] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 2022.
14

---

A Appendix
A.1 Proof of Proposition 3.1
The EM algorithm is an iterative method used for finding maximum likelihood estimates of parameters in statistical models with latent (unobserved) variables. consists of two main steps:
(1) Expectation Step (E-step): Compute the expected value the fog-likelihood function, with respect to the current estimate of the distribution of the latent variables.
(2) Maximization Step (M-step): Maximize this expected log-liklihood update the parameter estimates.
In the context of Gaussian Discriminant Analysis (GDA) for classification, the latent variables correspond to the true class labels of the data points, which are unobserved during testing.
Assumption A.1 Class-Condition Distributions). For each class k ... K), the data embedding follows Gaussian distribution:
P( y k) = ( Σx)
where 24 and Σ3 are the mean vector and covariance matrix of class k, respectively.
Assumption A.2 (Prior Probabilities). The prior probability for each class is uniform: Ply = k)
K
The objective is how estimate the parameters {Mr: ) using the EM algorithm, leveraging the zero-shot predictive probabilities (x xnn) as part of the expectation step.
E-Step: Compute Expected Log-Likelihood. In the E-step, compute the expectation of the log-likelihood with respect to the posterior distribution of the latent variables given the current parameter estimates. Given the zero-shot predictive probabilities PÆa (y le xr,), we treat them as the responsibilities (i.e., the posterior probabilities) in the E-step:
nuk
Pe (3 le x..)
The expected log- ligllikli Q is then:
N
1
Q((m Σx} current estimates)
Tak lag pa: Ex) + log
M-Step: Maximize the Expected Log-Likelihood. In the M-step, we maximize Q with respect to (Hy, E Maximizing with Respect to p4- To find the optimal μt. take the derivative of Q with respect to fy and set to zero. First, expand the Gaussian log- probability:
d
logN((x, P2: Ex)
(xn m&) T (xz, p4)) log Ex
(og((2))
2
where d is the dimensionality of æn. Focusing terms involving p4:
0₂ 2 n=1
(n 14 " (xo 14x
Take the derivative with respect to μ4 and set to zero:
aQ₂
Tuk
2 (Dn 14x)
Solving for p
N Σ 1=1
Tnk
n=1
N
1=
14
1=
Tia
15

---

This corresponds directly to the provided update formula for Paa-
i m= K (y Vn Jn
Em (yy = (.n))
Maximizing with Respect to Σg. Similarly, find the optimal Σx. consider the terms in involving
Σg:
QΣ
Tak
(xn (4x "E (z, - m4) + log Σx
2
n=1
Take the derivative with respect to Σx
(using matrix derivative identities) and zero:
N
aQr
Tak (xn (xx)( -1a)) - Eg = 0
2
n=1
Solving for Eg:
En=1 Tuk (nk Mx)(En 14k)
N
Σg
Σn Tnk
This aligns with the provided update formula for Σx:
pas y = k x (xn - (xx - pxx)
Σ
m=
ΣN
Pkat (y
σn)
Finally, the provided update equation is:
NN (y igg
k En Jnn
ΣN EN
PE (y = k (2) (x) pxx) Pe (y
P xn)
Σn
xnn)
are precisely the M-step updates obtained by maximizing the expected log lig-likelihood in the EM algorithm, where the E-step uses the zero-shot predictive probabilities responsibilities. This demonstrates that the parameter estimation process described is equivalent to performing a single EM iteration. This single EM iteration leverages the zero-shot predictions adjust the Gaussian parameters, effectively "reweighting" the contributions of different samples based on their inferred class probabilities.
A.2 Justification for the Gaussian Assumption
Our assumption that the embedding distribution of each class k follows Gaussian distribution, i.c., P(rly k) NNx,, Ex), grounded in both theoretical principles and empirical evidence.
Theoretical support: The Central Limit Theorem (CLT) provides general justification. A high-dimensional embedding can be viewed as an aggregation of numerous lower- and miv-level features. The CLT posits that the sum of many weakly correlated random variables will tend toward a normal distribution, making it reasonable to expect that the composite feature for a single class will approach multivariate Gaussian distribution.
Empirical validation: Our own ablation study validates the importance this assumption; omitting covariance updates, which relies on the Gaussian model, significantly degrades performance (c.g.. a 7.72% drop on EuroSAT), confirming the practical utility Gaussian modeling.
A.3 Analysis of Catastrophic Forgetting
key motivation our work to mitigate the catastrophic forgetting inherent in cache-based methods like TDA. In the context of TTA, forgetting manifests at "sample-level'ss due to a fixed-size cache, adapting to new samples necessitates discarding information from older ones. To rigorously validate this issue, conducted two targeted experiments. The core idea is have the model, after
16

---

a period of continuous adaptation, revisit and re-evaluate its performance on historical test samples. Our experiments revealed two levels of forgetting.
Forgetting without Significant Distribution Change. To assess this phenomenon, we designed two-stage experiment. This process first involves adapting model on ImageNet and recording its performance (Stage 1), after which we freeze the model's parameters and re-evaluate it on the early 25,000 test samples in the dataset (Stage 2). We hypothesize that if forgetting occurs, this initial adaptation stage should directly lead to an increase in the model's performance. Conversely, a performance decrease between stage and stage 2 would suggest that the model has experienced forgetting. Experimental results show that TDA suffers from obvious performance degradation, while our method does not.
Table 9: Performance comparaio in two-stage experiment.
Samples Seen
10000
20000
25000
TDA Stage 1
0.42
0.71
0.76
TDA Stage 2 (Frozen)
0.17
0.57
0.29
DOTA Stage 1
0.58
1.16
1.25
DOTA Stage 2 (Frozen)
2.34
2.30
2.25
Domain-Level Forgetting. We challenged the model by adapting it first to an original domain (ImageNet), then to new, shifted domain (ImageNettCC), and finally re-evaluating its performance the original domain, TDA's performance failed to improve upon returning to ImageNet, indicating had forgotten the original domain's features. DOTA, however, showed improvement, demonstrating it retained this knowledge.
Table 10: Domain-level forgetting experiment.
Method
ImageNet
briighet-CCbighhneess ImageNet (Frozen model)
Zero-Shot
68.34
56.98
68.34
TDA
69.55
58.22
69.35
DOTA
70.76
60.64
70.93
We reinterpreted Fig. 3 and added additional experiments. We found that catastrophic forgetting degrades model's generalization capabilities by eroding previously learned information. More strikingly, we found that under certain conditions, continued adaptation is actively counterproductive, yielding worse results than if adaption had stopped. This is because as we continue adapt, forget more than we learn. The Tab. IT illustrates this clearly. The standard TDA model's performance peaked after 20,000 samples and then degraded. A model whose training was frozen at that point (TDA frozen) consistently outperformed the continuously trained model thereafter. Our proposed method, DOTA, effectively mitigates this degradation and delivers robustly superior performance.
Table 11: Performance degradation during continuous adaptation.
Samples Sccn
5k
10k
15k
20k
25k
30k
35k
40k
45k
50k
Ave
TDA
0.14
0.70
0.76
1.22
0.96
0.98
1.02
0.68
0.46
0.24
0.72
TDA (frozen after 20k)
0.14
0.70
0.76
1.22
1.00
1.54
1.04
0.82
0.66
0.68
0.86
DOTA
-0.02
1.18
1.48
1.98
1.64
2.84
2.74
2.26
2.06
2.84
1.90
A.4 Results across multiple data ordering
We conducted experiments on five test data in different ordering multiple datasets. The experimen- tal results are shown in the Tab. 12 From the experimental results, we can see that the order of test data has little effect on the prediction performance.
Table 12: Experimental results across multiple datasets and test data orderings.
Dataset
Method ordering
2
3
4
Average
ImageNet
Dota
70.54
70.66
70.72
70.67
70.63
70.64
eurosat
Dota
62.74
62.56
63.01
62.46
63.28
62.81
OxfordPets
Dota
92.04
91.93
92.29
92.04
92.12
92.08
17

---

The function f*()). often referred as the discriminant function, measures how well data point fits the distribution of class k. It is derived from Gaussian Discriminant Analysis and consists of two main components. The first component is the Mahalanobis distance, - (x - (x pk)) A.5 Performance on non-i..d. data streams
We conducted additional experiments to evaluate the model's performance under non-i.i.d data distribution during testing, using the ImageNet dataset as a benchmark. By employing Dirichlet distribution, we simulated varying degrees of non-i..d. data streams, adjusting the concentration parameter and dividing the dataset into 5 and 10 time slices for analysis. The details of the experiments are shown as follows:
Time Slices: We divided the ImageNet dataset into and 10 time slices, where each slice contains varying numbers of samples and class distributions.
Concentration Parameter (())x): The concentration parameter of the Dirichlet distribution controls the uniformity class distributions across slices. Smaller a values (e-g.. 0.1) create highly uneven distributions, while larger values (e.g., 0.5 and 1) result in more uniform distributions.
Since the sizes of sub-datasets for each time slice are unequal, the final average accuracy is a weighted average based on the number of samples in each slice. The experimental results are summarized below.
Evaluation Setting:
Table 13: Performance non-ii.d. data streams (5 slices).
Slice
Slice 2
Slice 3
Slice 4
Slice 5
Average
0.1
68. 18
70.06
71.60
71.09
70.91
70.41
0.5
69.55
70.78
69.43
71.95
70.92
70.58
1
69.41
71.64
71.05
70.01
71.81
70.83
Table 14: Performance on no..i.i.d data streams (10 slices).
a
Slice
Slice
Slice
Slice
Slice
Slice
Slice 8
Slice
Slice 10
Average
0.1
69.99
69.31
67.34
70.23
71.49
68.8
73.13
70.35
69.89
72.39
70.39
0.5
68.74
69.32
72.17
71.27
69.97
69.8
70.3
70.78
72.48
70.78
70.61
67.33
70.29
70.92
68.43
69.85
71.14
72.03
72.52
71.29
71.66
70.68
From the experimental results, streams, with only minimal accuracy decline under small a (e.g., a 0.1).
that the model shows strong robustness to non-i.i.d data
A.6 More details and explanation about the fxx(x) Eq. 2
which calculates the squared distance between and the class mean μıs, scaled by the inverse of the covariance matrix Σx- This term captures the similarity of x to the center of the class, considering feature correlations. The second component is the normalization term, - log Exl, which accounts for the determinant of the covariance matrix Σς and reflects the spread (or volume) of the Gaussian distribution for class k. This ensures that classes with larger variances normalized appropriately, Intuitively, a larger value of fu (x) indicates a higher likelihood that æ belongs to class k. In classification, f& (x) is used within the sofumax function compute the posterior probability P(y = k x), which determines the most likely class for æ:
ex)(fx(x))
P(y = k x))
ΣR (x)(fx(x))
A.7 CLIP's Superior Calibration Performance
When estimating the distribution, we use zero-shot probabilities, which we argue here to be relatively well-calibrated. Specifically, we provide comparative data on the Expected Calibration Error (ECE) metric to substantiate CLIP's superior calibration performance. We draw on data from [47] and [7 to present the following comparative table of ECE metrics on the ImageNet dataset for different models.
18

---

Table 15: Datasets details.
Dataset
Classes
Validation Size
Test Size
Task
ImageNet
1,000
N/A
50,000
Classification
ImageNet-V V2
1,000
N/A
10,000
Generalization
ImageNet-S
1,000
N/A
50,000
Generalization
ImageNet-A
200
N/A
7,500
Generalization
ImageNet-R
200
N/A
30,000
Generalization
Aircraft
100
3,333
3,333
Aircraft recognition
Caltech101
100
1,649
2,465
Object recognition
Cars
196
1,635
8,041
Car recognition
DTD
47
1,128
1,692
Texture classification
EuroSAT
10
5,400
8,100
Remote sensing classification
Flowers 102
102
1,633
2,463
Flower recognition
Food101
101
20,200
30,300
Food classification
Pets
37
736
3,669
Pet classification
SUN397
397
3,970
19,850
Scene recognition
UCFI01
101
1,898
3,783
Action recognition
Kather
2
10,718
32,154
Colon classification
PanNuke
9
623
1,888
Tissue classification
WSSS4LUAD
2
1,009
3,028
Tissue classification
Table 16: Comparison of Expected Calibration Error (ECE) the ImageNet dataset. Lower values indicate better calibration. Note that CLIP achieves the lowest ECE without post-hoc calibration methods like Temperature Scaling (TS) or Vector Scaling (VS).
Method
DenseNet
DenseNet
DenseNet
ResNet
ResNet
152+VS
ECE
6.28%
2.24%
5.48%
2.23%
1.51%
In the table, "TS" refers to Temperature Scaling, and "VS" refers to Vector Scaling. These are calibration strategies applied DenseNet and ResNet reduce Expected Calibration Error (ECE), whereas CLIP adopts no such strategies. Experimental data were obtained on the ImageNet dataset.
The table demonstrates that CLIP's ECE is significantly lower than that of DenseNet and ResNet. Even after applying scaling strategies to reduce ECE for the baseline models, CLIP's ECE remains lower, confirming its superior calibration performance.
A.8 Statistical significance for results
To evaluate the stability and robustness of our proposed method, Dota, we conducted experiments across all datasets using five different random seeds. The aggregated results are presented in Tab. [7] Tab. 18. and Tab. 19 which detail the mean accuracy, standard deviation (stddev), and 95% confidence intervals. As shown, Dota exhibits highly consistent performance. This analysis confirms the robustness and reliability of our findings.
Table 17: Statistical analysis for cross-domain eeaemaiinggnnraiie generalization results.
Datact
Cattechtt
Flovers
Sand97
Averae
Dra beadel)
26.25
04 16
47
63.38
25.33
53.00
$2.01
60:37
72:54
09 71
(anme)
2015
59
09
2738
13.11
75335
57.06
91.11
60:83
72.69
09.77
contidenee inteval
125
(9422 9LMM
1974 9613)
(823)
pank
(9135 92.26
75, M491) (2.41
A.9 Implementation details
All the models in our experiments are built upon the pre-trained CLIP model [39 that consists of an image encoder and text encoder. Test-time adaptation set for single-image scenarios, using batch size of 1. For natural distribution shifts scenario, we tune our hyperparameters using the validation set. For the cross-domain generalization scenario, we perform hyperparameter search using the corresponding validation sets. We adjust o² within [0.001, 0.002. 0.004], then search for the best
19

---

Table 18: Statistical analysis for Natural Distribution Shift (NDS) results.
Dataset
ImageNet
ImageNet-A
ImageNet-R
ImageNet-S
Average
Dota (seed=1)
70.69
61.50
81.21
51.84
66.31
Dota (mean)
70.66
61.32
81.13
51.85
66.24
stddev
0.07
0.20
0.11
0.11
0.12
confidence interval
[70.57. 70.75]
[61.06, 61.57]
[81.00, 81.26]
[51.71, 51.99]
[66.08, 66.39]
Table 19: Statistical analysis for medical scenario results.
Dataset
Kather
PanNuke
WSSS4LUAD
Average
Dota (seed=1)
61.92
74.68
75.07
70.56
Dota (mean) stddev
61.94
74.65
75.13
70.57
0.14
0.14
0.15
0.14
confidence interval
[61.76, 62.11]
[74.49, 74.82]
[74.94, 75.32]
[70.40, 70.75]
across [0.2, 0.3, 0.4, 0.5] and across [0.005, 0.01, 0.02, 0.03), with the shrinkage parameter set 0.0001. We top-1 accuracy (%) evaluation metric. All experiments conducted using a single NVIDIA RTX 4090 GPU and a 12-core Intel Xeon Platinum 8352V CPU.
20

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
21

---

Justification:
the paper discusses the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: For each theoretical result, the paper provides the full set of assumptions and complete (and correct) proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer:[ Yes]
Justification: the code has be released.
Guidelines:
22

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code has been released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https ctipps.cc/public//uue
Codeeuumiiii wettsbbmmssiiioi ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
23

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The full details have been provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: we provide appropriate information about the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: we have provided time of execution.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp///eaapp...ccoooaa/////ott
Answer: [Yes]
Justification: The research carried out in the paper complies in all respects with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Foundational models are being widely deployed, but they do not always adapt perfectly to the distribution of test data. Collecting new data and fine-tuning models for specific applications can be costly and slow in response. Therefore, allowing models to adapt to unseen data during test time can enhance their generalization and adaptability.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
25

---

Justification:
the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators original owners of assets (e.g. code, data, models) used in the paper are properly credited, and the licence and terms of use are explicitly stated and properly respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, papprssii thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
26

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy hattoe//nearrjsscc//ooffeeeeeeee for what should or should not be described.
27


Reviews:
[{"review_id": "HP3A2xJbRG", "reviewer": "Reviewer_J9Lm", "summary": "This paper presents Dota, a method for adapting vision-language foundation models (e.g., CLIP) to distribution shifts during test time. Dota continuously estimates the underlying test data distribution for each class and dynamically updates class-specific mean and covariance matrices via an EM framework, leveraging zero-shot probabilities as soft weights.", "questions": "See my detailed comments.", "limitations": "See my detailed comments.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Dota replaces instance-level caching with continuous distribution estimation, enabling fuller utilization of test data and addressing the fundamental limitation of catastrophic forgetting in cache-based methods. This distribution-centric approach provides a more robust and scalable solution for dynamic environments.\n2.  Dota achieves comparable inference speed to cache-based methods while outperforming them in accuracy.\n3. Ablation studies validate the necessity of full-distribution estimation (covariance updates) and EM-based weighting. Hyperparameter analysis  shows stability across configurations.\n\nDisadvantages  \n1. The method assumes uniform class priors (\\(P(y=k) = 1/K\\)) in Eq. 2 (line 171), which is unrealistic in real-world scenarios where test data may exhibit significant class imbalance. For instance, if certain classes are underrepresented, the Gaussian estimates for those classes could be biased due to insufficient sample weighting, potentially degrading adaptation accuracy. The paper does not evaluate performance under imbalanced test streams, leaving open questions about robustness.  \n2. While Eq. 7 introduces a fusion of zero-shot and test-time classifiers via \\(\\cos(x, w_k)/\\tau + \\lambda f_k(x)\\), the rationale for adding \\(f_k(x)\\) in the exponent (instead of other forms of combination) is not thoroughly justified. The authors imply this stems from logit-level fusion (Sec. 3.3), but a deeper theoretical or empirical analysis is missing. This raises concerns about whether this design is optimal or merely heuristic.  \n3. Although Dota claims to mitigate catastrophic forgetting by estimating distributions (Sec. 3.2), the mechanism for how distribution updates prevent forgetting is not explicitly detailed. For example, does continuous covariance estimation inherently preserve earlier statistics better than cached samples?", "was_revised": true, "final_justification": "The authors address all of my concerns.", "rebuttal": "# 1. Considerations regarding the assumption of uniform class priors ($P(y=k) = 1/K$)\nWe sincerely acknowledge the reviewer for highlighting the limitation of assuming uniform class priors ($P(y=k) = 1/K$) in real-world test-time adaptation (TTA) scenarios with potential class imbalance. To address this concern, we have (1). clarified the prevalence and rationale of this assumption,(2). demonstrated Dota’s compatibility with non-uniform class priors, and (3). provided empirical evidence showcasing Dota’s robustness in handling imbalanced class data streams.\n\n## Prevalence and Rationale of Uniform Class Priors\nAssuming uniform class priors is a common and reasonable practice in TTA, particularly when prior information is unavailable. Several studies explicitly adopt this assumption as a starting point, achieving effective model initialization in the absence of test-time prior information [1-5]. \n\n## DOTA'S Compatibility with Non-Uniform Priors\nDota seamlessly integrates with non-uniform class priors when available in real-world scenarios. By incorporating class-specific weights into the probability computation, our method adapts to given prior distributions, demonstrating its flexibility and compatibility.\n\n## Robustness to Class-Imbalanced Data Streams\nTo evaluate Dota’s performance under class imbalance, we conducted experiments on the ImageNet validation dataset, simulating imbalanced data streams using a Dirichlet distribution with varying concentration parameters ($\\alpha$) and dividing the dataset into 5 and 10 time slices. The results are shown in Tables 1 and 2 below:\n\n**Table 1**\n| α | Slice 1 | Slice 2 | Slice 3 | Slice 4 | Slice 5 | Average |\n|----------|---------|---------|---------|---------|---------|---------|\n| 0.1      | 68.18   | 70.06   | 71.6    | 71.09   | 70.91   | 70.41   |\n| 0.5      | 69.55   | 70.78   | 69.43   | 71.95   | 70.92   | 70.58   |\n| 1        | 69.41   | 71.64   | 71.05   | 70.01   | 71.81   | 70.83   |\n\n**Table 2**\n| α | Slice 1 | Slice 2 | Slice 3 | Slice 4 | Slice 5 | Slice 6 | Slice 7 | Slice 8 | Slice 9 | Slice 10 | Average |\n|----------|---------|---------|---------|---------|---------|---------|---------|---------|---------|----------|---------|\n| 0.1      | 69.99   | 69.31   | 67.34   | 70.23   | 71.49   | 68.8    | 73.13   | 70.35   | 69.89   | 72.39    | 70.39   |\n| 0.5      | 68.74   | 69.32   | 72.17   | 71.27   | 69.97   | 69.8    | 70.3    | 70.78   | 72.48   | 70.78    | 70.61   |\n| 1        | 67.33   | 70.29   | 70.92   | 68.43   | 69.85   | 71.14   | 72.03   | 72.52   | 71.29   | 71.66    | 70.68   |\n\nResults show stable performance (average accuracy ~70.39–70.83%), with later slices often achieving higher accuracy. This indicates that class imbalance has a limited impact on overall performance.\n\nWe acknowledge that uniform priors may not fully capture real-world distributions. In the revised manuscript, we will expand Section 3.2 to discuss this limitation and explore adaptive prior estimation as a future direction to enhance robustness further.\n\n[1] Bayesian Test-Time Adaptation for Vision-Language Models\n\n[2] Improving CNN Classifiers by Estimating Test-Time Priors\n\n[3] Continual Test-Time Domain Adaptation\n\n[4] Posterior Adaptation With New Priors\n\n[5] The Hitchhiker’s Guide to Prior-Shift Adaptation\n# 2.Concerns regarding the rationale for the fusion of Eq. 7 (\\($\\cos(x, w_k)/\\tau + \\lambda f_k(x)$)\n\nThank you for your valuable feedback regarding the rationale for the additive fusion strategy in Eq. 7 ($\\cos(x, w_k)/\\tau + \\lambda f_k(x)$). To address this concern, we have elucidated the heuristic yet effective design of our logit-level fusion, underscored its widespread use in the test-time adaptation (TTA) literature, and outlined future directions to strengthen the theoretical and empirical foundations of the approach.\n\n## Rationale for Logit-Level Fusion: Heuristic yet Effective\nThe additive fusion in Eq. 7 ($\\cos(x, w_k)/\\tau + \\lambda f_k(x)$) is designed to balance the robust prior knowledge encoded in the pre-trained vision-language model (e.g., CLIP) with dynamic adaptation to test-time distribution shifts. This logit-level combination is computationally efficient, aligns with the softmax-based classification framework, and leverages the complementary strengths of both classifiers, as the zero-shot component provides generalization while the test-time component refines predictions based on test data. We acknowledge that this fusion strategy is largely heuristic. However, this dynamic and incremental approach ensures minimal interference with model outputs in early stages while effectively refining predictions as Dota’s per-class modeling becomes more accurate in later stages, proving highly effective in practical scenarios.\n\n## Prevalence of Additive Fusion in TTA\nSeveral studies collectively demonstrate that additive fusion at the logit level is a widely adopted and effective strategy in TTA [6-9]. This indicates that, despite its heuristic nature, our fusion strategy aligns with approaches commonly used in the field, particularly for vision-language models (e.g., CLIP), as it efficiently leverages pre-trained knowledge while incorporating test-time information.\n\n## Future Directions: Uncertainty and Ensemble Learning\nThe current logit-level fusion can be enhanced by incorporating real-time uncertainty estimation of online samples to dynamically adjust the fusion weights. Such sample-level adaptive weighting could better balance the capabilities of zero-shot and test-time classifiers, leading to improved performance. Additionally, leveraging ensemble learning principles, we can exploit the scalability of additive fusion to integrate multiple diverse classifiers, thereby combining their strengths for more robust outcomes.\n\n\n\n[6] Efficient Test-Time Adaptation of Vision-Language Models\n\n[7] Tip-Adapter: Training-Free Adaptation of CLIP for Few-Shot Classification\n\n[8] Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations\n\n[9] Bayesian Test-Time Adaptation for Vision-Language Models\n# 3. Analysis of strategies for mitigating catastrophic forgetting in continual learning\n\nWe appreciate the reviewer for highlighting the need for a clearer explanation of how Dota mitigates catastrophic forgetting. To address this concern, we: (1) first provide a theoretical analysis of Dota’s distribution updates to elucidate its mechanism, and (2) subsequently present experimental evidence that demonstrates its superior performance compared to cache-based methods like TDA.\n\n## Theoretical Analysis\nDota employs the EM algorithm to iteratively update the mean and covariance of class-conditional distributions, integrating statistical characteristics from early test samples into global parameters (Section 3.2). Unlike cache-based methods like TDA, which discard previously stored sample information during updates, Dota’s cumulative parameter updates preserve comprehensive statistical information, effectively reducing catastrophic forgetting.\n\n## Experimental Evidence\nAs shown in Figure 3 of the paper, Dota outperforms cache-based TDA in scenarios prone to catastrophic forgetting. The experimental results demonstrate that Dota progressively enhances model performance as the number of test samples increases. In contrast, TDA exhibits an initial improvement followed by a decline, as its cache update process discards representative samples, leading to performance degradation."}, {"review_id": "SbKDXPoFJ7", "reviewer": "Reviewer_EYv3", "summary": "This paper introduces DOTA, a CLIP-based test-time adaptation approach. \nUnlike existing cache-based methods that store a finite set of instance-level test samples and are prone to catastrophic forgetting, DOTA proposes to online estimate the distribution of test data streams. \nThe adaptation is based on Bayesian posterior computation, where the class-conditional distributions are modelled as Gaussians whose parameters are updated online using zero-shot predictive probabilities. \nThe method does not require backpropagation. \nExperiments show that DOTA provides superior accuracy, reduced forgetting compared to baselines, and faster inference.", "questions": "1. Could the authors provide statistical significance results (e.g., stddev or confidence intervals) for your main results in Tables 1–3, given that some improvements over strong baselines are modest?\n\n2. Could the authors provide some insight into setting or tuning the online estimation hyperparameters (such as $\\omega$, $\\sigma^2$, $\\rho$, $\\eta$) in practice?", "limitations": "No. As required by NeurIPS's submission policy, limitations should be discussed explicitly, but remain missing in the submission. The authors are advised to discuss the limitations of the Gaussian assumption.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well prepared and easy to follow. \n2. The technical contribution of continual test-time adaptation with online distribution estimation makes sense and is interesting. \n3. The performance improvements on the standard TTA benchmark are reasonable. The authors also demonstrate better efficiency of the proposed DOTA compared to other baselines.\n\n**Weaknesses**\n\n1. Main result tables (e.g., Table 1, Table 2, Table 3) lack standard deviations or confidence intervals, which makes it challenging to assess the robustness and statistical significance of reported gains. This is required by NeurIPS's policy (See Question #7 of the Checklist) but seems to be insufficiently supported.\n\n2. Proposition 3.1 is restricted to a single EM iteration with the soft zero-shot outputs. The effects of repeated updates or deviation from Gaussianity are not analyzed.\n\n4. The Gaussian assumption for embedding distributions (Section 3.2, Eq. 2, 3) is not deeply discussed in terms of its limitations. In highly multimodal or non-Gaussian settings, performance may degrade. Also, does the inaccurate assumption of Gaussian lead to the unsatisfactory cases (e.g., Aircraft, Caltech101) in Table 1?", "was_revised": true, "final_justification": "While minor points remain, the authors' revisions have sufficiently improved the manuscript to warrant acceptance.", "rebuttal": "# 1. Statistical significance for results in tables 1–3\n\nWe sincerely acknowledge the reviewer for suggesting the inclusion of statistical significance results, such as standard deviation or confidence intervals, to strengthen the evaluation of our results in Tables 1–3. To address your suggestion, we conducted additional experiments with five random seeds (1, 2, 3, 4, 5) across the cross-domain generalization (Tables 1), NDS (Table 2), and medical scenarios (Table 3).\nThe supplementary results include baseline performance, original results (seed=1), mean accuracy across seeds, sample standard deviation, and 95% confidence intervals for each dataset.\n\n**Table 1**\n\n| **Dataset**         | **Aircraft**   | **Caltech101** | **Cars**       | **DTD**        | **EuroSAT**    | **Flowers**    | **Food101**    | **Pets**       | **Sun397**     | **UCF101**     | **Average**   |\n|---------------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|---------------|\n| Zero-Shot           | 23.22          | 93.55          | 66.11          | 45.04          | 50.42          | 66.99          | 82.86          | 86.92          | 65.63          | 65.16          | 64.59         |\n| Dota(seed=1)        | 26.25          | 94.16          | 69.56          | 47.64          | 62.78          | 75.23          | 87.08          | 92.01          | 69.8           | 72.54          | 69.71         |\n| Dota(mean)          | 26.18          | 94.59          | 69.57          | 47.88          | 62.71          | 75.32          | 87.06          | 91.91          | 69.82          | 72.69          | 69.77         |\n| stddev              | 0.26           | 0.29           | 0.17           | 0.36           | 0.28           | 0.21           | 0.06           | 0.28           | 0.07           | 0.21           | 0.22          |\n| confidence interval | [25.86, 26.51] | [94.22, 94.96] | [69.35, 69.78] | [47.44, 48.33] | [62.37, 63.06] | [75.06, 75.58] | [86.99, 87.13] | [91.56, 92.26] | [69.73, 69.91] | [72.43, 72.94] | [69.50, 70.05] |\n\n\n\n\n**Table 2**\n| Dataset                | **ImageNet**   | **ImageNet-A** | **ImageNet-R** | **ImageNet-S** | **Average**   |\n|---------------------|----------------|----------------|----------------|----------------|---------------|\n| Zero-Shot           | 68.34          | 49.89          | 77.65          | 48.24          | 61.03         |\n| Dota(seed=1)        | 70.69          | 61.5           | 81.21          | 51.84          | 66.31         |\n| Dota(mean)          | 70.66          | 61.32          | 81.13          | 51.85          | 66.24         |\n| stddev              | 0.07           | 0.2            | 0.11           | 0.11           | 0.12          |\n| confidence interval | [70.57, 70.75] | [61.06, 61.57] | [81.00, 81.26] | [51.71 ,51.99] | [66.08 ,66.39] |\n\n**Table 3**\n| **Dataset**                | **Kather**     | **PanNuke**    | **WSSS4LUAD**  | **Average**   |\n|---------------------|----------------|----------------|----------------|---------------|\n| Zero-Shot           | 45.6           | 69.49          | 70.31          | 61.8          |\n| Dota(seed=1)        | 61.92          | 74.68          | 75.07          | 70.56         |\n| Dota(mean)          | 61.94          | 74.65          | 75.13          | 70.57         |\n| stddev              | 0.14           | 0.14           | 0.15           | 0.14          |\n| confidence interval | [61.76, 62.11] | [74.49, 74.82] | [74.94, 75.32] | [70.40,70.75] |\n\n**Experiment Results Analysis**: The standard deviation remains below 0.25 for most datasets, demonstrating Dota’s robust and stable performance across different random seeds. This stability arises because Dota avoids random initialization or backpropagation-based optimization, with seed variations only affecting the data stream order, which minimally impacts performance. We will incorporate these statistical results in the revised manuscript to enhance the reliability of our findings.\n\n# 2. Recommendations for hyperparameter tuning in practice\n\nIn response to the reviewer’s request for clarification on the setting and tuning of online estimation hyperparameters ($\\omega$, $\\sigma^2$, $\\eta$, $\\rho$), we address this by detailing the initialization and tuning process for these hyperparameters, emphasizing their practical implementation and generalizability.\n\n## Hyperparameter Initialization\nThe hyperparameter $\\omega$, used to initialize the distribution of each class in Dota, is set to $10^{-4}$ to ensure uniform and small initial values across classes, minimizing interference with subsequent EM algorithm updates. In practice, this value requires no further adjustment.\n\n## Tuning Strategy for $\\sigma^2$, $\\eta$, and $\\rho$\nWe recommend first conducting preliminary tuning for $\\sigma^2$, followed by a grid search to optimize $\\eta$ and $\\rho$. Given Dota’s strong generalization across datasets, hyperparameters tuned on one dataset, such as ImageNet validation, can be directly applied to similar datasets like ImageNet-C, ImageNet-A, and eta. Owing to Dota’s lightweight and efficient design, this tuning process incurs low computational cost and is straightforward to implement.\n\n\n\n# 3. Limitations of proposition 3.1 for a single EM iteration\n\n\nThank you for highlighting the limitation of Proposition 3.1, which focuses on a single EM iteration using soft zero-shot outputs without analyzing repeated updates or deviations from Gaussianity. To address this, we provide experimental results for the Dota model under single, double, and triple EM iterations.\n| **Datasets** | **Aircraft** | **Caltech101** | **Cars** | **DTD** | **EuroSAT** | **Average** |\n|--------------|--------------|----------------|----------|---------|-------------|-------------|\n| Zero-Shot    | 23.22        | 93.55          | 66.11    | 45.04   | 50.42       | 55.67       |\n| Iteration=1  | 26.25        | 94.16          | 69.56    | 47.64   | 62.78       | 60.08       |\n| Iteration=2  | 26.34        | 94.48          | 69.67    | 48.58   | 63.79       | 60.57       |\n| Iteration=3  | 26.37        | 94.48          | 69.64    | 48.76   | 63.87       | 60.62       |\n\n## Analysis of Experimental Results\n\n### Single Iteration\n**Advantages**: Compared to multiple iterations, a single iteration results in a more lightweight and faster model, making it suitable for rapid adaptation to downstream scenarios. \n\n**Disadvantages**: This may lead to slight performance degradation.\n\n### Multiple Iterations\n**Advantages**: Compared to a single iteration, multiple iterations enable Dota to better fit each class embedding distribution, improving classification performance. \n\n**Disadvantages**: Multiple iterations require repeated updates to model parameters, with each additional iteration increasing computational time by approximately 10%. Moreover, as the number of iterations increases, performance improvements become marginal.\n\n\n# 4. Limitations of the gaussian assumption in embedding distributions\n\nWe appreciate the reviewer for highlighting the need for a deeper discussion of the Gaussian assumption’s limitations. We address this by elaborating on the theoretical justification and empirical validation of our Gaussian assumption, while acknowledging potential limitations in certain datasets.\n\n## Theoretical Justification\nOur assumption that CLIP embeddings follow a multivariate Gaussian distribution per class is supported by prior work. Chupin and Dahyot (2024) [1] demonstrate that a single Gaussian model achieves high accuracy in CLIP embedding spaces (e.g., CIFAR100: 89.0%), closely matching Gaussian Mixture Models (G=2: 89.7%), indicating that a single Gaussian effectively captures class feature clustering, driven by CLIP’s contrastive learning optimizing embeddings for intra-class proximity.\n\n## Ablation Study on Covariance Updates\nOur ablation study (Table 7) further validates this assumption: omitting covariance updates significantly reduces performance (e.g., EuroSAT: 55.06% vs. 62.78%), demonstrating the effectiveness and importance of Gaussian modeling.\n\n## Limitations of the Gaussian Assumption\nAlthough prior studies validate that CLIP embedding spaces for different classes can often be modeled by a single multivariate Gaussian distribution, performance may still be suboptimal on certain datasets. The single Gaussian assumption may not fully capture multimodal distributions, potentially contributing to suboptimal performance in datasets like Aircraft and Caltech101.\n\n\n\n\n\n[1] Performance of Gaussian Mixture Model Classifiers on Embedded Feature Spaces"}, {"review_id": "peqtMWbl2l", "reviewer": "Reviewer_Roae", "summary": "This paper presents DOTA (Distributional Test-time Adaptation), a test-time adaptation (TTA) method designed for vision-language models (VLMs), particularly CLIP, under distribution shifts occurring at inference time. Most existing approaches attempt to tackle this problem using various cache-based mechanisms. In contrast, the authors adopt a Bayesian approach grounded in probability theory to update predictions dynamically as test data evolves.\n\nDOTA assumes that the feature embeddings of each class follow a Gaussian distribution. It continuously estimates these class-conditional distributions using test data. Leveraging Bayes' theorem, DOTA computes the likelihood of each new test sample belonging to a particular class based on the updated distributions. Specifically, the method employs a Gaussian Discriminant Analysis (GDA) framework, where the class-wise means and covariances of the embeddings are estimated and updated online in a streaming setting.\n\nA key contribution is the use of a single-step Expectation-Maximization (EM) algorithm, guided by CLIP’s zero-shot prediction probabilities, to incrementally refine these class-conditional parameters with each incoming test sample. This enables DOTA to adapt effectively to distribution shifts without relying on extensive caches or replay mechanisms.\n\nExtensive experiments across 10 benchmarks validate the effectiveness of DOTA across varied test-time distribution shifts.", "questions": "1. Can the authors provide empirical evidence or analysis to support the assumption of Gaussianity in the test-time embedding distributions?\n2. Why are Bayesian and continual learning-based TTA methods (e.g., SHOT, CoTTA) not included in the comparisons?\n3. How does the method perform when the initial zero-shot predictions are noisy or poorly calibrated?\n4. How does DOTA compare to prototype-based continual learning methods, particularly in handling evolving distributions?\n5. Can the authors provide visualizations to illustrate how the estimated class distributions evolve during test-time adaptation?", "limitations": "The limitations are not explicitly mentioned.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Major strengths:\n1.  The core idea of estimating distributional statistics of test embeddings, rather than caching discrete instances as in prior methods (e.g., TDA cache-based approaches), represents a substantive conceptual advancement.\n2. The method is well-grounded in Gaussian Discriminant Analysis (GDA), with a carefully motivated EM-based estimation procedure guided by CLIP’s predictive probabilities.\n3.  The algorithm is training-free, lightweight, and naturally suited to streaming test-time scenarios, making it highly practical for real-world deployment.\n4.  The approach is thoroughly evaluated on a large number of datasets with well-designed ablation analysis, demonstrating clear advantages in both accuracy and computational efficiency.\n\nMajor weaknesses:\n1.  The paper does not provide a strong justification or empirical evidence to support the assumption that class embeddings follow a Gaussian distribution.\n2.  The approach lacks a principled basis for determining the weighting between the original zero-shot classifier and the adapted test-time classifier.\n3.  The baseline comparisons are limited—key Bayesian and continual TTA baselines (e.g., O-TPT, SHOT, CoTTA) are missing, which would have strengthened the evaluation.", "was_revised": true, "final_justification": "Thanks for the rebuttal. I have adjusted my rating to BA", "rebuttal": "# 1. Empirical evidence supporting saussianity assumption\nThank you for requesting empirical evidence to support the Gaussianity assumption of test-time embedding distributions in Dota，we believe that elucidating the rationale for this assumption will enhance the research community’s understanding of our method. We address this by providing theoretical and experimental support to validate the reliability of our Gaussian assumption.\n\n## Theoretical Support for Gaussianity\nOur assumption that CLIP embeddings follow a multivariate Gaussian distribution is grounded in prior work. Chupin and Dahyot (2024)[1] show that a single Gaussian model achieves high accuracy in CLIP embedding spaces (e.g., CIFAR100: 89.0%), closely matching Gaussian Mixture Models (G=2: 89.7%), as CLIP’s contrastive learning promotes intra-class embedding clustering, approximating Gaussian distributions.\n\n## Experimental Validation\nTo validate this assumption, we conducted experiments on the ImageNet validation set (1000 classes), extracting 50 standardized CLIP embeddings per class. We employed: \n(1) Shapiro-Wilk tests to assess marginal normality across 512 dimension，as proposed by Shapiro and Wilk (1965) for evaluating normality in small to moderate sample sizes[2];\n(2) single Gaussian fitting to compute log-likelihoods with class-specific means $\\mu_c$ and shared covariance $\\Sigma$， following the methodology described by Duda et al. (2001) for multivariate Gaussian modeling[3].\nResults show a Shapiro-Wilk pass rate of 88.91% ± 3.98% (p > 0.05), confirming strong marginal Gaussianity, and a mean log-likelihood of -1206.84 ± 93.54, indicating a reasonable fit to $\\mathcal{N}(\\mu_c, \\Sigma)$, within the  typical range (-500 to -1500). These findings support Dota’s Gaussianity assumption. We will include this experiment in the revised manuscript to strengthen our analysis. \n\n[1] Performance of Gaussian Mixture Model Classifiers on Embedded Feature Spaces\n\n[2] An Analysis of Variance Test for Normality (Complete Samples)\n\n[3] Duda, R. O., Hart, P. E., & Stork, D. G. Pattern Classification (2nd ed.).\n# 2. More baselines\nThank you for noting the absence of Bayesian and continual learning-based test-time adaptation (TTA) methods, such as SHOT and CoTTA, in our comparisons. To strengthen the evaluation of Dota’s performance, we supplemented experiments with O-TPT and CoTTA baselines, with results sourced from the O-TPT paper (ViT-B/16) and our adapted CoTTA tests, respectively. Additionally, we clarify the rationale for not including SHOT as a baseline.\n\n## Experimental Setup for CoTTA\nTo align CoTTA with our experimental setup (ViT-B/16 backbone, no data augmentation, online streaming updates), we modified CoTTA by: \n\n(1) reducing batch size from 64 to 1; \n\n(2) removing image augmentation; \n\n(3) replacing ResNet50 with ViT-B/16 to match CLIP’s architecture.\n\n### Supplementary Results\n| **Dataset** | **Flowers** | **Food101** | **Pets** | **Sun397** | **UCF101** | **Average** |\n|-------------|-------------|-------------|----------|------------|------------|-------------|\n| Zero-Shot   | 66.99       | 82.86       | 86.92    | 65.63      | 65.16      | 73.51       |\n| O-TPT       | 70.07       | 84.13       | 87.95    | 64.23      | 64.16      | 74.11       |\n| Cotta       | 71.38       | 86.1        | 89.1     | 66.29      | 66.77      | 75.93       |\n| DOTA        | 75.23       | 87.08       | 92.01    | 69.8       | 72.54      | 79.33       |\n\n**Results indicate:**\n1. O-TPT optimizes text feature orthogonality to reduce expected calibration error (ECE), primarily addresses overconfidence in model predictions, resulting in modest accuracy improvements compared to baselines.\n2. CoTTA does not significantly outperform baseline models, likely due to its parameter updates during training, which may hinder full utilization of CLIP’s original classification capabilities, compounded by architectural changes from ResNet to CLIP.\n\n## The reason why SHOT not included in the comparisons\nSHOT was excluded as a baseline because it requires access to the entire target domain data for batch training, conflicting with TTA’s requirement for online streaming updates.\n\n\n# 3. Cases in which zero-shot predictions are noisy or poorly calibrated\n\nThank you for your insightful comments regarding Dota’s performance when initial zero-shot predictions are noisy or poorly calibrated. To address this, we discuss the cases of noisy and poorly calibrated zero-shot predictions separately below.\n\n## Noisy Zero-Shot Predictions\nDota’s design leverages distribution estimation to enhance the predictive capability of the base model, even under inaccurate zero-shot predictions. Experimental results in Table 1 demonstrate that, on datasets where CLIP’s zero-shot performance is weak, Dota consistently improves accuracy, such as Aircraft (26.25% vs. 23.22%) and EuroSAT (62.78% vs. 50.42%), highlighting the robustness of our distribution-based approach.\n\n## Poorly Calibrated Zero-Shot Predictions\nThe zero-shot classifier (CLIP) used in our model has been shown to be well-calibrated by prior studies [4]. CLIP’s strong calibration effectively reduces the influence of potentially inaccurate zero-shot predictions, as samples with lower confidence contribute less to the final distribution estimation. In practical testing, even in scenarios where CLIP’s calibration is suboptimal, calibration techniques such as Temperature Scaling or FLYP can be applied first to calibrate the predictions, followed by updating Dota’s model parameters using the calibrated results.\n\n[4]Revisiting the calibration of modern neural networks\n# 4. Comparison with prototype-based continual learning methods under evolving distributions\n\nThank you for requesting a comparison between Dota and prototype-based continual learning methods in handling evolving distributions. TDA can be considered a prototype-based continual learning method as it maintains a dynamic cache of representative test-sample features (keys) and pseudo labels (values), akin to prototypes, to adapt to evolving distributions.  In this experiment , we adopt TDA as a baseline for prototype-based continual learning methods. \n\n## Experimental Setup\nTo evaluate Dota’s superiority, we conducted an experiment on ImageNet-C (severity=5, datasets: brightness, fog, frost, snow) using a ViT-B/16 backbone. The setup involves sequential testing (brightness → fog → frost → snow) without resetting model parameters. Results are shown below:\n\n| **Method** | **Brightness** | **Fog** | **Frost** | **Snow** | **Average** |\n|------------|----------------|---------|-----------|----------|-------------|\n| Zero-Shot  | 56.98          | 38.25   | 32.32     | 34.03    | 40.40       |\n| TDA        | 58.68          | 38.78   | 33.59     | 35.47    | 41.63       |\n| DOTA       | 58.99          | 40.47   | 35.11     | 36.83    | 42.85       |\n\n**Experiment Results Analysis:** Dota consistently outperforms TDA across all datasets, achieving a higher average accuracy (42.85% vs. 41.63%) and greater stability during distribution shifts, demonstrating superior robustness in handling evolving distributions compared to prototype-based methods. We will include this analysis in the revised manuscript to further clarify Dota’s advantages. Additional details are available upon request.\n\n\n\n\n\n\n# 5. Visualization of evolving class distributions in test-time adaptation\nWe sincerely appreciate the reviewer for suggesting the visualization of estimated class distributions, which can help readers gain a deeper understanding of our method’s implementation process.\n\nWe conducted experiments on the ImageNet validation set to evaluate the mean cosine similarity between class embeddings modeled by DOTA at intervals of 5000 samples. The experimental results are presented in the table below:\n\n| **Samples seen**     | **0** | **5000** | **10000** | **15000** | **20000** | **25000** | **30000** | **35000** | **40000** | **45000** | **50000** |\n|----------------------|-------|----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n| Cosine Similarity (%) | 100   | 30.25    | 28.88     | 28.53     | 28.30     | 28.18     | 28.11     | 28.05     | 28.01     | 27.97     | 27.92     |\n\n**Analysis of Experimental Results**:\n1. After updating with 5000 samples, the cosine similarity of DOTA’s class embeddings drops to a relatively low value (0.3025), indicating that our method can effectively model each class accurately with few samples, quickly adapting to downstream tasks.\n2. As the number of updated samples increases, the cosine similarity continues to decrease steadily, demonstrating that our method can sustain learning in scenarios with abundant samples, further improving the modeling distribution quality for each class.\n\n# 6. Considerations for the weighting mechanism of zero-shot and test-Time adapted classifiers\n\nWe sincerely acknowledge the reviewer for highlighting the lack of a principled basis in our fusion strategy weight control. Our dynamic fusion weight method, inspired by practical experience, demonstrates heuristic yet effective performance, as evidenced by the experimental results in Tables 1–3 of the paper. Dota employs an adaptive fusion strategy (Section 3.3) that dynamically combines predictions from both classifiers based on the number of test samples, balancing the stability of zero-shot predictions with the accuracy of test-time adaptation.\n\nThe current weighting strategy can be enhanced by incorporating real-time uncertainty estimation of online samples to dynamically adjust the fusion weights. Such sample-level adaptive weighting could better balance the capabilities of zero-shot and test-time classifiers, leading to improved performance."}]

Author rebuttal:

Meta-review: This paper presents DOTA, where a vision-language foundation model (e.g., CLIP) is adapted to address distribution shifts during test time by continuously estimating the underlying test data distribution. The reviewers appreciated the motivation of the approach, its performance and extensive evaluation. All the concerns have been satisfactorily addressed in the author response. So the recommendation is to accept the paper.
Decision: Accept (poster)