Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment
Authors: Youjia Zhang
Youngeun Kim
Young-Geun Choi
Hongyeob Kim
Huiling Liu
Sungeun Hong
Venue: NeurIPS 2025 poster ()
Keywords: Test-time Adaptation
Vision-Language Models
Distribution Shifts
Primary area: general_machine_learning

Abstract: Test-time adaptation (TTA) enhances the zero-shot robustness under distribution shifts by leveraging unlabeled test data during inference. Despite notable advances, several challenges still limit its broader applicability. First, most methods rely on backpropagation or iterative optimization, which limits scalability and hinders real-time deployment. Second, they lack explicit modeling of class-conditional feature distributions. This modeling is crucial for producing reliable decision boundaries and calibrated predictions, but it remains underexplored due to the lack of both source data and supervision at test time. In this paper, we propose ADAPT, an Advanced Distribution-Aware and backPropagation-free Test-time adaptation method. We reframe TTA as a probabilistic inference task by modeling class-conditional likelihoods using gradually updated class means and a shared covariance matrix. This enables closed-form, training-free inference. To correct potential likelihood bias, we introduce lightweight regularization guided by CLIP priors and a historical knowledge bank. ADAPT requires no source data, no gradient updates, and no full access to target data, supporting both online and transductive settings. Extensive experiments across diverse benchmarks demonstrate that our method achieves state-of-the-art performance under a wide range of distribution shifts with superior scalability and robustness.

Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment
Youjia Zhang'
Youngeun Kim²
Young- Geun Choi
Hongyeob Kim
Huiling Liu'
Sungeun Hong
"Sungkyunkwan University
Amazonn
attps://aing skku w.w...ittuttu....... github. io/ AAADAP
Abstract
Test-time adaptation (TTA) enhances the zero-shot robustness under distribution shifts by leveraging unlabeled test data during inference. Despite notable advances, several challenges still limit its broader applicability. First, most methods rely on backpropagation or iterative optimization, which limits scalability and hinders real-time deployment. Second, they lack explicit modeling of class-conditional feature distributions. This modeling is crucial for producing reliable decision boundaries and calibrated predictions, but it remains underexplored due the lack of both source data and supervision at test time. In this paper, we propose ADAPT, an Advanced Distribution-A Aware and backaropagttion- adaptation method. We reframe TTA as Gaussian probabilistic inference task by modeling class-conditional likelihoods using gradually updated class means and a shared covariance matrix. This enables closed-form. training-free inference. To correct potential likelihood bias, we introduce lightweight regularization guided by CLIP priors and historical knowledge bank. ADAPT requires source data, no gradient updates, and no full access to target data, supporting both online and transductive settings. Extensive experiments across diverse benchmarks demon- strate that our method achieves state-of- the-art performance under a wide range distribution shifts with superior scalability and robustness.
Test-time
Introduction
While Vision-language models (VLMs) such CLIP [39] can recognize diverse visual concepts using only class names, their robustness drops significantly when test-time distributions differ from those seen during pre-training [33, 55). Test-time adaptation (TTA) offers promising solution to enhance the robustness of VLMs by adapting them using only unlabeled test data [49]. From a distributional perspective, TTA helps correct the mismatch between the class-conditional feature distributions learned during pretraining and those encountered at test time. Early approaches, including prompt tuning [33, 62, 31, 42] and adapter tuning methods [11, 68, 56, 46, 61], refine additional prompts adapter layers through iterative optimization. While more parameter-effice
than full fine-tume finettunigg these methods still depend costly backpropagation, hindering their deployment in real-time or streaming scenarios [63].
Recent advances in TTA have increasingly emphasized efficiency, leading to the development of training-free free approaches. Several methods leverage memory banks constructed from test samples [21, 64] or few-shot training data to adapt classifiers based sample similarities [63, 65], but often
"This work was completed prior to author joining Amazon Corresponding author: Sungeun Hong (csehong @ skku.edu)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: Comparison with existing TTA methods.
Task Setting
Method
BP-Free
Distribution-Are
Online
Transductive
Prompt Tuning [33, 62, 31, 54]
X
Adapter Tuning [11, 68, 56, 46, 61]
X
v
X
Similarity Score [21, 64, 65, 63, 471
X
V
Transductive Learning [20, 29, 59, 69, 51]
V
V
X
V
ADAPT (Ours)
BP-Pree: No backpropagation at test time.
require tuning task-specific fusion coefficients, limiting their generalizability across tasks, Other methods [20, 29] propagate labels from text prompts to test samples via static graph construction, yet frequently depend on external datasets and involve costly iterative updates.
Alongside these trends, transductive learning [66] has also gained attention. This setting assumes full partial access the target data during inference, enabling the model capture the global structure of the test distribution. It works well in offline scenarios such as batch processing of medical scans or document classification, where the model can benefit from support sets or feature clusters. However, such methods often require complete access to target data [59, 69, 60. 58], or even source data [51, 20], making them unsuitable for strict online or streaming environments where samples arrive and are processed sequentially, one by one, without access to future data.
Despite advances in efficiency and data accessibility, existing TTA approaches still overlook how class-conditional feature distributions are structured. Many rely solely on text-based prototypes define decision boundaries [33, 46, 61, 11, 42] adjust similarity scores using task-specific scaling [21, 64, 65, 63, 47], without explicit class distribution modeling. Moreover, since TTA operates without supervision on the target domain and lacks access source data, estimating class distributions remains challenging. As a result, existing methods often fail to capture intra-class variation and inter-class confusion (see Figure 3), which leads to unstable decision boundaries and reduced robustness under distribution shifts.
In this paper, introduce ADAPT, Advanced Distribution-4 ware method for baopPaaaggttion- free Test-time adaptation. Our approach reformulates test-time adaptation as probabilistic inference task by explicitly modeling class-conditional feature distributions under Gaussian assumption. Through the use of high-confidence historical predictions, we progressively estimate class means and shared covariance matrix without relying on supervision source data. This enables closed-form. training-foe free solution that supports both online and transductive settings, without requiring additional training or task task-specific tuning (see Table 1). Notably, our closed-forn approach enables test-time adaptation via explicit, non-iterative formulas derived from probabilistic assumptions, without any iterative optimization or backpropagation. ADAPT provides surprisingly strong, backpropagation-fe frree decision boundaries and achieves efficient and robust test-time adaptation.
wware
Our contributions summarized follows:
We propose ADAPT, backpropagation free test-time adaptation framework that models class- conditional feature distributions under a Gaussian assumption, enabling one- me-pass, closed-fom adaptation without iterative optimization.
We initialize the class means from CLIP prototypes and update them using high-confidence features stored fixed-size knowledge banks, enabling source-free, training-free adaptation in both online and transductive settings.
Extensive experiments across diverse tasks and multiple benchmarks show that ADAPT out- performs prior of-te-of-theaan methods under distribution shifts. Our method also offers strong scalability, faster inference, and stable performance across various test-time conditions.
2
Related Works
Online Test-Time Adaptation of VLMss Online TTA adapts pre-trained VL.Ms to downstream tasks in streaming scenarios. Existing approaches fall into two main directions: prompt tuning and

---

adapter tuning. trompt-based methods [33, 31, 42, 71, 62, 9, 55] adapt prompts by minimizing entropy across augmented views [33, 9, 31] or aligning multi-modal features using proxy data [42]. Adapter-based approaches [11, 61, 46] inject lightweight modules into VLMSs while keeping the backbone frozen. Despite their effectiveness, these methods rely on backpropagation, incurring high computation and memory costs. Recently, training- free TTA methods have gained attention for their efficiency. Approaches like Tip-Adapter [63] and SuS-X [47] utilize class-wist wise support samples from external sources for nearest-neighhor matching. Others [21, 65, 64] adapt predictions by retrieving historical features based on sample similarity. While gradient-free, they often require careful hyperparameter tuning and lack explicit modeling of class distributions, making them sensitive to distribution shifts and prone to unstable decision boundaries. We propose a distribution -awaee aware and propagation free TTA paradigm, achieving stable and efficient zero-shot performance in challenging scenarios involving severe distribution shifts and fine-grained tasks.
Transductive Zero-Shot Learning in VLMs. Transductive learning [19, 66), where all test samples are accessible during inference, was initially explored in the few-shot learning literature for leveraging both labeled and unlabeled samples to improve generalization. By modeling the structure of the test set, it often outperforms inductive methods [30, 72, 2]. Inspired by these successes, recent efforts have extended transductive learning the multimodal domain to improve the zero-shot adaptation of VLMs. ZLaP [20] and ECALP [29] build similarity graphs over test-time features to propagate labels but rely on costly iterative refinement steps. Wang et al. [51] employs Gaussian Discriminant Analysis classifier by estimating distributional statistics from external source data. Frolic [69] adjusts predictions by learning class-wise prompt distributions inferred from target data moments. TransCLIP [59], Histo-TransCLA. [60] and StatA [58] model each class with a multivariate Gaussian and penalize the deviations from pseudo-labels and text-driven priors. While effective, these methods often require access to all or large batches of test data, limiting their use in online or streaming settings. To overcome this, we propose an adaptive method that builds class-wise Gaussian distributions using reliably accumulated high- high-confidence predictions. This supports both transductive and online adaptation, offering strong deployment flexibility.
3 Method
3.1 Preliminaries
Zero-Shot Classification with VLMs. Vision-language models, such as CLIP, consist of two parallel encoders: an image encoder 0g,() and a text encoder (-(.). Given a set of K candidate classes [tx]/= and unlabeled images {x their embeddings are computed as x; = (2,)) E Rd and (t)) E Rdd The text-derived class prototypes t, serve as fixed semantic anchors that define implicit decision boundaries in the shared embedding space. Zero-shot predictions are made by computing the cosine similarity between image x; and class prototypes tk, scaled by parameter ::
exp(tTxx.71)
Di,k
K E exp(txxxx/) x//T)
Constructed Knowledge Banks. To enable effective test-time adaptation without access to source data, we maintain a set of class-wise knowledge banks B= (B } == where each B₁ stores a small set of high-confidence samples associated with pseudo-class k. These banks act as lightweight memory modules, continuously accumulating reliable evidence from previously seen test samples. To determine which samples are sufficiently reliable for inclusion, we compute a confidence score for each test input x; based on the negative entropy of its CLIP-derived prediction $y:
Conf(x,) = -K 05.A log Úi.
(2)
We maintain a fixed-size buffer B& with capacity L (L < N) for each class. The impact of L discussed in Section 4.3. A new test sample x; with pseudo-class is inserted only if B3 is not full or its confidence score exceeds the lowest-confidence entry in Bg. This threshold-free priority-based mechanism ensures that the banks remain fresh, compact, and focused on the most informative observations. By adaptively collecting representative and high-confidence samples without supervision or backpropagation, the knowledge banks provide a reliable basis for downstream estimation of test-time class-conditional distributions.
3

---

Mi ripl +11 -0
photo of Seleses
E (xy polto MA)
Mipe
Initalize Nijo, Be)
Knowlolge Banks
Conf(x))
f
Sampleak
Len(&) =
Choffia Coofie))
Update pi
Sex exp (Bia Eyen "alks
Lower Coet
20108 M.
E the exp (Mi Ijen midhe))
Bi (E (E-1A)) x 2 "
Sample
A Caren Sarria G: Proviceas Sample O+ Lowws
Closal-Fumm Solution
Figure 1: Overview of Online ADAPT. We perform TTA by modeling class-conditional feature distributions under a Gaussian assumption with shared covariance across classes. Class means are initialized from CLIP prototypes and refined using high-confidence samples in fixed-size per-class knowledge banks. To avoid eπror accumulation, the current test sample is excluded from updates. Predictions are made via a closed-form, backpropagation- free solution. In the transductive setting, the knowledge bank is built using the top-L most confident samples per class from the full test set.
3.2 ADAPT: Backpropagation-fiee and Distribution-aware Test-time Adaptation
Backpropagation-free TTA via Gaussian Discriminant Analysis. Gaussian Discriminant Anal- ysis (GDA) is classical generative model that assigns class labels based on the likelihood of class-conditional distributions [13]. Recent studies have shown that CLIP features exhibit class- conditional clustering and can be well approximated by Gaussian distributions [51, 69, 59]. Motivated by these findings, we adopt GDA as backpropagation-fiee probabilistic classifier for test-time adap- tation (TTA). Specifically, to achieve efficient, real-time adaptation, we simply assume that features conditioned on class k follow a Gaussian distribution with a shared covariance matrix:
Σ exp (-2 (- 1 -1:))
P.A P(x:ly))= = (xx;;; Mu: Σ)
(2TT)
where μ4 and Σ denote the class mean and shared covariance, respectively. While this assumption may oversimplify real-world distributions, it enables closed-form inference with minimal computa- tional overhead, making it well-suited for real-time, cesutrce-conttrained deployment.
From the Bayes' rule, the posterior probability of the k-th class given image is P(ys P(xx) α (x; Ma: E). Assuming uniform class prior T& = K. then the Bayes opti- P(x)
mal prediction label for x; is known as the argmax of Di, over k = 1, .,.... K:
1
Uı.k w.[ + bbs where Wi Σ Hg. by =E''
24 -
H Mg.
Mk.
(4)
The estimation of μ4 (k l..... K) and E can be optimization-fret since their maximum likelihood estimators under the i.i.d. assumption are just the class-wise empirical averages and the pooled sample covariance matrix of x (proof Section A.
Correcting Online Likelihood Bias via Constructed Knowledge Banks. While GDA enables label prediction based on estimated class statistics, directly applying it in an online test-time setting results in biased likelihood estimates due to limited samples and overconfident early predictions. Inspired by prior works on transductive adaptation and regularized inference [72, 59, 58], we propose bias correction framework that minimizes regularized objective combining three components: (i) Online Negative Log-Likelihood = log P,, (ii) CLIP Prior-based Regularization R(2; 9:), and (iii) Knowledge Bank-guided Consistency Regularization R(; B), as follows:
Contine (z,, μ, Σ) = 2; log + R(z 0.) R(2;; B),
(5)
where R(2; 91) = KL (ZL(2>110)) (1))) + + K KL (N(P ENIL Σ))
(6)
R(2, 13) = Eee 0J log P3 Ejen Wizz 0;-
(7)
Here, 2; E K denotes the predicted class distribution for test input x,, and P, is the GDA likelihood vector for test sample x; defined in Eq. (3). The three terms are explained below:

---

Algorithm ADAPT: Online TTA
Algorithm 2 ADAPT: Transductive TTA
1: Input: Test data Du. class prototypes t and knowledge bank size L
1: Input: Test data D. = (x, J class proto-
types t and knowledge bank size L
2: Initialize: fa +- t
2: Initialize: fa <- t
3: for X; Dr, do
3: Compute Conf(x) for all data by Eq. (2)
4: Compute Conf(x;) by Eq. (2)
4: for B& B do
5: Update Bg with x; if high-confidence
5: Cache Top-L. confidence samples
6: Update 1e and Σ by Eq. (9)((10)
6: end for
7: Compute 2 by Eq. (8)
7: Update E and μr" by Eq. (10)-(67)
8: end for
8: Compute x" 1-1 by Eq. (8)
9:
return {
9: return zz
Online Negative Log-Likelihood: It encourages z; align with the class-conditional Gaussian likelihoods P., playing the role of an online maximum likelihood estimation objective. However, due the streaming nature of test-time inference, early predictions may be unreliable, motivating the need for additional regularization.
CLIP Frior-based Regularization: This term introduces prior over 2; using the zero-shot CLIP prediction Ds. enforcing semantic consistency between the predicted label and pretrained vision- language alignment. In addition, softly constrains the learned GDA parameters (F4g Σ) to remain close to the deriied estimates (ft: Σx)² stabilizing adaptation across distribution shifts. The balancing factor B3 controls the strength of this prior.
Knowledge Bank- monk-guided Consistency Regularization: To mitigate the bias introduced by single- sample updates in online scenarios, this term leverages the constructed knowledge banks B to provide stable, context-aware regularization. Instead of enforcing alignment solely between the current prediction 2; and its estimated likelihood, which may be unreliable due to early-stage noise or distribution shift (see Table 5), we incorporate additional supervision from reliable historical samples. Specifically, the first component enforces consistency between the GDA likelihoods P and their pseudo-labels Új within B. The second component aligns the current prediction with these pseudo-labelt weighted by cosine similarity whj max (0, f. fz) to reflect semantic proximity. This memory-base regularization stabilizes learning dynamics, corrects transient errors, and promotes smoother decision boundaries, particularly under shifting data distributions.
Closed-form Solution without Sub-iterations. In streaming scenarios, where test samples arrive sequentially and inference must be performed online, iterative optimization (e.g., Majorization- Minimization (MM) algorithms) is prohibitively expensive. As shown in Figure 1, we therefore derive a closed-form label estimator by minimizing the regularized online objective Eq, (5) that enables one-pass, sub-iteration-free inference as follows (proof in Section A.2):
Di,k exp Ji.K + EjEB, WirDj,K)
(8)
Ek 0r. exp
Ji.A + Ejes, Weylik
which fuses three sources of information: CLIP zero-shot logits ÿi. in Eq. (1), GDA predictions Ji.K in Eq. (4), and condiiooniiional consistency via the knowledge bank.
We estimate the mean vectors by taking the derivative and setting to zero, which yields (proof in Section A.2): 14 - (z,,*x,+) + EjeB, ÚjæX, + 1+3k)) + +). To enhance robustness against noisy predictions at test time and prevent early overfitting to individual samples, we exclude the current instance x; from the class mean N4 estimation. Instead, we rely on the accumulated high-confidence features stored in the knowledge bank B3 and the prior mean jty. It ensures stable and adaptive class distribution estimate that strikes a balance between efficiency and accuracy for real-time inference, while avoiding the need for iterative mean and prediction optimization. The final mean vector can be expressed using a scalar a E [0, 1] (see Section 4.3 for analysis) as:
EjeB, ÚjAX,
EjeB. Ú3A
# + aqk (1 a) where 24 Eje8, Új.k
(9)
EjeB, H4. +
We define the distribution with j from class prototypes and E from prompt variance fixed diagonal.
5

---

In high-dimensional feature spaces, directly inverting the empirical covariance matrix is often ill-conditioned and leads to biased estimates [51]. To address this, we adopt Bayesian ridge estimator [23] with shrinkage regularization to approximate the shared covariance:
K Σ NB kE
Σ
x
mX)( P E- d d((N 1) + tr(E)
(10)
Where is the feature dimension, and NB S K x L the total number of cached samples in dynamic knowledge banks B. This regularization ensures numerical stability and reliable density estimates without requiring access to source data.
Extension to the Transductive Optimization. In the transductive setting, the entire test set Dr = {x; ]x available during inference, allowing us to jointly optimize label distributions over all test instances rather than updating them sequentially. We extend the online regularized objective
in Eq. (5) to transductive objective as follows:
Luans( (2,1,, D) = Σi= of log P, E R(260)) + EE R(z;
(11)
Similar to the online case, the optimization in Eq. (11) is convex with respect to and admits a closed-form label estimator. Because each sample remains conditionally independent, the form of the solution is unchanged from the online case in Eq. (8) (proof in Section A.3).
Taking the derivative of Eq. (11) with respect to Ma and setting it to zero yields the following solution (proof in Section A.3): N4 = -E>r + 0jAX, 3/2 / 21,x + In practice, to further improve efficiency and avoid the iterative updates required by MM-like algorithms, we substitute 2; with the CLIP zero-shot predictions & in Eq. (1), yielding a one-pass estimate for class means 14g-
Új.A + B).
EN Di,EX, + EjEB, U5
Σm Vi.k Ejes, Dj.K
1 apk + (1 (1-a) He
E di, + EjEB, UjA
Di, + EjEB, 02
(12)
We apply the same covariance matrix estimation strategy as in the online setting (Eq. 10). We summarize the procedure for online test-time adaptation in Algorithm 1, and present the overall transductive adaptation process in Algorithm Notably, both procedures are fully optimization-fre relying on one-pass closed-form updates for efficient adaptation.
To enable efficient and stable adaptation in streaming settings, we exclude the current test sample x; from updating class means. This prevents noise from low- or mid-confidence predictions and avoids error propagation during early stages. Only high-confidence samples are stored in the knowledge bank for future updates. As shown in Table 8, this achieves similar performance with reduced computation. In the transductive setting, the class means μæ are estimated using the full unlabeled test set and accumulated confident samples. Instead of optimizing latent labels, apply CLIP's soft predictions y; for one-pass, closed-form estimation, This avoids iterative overhead while leveraging globally consistent signals. Aggregated soft labels offer reliable proxy, matching outperforming iterative methods in both accuracy and stability (see more details in Section A.4).
Remark.
4 Experiments
4.1
Setup
Dataset. We evaluated the proposed ADAPT on three different tasks: natural distribution shift, fine ume-grained categorization, and corruption robustness. Specifically, for natural distribution shift, we use multiple datasets including ImageNet [4], ImageNet-A [17], ImageNet-R R Image.maaeNee-VV [40], and ImageNet-Sketch [50]. The corruption robustness task is evaluted on ImageNet-C [16], which contains 15 corruption types covering noise, blur, weather, and digital artifacts. We also evaluate performance on 10 fine-grained recognition datasets: Aireraft [32], 1altech101 [8], Cars [22].
DTD [3], EAr [14], Flower102 [36], Food101 [1], Pets [37]. SUN397 [53] and UCF101 [45].

---

Table 2: Top-1 accuracy (%) comparison on natural distribution shift.
Methed
BP-free
ImageNet
ImageNet-A
ImageNet-V
ImageNet-R
ImageNes-S
OOD Avg.
Avg.
CLIP [39]
66.74
47.79
60.89
73.99
46.12
57.20
59.11
Tip=Adapter (63)
70.75
51,04
63.41
77.76
48.88
60.27
62.37
TPT [33]
68.98
54,77
63.45
77.06
47.97
60.81
62.45
DisTPT
70.30
75.00
46.80
60.65
C-TPT
68.50
51.60
62.70
76.00
47.90
59.55
61.34
DMN [65]
72.25
58.28
65.17
78.55
53.20
63.80
65.49
DPB [61]
71.91
59.63
65.44
80.40
$2.26
64.43
65.93
[46]
70.38
77.49
62.52
DynaPrompt [54]
69.61
56.17
64.67
78.17
48.22
61.81
63.37
B²T [34)
69.57
55.26
65.40
78.64
49.53
62.21
63.68
MTA [57]
70.08
58.06
64.24
78.33
49.61
62.56
64.06
TDA (21)
69.51
60.11
64.67
80.24
50.54
63.89
65.01
ZERO
69.31
59,61
64.16
77.22
48.40
62.35
63.74
AWT [70]
71.32
60.33
65.15
80.64
$1.60
64.43
65.81
RA-TTA [24]
70.58
59.21
64.16
79.68
$0.83
63.47
64.89
BCA (67]
70.22
80.72
TCA [52]
68.88
50,13
62.10
77.11
48.95
59.57
61.43
Dota (12)
70.68
61.19
64.41
81.17
$1.33
64.53
65.76
ADAPT
70.91
63.32
64.64
80.66
53.13
65.44
66.53
GDA-CLIP [51]
64.13
19.72
55.67
55.30
34,32
41,25
45.83
TranSLLIP [59]
70.30
49.50
62.30
75.00
49.70
59.13
61,36
Frolic [69]
70.90
60.40
64.70
80.70
53.30
64.78
66.00
TIMO [28]
64.63
22.06
56.40
58.47
35.96
43.22
47.50
ADAPT
71.56
63.77
65.59
80.64
$3.87
65.97
67.09
Table 3: Top-1 accuracy (%) comparison on corruption robustness.
Blur
Weather
Digital
Noise
Method
Avg.
Defo.
Glas.
Zoom
Snow
Fog
Brig.
Cont.
Blas.
JPEG
Gauss.
Shot
Impu.
CLIP [39]
24.25 15.71 24.46
22.60
33.08
31.06
37.61
55.62
17.11
13.43
33.04
33.70
13:25
14.16
13.48
25.50
TPT [33]
27.56 15.48
26.16 26.94
36.74
34.28 39.38 60.22
16.96
15.64
40.74
37.90
10.64
11.94
10.92
27.43
Online
DiffTPT [9]
25.63
16.96
26.74
25.40
35,99
34.57
39.83
59.01
17.32
17.16
38.43
35.47
12.97
13,60
13.21
27.49
TDA [21]
26.53
17.91
27.35
25.90
36.50
34.84
40.53
58.57
20.16
16.62
35.65
36.69
15.42
16.46
16.03
28.34
[65]
34.81
15.40
36.49
14.69
ADAPT
26.30
18.01
27.31
25.54
36.19
34.67
40.96
60.29
19.95
16.09
37.44
37.22
15.76
16.84
15.90
28.56
ZLaP [20]
24.88
16.13
25.77
24.36
34.43
32.63
38.56
58.42
17.53
14.21
33.72
35.52
12.83
14.03
13.27
26.42
Trans.
TransCLIP
25.35
16.40
25.53
23.22
34.58
32.47
39.65
59.04
17.72
14.76
35.22
35.53
14.82
16.11
15.60
27.07
StatA [58]
13.29
20.38
18.84
31.30
29.80
34.58 54.79
11.24
11.80
26.31
33.20
9.58
10.52
10.12
ADAPT 27.98 19.78 29.00 27.38
38.09 36.44 42.43 62.21
21.94
18.40 39.89 38.23
17.71
18.81
18.09
30.29
Evaluation Protocol and Implementation Details. We evaluate ADAPT under two TTA protocols:
Online (batch size=1) and Transductive. In the online setting, the model processes one sample at time with access to current and past inputs, following the sequential setup in [33, 21J. In the transductive setting [59, 60], all test samples are accessible at once. We use CLIP [39] ViT-B/16 as the backbone. We set the coefficient to 0.9, and assign the knowledge bank size L as 16 for online and for transductive evaluation. All experiments are conducted on an NVIDIA RTX A6000 GPU.
4.2 Main Results
Task 1: Natural Distribution Shift. As shown in Table 2, ADAPT achieves the highest average accuracy of 66.53% in the online setting, outperforming all optimization- based and backpropagation free baselines. It also surpasses strong transductive methods like TransCI.IP and Frolic, despite not accessing the full target set. In the transductive setting, ADAPT further improves to 67. 67.09%, ranking first on three out of four OOD benchmarks and showing consistent robustness under domain shifts.
Task 2: Corruption Robustness. Table reports results under variety of synthetic corruptions. ADAPT achieves the best performance with 28.56% accuracy in the online setting and 30.29% in the transductive setting, consistently surpassing all baselines, including transductive methods. These results show the robustness and adaptability of ADAPT even under severe visual degradations.
Task 3: Fine-Grained Categorization. As shown in Table 4, ADAPT outperforms the CLIP zero- shot baseline by 7.31% 11% (online) and 8. 8.98% (transductive), demonstrating strong generalization to fine-grained variations. It achieves the highest overall accuracy among backpropagation-ffee methods without labels or gradients, and remains competitive with transductive approaches using full target data. Despite using neither source data external supervision, ADAPT narrows the gap to Oracle performance within 5%, which is obtained by estimating target distributions using ground-trutt labels and full access to the test set, offering unified and practical solution for real-wor world TTA.
7

---

Table 4: Top-1 accuracy (%) comparison on fine-grained categorization.
Method
BP-free Aircraft
Caltech
Cars
DTD
EuroSAT
Flower
Food101
Pets
Sun397
UCFI01
Avg.
CLIP [39]
23.70
92.98
65. 24
44.44
41
67.28
83.80
87.98
62.55
65.08
63.45
TPT [33]
x 24.78
94.16
66.87
47.75
42.44
68.98
84.67
87.79
65.50
68.04
65.10
DITIPT [9]
X 25.60
92.49
67.01
47.00
43.13
70.10
87.23
88.22
65.74
68.22
65.47
C-TPT [55]
X
24.00
93.60
65.80
46.00
43.20
79.80
83.70
88.20
64.80
65.70
64.48
DMN [65]
30,03
95.38
67.96
55.85
59.43
74.49
85.08
92.04
70.18
72.51
70.30
TPS [29)
X
26.27
94.56
67.00
53,80
42.11
71.69
84.78
87 82
68.
71.18
66.75
28.95
94.81
67.31
54.20
55.79
75.07
86.17
70.07
70.44
69:40
Online
HisTPT [62]
K
26.90
94.50
69.20
48.90
49.70
71 71.20
89.30
89.10
67.20
70.10
67.61
DynaPrompt [54]
24.35
94.32
67.65
47.
42.28
69.95 69.95
85
88. 88.28
66.32
68.72
65.52
MTA [57]
25.32
94.13
66.36
45.59
38.71
68. 08.26
84,95
88, 22
64,98
68.11
64.46
TDA [21)
23.91
94.24
67.28
47.40
58.00
71.42
86,14
88.63
67.62
70.66
67.53
7L.aP [20]
25.40
93.10
65.60
48.60
55.60
73.50
86.90
87.10
67.40
71.50
67.47
ZERO [7]
25.21
93.66
68.04
46.12
34.33
67.68
86.53
87.75
65.03
67.77
64.21
BCA [67]
28.59
94.69
66.86
53.49
56.63
73.12
85.97
90.43
68.41
67.59
68.58
OGA [10]
23.20
93.60
68.10
47.90
$4.20
69.20
85.60
89.40
67.90
71.40
67.05
[52]
24.87
93.63
70.43
73.33
85.31
89.53
72.38
Dota [12]
V
25.59
94.12
69.48
47.87
57.65
74.67
87.02
91 .69
69.70
72.06
69.01
ADAPT
28.95
94.48
68.19
55,20
68.19
75.56
83.81
92.01
70.57
70.66
70.76
GDA-CLIP [51]
18.69
87.53
60.78
46.81
49.92
72.65
78.25
89
63.60
68.70
63.68
ZLaP [20]
26.30
91.80
66.80
46.00
57,70
67.90
87.20
87,90
67.80
73.80
67.32
Trans.
TransCLIP [59]
26.90
92.70
69:40
49.50
65.10
76.70
87.10
92.60
68.90
74.40
70.33
Frolic [69]
31.40
95.10
69.10
56.10
58.50
74.80
87.10
92.90
70.80
75.20
71.10
StatA [58]
24.70
94.20
68.00
48.40
67.30
75.20
87.10
92.40
68.70
73.50
69.95
ADAPT
30.81
95.46
71.32
56.86
65.93
80.11
85.15
92.59
72.25
73.86
72.43
Oracle ADAPT
41.88
98.26
82.89
60.87
56.51
81.93
85.74
92.61
80.04
90.14
77.09
Table Ablation study on components.
Online Task 1
Online 2
Online Task
Transductive Task 1 .
Trensductive Task 2 *
Transductive Tesk
8
Update μ Update E
Task
Task
Task
75
75
x
x
59.11
25.50
63.45
x
x
49.64
9.58
60.02
Accurcy
65
X
61.54
25.42
67.03
55
60
X
49.65
9.58
60.04
30
30
V
64.89
25.08
67.06
20
x
64.05
19.49
67.95
20
12
16
20
0.0
0.2
0.4
0.6
0.8
1.0
65.27
25.67
67.43
Bank Size L
Scalar
66.53
28.56
70.76
Figure 2: Hyperparameter analysis.
4.3 Ablation Studies and Further Analysis
Ablation Study. We ablate the key components in ADAPT. As shown in Table 5, removing the knowledge bank B causes notable performance degradation, especially when updating Σ, due to noisy statistics from test-time predictions. This confirms that our bank-guided regularization in Eq. (7) effectively mitigates likelihood bias in the online setting. With B, adapting either μ or Σ provides moderate gains, while jointly updating both yields the best results. These findings highlight the complementary roles of adaptive statistics and memory- guided estimation.
Hyperparameter Analysis. We analyze the impact of the knowledge bank size L and the coefficient in (9) and (67),, where a controls the balance between the CLIP prior and the empirical mean of confident test features. Figure 2 (left) shows that online performance is unstable when due to limited statistics, but stabilizes once L28. In contrast, the transductive setting is robust to L as all target data are available. We use L=16 for online and L=6 for transductive. As shown in Figure (right), performance steadily improves with larger a. confirming that ADAPT benefits from reliable historical samples over static CLIP priors. We set a=0.9 to balance prior knowledge and adaptivity.
small
Cost Comparison. In Table 6, we evaluate all methods under the same protocol and computational resource (NVIDIA RTX A6000) to ensure a fair comparison. In the online setting, ADAPT achieves the highest accuracy with just Ih 1m and 0.93GB, notably lower than TPT. In the transductive setting, builds the knowledge bank once and runs in single pass, requiring only 0.73 minutes and 3.37GB. These results highlight ADAPT as a practical solution balancing accuracy and efficiency.
Comparison with Different Mean Initialization çt. We study how different initializations of the mean fa in Eq. (9) affect performance. We compare 6 types of text-derived prototypes: CLIP's vanilla template, 7 generic templates, 80 ImageNet-speciff templates, GPT GPPT-generated descriptions in [70], and its variants. As shown in Table 7, diverse and descriptive initializations improve performance, though ADAPT remains robust across all settings. We use GPT-based descriptions in our main results.
8

---

Table 6: Efficiency comparison ImageNet.
Table 7: Mean initialization comparison.
BP-free
Acc (%)
(S)
4 Mem.GGB
Mean initialization A
Task
Task
Tusk
CLIP [39]
66.74
8m
0.79
TPT (33]
68.95
2.21
45m
4,29
Vanilla [39]
64.70
27.52
67.43
Online
DIFTPT 19]
70.30
3.56
> 20h
4,60
TDA 121]
69.51
2.77
50m
0.84
Ensemble [39]
66.56
28.54
67.74
ADAPT
TPS [46]
70.38
70.91
4.17
3.64
lb 11m
19m
0.93
1.21
CLIP Template [39]
66.51
28.44
67.62
GPT [70]
66.53
28.56
70.76
GDA-CLIP 181]
64,13
2.661
1.31m
10.03
Trank
TransCLIP (59)
70.30
3.56
1.34m
16.17
GPT & Ensemble
66.57
28.98
69.95
StatA 158)
69.90
3.16
1.5m
20.74
ADAPT
71.56
4.82
0.73m
3.37
GPT & CLIP Template
66.58
28.91
69.90
Table 8: Closed-form vs Iterative optimization.
Table 9: Effect of test-time input order in the online adaptation setting.
Sub-iter.
Task 2
Task
Time
Online
Iterative Closed
66.74 66.53
28.68
71.05
4h 32m
Order
Task
Task
Task
Time
28.56
70.76
1h 11m
Hard-to-Fare
65.37
27.29
68.29
1h 57m
Iterativv
66.88
27.95
73.98
0.89m
Easy-to-Hard
66.79
29.09
70.97
30m
30.29
72.43
Random (Ours)
66.53
28.56
70.76
1h 11m
CLIP
TPT
TDA
ADAPT (Online)
ADAPT (Transdacivv)
Figure 3: Visualization of decision boundaries on ImageNet-A. The colors indicate different classes.
Comparison with Iterative Optimization. To avoid costly iterative optimization, we proposed a closed-form, one-pass approximation as in Eq. (9) and Eq. (67). We evaluate it across three tasks both settings. As shown in Table 8, our closed-form solution matches the accuracy of iterative methods while achieving 4x speedup on ImageNet in the online scenario. The transductive results further confirm that our method offers an optimal trade-off between efficiency and effectiveness.
Effects of Online Input Order. We study how test-time input order affects online adaptation, comparing random (default), hard-to-easy, and easy-to-hard confidence (Eq. (2)) orderings. As shown in Table 9, easy-to-hard ordering performs best, as early high-confidence samples help build reliable class statistics. Despite this, our default random order still yields competitive results, showing the robustness and practicality of ADAPT without relying on input scheduling.
Visualization. Figure illustrates the decision boundaries different TTA methods. CLIP, TPT, and TDA rely on feature similarity, lacking explicit class distribution modeling. Their boundaries are thus irregular and unstable near class overlaps, leading to high intra-class variance and poor separability. In contrast, ADAPT builds a Gaussian- based model that explicitly estimates the class distribution without training, yielding more compact clusters and smoother boundaries.
5 Limitation and Conclusion
Limitation. Our method provides closed-form, backpropagation-free updates, making suitable for scenarios with low latency or limited resources. This efficiency comes from modeling class- conditional features using a Gaussian assumption with shared covariance, While this strategy performs well across diverse benchmarks, it may struggle to capture complex multi-modal structures in challenging real-world data. Exploring more flexible distribution models, such as Gaussian mixtures or pon-maraettiic estimators, without compromising efficiency remains promising direction.
Conclusion.
We propose ADAPT, distribution-awan awaree and backpropagation free TTA framework. Unlike prior methods that rely on gradient updates or iterative optimization, ADAPT performs efficient, training-free adaptation by estimating class-conditional distributions from test-time features using closed-form updates. Extensive experiments show that ADAPT is robust and scalable, making it suitable for deployment in dynamic or resource-constraine settings.

---

This work was partly supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government(MSIT) (No.RS- 2025-0221 7960, Development of a human-oriented Al model that imitates the human growth process based cognitive science). This research was partly supported by the MSIT(Minnstry of Science, ICT), Korea, under the Global Research Support Program in the Digital Field program(RS-2022 00425354) supervised by the (Intitit for Information & Communications Technology Planning & Evaluation). This research was partly supported by the MSIT(Ministry of Science and ICT), Korea, under the Graduate School of Metaverse Convergence support program(1TTP20055 (000T--2022-R-22223 00254129) supervised by the ITTP(Istittte for Information & Communications Technology Planning Evaluation).
Acknowledgement.
References
[1] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101-Mining Discriminative Components with Random Forests. In ECCV, 2014.
[2] Malik Boudiaf, Imtiaz Ziko, Jérôme Rony, José Dolz, Pablo Piantanida, and Ismail Ben Ayed. Information maximization for few-shot learning. In NeurlPS, 2020.
[3] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing Textures in the Wild. In CVPR, 2014.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li. and Li Fei-Fei. Imagenet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
[5] Sara El Bouch, Olivier Michel, and Pierre Comon. A normality test for multivariate dependent samples. Signal Processing, 201:108705, 2022.
[6] Sara ElBouch, Olivier JJ Michel, and Pierre Comon. Joint normality test via two-dimensional projection. In ICASSP, 2022.
[7] Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, and Elisa Ricci. Frus- tratingly easy test-time adaptation of vision-language models. In NeurIPS, 2024.
[8] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. In CVPRW, 2004.
[9] Chun-Mei Feng. Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmenta- tion with diffusions for effective test-time prompt tuning. In ICCV, 2023.
[10] Clément Fuchs, Maxime Zanella, and Christophe De Vleeschouwer. Online gaussian test-time adaptation of vision-language models. arXiv preprint (rXXi:25001.43522 2025.
[11] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. IJCV, 132(2), 2024.
[12] Zongbo Han, Jialong Yang, Junfan Li, Qinghua Hu, Qianli Xu, Mike Zheng Shou, and Changqing Zhang. Dota: Distributional test-time adaptation of vision-language models. arXiv preprint arXiv:2409. 19375, 2024.
[13] Trevor Hastie and Robert Tibshirani. Discriminant analysis by gaussian mixtures. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):155-176, 1996.
[14] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-
2226, 2019.
[15] Dan Hendrycks, Steven Basart, Norman MU, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The Many Faces of Robustness: Critical Analysis of Out-of- Disttribtttt Generalization. In CVPR, 2021.
10

---

[16] Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR, 2019.
[17] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural Adversarial Examples. In CVPR, 2021.
[18] Norbert Henze and Bernd Zirkler. A class of invariant consistent tests for multivariate normality. Communications in statistics-Thett Theory and Methods, 19(10):3555-3617, 1990.
[19] Thorsten Joachims. Transductive inference for text classification using support vector machines. In ICML, 1999.
[20] Yannis Kalantidis, Giorgos Tolias, et al. Label propagation for zero-shot classification with vision-language models. In CVPR, 2024.
[21] Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb EI Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In CVPR, 2024.
[22] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D Object Representations for Fine-Grained Categorization. In CVPRW, 2013.
[23] Tatsuya Kubokawa and Muni S Srivastava. Estimation of the precision matrix of a singular wishart distribution and its application in high-dimensional data. 1906 2008
[24] Youngjun Lee, Doyoung Kim, Junhyeok Kang, Jihwan Bang, Hwanjun Song, and Jac-Gil Lee. Ra-tta: Retrieval-augmenne test-time adaptation for vision-language models. In /CLR, 2025.
[25] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In ICML, 2022.
[26] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align Before Fuse: Vision and Language Representation Learning with Momentum Distillation. In NewrIPS, 2021.
[27] Tao Li, Shenghuo Zhu, and Mitsunori Ogihara. Using discriminant analysis for multi-class classification: an experimental investigation. Knowledge and information systems, 10:453-472, 2006.
[28] Yayuan Li, Jintao Guo, Lei Qi, Wenbin Li, and Yinghuan Shi. Text and image are mutually beneficial: Enhancing training-free few-shot classification with clip. In AAAI, 2025.
[29] Yushu Li, Yongyi Su, Adam Goodge, Kui Jia, and Xun Xu. Efficient and context-aware label propagation for zero-/few-shot training-free free adaptation of vision-language model. In JCLR, 2025.
[30] Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In ICLR, 2019.
[31] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vasion-language models. In NeurlPS, 2023.
[32] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Grained Visual Classification of Aircraft. arXiv preprint 30. 2013.
[33] Shu Manli, Nie Weili, Huang De-An, Yu Zhiding, Goldstein Tom, Anandkumar Anima, and Xiao Chaowei. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurlPS, 2022.
[34] Fan'an an Meng, Chaoran Cui, Hongjun Dai, and Shuai Gong. Black-box test-time prompt tuning for lasion-language models. In AAAI, 2025.
[35] Alicia Reyto- Juan Antonio Cuesta- Aaastt-Albetto,, and Fabrice Gamboa. A random-projectio based test of gaussianity for stationary processes. Computational Statistics Data Analysis 75: 12:114-141, 2014.
11

---

[36] Maria-Elema Nilsback and Andrew Zisserman. Automated Flower Classification over Large Number of Classes. In ICVGIP. IEEE, 2008.
[37] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and Dogs. In CVPR, 2012.
[38] Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. University of Denmark, 7(15):510, 2008.
Technical
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[40] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
[41] J Patrick Royston. An extension of shapiro and wilk's w test for normality to large samples. Journal of the Royal Statistical Society: Series C (Applied Statistics), 3 31(2):155-124 1982.
[42] Jameel Hassan Abdul Samadh, Hanan Gani, Noor Hazim Hussein, Muhammad Uzair Khattak, Muzammal Naseer, Fahad Khan, and Salman Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. In NeurIPS, 2023.
[43] Shaphiro and MBJB Wilk. An analysis of variance test for normality, Biometrika, 52(3):591- 611, 1965.
[44] Houssem Sifaou, Abla Kammoun, and Mohamed-Slim Alouini. High-dimensional linear discriminant analysis classifier for spiked covariance model. Journal of Machine Learning Research, 21(1 12):1224, 2020.
[45] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCFI01: A Dataset of 101 Human Actions Classes from Videos in the Wild. arXiv preprint
2012.
[46] Elaine Sui, Xiaohan Wang, and Serena Yeung-Levy. Just shift it: Test-time prototype shifting for zero-shot generalization with vision-language models. In WACV. IEEE, 2025.
[47] Vishaal Udandarao, Ankush Gupta, and Samuel Albanic. Sus-x: Training-fre name-only transfer of vision-language models. In /CCV, 2023.
[48] Raquel Urtasun and Trevor Darrell. Discriminative gaussian process latent variable model for classification. In /CML, 2007.
[49] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
[50] Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning Robust Global Representations by Penalizing Local Rredictive Power. In NeurlPS, 2019.
[51] Zhengbo Wang, Jian Liang, Lijun Sheng. Ran He, Zilei Wang, and Tieniu Tan. A hard-to-beat baseline for training-free clip-based adaptation. In ICLR, 2024.
[52] Zixin Wang, Dong Gong, Sen Wang, 7i Huang, and Yadan Luo. Is less more? exploring token condensation as training-free adaptation for clip. In ICCV, 2025.
[53] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun Database: Large-Scale Scene Recognition from Abbey to Zoo. In CVPR, 2010.
[54] Zehao Xiao, Shilin Yan, Jack Hong, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiayi Shen, Qi Wang, and Cees GM Snock. Dynaprompt: Dynamic test-time prompt tuning. In ICLR, 2025.
[55] Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark Hasegawa-Johnson, Yingzhen Li, and Chang Yoo. C-tpt: Calibrated test-time prompt tuning for vision-languan models via text feature dispersion. In ICLR, 2024.
12

---

[56] Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang, Task residual for tuning vision- language models. CVPR, 2023.
[57] Maxime Zanella and Ismail Ben Ayed. On the test-time zero-shot generalization of vision- language models: Do we really need prompt learning? In CVPR, 2024.
[58] Maxime Zanella, Clément Fuchs, Christophe De Vleeschouwer, and Ismail Ben Ayed. Realistic test-time adaptation of lanion-language models. In CVPR, 2025.
[59] Maxime Zanella, Benoît Gérin, and Ismail Ayed. Boosting vision-language models with transduction. In NeurIPS, 2024.
[60] Maxime Zanella, Fereshteh Shakeri, Yunshi Huang. Houda Bahig, and Ismail Ben Ayed. Boosting usion-language models for histopathology classification: Predict all at once. In J. Multivar. Anal., 2024.
[61] Ce Zhang, Simon Stepputtis, Katia Sycara, and Yaqi Xie. Dual prototype evolving for test-time generalization of vision-language models. In NeurlPS, 2024.
[62] Jingyi Zhang. Jiaxing Huang, Xiaoqin Zhang, Ling Shao, and Shijian Lu. Historical test-time prompt tuning for vision foundation models. In NeurIPS, 2024.
[63] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In ECCV, Springer, 2022.
[64] Taolin Zhang, Jinpeng Wang, Hang Guo, Tao Dai, Bin Chen, and Shu-Tao Xia. Boostadapter: Improving vision- language test-time adaptation via regional bootstrapping. NeurlPS, 2024.
[65] Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision-language models. In CVPR, 2024.
[66] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Berhard Schölkopf. Learning with local and global consistency. In NeurIPS, 2003.
[67] Lihua Zhou, Mao Ye, Shuaifeng Li, Nianxin Li, Xiatian Zhu, Lei Deng, Hongbin Liu, and Zhen Lei. Bayesian test-time adaptation for vision-language models. In CVPR, 2025.
[68]
Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. In ICCV, 2023.
[69] Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Enhancing zero-shot vision models by label-free prompt distribution learning and bias correcting. In NeurIPS, 2024.
[70] Yuhan Zhu, Yuyang Ji, Zhiyu Zhao, Gangshan Wu, and Limin Wang. Awt: Transferring vision-language models via augmentation, weighting, and transportation. In NewrIPS, 2024.
[71] Yuhan Zhu, Guozhen Zhang, Chen Xu, Haocheng Shen, Xiaoxin Chen, Gangshan Wu, and Limin Wang. Efficient test-time prompt tuning for vision-language models. arXiv preprint 20X.5.2408.05775 2024.
[72] Imtiaz Ziko, Jose Dolz, Eric Granger, and Ismail Ben Ayed. Laplacian regularized few-shot learning. In ICML, 2020.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
This work approaches test-time adaptation (TTA) from a probabilistic perspec- tive and introduces ADAPT, distribution-awan and backpropagation-Fii method tailored for vision-languen Isionnlanguage models (VIMSs). The proposed approach theoretically justified under both online and transductive settings. Extensive experiments confirm that ADAPT achieves strong trade-off between accuracy and efficiency, aligning well with the claims made in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes "Limitation and Conclusion" section in the main paper, which reflects on the theoretical assumptions underlying the proposed method and the potential limitations in practical scenarios.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
14

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: This paper contains some formulas and inferences, the proofs of which are given in the appendix. All theoretical results are supported by empirical evidence.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofss provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides implementation details in the "Experiments" section, including training configurations such as hyperparameters and hardware setup. It also outlines the evaluation protocol for both online and transductive test-time adaptation and specifies the datasets used. These details are sufficient to reproduce the main experimental results and verify the core claims.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets we have used are all publicly available, including the data splits. The code will be fully publicly available upon acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/gun heaiic/ggiis////e htte://iid///ecddiiieeeeee (CodeSummissionF ollicy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wwtppbccccccc/ccccc /itii//uudie//oouuubbiiiiii CodeSubmisse wwwsssssiiiooooeeeoooooeiiiiipii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details explicitly reported in the "Experiments" section of the main paper, including hyperparameters and data splits.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Docs the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The paper reports mean aceuracy across three independent runs for main
results and ablation studies, providing insight into the robustness of the method. In addition to empirical top-I accuracy, the paper includes closed-form expressions and corresponding theoretical upper bounds, further supporting the validity of the reported performance.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the "Experiments" section, the paper explicitly states that all experiments were conducted on an NVIDIA RTX A6000 GPU. It also includes a runtime memory comparison across methods, offering transparency regarding computational resources and efficiency.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///neuipppp wttpppppp..ppppiaae httppppwwww..piiciiiiieieeeeeeee
Answer: [Yes]
Justification: We have carefully reviewed the Code Ethics and affirm that our work fully complies with its principles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
17

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
The paper primarily focuses on algorithmic improvements for test-time adapta- tion in vision-language models and does not explicitly discuss broader societal impacts. The proposed method improves efficiency and robustness under distribution shifts. This may benefit real-world applications such as medical imaging or autonomous systems. We do not foresee any direct societal risks introduced by our method beyond those already inherent to the use of vision-language models.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe the contributions of this paper do not pose a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
18

---

Answer: [Yes]
Justification: Yes, the paper properly credits the original creators of all assets used, including datasets, models, and code. Each benchmark dataset (e.g., ImageNet, ImageNet-A, etc) is cited appropriately with references to the original papers. The use CLIP and other pre-trained models respects their respective licenses, and the terms of use acknowledged. Additionally, any external codebases used for baselines or comparisons are credited in the main text supplementary material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow/coo///ooooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-par re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The work builds entirely upon existing publicly available resources (e.g., CLIP benchmark datasets) and focuses on a new algorithmic method. Therefore, the question of documenting new assets does not apply. However, all implementation details of the proposed method clearly described, ensuring reproducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
19

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We use GPT-based descriptions proposed in prior work solely for comparative analysis in ablation studies. These descriptions are not generated by us, nor are they an original or non-standard component of our method. Our core methodology does not involve the use of L.LMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https /ttpr///neeu../eeeeeee cc/Conferences/22222 for what should or should not be described.
20

---

Technical Appendices and Supplementary Material
This appendix provides detailed theoretical analysis of our method, along with additional experi- mental results. The contents are organized as follows:
Appendix Theoretical Analysis
A.1 Proof of Gaussian Discriminant Analysis
A.2 Online Test-time Distribution Estimation
A.3
Transductive Test-time Distribution Estimation
A.4
Limitations and Discussion
Appendix Additional Experimental Results
B.1
Experiments with CLIP-RNSO Backbone
B.2
Few-shot Adaptation
B.3
Evaluation with Different VLMs
B.4
Additional Analysis
A Theoretical Analysis
A.1 Proof of Gaussian Discriminant Analysis
This subsection provides the Gaussian Discriminant Analysis (GDA) decision rule under class- conditional Gaussian assumptions and serves the theoretical justification for Eq. (4) in the main paper. Specifically, we derive the discriminant function of GDA under the assumption that class- conditional distributions follow multivariate Gaussians with a shared covariance matrix:
P(x Nxxx; M4. Σ)
< cxp (- (x
(-
M4) M (x ua))
(13)
2mT)
Using Bayes' rule, the posterior probability can be expressed as: P()x|x) P(xlus ( P(yx)
P(x)
x
Tx;(x; p4: Σ), which reformulates the K-way classification problem into maximum likelihood estimation task. Considering that the target source domain generally Hass-balanced, we set the
prior class distribution to be uniform: P(y)) 1
K Then, we can derive the GDA classifier by
maximizing the likelihood following:
1
1 (x (2)) - (x 13x)) + log T
log.N(x; 14, E) nTk log
exp
(14)
v
/(2T)
Σ
d
1
1
log 2m
log
[>]
(x
u²) T --1((x - m4) + log Ti
(15)
2
2
2
d
1
1
log 2T
log
Σ
x
- + u1 -
(16)
2
2
2
2 -
Σ
log Tk
(17)
mg -
2 14 Σ
14
C
(18)
(a)
14 / # -ー1 14
T
(19)
w by-
(20)
(a)
holds as we can remove all constant terms C log TT 1 llg 2T - log Σ - x E x.
Then, the GDA prediction can be expressed as: Øi.A w x; + bb. where = Σ #c. by =
/o m Σ pk..
21

---

A.2 Online Test-time Distribution Estimation
While GDA offers a principled framework for label estimation using class-conditional statisties, its direct application in the online setting may result in biased likelihood estimates particularly in early stages where data scarce and predictions tend to overconfident, To address this, we construct regularized objective (Eq. (5) in our main paper) that integrates three complementary sources of information: (i) the online negative log-likelihood, = log I:; (ii) prior regularization term based on CLIP zero-shot predictions, R(2 it); and (iii) consistency regularization term guided by the knowledge bank, R(;; B). We formulate the full objective as follows:
Lontine (Zi, ,r, E) -2g log Pi + R(z; 0s) + R(2;; B)
(21)
24 log P. (L(=:l):)) + 3 Σ KL (Ni Ex)I E))
(22)
EJEB 0j log E jEB Wasz Úz.
(23)
For the third term, we assume that follows Gaussian distribution No Nis,, Ex). According
Matrix Cookbook [38] and [58], the following identity holds: IENG [[<: mx) 2 px)]
(px- us)) Σ- (As pk) + Tr(==111
Based on this, the KL divergence between the distributions No = (ir,, Σx) and Ni = (u,, Σ)) is given as:
(A E Σ)) = No(z)) log No (z)
dx = ENollogNN((r) - log.Ni (z)]
(24)
Σ
sx (xx) Sxg (x fax) + 1 _ ux)
=
ENA
log E
T (x _
mxx)
(25)
2
log
[
ENo
(x (ux) Sxz (x - jsm) + ENo [(x - uu) " - - mill
-mai]]
(26)
2
|>
(b) 1
log
Σ>
d + (āx - 1x) T T (fu - mn) + Tr(E (Su)
(27)
2
where df the feature dimension and holds as:
ENo [( T (x [ao]
ENo Tr (Ir fxx) - (ax)) T EEE ]
(28)
-TrEENG - (#xx= - p ET
(29)
-T(SS
(30)
Tr (Uk)
(31)
d
(32)
22

---

A.2.1 Estimation of Class Mean
To enable efficient distribution estimation in streaming scenarios where test samples arrive sequen- tially, we estimate the class means μ4 by taking the derivative of Conline w.r.t. pé:
oolline
0
(2f log Pi) + Ø
(B KL(N(A E E)))
(33)
0pk
0pp
Opc
0
(EjEB U/ log P,))
(34)
0ph2
0
1
(zi log(
(2 exp( 2x(x= (x: 14x) /x)/ - 1x)))))
(35)
0pp
0
+ 0p
(BKL(N Ex)( Σ)})
(36)
(EjeB, 07 log( ((27)/E exp( 1 1xxT (x,+(x))+,)))
(37)
0p4k
= -i,k E
[xx: - -14) + (SKI(N E6) Σ)))
(38)
0pk
-E JEE 11ik2-16 - -wa)
(39)
(
2 (xx 14x) - -EEE, Új.k E 1x; - 14xz)
(40)
1
|3]
+
B
(log
d + (ir - -Ha) T - (ba) +
(41)
0ppk
2
[Σ
kikk Σ 1 (x: - (4)) - EjEB, Dj Σ (x, - (bx) - ------1- - mk).
(42)
We obtain from Eq. (27). And, setting the partial derivative to zero, we can get:
21,:X; + EjeB Új.X, + Bi f1
(43)
21 + + J1,k + B
(d) EjeB, Uj.KXj + 8i4
(44)
ΣjeB, 8s.k + 3
14
(45)
To enable efficient one-pass, closed-form online inference without iterative updates, we estimate the class mean by excluding the current instance x; in (d) helping prevent early overfitting and eliminating sub-iteration. The final mean vector is then expressed as weighted combination with a scalar a E [0, 1]:
EjeB, Új,83 1 + apk' + (1 - a)p where u
EjeB, Új.2 a E₁EB, 83.k + B
(46)
EjeB, Ú5.8
A.2.2 Proof of Our Closed-form Solution
We propose closed-form label estimator (Eq. (8) in our main paper) for streaming scenarios by minimizing regularized objective in Eq. (21). To derive one-pass solution suitable for online inference, we compute the partial derivative of the objective with respect to the soft label assignment 24. Since the Laplacian regularization term in R(2 D.) is concave, we incorporate the simplex
23

---

constraint 2, A K to ensure that the solution lies within the probability simplex.
Lonline
log P;) + d (-2
(T(2s|lūs)
(47)
Ōzi
Ōzi
dzi
①
Ejes Wii 2, D3 +
8
21
(48)
Uzin
ØELK
=loog Pr. + log 21,2 - log Di.k
21,
Σ
wiyUskk + A
(49)
JEBk
- + log Zi,k log Vi. +
Wifūs,R
Ai.
(50)
JEBr
The first term in Eq. (49) is replaced by -s. because we can obtain Yi,k x log Pi, from Section A.I. And, setting the partial derivative to zero, we can get:
24,b
Di,k
exp
Vi.k + EjeB, Wiyde, (1 (1+1i))
(51)
s.t.
2 =
(52)
We can get exp(-(1 + Xx)) =)))= dh explUix + EJEB, wiyūs,)). Bring into Eq. (51),
can obtain the final optimal solution:
D1,K - exp (iix + EJEB, wer(s..))
(53)
2i,
Ex Di.e exp Uū Ejer, WiyHihh
A.3 Transductive Test-time Distribution Estimation
In the transductive setting, the entire test set D., = {x, is available during inference, allowing us to jointly optimize label distributions over all instances rather than updating them sequentially. We extend the online regularized objective in Eq. (21) to transductive objective as follows:
Crrons p1 E) = - E 2 log P. + ΣM R(z D.) + Σ B)
(54)
EN 25 P. + ΣN KL (1) + 3 BK KL (N( (N(H
Σ)
(55)
EjeB 0J log - EN E wirzT Úz
(56)
Similar the online case, we obtain the closed-form label estimator by taking the derivative of Luans w.r.t. zi
ALuans
(- List 2T log P: + E R(zE()) + Ein R(2;:
(57)
0zi,k
0zikk
eontine
(-z,] log IP + R(z; 0:) + R(z;;) B)) Ozik
(58)
Zzikk
With the results of Section A.2.2, we have:
D exp Thi + EyEB, wirDs,k)
2i,k
(59)
K Ek= Di. exp (iLN + EjeB, wiyDin
24

---

Then, br is estimated by taking the derivative of Luans w.r.t. Ma:
Uuans
0
-ii
of log Pi) +
a
(B> (L(N E 2))
(60)
0pp
ppp
EjEn D/ log
(61)
--
=
a
(2, log +
a 3rt
(3 (BE KL(V(A ER)IVV Σ)))
(62)
0
CjER D3 log
(63)
0pp
- Ei,k Ei E (x: m4) - 3E
- - - -1%) Ejes, 15 -12)-
(64)
Setting the Eq. (64) zero, we have:
E 2 + Σ jEB, U5,kX, + 8pa
Pk 2i Ri,k + EjEB, 0j.2 + ()
(65)
(c) E 1= Dix, + E JEB Dy,NXj + +Bu pig.
(66)
Em di EjeB. Dj +
In practice, to further improve efficiency and avoid the iterative updates required by MM-like algorithms, obtain [ by substituting 2 with the CLIP zero-shot predictions ÿt. yielding one-pass estimate for class means:
EE D4AX, + EjEB, D1 Ei= Di t Hi.
E H1.k EjEB, Új.A
μi + auk + (1 - a) 11
Ei=1 Di,k +jjB, Di +B
(67)
A.4 Limitations and Discussion
Limitation. Our method assumes that class-conditional features follow Gaussian distributions with shared covariance matrix. While this assumption simplifies the underlying data distribution and may not fully capture complex, multi-modal, or highly skewed patterns in real-world scenarios, it enables key practical benefit. Specifically, the Gaussian assumption allows us to derive closed-form: backpropagation-fig -ree updates for both online and transductive test-time adaptation. This property is important for deploying vision-language models in environments that require low latency or have limited computational resources. Examples include mobile devices, robotic systems, and real- time inference settings, where gradient-basee based optimization is often infeasible.
Discussion. To verify the core assump- tions of our method that class-conditional features follow Gaussian distributions with shared covariance matrix, we conduct two empirical analyses.
Table 10: Projection based normality test results across class-conditional features.
Low-dim
Freq of p>0.05 (%) t p-value Avg.
Henze Honze-Zirkler Zirkler
2
100
0.39
First, examine the Gaussianity of class- conditional features, we adopt a projection- based testing strategy inspired by prior sta- tistical literatures [35, 5, 6]. Classical mul- tivariate normality tests [18, 43, 41] are known to break down when the feature di- mension large relative to the sample size (e.g., 512/50 > 0 in our case using CLIP- ViT/B-16 on ImageNet). To address this, we project high-dimensional data into low- dimensional subspaces by randomly select- ing small number of feature dimensions
99.90 4
0.32
99.00 6
0.27
96.30 8
0.22
10
92.90
0.19
Shapiro- Wilk
2
100
0.31
4
100
0.21
6
99.50
0.16
8
96.30
0.13
92.20
0.11
25

---

20
70
25
Covariance Dissimilarity
Number of Classes
60
70
50
Class
40
15
30
10
0.02 0.04
0.08 8.10 0:14 0.16 0.18 0.20 Frobenius Distance
29
Class
Figure 4: Comparison of covariance properties ImageNet: (Left) shows Frobenius distance
between class-wise Σg and shared Σ, and (Right) shows class-wist wise covariance dissimilarity.
class, and repeat this procedure 100 times. For each projected subset, we apply the Henze test [18] and Shapiro- Wilk test [41] for normality and report the average p- p-values across repetitions. As shown in Table 10, the average p-values frequently exceed 0.05, suggesting that class-conditional CLIP features are approximately Gaussian in projected subspaces. Furthermore, even when the data deviates from strict Gaussianity, our method remains effective evidenced in Table 13, where the-hhhrt art performance. Existing works [27, 44, 48] also support the robustness of Gaussian discriminant analysis (GDA) against slight normality violations.
consistently achieves state-of saate-of-the-ardd
Second, we assess the validity of the shared covariance assumption. For each class, we compute its empirical covariance matrix and compare it to the pooled covariance using the Frobenius norm. As shown in Figure (left), over 99.3% of the classes exhibit Frobenius distance smaller than 0.1 from the pooled covariance, indicating strong alignment. Additionally, we visualize covariance matrices from several randomly selected classes in Figure 4 (right), and further quantify dissimilarity by comparing the trace values of class-wise covariance matrices. These heatmaps reveal consistent structural patterns across classes, providing qualitative evidence for the shared covariance structure. In summary, these findings collectively provide strong empirical support for the Gaussianity and shared covariance assumptions underlying our method.
To further compare with class-wise separate covariance matrices, we con- ducted additional experiments on Ima- geNet by replacing our shared covari- ance with class-specific covariance matrices, keeping all other settings identical. The results, summarized in Table 11, show that this leads to a sig- nificant drop in accuracy, alongside a sharp increase in computation time and memory usage. We attribute this performance degradation data sparsity. Estimating full covariance matrix for each class becomes unreliable with few high-confidence samples, leading to poor generalization and overfitting. In contrast, the shared covariance pools information across all classes, resulting in a much more stable and robust estimate, which is critical in online or data-scarce settings. Ultimately, this ablation confirms that our shared covariance assumption not limita- tion, but a crucial design choice. It strikes an effective trade-off between accuracy, robustness, and computational efficiency, making our method practical for real-world, eesourceconstrraieed scenarios.
Table 11: Evaluation with class-wise separate covariance matrices on ImageNet.
Method
Time 4
Ace (%)
Gain (%)
Mem.(GB)
CLIP
8m
66.74
0.79
ADAPT (Online)
1h 11m
70.91
+4.17
0.93
w/ Separate Cov.
1h 44m
53.80
-12.94
2.89
ADAPT (Trans.)
0.73m
71.56
+4.82
3.37
w/ Sepurate Cov.
1.40m
66.16
-0.58
4.62
Remark. To balance efficiency and stability streaming scenarios with evolving data distributions, we exclude the current test sample from online class mean updates. This decision is based on three key observations: (i) high-confidence samples are stored in knowledge bank and will be incorporated later; (ii) low-confidence predictions contribute little and add noise; and (iii) moderately confident predictions are prone to errors, distorting class statisties. By omitting X;, we prevent carly-stage prediction errors from propagating into the model. As shown in Table 8 (see main paper),
26

---

this approach offers significant computational savings with minimal accuracy loss, especially in the early stages of high uncertainty. In the transductive setting, we estimate class means Nr using the entire test set and accumulated high-confidence samples. Instead of relying on latent assignments " optimized through iterative procedures (as in MM-based methods), we replace them with CLIPPs zero-shot predictions Ù;- This design is motivated by three factors: (i) it avoids costly sub-iterations, enabling a one-pass, closed-form estimation of μ4i (ii) with all test samples available in advance, computing Di for the entire set is efficient and supports globally consistent estimation; (iii) although noisy, CLIP's soft predictions provide a reasonable proxy for class membership when aggregated. As shown in ablations (Table 8 in the main paper), this heuristic achieves comparable or better performance than MM- based optimization, while being more stable and computationally efficient.
Comparison with Other Distribution Estimation Methods. We compare existing methods for estimating test-time distributions and demonstrate their limitations or inapplicability under the realistic online setting.
GDA-CLIP [51]: This method estimates the class-wise distribution by computing the empirical mean and inverse covariance from an external training set D3 = (x, (x The class mean is calculated as:
EjED, I,);
11 ΣJED, ((,==))
(68)
Although the covariance is estimated using an empirical Bayes ridge-type estimator, this ap- proach performs well only when the test distribution closely resembles the source data in its statistical properties. Crucially, it inherently assumes access a labeled and stationary source distribution, which is unavailable in the TTA setring. As result, it fails to adapt under distri- bution shifts, and the estimated statistics can quickly become outdated or misaligned with the continuously evolving test data stream.
Frolic [69]: This method directly adopts CLIP's class prototypes class-wise means, i.e., My tg. and estimates the shared covariance E from the marginal distribution via the expectation and second-order moment, follows:
Σ
Σ X VED
(69)
However, this method requires complete access to the entire target dataset D, )x it unsuitable for online or streaming scenarios where test samples arrive sequentially and future data is inaccessible.
-== making
TransCLIP [59]: This method estimates the parameters f4 and Σ via a Majorize-Minimize (MM) optimization procedure, which iteratively refines the predicted soft-assignments and the distribution parameters:
ΣiED, 2i,88i Pk
(70)
EiED, Zi,
diag ()- EIED, 2i,a(x; pxx
(71)
It simultaneously relies on full access to the entire target dataset D,, and involves computationally expensive inner-loop MM iterations achieve convergence. These two limitations significantly hinder its practicality in real-time or resource-constraing online scenarios, where access to future samples restricted and fast adaptation essential.
ADAPT (Ours): In contrast the above methods, ADAPT progressively estimates class-wise means and a shared covariance matrix without requiring supervision or access to source data (see Section A.2 and Section A.3). This leads closed-fone formm training-fres solution that is naturally compatible with both online and transductive test-time adaptation settings. Moreover, ADAPT incurs minimal computational overhead and seamlessly adapts distributional shifts in streaming scenarios.
27

---

Table 12: Top-1 accuracy (%) comparison natural distribution shift task with CLIP- backbone under both online and transductive protocols.
Method
BP-free
ImageNet
ImageNet-A
ImageNet-V
ImageNet-R
ImageNet-S
OOD Avg.
Avg.
CLIP [39]
58.16
21.83
51.41
56.15
33.37
40.69
44.15 18
TPT [33]
60.74
26.67
54,70
59.11
35.09
43.89
47.26
DifTIT [9]
x
60,80
31,06
55.80
58.80
37.10
45.69
48.71
C-TPT [55]
x
60.20
23.40
54,70
58.00
35.10
42,80
46.28
DMN [65]
63.87
28.57
56.12
61.44
39.84
46.49
49,97
Online
TPS [46]
62.27
29,80
60.04
55.49
35.74
45.27
48.67
DPE
63.41
30.15
56.72
63.72
40.03
47.66
50.81
DynoPrompt [54]
61.56
27.84
55.12
60.63
44.81
16
BCA (67]
61.81
30.35
56.58
62.89
38.04
46.97
49.93
TDA [21)
61.35
30.29
55.54
62.58
38.12
46.63
49,58
Dota [12]
61.82
30.81
55.27
62,81
37.52
46.60
49.65
ADAPT
62.16
33,08
55.97
62.69
40.21
47.99
50,82
TransCLIP
58,00
21.93
51.54
35.15
$2.79
40.35
43.88
56.57
63.11
Table 13: Top-1 I accuracy (%) comparison corruption robustness task with CLIP-RN50 backbone under both online and transductive protocols.
Blur
Weather
Digital
Noise Shot
Method
Ávg.
Defo.
Glas.
Moti.
Zoom
Snow
Fros.
Fog
Brig.
Cont.
Elas.
Pix.
JPEG
Gauss,
Impo
CLJP
9.54
3.40
7.46
12.62
12.29
15.72
22.08
41.69
6.24
4.67
11.01
14.24
2.43
3.07
2.52
11.27
TPT [33]
8.02
2.74
5.34
10.97
10.59
12.92
16.17
35.67
4.45
3.73
11.56
16.68
1.43
1.94
1.42
9.58
Online
DifTIPT [9]
10.50
3,90
8.62
12.74
11.31
15.03
19.64
37.73
4.98
4.74
13.58
16.56
2.69
3.59
2.08
11.18
TDA [21]
9.84
4.4
7.38
13.74
13.74
17.16
23.76
44.16
7.00
5.79
11.24
15.26
2.54
3.26
2.72
12.13
ADAPT
10.54
4.44
8.57
14.34
13.85
17.84
24.56
45.67
7.76
5.85
11.96
15.86
2.91
3.77
2.92
12.72
Trans,
ZLaP [20]
10.3
3.54
7.99
13:47
13,66
17.15
23.2
44.67
6,55
5.15
11.61
14.23
1.17
2.33
1.65
11.82
TransCLIP [59]
9.38
7.14
12.42
11.87
15.04
23.93
44.33
6.45
11.03
14.85
2.87
3.14
11.72
ADAPT
12.26
6.11
10.92
17.02
16.67
20.62
27.77
47.31
9.20
8.56
14.61
18.15
4.05
4.80
4.03
14.81
B More Experimental Results
B.1 Experiments with CLIP-RN50 Backbone
Task 1: Natural Distribution Shift. Table 12 reports the performance on natural distribution shift benchmarks using the CLIP-RN50 backbone under both online and transductive settings. In the online setting, ADAPT outperforms all prior backpropagation-fie
and optimization-ban based methods, achieving the highest average accuracy of 50.82%. It even surpasses strong transductive methods such as TransCLIP, despite not accessing the full target set. In the transductive setting, where all test samples available, ADAPT further improves to 51.51% 1.,%,,%&6.. exceeding TransCLIP by 7.46%. also ranks first on OOD benchmarks, demonstrating consistent robustness under domain shifts.
Task 2: Corruption Robustness. We further evaluate the performance on corruption robustness benchmarks, with results presented in Table 13, Using the CLIP-RN50 backbone, ADAPT achieves the highest average accuracy 12.72% in the online setting, outperforming all prior backpropagation- free and optimization-base
methods. In the transductive setting, ADAPT further improves to 14.81% and surpasses all compared methods. It consistently ranks first across all 15 corruption types, demonstrating strong resilience to diverse perturbations and confirming its robustness under severe domain shifts.
Task 3: Fine-Grained Categorization. Finally, we evaluate the ADAPT 's adaptation on fine- grained categorization benchmarks, as shown in Table 14. In the Online setting, ADAPT achieves 62.82% average accuracy, which is competitive with the strongest methods like DMN, while main- taining the advantage of being backpropagation-fin -free. In the transductive setting, ADAPT achieves highest accuracy of 64.64%, outperforming prior state-of-the-art methods such as TransCLIP and StatA. It ranks at the top in most datasets, demonstrating strong generalization to subtle inter- class differences. These results highlight the effectiveness and robustness of our framework under fine-graid finengrained and iestribttion-siffee conditions.
28

---

Table 14:
Top-1 accuracy (%) comparison fine-grained categorization task with CLIP-RN50 backbone under both online and transductive protocols.
Method
BP-free Aircraft
Caltech
Cars
DTD
EuroSAT
Flower
Food101
Pets
Sun397
UCFI01
Avg.
CLIP [39]
15.66
85.88
55.70
40.37
23.69
61.75
73.97
83.57
58.80
58.84
55.82
TPT [33]
X
17.58
87.02
58.46
40.84
28.33
62.69
74.88
84.49
61.46
60.82
57.66
DifFIPT [9]
17.60
86.89
60.71
40.72
41.04
63.53
79.21
83,40
62.72
62.67
59.85
C-IPT [55)
17.00
86.90
56.50
42.20
27.80
65.20
74.70
84.10
61.00
59.70
57.51
Online
DMN [65]
22.77
90.14
60.02
50.41
48.72
67.93
76.70
86.78
64.39
65,34
63.32
TPS [29]
18.30
59,40
68.20
76.20
84,40
62.70
64,30
59.60
DPE [61]
19.80
90.83
59.26
50.18
67.60
77.83
85.97
64.23
61.98
61.94
BCA [67]
19.89
89.70
58.13
48.58
42.12
66.30
77.19
85.58
63.38
63.51
61.44
[21]
17.61
89.70
57.78
77.75
86.
62.53
64.18
[7]
18.06
58.72
78.61
87.33
63.89
65.08
ADAPT
18.00
89.37
58.38
70.04
86.43
64.94
63.12
62.82
Trans
TransCLIP [59]
16.60
88.60
57.90
47.80
$9.60
72.20
78.00
89.30
64.20
68.80
64.30
StatA [58]
16.00
87.30
58.20
48.50
50.50
67.70
77.90
87.70
64.30
67.50
62.56
ADAPT
19.53
90.99
61.46
55.73
46.26
74.14
76.97
87.95
66.66
66.75
64.64
VIT-B/16 Online
VIT-B/16 Transductive
Online
RN50 Transductive
Avg. 30 datasets
Task
Task
Task
79
Aceuncy (56)
Accuacy (16)
Accuacy (16)
Avcuracy (36)
49
61
shots per class
shots per class
#abots per class
# shots per class
OOD Avg.
ImageNet
Caltech101
Pets
Accuracy (%6
Accuracy (%)
Accuracy (%)
Accuracy (%)
93
55
88
15
612
012
shots per class
# shots per class
# shots per class
# shots per class
Stanford Cars
Flowers 102
Food101
Aircraft
Accurecy (%)
Acemracy (%)
Acemncy (9)
Couunan (9))
012
per class
shots per class
per class
per class
Sun397
DTD
EuroSAT
UCF101
75
81
Acuimey (9)
Aconzcy (%)
Acouncy (%)
Accuracy (%))
76-
0
15
012
01.
# shots per class
shots per class
sbotts per class
# shots per class
Figure 5: Results of few-shot classification across 30 datasets. We evaluate our method under both online and transductive protocols with 0, 1, 2, 4, 8, and 16-shot settings.
29

---

Biscline
Online
Transhucive
Acceracy (%)
Acceravcy (%)
Accmnsyy
CLIPVII-- CLIP-NNI
KL.FP
ALibE
CLR-iicads CLIP-INN
8L.PP
ALHEZ
CLIP-YY--RS
CLIPANNE
RLAP
ALA
Figure 6: Performance comparison of proposed ADAPT on different VLMs.
B.2 Few-shot Adaptation
Figure 5 presents few-shot classification results of our ADAPT across three tasks covering 30 datasets. We evaluate performance under both online and transductive adaptation protocols, using 0, 1, 2, 4, 8, and shots per class. assess architectural generality, we conduct experiments with two backbone variants: CLIP-VIT-B/16 and CLIP-RN50.
Overall, results show consistent performance increase with more labeled samples, confirming the scalability and effectiveness of our method. Results with CLIP-ViT-B/IK consistently outperform RN50 across all shot counts, particularly in low-shot regimes, suggesting stronger feature represen- tations. Additionally, transductive adaptation variants exhibit consistently better performance than their online counterparts. This improvement can be attributed to the transductive setting's 's ability to leverage the entire test set global context, thereby facilitating more informed label predictions and reducing uncertainty in classification. Notably, Task 3 requires finer-grained discrimination due to subtle intra-class variations, making accurate classification particularly dependent detailed visual cues. In this context, the few-shot setting offers class- class-specific supervision that enhances the model's ability to capture these nuances. As result, our method exhibits substantial performance gains Task 3, with consistent improvements observed across 10 representative datasets. These findings underscore method's capacity adapt to complex recognition challenges and its effectiveness across varying adaptation regimes.
B.3 Evaluation with Different VLMs
We conduct a comprehensive evaluation of our ADAPT across a diverse set vision-language models (VLMs), including CLIP (ViT-B/16 and RN50) [39], BLIP [25], and ALBEF [26]. Figure presents the performance comparison across three representative tasks: Task assessing natural distribution shifts, Task 2 evaluating robustness synthetic corruptions, and Task focusing on fine-grained categorization challenges.
Our results demonstrate that ADAPT consistently improves performance across all tasks and model architectures. The online adaptation variant already delivers notable gains by leveraging sequential test data, while the transductive variant further enhances results by exploiting global information from the entire test set. Importantly, these improvements hold true not only for stronger VLMs like
baseline methods
but also for comparatively lighter models such as ALBEF, indicating that ADAPT is broadly applicable and effective regardless of the underlying model capacity. This versatility highlights the practical value of our approach for enhancing test-time adaptation across diverse vision-language architectures and real-world scenarios.
B.4 Additional Analysis
Further Analysis on Ablation Study. Table 15 presents a detailed ablation study of three key components in ADAPT: (i) the knowledge bank B, (ii) the adaptive class-wise mean f4, and (iii) shared covariance Σ. We compare eight configurations by selectively enabling or disabling these components and report performance across all three tasks. In this setup, disabling i means using CLIP's original class prototypes as the fixed class means, while disabling Σ corresponds to using fixed identity matrix as the shared covariance.
30

---

The upper block of the table (Rows 1-4) corresponds to knowledge bank-free set- tings, where B is not used. The first row
Table 15: Ablation study on components.
B
Update
Update Σ
Task
Task
Task
represents the baseline CLIP model with-
out any adaptation. Updating only ja (Row 3) provides modest improvements (e.g.,
x
X
59.11
25.50
63.45
X
X
49.64
9.58
60.02
+2.43% on Task 1), likely due to better- aligned class centers. However, updating only E (Row 2) leads to substantial per- formance drops (e.g., down to 9.58% on
X
V
X
61.54
25.42
67.03
49.65
9.58
60.04
X 64.89
25.08
67.06
Task 2), indicating that estimating covari- ance from noisy test-time predictions alone highly unstable and unreliable.
V 64.05
19.49
67.95
X
65.27
25.67
67.43
66.53
28.56
70.76
The lower block (Rows 5-8) introduces the knowledge bank B. Even without updating μ E (Row 5), the model significantly outperforms all memory-frec variants, validating the effectiveness the regularization term R(z;; B), which encourages alignment with high-confidence stored features. Incorporating adaptation for either μ or E further improves results, showing that both distributional statistics offer complementary cues for more accurate estimation. The best performance is achieved by jointly adapting both μ and E, which fully leverages the knowledge bank for robust distribution modeling, with Task reaching 28.56%. These results highlight the synergy between adaptive statistics and memory -ooooded estimation for stable TTA.
Robustness under Significant Shifts.
Table 16: Robustness evaluation on ImageNet-CC across five levels of synthetic corruption.
To evaluate our method's robustness un- der significant distribution shifts, bench- marked ADAPT against TDA [21]. a state- of-the-art memory-based method. The eval- uation was conducted using synthetic cor- ruptions across five levels of increasing severity, with detailed results presented in Table 16. The results show that while all methods degrade as the shift intensifies, ADAPT consistently outperforms TDA across all severity levels. This provides strong evidence that our proposed mechanism is more resilient to noisy pseudo-labels, validating its superior robustness for practical applications involving severe, real-world distribution shifts.
Severity Level
3
4
5
CLIP
59.68
52.97
46.79
37.51
25.50
TDA
60.42
54.15
48.35
36.84
28.34
ADAPT (Online)
61.37
54.70
48.61
39.28
28.56
ADAPT (Trans.)
62.04
55.51
49.68
40.76
30.29
Robustness under Low-confidence Sce- narios. To assess the impact of having a limited number of high-confidence samples per class (denoted as N), we analyze our method's performance in this constrained setting. The results, presented in Table 17, reveal distinct behaviors for our online
Table 17: Evaluation with very few high-confidence samples. Where N means the number of high- confidence samples per class.
N
2
4
8
10
Online
Task
59.61
64.19
65.50
66.10
66.33
and transductive approaches. Our online
Task 2
22.95
26.00
27.26
27.93
28.22
method, while sensitive to extremely low sample counts (N = 2), recovers quickly and achieves robust performance with few as 4-6 samples per class. However, acknowledge potential efficiency chal- lenge: the test stream begins with many low-confidence predictions, it takes longer to collect reliable samples for the memory bank, which may slightly delay adaptation and degrade performance. Notably, our transductive ADAPT demonstrates exceptional robustness, maintaining high and stable performance even with just two high-confidence samples per class (N = 2). We attribute this resilience its ability to leverage the global structure of the entire batch, which regularizes parameter estimates and ensures strong performance even under severe data scarcity.
Task
63.87
67.46
69.04
69.89
70.31
Task
67.02
67.11
67.09
67.10
67.04
Trans.
Task 2
30.26
30.29
30.29
30.23
30.23
Task 3 72.15
72.21
72.43
72.24
72.25
31


Reviews:
[{"review_id": "bjMrvtOtsq", "reviewer": "Reviewer_uiWy", "summary": "The paper proposes a gradient free method for adapting vision–language models at test time. The prediction is treated as Gaussian mixture class assignment, where the parameters of each class are updated based on a set of highly confident representatives of that class. \nThe paper treats two settings: (i) online: as each test sample arrives, its embedding is scored, and if confident, added to the per-class bank. Then, the class Gaussian parameters are recomputed in closed form before making that sample’s prediction. (ii) once the entire unlabeled test set is available, the method builds the banks from the most confident samples.\nThis allows the predictions, modeled as GMM assignment, to adapt to test time distribution changes.", "questions": "- Have the authors considered a bank-pruning step (periodically removing samples whose likelihood falls below a threshold)? This can help addressing some of the weaknesses.\n- The shared covariance assumption seems very restrictive. Have the authors considered experimenting with separate covariance matrices, even on a small scale example?", "limitations": "See weaknesses above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\nThe paper is well written and it proposes a simple and clean solution to test time adaptations. The empirical evaluation is extensive and convincing.\n\n**Weaknesses:**\n- The memory banks, build from high confidence samples, are central to the method’s efficiency, but they also introduce a potential failure, especially if significant shift occurs. If the initial CLIP zero-shot predictions are substantially misaligned with the new domain, their confidence scores will become unreliable, causing the memory banks to be built from misclassified examples. This, in turn, will corrupt the estimated class means and shared covariance.\n- Similarly, the question of how having classes with very few high-confidence samples affects the estimation of the parameters, and therefore the results.\n-  The paper offers no formal guarantees on how quickly the estimated parameters converge, or bound on the divergence from the true distribution. \n- Evaluation relies exclusively on CLIP’s embedding space.  It’s unclear how well its assumptions and mechanisms transfer to other models.\n- The method may be  highly dependent on the randomized order of samples in the online case. However, no standard error or other form of confidence is reported for repetitions of each experiment. \n\nDespite these limitations, the paper offers a simple, elegant solution to a significant problem. I would be happy to revisit my evaluation—and potentially raise my score, if the authors address the identified weaknesses, particularly by reporting standard deviations or confidence intervals for their results.", "was_revised": true, "final_justification": "The authors successfully addressed my concerns, and accordingly I am raising my score.", "rebuttal": "We thank the reviewer for the constructive comments. We provide our feedback as follows.\n\n> **W1: Performance and robustness under significant shifts.**\n\n We agree that robustness against unreliable predictions under severe domain shift is essential. To further assess our method under significant shifts with synthetic corruptions, we benchmarked ADAPT against TDA, a state-of-the-art memory-based method, **across five levels of increasing shift severity**. As shown in the table below, while all methods degrade as the shift intensifies (an expected trend in an unsupervised setting), **our ADAPT consistently outperforms TDA across all levels**.\n\nNotably, ADAPT's performance advantage is even more significant under severe shifts (e.g., Level 4 and Level 5), where the risk of memory corruption is greatest. This provides strong evidence that our proposed mechanism is more resilient to noisy pseudo-labels, validating **its superior robustness for practical, severe distribution shifts**.\n\n| Shift Level | 1 | 2 | 3 | 4 | 5 |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| CLIP | 59.68 | 52.97 | 46.79 | 37.51 | 25.50 |\n| TDA | 60.42 | 54.15 | 48.35 | 36.84 | 28.34 |\n| ADAPT (Online) | 61.37 | 54.70 | 48.61 | 39.28 | 28.56 |\n| ADAPT (Trans.) | 62.04 | 55.51 | 49.68 | 40.76 | 30.29 |\n\n---\n> **W2: How having classes with very few high-confidence samples affects the results.**\n\nWe analyze how the number of high-confidence samples per class (denoted as **N**, ranging from 2 to 10) influences performance. As shown in the table, while **our online method** is somewhat sensitive to extremely low sample counts (e.g., 2 per class), it recovers quickly and achieves robust performance  **with only 4 to 6 samples per class**.\n\nNotably, **our transductive ADAPT** maintains high and stable performance  **even with just 2 high-confidence samples per class**. We attribute this to its ability to leverage the global structure of the full test batch, which regularizes parameter estimates and ensures robustness under data scarcity.\n\n|  | N | 2 | 4 | 6 | 8 | 10 |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Online | Task 1 | 59.61 | 64.19 | 65.50 | 66.10 | 66.33 |\n| | Task 2 | 22.95 | 26.00 | 27.26 | 27.93 | 28.22 |\n| | Task 3 | 63.87 | 67.46 | 69.04 | 69.89 | 70.31 |\n| Trans. | Task 1 | 67.02 | 67.11 | 67.09 | 67.10 | 67.04 |\n| | Task 2 | 30.26 | 30.29 | 30.29 | 30.23 | 30.23 |\n| | Task 3 | 72.15 | 72.21 | 72.43 | 72.24 | 72.25 |\n\n---\n> **W3: On the convergence and divergence bounds of estimated parameters.**\n\nWe appreciate the reviewer’s insightful question. Providing formal guarantees for convergence is inherently difficult, as our regularization term $\\mathcal{R}(z_i;\\mathcal{B})$ relies on a knowledge bank and introduces non-decomposable dependencies. This type of regularizer poses fundamental challenges, and to the best of our knowledge, establishing general convergence theory for such structures remains an open problem in the broader learning theory community.\n\n\nGiven these theoretical challenges, we instead provide a thorough empirical validation of our parameter estimation. We **conducted experiments on 100 randomly selected ImageNet classes to measure the divergence between our estimated distribution parameters ($\\mu_k$ , $\\Sigma$) and those of the ground-truth empirical distribution**.\n\nThe results, averaged over **10 runs** and summarized below, demonstrate the high accuracy of our estimations. As the table shows, the estimation errors are consistently low and stable. The small Frobenius distances (F-dist) for the mean ($\\mu$) and covariance ($\\Sigma$), along with the low overall error, confirm that the estimated distribution closely aligns with the ground-truth. This strong empirical evidence demonstrates that, while a formal proof is an open challenge, our method effectively converges to the true underlying data distribution in practice. It achieves this highly accurate and robust estimation without requiring any supervision.\n\n| F-dist μ | F-dist Σ | Error |\n| :---: | :---: | :---: |\n| 0.1466 ±0.0055 | 0.2123 ±0.0003 | 0.2897 ±0.0174 |\n\n---\n > **W4: Evaluation with different VLMs.**\n\nWe have already evaluated ADAPT with different VLMs in **Appendix B.3**. For reference, we present the summary table below, and more detailed results can be found in **Figure 3 of the Appendix**. ADAPT delivers consistent gains across various models, including **CLIP (ViT-B/16, RN50)**, **BLIP** [1], and **ALBEF** [2]. This demonstrates that ADAPT is model-agnostic and broadly applicable across different VLM architectures. \n\n| Method | Task 1 | Task 2 | Task 3 |\n| :--- | :--- | :--- | :--- |\n| **BLIP** [1] | 46.04 | 22.06 | 49.81 |\n| w/ ADAPT (Online) | 56.67 (+10.63) | 26.70 (+4.64) | 62.23 (+12.42) |\n| w/ ADAPT (Trans.) | 57.59 (+11.55) | 27.48 (+5.42) | 62.41 (+12.60) |\n| **ALBEF** [2] | 33.88 | 15.21 | 31.16 |\n| w/ ADAPT (Online) | 40.27 (+6.39) | 19.79 (+4.58) | 43.34 (+12.18) |\n| w/ ADAPT (Trans.) | 43.26 (+9.38) | 20.79 (+5.58) | 45.78 (+14.62) |\n\n[1] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (ICML 2022)\n\n[2] Align Before Fuse: Vision and Language Representation Learning with Momentum Distillation (NeurIPS 2021)\n\n---\n> **W5: Report the standard deviation of the results.**\n\nTo address the concern regarding stability, we evaluated our method **over 10 random permutations** of the test stream, reporting the average performance and standard deviation across **three tasks**. The results confirm our method's high stability, with a **consistently low standard deviation** across all tasks. These standard deviations will be included in the tables of our final manuscript to formally report the method's reliability.\n\nTask1:\n| ImageNet | ImageNet-A | ImageNet-V | ImageNet-R | ImageNet-S | OOD Avg. | Avg. |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| 70.72 ±0.10 | 63.45 ±0.21 | 64.56 ±0.16 | 80.55 ±0.09 | 53.08 ±0.09 | 65.41 ±0.06 | 66.47 ±0.06 |\n\nTask2:\n| Gauss. | Shot | Impul. | Defoc. | Glass | Motion | Zoom | Snow | Frost | Fog | Brig. | Contr. | Elastic | Pixel | JPEG | Avg. |\n|  :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :-- | :--- | :--- | :--- | :--- | :-- | :--- | :--- |\n| 15.71 ±0.05 | 16.82 ±0.05 | 15.81 ±0.07 | 26.34 ±0.06 | 17.90 ±0.07 | 27.26 ±0.09 | 25.42 ±0.08 | 36.25 ±0.07 | 34.66 ±0.07 | 40.89 ±0.06 | 60.28 ±0.06 | 19.84 ±0.22 | 16.10 ±0.06 | 37.30 ±0.10 | 37.23 ±0.08 | 28.52 ±0.03 |\n\nTask3:\n| Aircraft | Caltech101 | Cars | DTD | EuroSAT | Flower | Food101 | Pets | Sun397 | UCF101 | Avg. |\n|:--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| 28.93 ±0.21 | 94.60 ±0.15 | 68.05 ±0.27 | 53.87 ±0.61 | 67.24 ±0.71 | 76.28 ±0.34 | 83.91 ±0.04 | 90.37 ±0.58 | 71.26 ±0.26 | 71.82 ±0.45 | 70.64 ±0.10 |\n\n---\n> **Q1: Experiment with the bank-pruning step.**\n\nThank you for this constructive suggestion. We implemented a bank-pruning mechanism as a new ablation study. We **periodically (every 1k steps) remove a portion of the lowest-confidence samples (e.g., 10%, 30%, 50%) from the memory bank**. This pruning is done on a per-class basis, as a single global confidence threshold can be unfair to naturally 'harder' classes. The results, presented below, show that this pruning strategy consistently degrades performance, especially on the more challenging Image-Net-A dataset with distribution shift.\n\nThis outcome suggests that **ADAPT inherently performs a form of soft dynamic filtering, effectively mitigating the impact of noisy samples without requiring explicit pruning**. Through its regularized parameter estimation process, the method naturally down-weights the influence of unreliable samples rather than discarding them entirely. Introducing an explicit or aggressive pruning mechanism may, in fact, be counterproductive, as it risks eliminating samples that could become informative as adaptation progresses. This finding highlights the robustness and stability of our original design, which avoids premature decisions while remaining resilient to noise.\n\n| | ImageNet | ImageNet-A |\n| :--- | :--- | :--- |\n| ADAPT (Online) | 70.91 | 63.32 | \n| w/ 0.1 pruning | 69.16 | 50.69 |\n| w/ 0.3 pruning | 69.03 | 50.60 |\n| w/ 0.5 pruning | 69.02 | 50.56 |\n| w/ 0.7 pruning | 68.97 | 50.52 |\n\n\n---\n> **Q2: Using separate covariance matrices.**\n\nAs we detail in Appendix A.4 (Discussion), we provide strong empirical evidence that the class-conditional features approximately follow Gaussian distributions with a shared covariance matrix.\n\nTo further compare with class-wise separate covariance matrices, we conducted additional experiments on ImageNet by **replacing our shared covariance with class-specific covariance matrices**, keeping all other settings identical. The results, summarized below, show that this **leads to a significant drop in accuracy, alongside a sharp increase in computation time and memory usage**.\n\nWe attribute this performance degradation to data sparsity. **Estimating a full covariance matrix for each class becomes unreliable with few high-confidence samples, leading to poor generalization and overfitting**. In contrast, the shared covariance pools information across all classes, resulting in a much more stable and robust estimate, which is critical in online or data-scarce settings.\n\nUltimately, this ablation confirms that **our shared covariance assumption is not a limitation, but a crucial design choice**. It strikes an effective trade-off between accuracy, robustness, and computational efficiency, making our method practical for real-world, resource-constrained scenarios.\n\n| Method | Time | Acc | Gains | Mem. (GB) |\n| :--- | :--- | :--- | :--- | :--- |\n| CLIP | 8m | 66.74 | - | 0.79 |\n| ADAPT (Online) | 1h 11m | 70.91 | +4.17 | 0.93 |\n| w/ Separate Cov. | 1h 44m | 53.80 | -12.94 | 2.89 |\n| ADAPT (Trans.) | 0.73m | 71.56 | +4.82 | 3.37 |\n| w/ Separate Cov. | 1.40m | 66.16 | -0.58 | 4.62 |"}, {"review_id": "a106buHFcn", "reviewer": "Reviewer_Hyj2", "summary": "This paper introduces ADAPT, a novel backpropagation-free and distribution-aware test-time adaptation method designed for VLMs. By modeling class-conditional feature distributions via gaussian discriminant analysis and updating them with high-confidence test samples in a constructed knowledge bank, the method provides closed-form, training-free and efficient adaptation. ADAPT operates under both online and transductive settings and outperforms both backpropagation-based and training-free baselines across natural shifts, corruption robustness and fine-grained categorization tasks.", "questions": "1. What is the effect of significant noise in the early high-confidence predictions on long-tail adaptation?\n2. What kind of historical samples or priors does ADAPT need?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. ADAPT avoids iterative updates or backpropagation at test time by formulating TTA as a probabilistic inference task with gaussian assumptions, enabling low-latency adaptation.\n2. ADAPT explicitly estimates class-conditional feature distributions with a shared covariance matrix and uses knowledge banks for regularization.\n3. ADAPT achieves SOTA performance on ImageNet variants and 10 fine-grained datasets under both online and transductive settings, while also being more efficient in time and memory usage.\n\nWeaknesses:\n\n1. While the Gaussian Discriminant Analysis formulation enables efficient closed-form inference, the assumption of unimodal Gaussian distributions with a shared covariance across classes may not capture the complexity of real-world data, especially in multimodal or highly structured feature spaces. This simplification can result in biased or suboptimal predictions in challenging domains (e.g., fine-grained classification or long-tail distributions), potentially limiting ADAPT's effectiveness in practical deployment scenarios.\n2. The method depends on confidence thresholds, bank sizes and prior means from CLIP templates, which may require careful tuning.", "was_revised": true, "final_justification": "All my concerns have been resolved, and I am now inclined to recommend acceptance of the paper.", "rebuttal": "We thank the reviewer for the valuable feedback.\n\n---\n> **W1: Assumption of Gaussian distributions with a shared covariance.**\n\nThank you for this insightful question. We agree that the unimodal Gaussian assumption introduces a simplification and may be limiting in hypothetical scenarios involving highly complex feature distributions. However, as demonstrated in the main paper, **ADAPT consistently achieves SOTA performance across diverse and challenging settings**, including **domains with severe visual corruption (Table 3)** and **fine-grained classification tasks (Table 4)**. These empirical results suggest that, despite the assumption, our method remains robust and effective in real-world applications.\n\nTo further investigate this design choice,  we conducted an ablation study where we **replaced the shared covariance matrix with class-wise covariance matrices**, keeping all other settings fixed. This led to a **notable drop in accuracy, along with increased memory usage and inference time**. We attribute this to data sparsity: in many TTA settings, only a few high-confidence samples are available per class, making class-wise covariance estimation unreliable and prone to overfitting. \n\nIn contrast, our shared covariance pools statistics across classes, producing a more stable and generalizable estimate, which is crucial in online or data-scarce environments. This further supports our design choice: **the shared covariance assumption is not merely for reducing computational overhead but is crucial for improving stability and generalization**. As the reviewer also noted, a key strength of ADAPT lies in its broad applicability, particularly in practical TTA scenarios where low latency and limited resources are critical.\n\n| Method | Time | Acc | Gains | Mem. (GB) |\n| :--- | :--- | :--- | :--- | :--- |\n| CLIP | 8m | 66.74 | - | 0.79 |\n| ADAPT (Online) | 1h 11m | 70.91 | +4.17 | 0.93 |\n| w/ Separate Cov. | 1h 44m | 53.80 | -12.94 | 2.89 |\n| ADAPT (Trans.) | 0.73m | 71.56 | +4.82 | 3.37 |\n| w/ Separate Cov. | 1.40m | 66.16 | -0.58 | 4.62 |\n\n---\n> **W2: Hyperparameter analysis.**\n\nWe already provided a detailed hyperparameter analysis in **Section 4.3** of our main paper, particularly in **Figure 2 and Table 7**. For clarity, we summarize our response as follows:\n\n(1) **Confidence Threshold:** Our method **does not use any manually defined confidence threshold**. Instead, we adopt a fixed-size memory bank that is dynamically updated; new samples with higher confidence scores replace those with the lowest scores. This adaptive mechanism eliminates the need for hand-crafted thresholds and ensures that only the most reliable samples are retained.\n\n(2) **Bank Size (L):** ADAPT exhibits strong robustness to this parameter. As shown in **Figure 2 (left)**, performance in the transductive setting remains stable across a wide range of values for L, while in the online setting, accuracy plateaus once L exceeds 10. We thus use a single, fixed configuration across all datasets (L=16 for online, L=6 for transductive), **with no per-dataset tuning required**.\n\n(3) **CLIP Prior Means:** ADAPT is not sensitive to the initialization of the CLIP prior means. **Table 7** verifies the robustness of our method under various mean settings. Furthermore, **Figure 2 (right)** shows that **even when the CLIP prior mean is entirely removed (α=1.0), ADAPT still achieves state-of-the-art performance**.\n\nIn summary, **ADAPT does not rely on dataset-specific hyperparameter tuning or static priors**. Instead, it leverages dynamic, data-driven mechanisms to adaptively maintain a reliable knowledge bank. This design choice ensures strong performance, high robustness, and ease of deployment in real-world settings.\n\n---\n> **Q1: What is the effect of significant noise in the early high-confidence predictions on long-tail adaptation?**\n\nTo assess the impact of early-stage prediction noise under long-tailed distributions, we conducted a controlled experiment with the following setup:\n\n(1) **Long-tail dataset construction**: We constructed a **long-tailed subset from ImageNet-1K by sampling classes according to a power-law distribution**, resulting in:\n\n* **Head class**: 61 classes (≥40 shots, 2,719 images)\n* **Medium class**: 338 classes (10–40 shots, 7,396 images)\n* **Tail class**: 601 classes (≤10 shots, 2,416 images)\n\n(2) **Worst-case noise injection**: To simulate an extreme scenario, we deliberately injected noise during the **early 30% phase of adaptation**. Specifically, we **corrupted 30% of the high-confidence predictions by randomly reassigning them to incorrect classes**. This models a challenging case where early memory is contaminated with misleading information.\n\n(3) **Evaluation and Findings**: We compared the baseline CLIP model with ADAPT in this adverse setting. The results, summarized in the table below, yield three key observations:\n\n* While both methods degrade under heavy noise, **ADAPT consistently achieves higher final accuracy (64.27% vs. 63.25%)**, demonstrating superior overall robustness.\n\n* For the **extremely few-shot tail classes**, both methods are highly vulnerable to the strong initial misguidance, exhibiting a comparable **performance drop**. This is an expected outcome, as a 30% initial corruption can heavily skew parameter estimates when very few samples are available.\n\n* ADAPT demonstrates **self-correction capability**: As adaptation progresses, the accumulation of correctly identified samples (especially from head/medium classes) allows the model to progressively refine its distribution estimates, effectively mitigating the influence of the initial noisy predictions. **This confirms that ADAPT is a more reliable solution for practical, noisy, long-tailed scenarios**.\n\n| Method | Head class | Med class | Tail class | Avg. |\n| :--- | :--- | :--- | :--- | :--- |\n| CLIP | 68.44 | 69.69 | 68.42 | 69.33 |\n| w/ Noise | 62.16 (-6.28) | 64.03 (-5.66) | 62.09 (-6.33) | 63.25 (-6.08) |\n| ADAPT (Online) | 69.22 | 70.31 | 69.08 | 69.83 |\n| w/ Noise | 63.55 (-5.67) | 65.20 (-5.11) | 62.25 (-6.83) | 64.27 (-5.56) |\n---\n > **Q2: Clarification on the required priors and historical samples for ADAPT.**\n\nADAPT operates in a **training-free and label-free** manner. The only information it uses is derived from (1) a small set of high-confidence samples from the test stream itself and (2) optional class-mean priors from CLIP.\n\n(1) **Historical Samples:** ADAPT maintains a fixed-size memory bank containing previously seen test samples with high-confidence predictions. These are not labeled samples, but are **selected based on prediction confidence during inference**. The memory bank is **continuously updated, replacing the lowest-confidence entries with more reliable samples**, and is used to guide the adaptation of incoming samples. This strategy eliminates the need for manual thresholds, making ADAPT a fully self-supervised approach.\n\n(2) **CLIP Prior (Optional):** We optionally initialize the class-wise means using CLIP’s textual priors, but this prior is not essential. As shown in **Table 7** and **Figure 2** of the main paper, **even when these CLIP priors are removed (α = 1.0), ADAPT still achieves strong performance**. This highlights the robustness of our approach to the choice of prior.\n\nIn essence, ADAPT is a self-contained method that adapts directly to the test stream, eliminating the need for external labeled data or strong, fixed priors."}, {"review_id": "ewJ5G1bmCb", "reviewer": "Reviewer_c5tn", "summary": "This paper proposes a Gaussian Discriminant Analysis (GDA)-based test-time adaptation (TTA) method for CLIP. The GDA framework is initialized with textual embeddings of prompted class names. At test time, it models visual features from the image encoder and yields predictions in a backpropagation-free manner. The evaluation includes commonly used benchmarks.", "questions": "1. The authors may need to compare ADAPT with previous memory-based methods under comparable prior knowledge from textual prompts to more convincingly demonstrate the superiority of the proposed algorithm.\n2. Given that ADAPT ultimately adopts prompts from AWT [57], AWT may need also be compared and discussed.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The work is well-motivated, and the paper is well-written.\n2. The method design appears principled and is supported by theoretical analysis.\n3. The evaluation includes sufficient benchmarks, and the ablation studies are well-structured.\n\n**Weaknesses**\n\nThe major concern lies in the exact performance advantage of the proposed method compared to existing backpropagation-free TTA baselines, especially memory-based TTA methods like TDA [18]. My understanding is that the main difference between ADAPT and memory-based methods is that memory-based approaches use a KNN-like strategy to model test-time visual features and yield predictions, while ADAPT uses a GDA-based approach to do the same. Since the text embeddings are fixed, both can incorporate prior knowledge from textual prompts. Although Tables 2 and 4 show performance gains of ADAPT over TDA, Table 7 indicates that ADAPT benefits significantly from GPT-generated prompts. To my knowledge, TDA employs hand-crafted prompts similar to the “CLIP Template” in Table 7. If we compare ADAPT and TDA using similar prompts, their performances seem comparable. This suggests that the observed performance gains in Tables 2 and 4 may largely result from the use of stronger GPT-generated prompts rather than from inherent advantages of the GDA-based method itself. Therefore, the advantage of the proposed GDA-based approach over memory-based methods remains uncertain based on the current manuscript.", "was_revised": true, "final_justification": "My initial concern was that the manuscript does not clearly demonstrate raw performance improvements over previous backpropagation-free TTA methods, given that different prompts significantly influence the results of such methods, as shown in the authors' rebuttal. \n\nRegarding raw performance improvements, the authors argue that recent backpropagation-free TTA methods have saturated on specific benchmarks, which partially supports the claim that the proposed method offers improvements. This reasoning is somewhat convincing.\n\nConsidering the strengths in other aspects of the work, I maintain my positive rating. I hope the authors can add the discussion on the impact of prompts and the saturating performance of backpropagation-free TTA methods, to help readers clearly understand the current state and challenges in this line of research.", "rebuttal": "We thank the reviewer for their thoughtful comments and feedback.\n\n---\n> **W1 & Q1: Clarification on the superiority of ADAPT over memory-based methods under comparable prompt conditions.**\n\nWe would like to clarify that **the superior performance of ADAPT stems not from advanced prompt engineering, but from its core methodology**: distribution-aware adaptation.\n\n(1) **Methodological Advantage**: As the reviewer correctly noted, prior memory-based methods like TDA rely on a non-parametric, instance-wise k-NN approach, making their predictions highly sensitive to individual noisy samples in online scenarios. In contrast, ADAPT employs a parametric, GDA-based strategy to explicitly model the class-conditional distributions. This modeling not only reduces the influence of individual outliers but also leads to more stable and robust decision boundaries, as illustrated in Figure 3 of our main paper.\n\n(2) **Performance Beyond Prompt Engineering:** To directly address this concern, we evaluated ADAPT against TDA using a wide array of identical prompting strategies. The results summarized in the table below highlight that our ADAPT **consistently and significantly outperforms TDA across all tasks and prompting settings**. This confirms that **the gains stem from the method itself, not from stronger textual prompts**.\n\n(3) **Robustness to Prior Removal:** As further proof, Figure 2 (right) in our main paper shows that ADAPT maintains SOTA performance even when the prompt-based CLIP prior is **completely removed** ($\\alpha=1.0$). This highlights that ADAPT’s strength lies in its data-driven, distributional modeling, not in external textual guidance.\n\n|  | Method | Vanilla | Ensemble | CLIP Template | GPT | GPT & Ensemble | GPT & CLIP Template |\n| :---: | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Task 1 | TDA | 63.92 | 66.01 | 65.93 | 65.92 | 65.94 | 65.96 |\n| | ADAPT (Online) | 64.70 (+0.78) | 66.56 (+0.55) | 66.51 (+0.58) | 66.53 (+0.61) | 66.57 (+0.63) | 66.58 (+0.62) |\n| | ADAPT (Trans.) | 65.53 (+1.61) | 67.16 (+1.15) | 67.23 (+1.30) | 67.09 (+1.17) | 67.10 (+1.16) | 67.12 (+1.16) |\n| Task 2 | TDA | 27.17 | 28.25 | 28.05 | 28.38 | 28.76 | 28.69 |\n| | ADAPT (Online) | 27.52 (+0.35) | 28.54 (+0.29) | 28.44 (+0.39) | 28.56 (+0.18) | 28.98 (+0.22) | 28.91 (+0.22) |\n| | ADAPT (Trans.) | 29.45 (+2.28) | 30.00 (+1.75) | 29.96 (+1.91) | 30.29 (+1.91) | 30.51 (+1.75) | 30.48 (+1.79) |\n| Task 3 | TDA | 66.92 | 67.39 | 64.46 | 69.62 | 69.04 | 69.18 |\n| | ADAPT (Online) | 67.43 (+0.51) | 67.74 (+0.35) | 67.62 (+3.16) | 70.76 (+1.14) | 69.95 (+0.91) | 69.90 (+0.72) |\n| | ADAPT (Trans.) | 69.55 (+2.63) | 70.00 (+2.61) | 70.38 (+5.92) | 72.43 (+2.81) | 72.05 (+3.01) | 71.91 (+2.73) |\n\n---\n> **Q2: Comparison with AWT.**\n\nWhile ADAPT uses the same handcrafted prompts as AWT, their core adaptation strategies differ fundamentally. AWT aligns multiple augmented views of each test image with textual prompts via Optimal Transport, making its performance **highly dependent on visual augmentations**.\n\nTo illustrate this, we compare ADAPT with both original  AWT and **a variant (AWT\\*) without visual augmentations**. As shown in the table below, AWT suffers a **significant performance drop without augmentation**, confirming this reliance. In contrast, ADAPT achieves consistently strong performance without requiring such augmentations.\n\nThis highlights ADAPT's ability to perform source-free, training-free adaptation without relying on computationally expensive visual augmentations. This makes ADAPT a more streamlined and versatile solution, which is readily applicable to both online and transductive settings.\n\n| Method | ImageNet | Aircraft | Clatech101 | Cars | DTD | EuroSAT | Flower102 | Food101 | Pets | Sun397 | UCF101 | Avg. |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| AWT | 71.32 | 29.22 | 95.54 | 69.93 | 55.56 | 58.61 | 75.07 | 85.54 | 92.53 | 70.58 | 72.51 | 70.58 |\n| AWT \\* | 69.03 | 27.48 | 94.77 | 65.94 | 53.96 | 60.41 | 75.11 | 84.76 | 91.88 | 68.18 | 69.81 | 69.21 |\n| ADAPT (Online) | 70.91 | 28.95 | 94.48 | 68.19 | 55.20 | 68.19 | 75.56 | 83.81 | 92.01 | 70.57 | 70.66 | 70.78 |\n| ADAPT (Trans.) | 71.56 | 30.81 | 95.46 | 71.32 | 56.86 | 65.93 | 80.11 | 85.15 | 92.59 | 72.25 | 73.86 | 72.35 |"}, {"review_id": "GYCpJO4I13", "reviewer": "Reviewer_o7Us", "summary": "This paper presents ADAPT, a novel method for test-time adaptation (TTA) that is both backpropagation-free and distribution-aware. The core idea is to reframe TTA as a probabilistic inference problem, where class-conditional feature distributions are modeled as Gaussians. This allows for a closed-form, training-free solution that efficiently adapts vision-language models to new test distributions using dynamically updated \"knowledge banks\" of high-confidence samples.", "questions": "1. What is Oracle ADAPT and how does it differ from ADAPT in the transductive setting?\n2. Consider adding memory and computation analysis to strengthen ADAPT's contribution. This should favor ADAPT since it avoids backpropagation.\n3. Have you explored class-specific covariance matrices? What's the accuracy vs. computational complexity trade-off?\n4. Why exclude the current test sample from class mean updates in online but include all samples (via soft labels) in transductive settings?\n5. How does the knowledge bank perform on high inter-class similarity datasets? Could low confidence across all classes stall updates and hurt adaptation?\n6. How did you tune the regularization parameter β in Eq. 5, and how sensitive is performance to its value?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\nI think the main strength of this work lies in its efficiency and practicality. By eliminating the need for backpropagation and iterative optimization, ADAPT presents a compelling solution for real-time or resource-constrained scenarios, a significant advantage over many existing TTA methods. I also found the framework's flexibility to be a major plus; its ability to operate in both online and transductive settings without requiring source data makes it broadly applicable.\n\n# Weaknesses\n\nDespite the strong results, I have a few concerns:\n\n1. The method's core assumption is that class features follow a Gaussian distribution with a shared covariance matrix (lines 128-129) and a uniform class prior (line 134),  it may be too simplistic for real-world data distributions that could be multi-modal or highly skewed. As noted in prior work [1, 2, 3], the assumption of uniform prior is often violated in the real world, which could degrade the performance of methods requiring such an assumption. An analysis of potential failure cases would strengthen the paper.\n\n2. The literature review contains several critical gaps that weaken the paper's novelty claims. The backpropagation-free TTA approach using pre-trained classifier weights as prototype anchors closely parallels [4]'s vision-only methodology—creating linear layers from CLIP's text embeddings is essentially a direct extension of their approach. Other backpropagation-free TTA methods [6, 7] for vision or vision-language model are missing. Additionally, the \"Knowledge banks\" mechanism resembles those in [3, 5] ([5] is cited but not discussed), and ADAPT's update strategy (e.g., Eq. 9) appears very similar to existing work. The paper requires proper citations and clearer differentiation from these prior works to establish its contributions.\n\n[1] NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation (NeurIPS 2022)\n\n[2] Robust Test-Time Adaptation in Dynamic Scenarios (CVPR 2023)\n\n[3] DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation (ICML 2025)\n\n[4] Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization (NeurIPS 2021)\n\n[5] DynaPrompt: Dynamic Test-Time Prompt Tuning (ICLR 2025)\n\n[6] Test-Time Model Adaptation with Only Forward Passes (ICML 2024)\n\n[7] Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025)", "was_revised": true, "final_justification": "The rebuttal and response addressed most of my concerns. I maintain my positive rating.", "rebuttal": "We thank the reviewer for their thorough analysis and constructive feedback.\n\n---\n> **W1: Discussion on the assumption of Gaussian distributions with shared covariance.**\n\nWe acknowledge that assuming a unimodal Gaussian distribution with a shared covariance may not capture the full complexity of all hypothetical, multi-modal data structures. For instance, the performance on the ImageNet-C corruption benchmark is significantly lower compared to clean ImageNet. However, as we discuss in **Appendix A.4** (Limitations and Discussion), this assumption was a deliberate design choice made to prioritize practicality and efficiency, and it is supported by strong empirical evidence.\n\nFurthermore, as we show later in **Q3 of our rebuttal**, a more complex model does not inherently yield better results. We found that adopting class-specific separate covariance matrices actually degrades performance, while significantly increasing memory usage and inference time. This suggests that **our simpler assumption strikes a crucial and effective trade-off between model fidelity, robustness, and computational efficiency**. This balance is precisely what enables ADAPT to be, as the reviewer insightfully highlighted, \"a compelling solution for real-time or resource-constrained scenarios, a significant advantage over many existing TTA methods.\"\n\n---\n> **W2: More comparisons with prior works.**\n\nWe extend the comparisons to include most of the works mentioned by the reviewer, excluding DPCore [1] and FOA [4], which require source data and thus violate ADAPT's source-free protocol. Instead, recent SOTA training-free methods such as AWT [6] and RA-TTA [7] are included. A more detailed comparison and discussion will be provided in the final version.\n\n| | Method | Venue | ImageNet | ImageNet-A | ImageNet-V | ImageNet-R | ImageNet-S | OOD Avg. | Avg. |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| | T3A [2] | NeurIPS 2021 | 66.09 | 47.54 | 61.34 | 69.24 | 46.82 | 56.24 | 58.21 |\n| | DynaPrompt [3] | ICLR 2025 | 69.61 | 56.17 | 64.67 | 78.17 | 48.22 | 61.81 | 63.37 |\n| | B²TPT [5] | AAAI 2025 | 69.57 | 55.26 | 65.40 | 78.64 | 49.53 | 62.21 | 63.68 |\n| | AWT [6] | NeurIPS 2024 | 71.32 | 60.33 | 65.15 | 80.64 | 51.60 | 64.43 | 65.81 |\n| | RA-TTA [7] | ICLR 2025 | 70.58 | 59.21 | 64.16 | 79.68 | 50.83 | 63.47 | 64.89 |\n| | ADAPT (Online) | | 70.91 | 63.32 | 64.64 | 80.66 | 53.13 | 65.44 | 66.53 |\n| | ADAPT (Trans.) | | 71.56 | 63.77 | 65.59 | 80.64 | 53.87 | 65.97 | 67.09 |\n\n**References**:\n\n[1] DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation (ICML 2025)\n\n[2] Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization (NeurIPS 2021)\n\n[3] DynaPrompt: Dynamic Test-Time Prompt Tuning (ICLR 2025)\n\n[4] Test-Time Model Adaptation with Only Forward Passes (ICML 2024)\n\n[5] Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025)\n\n[6] AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation (NeurIPS 2024)\n\n[7] RA-TTA: Retrieval-Augmented Test-Time Adaptation for Vision-Language Models (ICLR 2025)\n\n---\n> **Q1: Clarification on the Oracle ADAPT.**\n\nAs noted in **Section 4.3**, Oracle ADAPT uses the **same setup as transductive ADAPT, but with one key difference: it leverages test set ground-truth labels to estimate the target distribution**. While this is not feasible in real-world TTA scenarios, **this setting serves as an upper bound for performance**. In contrast, our standard ADAPT is fully unsupervised and label-free.\n\nAs shown in **Table 4** of our main paper, our standard method achieves results within **5%** of this oracle, showing that ADAPT can closely approximate ideal adaptation performance without supervision. This highlights the strength and practicality of our approach in real-world test-time adaptation settings.\n\n---\n> **Q2: Efficiency analysis.**\n\nWe already provided efficiency analysis in **Table 6** of our main paper. Compared with optimization-based methods (e.g., TPT, DiffTPT, and TPS) and training-free methods like TDA, ADAPT achieves the largest performance gains on ImageNet with **just 1h 11m and 0.93GB in the online scenario**.\n\nNotably, our **transductive ADAPT** builds the knowledge bank once and predicts all samples at once, requiring only **0.73 minutes** and **3.37GB**. These results highlight ADAPT as a practical solution balancing accuracy and efficiency.\n\n---\n> **Q3: Comparison with separate covariance matrices.**\n\nWe conducted an additional experiment on ImageNet by **replacing our shared covariance matrix with class-specific covariance matrices**, while keeping all other settings identical.\n\nAs shown in the table below, this variant leads to **a noticeable drop in accuracy, along with substantial increases in both computation time and memory usage**. We attribute this performance degradation to data sparsity. When only a few high-confidence samples are available for each class, estimating a full covariance matrix becomes unreliable. This can lead to overfitting and poor generalization.\n\nIn contrast, using a shared covariance matrix allows the model to pool information across all classes, yielding a more stable and robust estimate. This design is especially beneficial in online or data-scarce scenarios, where efficiency and generalization are crucial.\n\nThese results further validate our design choice. **Our shared covariance not only enhances performance but also supports ADAPT's efficiency, low latency, and applicability across diverse settings, making it a practical solution for real-world deployment**.\n\n| Method | Time | Acc | Gains | Mem. (GB) |\n| :--- | :--- | :--- | :--- | :--- |\n| CLIP | 8m | 66.74 | - | 0.79 |\n| ADAPT (Online) | 1h 11m | 70.91 | 4.17 | 0.93 |\n| w/ Separate Cov. | 1h 44m | 53.80 | -12.94 | 2.89 |\n| ADAPT (Trans.) | 0.73m | 71.56 | 4.82 | 3.37 |\n| w/ Separate Cov. | 1.40m | 66.16 | -0.58 | 4.62 |\n\n---\n> **Q4: Why exclude the current test sample from class mean updates in the online but include all samples in the transductive settings?**\n\nWe already analyzed and discussed this in the last paragraph of **Section 3.2 (Remark)** of our main paper, with further details provided in **Appendix A.4 (Remark)**. We summarize the key points as follows.\n\nIn the online setting, we exclude the current test sample from class mean updates to **enable a one-pass, iteration-free solution**. High-confidence samples are automatically stored in the knowledge bank and used for future updates without additional processing. So, this design also avoids injecting noise from low- or mid-confidence predictions. As shown in **Table 8** of our main paper, this strategy **achieves comparable accuracy to the iterative version (with the current test sample) while achieving a 4× speedup on ImageNet in the online scenario**.\n\nIn contrast, **the transductive setting assumes access to the entire test set beforehand, allowing all samples to contribute to a more accurate estimate of the class distribution**. To leverage global consistency without iterative optimization procedures such as Majorization-Minimization, we replace hard latent assignments with soft labels.\n\nAs demonstrated in **Table 8** of our main paper, this choice leads to an effective balance between accuracy and efficiency.\n\n---\n> **Q5: On the robustness to high inter-class similarity and low-confidence scenarios.**\n\nThank you for raising this question regarding challenging cases with high inter-class similarity and limited confident samples.\n\nAs demonstrated in the **fine-grained categorization results in Table 4** of the main paper, where distinguishing between classes is particularly difficult, ADAPT achieves state-of-the-art results, highlighting its effectiveness even when classes are highly similar.\n\nTo further assess the impact of **limited high-confidence samples**, we vary the number of available samples per class (denoted as **N**) in this constrained setting. The results in the table below show that our method remains robust. Specifically, **Online ADAPT performs reliably with as few as 4–6 confident samples per class, and Transductive ADAPT maintains competitive accuracy even when provided with only two confident samples per class**.\n\nWe acknowledge, however, that extremely low confidence across all classes presents a challenge, especially for the efficiency of our ADAPT in the online scenario. As shown in **Table 9** of our main paper, if the test stream begins with many low-confidence predictions, it takes longer to collect reliable samples for the memory bank, which may slightly delay adaptation and degrade performance.\n\n|   | N | 2 | 4 | 6 | 8 | 10 |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Online | Task 1 | 59.61 | 64.19 | 65.50 | 66.10 | 66.33 |\n| | Task 2 | 22.95 | 26.00 | 27.26 | 27.93 | 28.22 |\n| | Task 3 | 63.87 | 67.46 | 69.04 | 69.89 | 70.31 |\n| Trans. | Task 1 | 67.02 | 67.11 | 67.09 | 67.10 | 67.04 |\n| | Task 2 | 30.26 | 30.29 | 30.29 | 30.23 | 30.23 |\n| | Task 3 | 72.15 | 72.21 | 72.43 | 72.24 | 72.25 |\n\n---\n> **Q6: Analysis of parameter $\\beta$.**\n\n**The parameter $\\beta$ in Eq. 5 serves to map the scalar $\\alpha$ into the range [0, 1]**, which is then applied in computing the class mean in Eq. 9 and Eq. 12. The value of $\\alpha$ controls the balance between the CLIP prior and the empirical mean of confident test features. **We already provided a detailed analysis of α in Section 4.3 and Figure 2 (right) of the main paper**."}]

Author rebuttal:

Meta-review: ADAPT is a method for test-time adaptation, or updating on new data during testing to improve accuracy on distribution shifts, which does not require gradient optimization. Like other methods for TTA, ADAPT does not need the source data or offline access to the target test data. The experiments cover standard evaluations for test-time adaptation, like ImageNet variants including corruptions and renditions, and additional fine-grained image recognition benchmarks. The results rival or improve on existing methods with and without gradient optimization. Ablation and sensitivity analysis of the hyperparameters justify the inclusion of each part of the method.

Four expert reviewers with backgrounds on domain adaptation, transfer learning, meta-learning, and model editing vote for borderline acceptance (4, 4, 4) and borderline rejection (3). After the initial reviews the authors provided a rebuttal, and all reviewers then engaged in discussion and finalized their reviews. Following the rebuttal Hyj2 and uiWy raised their scores and o7Us and c5tn maintained their positive scores resulting in final ratings of one accept / 5 and three borderline accept / 4. While there is a shared concern about the weakness of the Gaussian assumption for class conditional distributions, there is nevertheless shared agreement on the thoroughness of the experiments and the positive results, and the choice of modeling is adequately addressed by the rebuttal. Further issues like missing related work (as highlighted by oyUs) and the possibility that results could be explained away by better prompting with a more recent model (as raised by c5tn) have also been addressed during the rebuttal. Once incorporated into the revision, the issues identified by review will have all been resolved. The area chair agrees with the final positive consensus of the reviewers on acceptance.
Decision: Accept (poster)