Bilateral Information-aware Test-time Adaptation for Vision-Language Models
Authors: Jingwei Sun
Jianing Zhu
Jiangchao Yao
Gang Niu
Masashi Sugiyama
Bo Han
Venue: ICLR 2026 Poster (2026)
Keywords: Test-time Adaptation
Vision Language Model
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Test-time adaptation (TTA) fine-tunes models using new data encountered during inference, which enables the vision-language models to handle test data with covariant shifts. Unlike training-time adaptation, TTA does not require a test-distributed validation set or consider the worst-case distribution within a given tolerance. However, previous methods primarily focused on adaption-objective design, while the data tend to be fully utilized or simply filtered through a fixed low-entropy selection criteria. In this paper, we analyze the weakness of previous selection criterion and find that only selecting fixed proportion of low-entropy samples fails to ensure optimal performance across various datasets and can lead the model to becoming over-confident in wrongly classified samples, showing unexpected overfitting to atypical features and compromising effective adaptation. To improve upon them, we propose \textit{Bilateral Information-aware Test-Time Adaptation} (BITTA), which simultaneously leverages two distinct parts of the test inputs during adaptation. Specifically, a dynamic proportion of low-entropy samples are used to learn the core representation under covariant shifts, while high-entropy samples are adopted to unlearn atypical features. This dual approach prevents the model from undesired memorization and ensures  extensive optimal performance. Comprehensive experiments validate the effectiveness in various datasets and model architectures. The code is publicly available at: https://github.com/tmlr-group/BITTA.

Published as a conference paper at ICLR 2026
BILATERAL INFORMATION-AWARR TEST-TIME ADAP- TATION FOR VISION-LANGUACE MODELS
Jingwei Sun 1,4 Jianing Zhu !+
Jiangchao Yao? Gang Niu
Masashi Sugiyama
Bo Han 1.3.
TMLR Group. Department of Computer Science, Hong Kong Baptist University
2 CCopperative Medianet Innovation Center, Shanghai Jiao Tong University
RIKENN Center for Advanced Intelligence Project
"The University of Tokyo
ABSTRACT
Test-time adaptation (TTA) fine-tunes models using new data encountered during inference, which enables the vision- -aiiuage models to handle test data with covari- ant shifts. Unlike training-time adaptation, TTA does not require a test-distributed validation set or consider the worst-case distribution within a given tolerance. How- ever, previous methods primarily focused on adaption-objective design, while the data tend to be fully utilized or simply filtered through a fixed low-entropy selection criteria. In this paper, we analyze the weakness of previous selection criterion and find that only selecting fixed proportion of low-entropy samples fails to ensure optimal performance across various datasets and can lead the model to becoming over-confident in wrongly classified samples, showing unexpected overfitting to atypical features and compromising effective adaptation. To improve upon them, we propose Bilateral information-awwan awwre Test-Time Adaptation (BITTA), which simultaneously leverages two distinct parts of the test inputs during adaptation. Specifically, a dynamic proportion of low-entropy samples are used to learn the core representation under covariant shifts, while high-entropy samples are adopted to unlearn atypical features. This dual approach prevents the model from undesired memorization and ensures extensive optimal performance. Comprehensive experi- ments validate the effectiveness in various datasets and model architectures. The code is publicly available at: https:/ //gthtb...uu//gg com/ /tom/tmlreee www.inirrrou//IITTT
INTRODUCTION
Vision-language models (VL.s), such as CLIP (Contrastive Language Image Pre-training) (Radford al., 2021), have gained widespread attention in recent years foundation models. Due to the strong prior knowledge acquired through training on massive vision and language pairs, VLMs demonstrated outstanding zero-shot generalization performance in downstream tasks (Zhang et al., 2024b; Liang et al., 2024b; Cui et al., 2024). However, in real-world applications, the inevitable distribution shift between the source and target domains (Liang et al., 2024a; Wang Deng. 2018), caused by natural variations or data corruption (Hendrycks & Dietterich, 2019) (e.g., weather changes and digital disturbances), often leads to significant performance degradation. This poses great challenges to VLMs, especially in safety-critical fields, such as autonomous driving (Zablocki et al., 2022) and medical diagnosis (Guan & Liu, 2021). To handle this problem, researchers have proposed Test- Time Adaptation (TTA), which leverages the new data encountered during inference in unsupervised way (Shu et al., 2022; 2023; Karmanov et al., 2024; Ma et al., 2024) to adapt to shifted domains.
Despite the promising results achieved by previous research, we observe that most existing TTA algorithms focus on improving optimization techniques while overlooking the adaptation data (Shu et al., 2022: Sui et al., 2024; Osowiechi al., 2024). In test time, the ground-truth labels are unavailable, the models are fine-tuned with an unsupervised loss on those unlabeled data to adapt to shifted distributions. Generally, as shown in the left part of Figure 1, all the input data will be
Equua Contribution, &Trainee at RIKEN AIP. Correspondence Bo Han (bhanml @ comp hkbu.edu.hk)

---

Published as a conference paper at ICLR 2026
Previous Work:
Oun
Encoder
Eccodar
photoo-ff a (rlond
photo of (clas)
Text
Eraxddrr
Gosoder
Dog
Radio
Selectio
Natio
Selection
Dog
(Bed
Figure 1: Comparison of previous low-entropy selection criteria and our bilateral information- aware variant. We pay attention to bilateral information for simultancously selecting low entropy samples for learning and high entropy samples for unlearning.
considered as the adaptation target, then a low entropy selection criterion will be used to select a fixed proportion of confident samples to minimize the learning objectives (Shu et al., 2022; Niu et al., 2022; 2023; Feng et al., 2023). The intuition is based on an assumption that low-entropy samples contain typical features, from which the model can learn representative information about the adaptation target. However, it naturally raises the two questions: Does the fixed low-entropy selection beneficially affect adaptation on all input data? Are those high-entropy samples all useless?
In this paper, we start by highlighting two concerning phenomenons: learning on low-entropy samples can exacerbate the errors on certain data, and the fixed selection ratio cannot adapt to all data. As shown in Figure 2, we analyze the predictions of misclassified samples under TTA with CLIP model after applying previous selection criteria (Shu et al., 2022). The results indicate that while some initially misclassified samples are correctly classified during TTA, with lower output entropy (i.e., higher confidence), a significant number misclassified samples are still incorrectly classified. More critically, these samples also exhibit lower output entropy along the process, suggesting that the model memorizes certain atypical features during optimization on the confident samples, which compromise the adaptation. Fortunately, encompassing various atypical features, the high-entropy samples which the current model is unconfident can exactly serve as a potential candidate for appropriate regularization during adaptation. Additionally, the fixed selection ratio also restricts the model's 's adaptation performance. The ablation experiment in Figure 9 confirms that data with different distributions have different optimal selection ratios. Through a closer look, we found that the entropy values corresponding to different optimal selection ratios exhibit high consistence, which provides an important basis for generating standardized signals to indicate the optimal selection ratio.
Based on our findings in the analysis, we propose Bilateral Information-avee aware and Test-Time Adaptation (BITTA) to adapt VLMs during test time. In general, utilizes data distribution informa- tion to calculate a dynamic proportion of low-entropy samples to learn representative information and uses high-entropy samples to unlearn atypical ones (as shown in the right part of Figure 1). On the one hand, considering the scattered feature distributions in corrupted data, we utilize low-entropy samples for model learning by minimizing entropy to reduce the uncertainty of output results, enhancing the consistency between vision and text features, and improving the separability between classes. On the other hand, considering the memorization on atypical features in low low-entropy samples, we conduct unlearning on high-entropy samples to prevent misleading adaptation with atypical features. To demonstrate the effectiveness of our method, we conduct extensive experiments on three commonly used corrupted datasets, including CIFAR- 10-C (Krizhevsky et al., 2009), CIFAR-100-C (Krizhevsky et al.. 2009), and ImageNet-C (Deng et al., 2009), to improve VLMs' robustness to common im- age corruptions in safety-critical fields. We also provide further discussions on more cross-domain datasets and various ablation to justify the method's rationality. Our main contributions are as follows:
Conceptually, we reveal two critical phenomenons in TTA with VLMs from new perspective, i.e., the fixed previous low-entropy selection criteria causes unexpected overfitting on atypical features, and cannot adapt to different data distribution. (in Section 3.1)
Empirically, we recognize high-entropy samples can function as auxiliary set to effectively offset the atypical features memorized during the adaptation process and the entropy values can serve as standardized signals to indicate optimal selection ratio. (in Section 3.2)
Technically, we propose BITTA, which innovatively selects bilateral information with dynamic ratio and simultaneously implement learning on low-entropy samples to fit core representations and unlearning on high-entropy samples to avoid overfitting. (in Section 3.3)
Experimentally, we conduct extensive explorations to verify the effectiveness of BITTA under different scenarios, including the significant improvement on various datasets, com- patibility with diverse methods and model architectures, etc. (in Section 4)
2

---

Published as a conference paper at ICLR 2026
<bar TIA
droobbo CLP
ATW T1A
APal ETA
AND SPF
Afle TEA
OlAa LOLC 01.333
Scoe
Sceee
2422
22:43
Figure 2: Entropy dynamics different data during adaptation. The top line shows the entropy values when optimizing using low-entropy selection, while the bottom line shows the changes in the highest similarity score between samples and labels. It demonstrates that only learning on confident samples can induce unexpected overfitting (as decreasing entropy values) on misclassified samples.
2 PRELIMINARIES
In this section, we review the preliminaries of zero-shot prediction with CLIP and some commonly used techniques for test-time adaptation. In Appendix B, we provide discussions on related work.
Zero-Shot Prediction. The CLIP model (Radford et al., 2021) utilizes two pre-trained encoders to map images and textual descriptions into shared feature space IRd, For K-elass classification task, test images x are sequentially fed into CLIP's visual encoder E, to generate the image feature fv, and textual representations {ft, ) for each category are generated through the text encoder E. CLIP can perform zero-shot predictions by computing the similarity between the extracted
(ospeii((fr.fs))) image feature and textual representation as follows, p(y c/a)) = Σ, exp(sim(f-JJe, where is
specific category. The pairwise similarities sim(:--) are calculated using cosine similarity, and represents the temperature parameter in the softmax function.
Test-Time Adaptation (TTA). TTA allows models be fine-tuned based on new data encountered during inference, which enable the generalization of vision-langusan models on data with covariant shifts (c.g., common corruptions (Hendrycks & Dietterich, 2019)). TTA is generally realized by conducting entropy minimization (Niu et al., 2022; 2023; Wang et al., 2020) on low-entropy predictions to filter out the noise from test data:
1
min
PM Σ [H(x)))) < rl Σ ply p(y= = cl: log cla,,,
(I)
where M is the batch size of test data and K is the number of categories which is assumed to be known. H measures the output entropy and 7 represents the threshold corresponding to the pre-defined selection ratio For cases where batch size is 1, a family of random augmentations A (Shu et al., 2022) typical used to generated M augmented views (A1, ...
METHOD
In this section, we present and discuss the key observations about unexpected overfitting and pro- portion adaptation problem that motivate our approach (in Section 3.1). Second, we discuss the key intuition behind the dynamic selection of bilateral information about input data during TTA (in Section 3.2). Lastly, we provide a detailed implementation of our proposed algorithm (in Section 3.3).
3.1 DRAWBACKS OF LOW-ENTROPY SAMPLES SELECTION.
Overfitting on atypical feature. We adopt the confidence selection strategy (Shu et al., 2022) to pick the top 10% of samples with the highest confidence and optimize them using the entropy minimization strategy (Shu et al., 2022; Maharana et al., 2025), and then analyze the entropy changes four types of samples. As shown in Figure 2, the samples that remain correctly classified and those which were initially misclassified in the zero-shot CLIP but correctly classified after TTA exhibited lower entropy and higher confidence, indicating that the model has learned some typical features. This not only
3

---

Published as a conference paper at ICLR 2026
SEANUIC
Ream Enarop
13 et
202
10
Time Stop
(a) Atypical sample (b) Entropy change during TTA
(c) Entropy values under different selection ratios
Figure 3: Motivation behind regularization and standardization. (a) Visualizations of the misclassified low low-entropy samples (top) and indistinguishable high-entropy samples (bottom), which both stem from noise information; (b) Entropy value change of the misclassified low-entropy samples when unlearning those high-entrops counterpart, which demonstrates the correlation between this two parts. (c) The entropy values corresponding to different selection ratios remain remarkably close.
helps correct previous errors but also boosts the model's ' confidence in subsequent classification. However, for the other two types of samples (i.e., those that remain misclassified or those that were originally correctly classified but misclassified after TTA), the model keeps wrong predictions with higher confidence. This suggests that the model has memorized some atypical features from selected samples (i.e., the noise information contained in the high confident incorrect predictions), which compromise the effective adaptation during inference.
Incompatibility of fixed selection ratio. The sample selection ratio is also a crucial and noteworthy question which previous works often overlook. Typically, a pre-set threshold is introduced to select a fixed proportion of low-entropy samples. To illustrate the drawbacks of this strategy, we conduct optimization using different selection ratios on multiple datasets and count the corresponding optimal selection ratio. As shown in Figure 9 and Table 8, a remarkable drawback of this strategy is that the pre-set ratio cannot enable optimal performance for different datasets. Even within the same dataset, the optimal selection ratios for different corruption types are inconsistent. This indicates that there are inherent differences in the effective information density of data with different distributions, and is necessary to dynamically adjust the selection ratio to achieve the optimal performance.
3.2 REGULARIZATION AND STANDARDIZATION
The aforementioned overfitting problem means that some misclassified cases within low-entropy samples cause the model to unconsciously memorize some atypical features, leading to overly confident decisions when facing wrongly classified ones. Therefore, additional regularization should be considered. However, the atypical features seems to be entangled into the low-entropy samples during TTA which is hard to be filtered out regarding the sample-level selection. Fortunately, we also have a part of high- samples, containing various atypical features which can be used to regularize the adaptation process. Moreover, for the incompatibility of fixed low-entropy selection ratio, we need to identify standardized characteristics which can reflect the optimal selection ratio.
Regularization with atypical feature. To verify whether is feasible to utilize nigh-entropy samples for regularization, we visualize some misclassified samples from both high-entropy and low-entropy part Figure 3(a). As can be seen, for the misclassified low-entropy samples, the model mistakenly identifies some noise information as key features of the wrong category. Similarly for the high-entropy part, the key features that could originally be used to distinguish categories are masked by massive indistinguishable noise. It can be seen that both the indiscriminability of high-entropy samples and the misclassification of some low-entropy samples stem from noise information. Therefore, the atypical feature from both parts is similar. More visualization of atypical samples are shown in Appendix H.3. To further validate the feasibility of using high-entropy samples to regularize the overfitting on atypical features, we conduct active unlearning on the high-entropy samples. By selecting the bottom 10% of samples with the highest entropy values, we increase the model's prediction entropy for these samples to further increase uncertainty. Mcanwhile, we track the entropy changes of those misclassified low-entropy samples. As shown in Figure 3(b), the increase in entropy of high-entropy samples also triggers the increase in entropy of low-entropy misclassified samples. demonstrates the close correlation between that two parts and the potential capability of using high-entropy samples as a representative auxiliary set. To further corroborate the rationality utilizing high-entropy samples for unlearning, we provide the following theorem to demonstrate their indistinguishabiiity.
4

---

Published as a conference paper at ICLR 2026
SmaAE of Compunec Dara
Ourput Errrppp Sert
Meae
TPn Eutrice
Entragy misskon
*.*
Sa
C11 Feaner Space
Iragg Feature
ner excitionn SSpp abditt
Tuane
Wanding
Figure 4: Overview of BITTA: Left: Given stream of corrupted data, BITTA conduct inference using image features and text embeddings generated by the CLIP model. Lest-middle: Estimating appropriate sample selection ratio according to the output entropy and number of categories. Right- middle: High-entropy samples will be used for unlearning to avoid overfitting. Right: Low-entropy samples will be used for learning core representations, with various methods available for this process.
Theorem 3.1 Assuming there exists one generation function g(c, s). utilizing class and distribution information to generate image z. Now we have one image Zigt from target domain belongs to class Ck. we assume that it experiences huge distribution shift h from source domain Sare i.e. Tigt = -(Ck, Sare + h.) we then have that class Og can be strictly distinguished from all the other D(cr,eg) where D(ca, Cw)) is the minimal distance between Cj and ej in the
classes cæ when h < 2Ju source domain, Ju is an upper bound of the Jacobian spectrum norm of g.
find that it is theoretically difficult to distinguish the samples that deviate significantly from the source domain and high-entropy samples that deviate severely from the source domain due excessive noise exactly belong to this category of samples. Therefore, to prevent the model from falling into overfitting, it is wiser to avoid learning these unlcarnable features.
From this theorem,
Standardization of selection ratio. To identify common characteristics across different selection ratios, we analyzed the entropy values of the samples at these optimal selection ratios, As shown in Figure 3(c), within the same dataset, the samples exhibit similar entropy values across these different ratios. This leads us to hypothesize that the initial entropy value may reflect the optimal selection ratio. However, we also observe that the initial entropy values differ across datasets. By analyzing the initial data obtained during the adaptation process (i.e., the entropy values of the first batch), we noticed a certain regularity. Specifically, there exists a linear relationship between the initial entropy and the number of categories (which will be discussed in detail in Section 3.3). Based this relationship, we are able to dynamically predict the selection ratio, yielding reasonable adjustments. To provide theoretical support for this prediction, introduce the following theorem.
Theorem 3.2 Let X, denote a butch of data, represent the predictive function fitted the training set data, and Y, = f(xi) denote the low-entropy sample selection ratio predicted by this function. Now, we have the prediction error {S; = IY: Yil]i=i-n of the selection mod- ule calculated on calibration set (X1. Y1)...(X,,, Yn), measuring the difference between the pre- dicted value Y, and the ground-truth value Y, and the corresponding quantile Q(1 a) = quantie(1-o)(--1/0) SS, satisfies the marginal coverage guarantee P(Yx+1 E @(Y++ 10t---Q(1-0 (1), a), 8n+1 + +Q11111000 a)) 2 1 a.
S2,,.,, Then, for test data n+ its corresponding ground-truth Yn+
From this theorem, the ground- tround-truth optimal low-entropy sample selection ratio has at least probability to fall within the aforementioned interval. Therefore, as long as the width of this interval remains low level when a takes small, the predicted result can be theoretically guaranteed to be strictly close to the true optimal value. Due to space constraints, the completed proof of Theorems 3.1 and 3.2, as well as the calculation for the interval width, can be found in Appendix C and E.
3.3 METHIO REALIZATION
Based on the above insights, we present the framework and learning objectives of our proposed BITTA. The overall algorithms of the method is summarized in Appendix D.
5

---

Published as a conference paper at ICLR 2026
Framework. As shown in Figure 4, our framework consists of the following four core components: (I) given a stream of corrupted data, we adopt well-trained VLMs (e.g., CLIP), extracting image features and text embeddings to calculate cosine similarity; (2) then estimate the required sample selection ratio during the process hased on the distribution of data categories; (3) using high-entropy samples to unlearn atypical features contained in low-entropy samples; (4) simultaneously using low-entropy samples to learn core representations. Since our framework is designed via a data perspective, is general and compatible with different specific adaptation learning parts (Shu et al., 2022: Sui et al., 2024; Maharana et al., 2025).
Bilateral information-awane adaptation. As aforementioned, the adaptation with low-entropy selection results overconfidence when handling misclassified samples. Thus, we consider using the high-entropy samples alleviate overfitting, for which the overall objective is minimize Ceaamnng + >Cunteamiigg where objective for high-entropy samples, which targets prevent the model from overfitting due to memorizing excessive atypical features, Lleaming follows the same setup as previous TTA algorithms, which is responsible for learning core representations. Concretely, the unlearning loss is defined as,
is a balanced weight, Lunkeaming indicates the newly-introduced
K p(usl (nigh) log(p(yi Xhigh) ))-
Luntearning
where Xnigh indicate the selected high-entropp samples among input data. With reverse optimization direction, it is designed to increase the predictive entropy of high-entropy samples, thereby enhancing the model's uncertainty in these predictions with atypical features.
For the learning part in TTA with CLIP model, various traditional methods can adopted. Here we adopt the advanced adaptation-objective proposed by BAT (Maharana et al., 2025) as an instance, which fine-tune the layernorm layers to adapt VLMs. It consists of the following three components:
K
K
Lleaming
p(ui (oow) log(p(us)
Se
T[i + +J(1 sim Ry)),
K
which is to reduce prediction uncertainty, enhance image-text alignment, and improve discrimination ability across different categories, respectively. X1ow is the selected low-entropy samples and G. is computed by averaging those visual embeddings whose corresponding pseudo label is c.
Dynamical estimation threshold. As we analysed before, the sample selection ratio is also a crucial and noteworthy question which previous works often overlook. A pre-set fixed low-entropy sample selection ratio cannot enable optimal performance for different datasets. Considering the entropy values of the initial data corresponding to different optimal selection ratios are similar, we extract a subset of images from the training set of several different datasets, and find the following relationship between the category number K, prediction entropy H(xx), and the corresponding entropy Tf of the optimal selection ratio:
Mox(H(a;) - 0.00038 + 0.83.
Subsequently, we use 7/' = = EM I[H(),;) < <T" to convert the entropy into the predicted optimal ratio, which yields reasonable adjustments. However, for high-entropy samples, we adapt a fixed selection ratio T as we have observed that fixed ratio of high-entropy samples is sufficient to achieve optimal performance, which indicates that, unlike the requirement for diverse low-entropy samples, small amount of high-entropy samples provide sufficient regularization signals. More detailed explanations of our dynamical estimation module can be found in Appendix
EXPERIMENT
In this section, provide comprehensive verification. First, introduce details of experimental setups (in Section 4.1.. Second, we provide performance comparison and compatibility experiments (in Section 4.2). Third, conduct extensive ablation studies understand BITTA (in Section 4.3).

---

Published as a conference paper at ICLR 2026
Table I: Comparison of TTA mean accuracy (%) across different corruption types, using two different CLIP models. A highlighted the improvement in green over the rop-performing baseline.
Commec
SPEN
Mothod
Meun
VIT-B-16
57.99
41:73
54.411
71.70
40.91
67.92
73.61
73.99
77.41
70.26
84.47
62.29
$3.82
47.60
59.39
61.16
TPT
39.91
44.91
58.76
72.25
43.53
70.03
74.77
75.48
76.51
72.71
85.06
70.54
57:25
52.29
61.27
63.84
TDA
42.94
46.04
63:27
72.27
46.59
69:76
74.65
77.46
12:35
71.57
62:38
60.4%
63.72
02.94
65.30
RCA
$1.51
54-53
54.17
66.35 50
35
64.93
701 59
73.00
26.27
8+02
55.42
51.52
55.80
55.95
58.61
DMN-ZS
44.97
40:50
59.77
73.60
46.75
70.5
74.80
27.05
77
2005
8523
62:55
62:35
55.30
62:05
65.38
DPC
34.78
38.66
57.81
73.04
46.30
68.59
75.89
74.26
78.89
71.17
8550
64.43
57.49
52.37
58.54
62.45
CIPAR-10-C
BAT
60:47
65.48
63.38 40
50.09
52.34
80 40
82.00
83.05
83.55
61.25
80.63
82.19
67.54
60.52
68.06
73:34
BAT+BITTA
62.59
07.42
64.97
80.63
56.25
80.74
82.39
83.52
84.29
81.92
89.50
83.25
69.13
65.50
69.26
74.78
+2.12
+1.94
+1.57
+0.54
+3.91
+0.34
+0.29
+0.47
+0.74
+0.67
+0.25
+0.96
+1.59
+1.78
+1.30
+1.44
VIT-B-32
55.58
40.07
45. 16
69 96
41 10
64.51
70.19
70,80
72. 34
66.65
81.36
64.51
59 66
46.16
56. 56
59.01
TPT
43.00
57.25
46.77
71.24
46.46
65.01
72.67
73.68
75.75
63.94
83.86
75.51
62:55
50.39
57.80
63.46
TDA
47.40
51.38
52.15
72.68
56.42
69.19
75.08
75.77
76.01
72.27
84.70
63.35
66.35
56.54
66.55
65.40
BCA
47.88
51.13
51.5 54
71,65
52.97
68.64
75.75
76.14
77.55
70.76
84.43
65.07
65.76
56.36
60.39
64.83
DMN-25
45.90
48.60
43:85
70.30
48.30
65.00
72.20
72.10
73.05
67.95
81.40
67.70
63.10
53.20
59.05
62.33
DPE
38,83
41.34
43.76
72.49
42.00
65.36
73.05
73.99
75.19
67.45
83.45
65.56
02.58
54.25
57.30
61.15
BAT
49.40
54.79
52.40
76.11
53.08
75.06
76.29
76.81
76.32
74.83
85.35
78.33
66.58
53.06
60.39
67.49
BATABITTA
52.34
57.61
52.28
76.53
55.99
75.79
76,83
77.64
78.68
76.79
86.50
M0.41
67.44
58.45
61.69
69,92
+2.94
+0.38
-0.12
+0.67
+1.57
+0.73
+0.54
+0.36
+1.96
+1.58
+0.36
+1,59
+1.14
+1.53
VIT-B-16
19.57
21.39
25.
26
42,46
20.08
43.19
47.9%
48.44
40.70
41.69
$7.00
34.37
29.21
23.03
32.47
35.79
TPT
17.21
18.96
25.4 43
42.73
20.04
42.68
46.27
49.23
50.22
42.49
57.75
37.99
30.20
25.15
32.46
36.06
TDA
23.36
24.34
32 60
42.31
20.17
42.04
46.93
48.45
45. 26
41.46
$7.00
$1.95
32.95
33.39
33.41
37.34
RCA
12.48
13.51
25.93
39.85
19.10 10.16
40.54
45.34
47.95
46.75
20.66
55.77
27.07
31.5 59
$1.59
31.33 11.38
33.07
DMN 29
19.75
21.95
29.80
42.50
18.30
43.40
48.30
69.10
51.05
43.85
56.70
35.80
31.80
25.45
31.2
36.67
DPC
22.74
26.18
29.95
45.48
22. 18
45.83
50.67
69.97
90.84
43.57
59.55
36.61
31.3 11.55
27.59
34.3
CIPAR-IO-C IIL-C
29.25
54.85
50.69
45.34
34.7%
37.17
BAT+BITIA
35.19
26.06
48.95
55.24
$2.56
51.44
48.66
35.32
37.79
42.56
+2.59
+0.36
+0.39
+0.39
0.62
VIF-B-32
16.20
17.82
17:55
29.06
17.68
38.59
43.83
42.30
43.37
39.59
50.38
29.36
28.7%
29.40
31.78
IPT
14.76
16.37
15:56
18.62
42:82
43.01
44.16
51.36
29.35
28.69
31.35
19.30
22.04
45.38
45.7%
40.38
33.08
31.85
34.60
14.39
15:38
46
44.55
34.39
30.39
32.76
DMN-ZS
16.90
15.70
52.25
25.85
18.33
19.33
31.72
31.97
34.58
14.76
44.24
58.54
34.74
36.20
BAT+BITTA
21.50
23.42
23.81
50.76
47.92
47.12
35.03
33.81
+0.87
+0.60
+0.93
+0.59
+0.75
13,00
13:68
15.55
26.60
30.70
17.58
13:60
9.64
16.20
23.98
57.76
19:10
14:26
25.04
35.06
14:30
6.74
7.38
26.00
14.96
24.90
25.88
33.26
22.94
37.22
56.14
17.22
14:34
39.00
35:90
25.42
DMN 25
11.90
12.24
12:20
23.20
14.96
25.04
23.10
29.06
53.06
1696
12:10
31.56
33.10
25,40
DPF
10:26
13:46
17.28
26.36
17.96
27.46
24 86
35.20
33.24
30.00
20.16
15.06
36.78
27.37
Image-Nesc
BAT+BITTA
20:18
22.46
24.10
23.48
22.44
19.92
27.36
28.00
23.94
21.02
31.88
31.34
29.30
28.66
36.92
32.22
31.48
40.8
41.86
56.36
55.88
26.42
26.25
23.90
22.68
39.90
40.38
39.68
30.22
31,88
31.06
+10.62
+2.52
+0.54
+2.92
+0.64
+0.64
+0.66
-1.02
+0.38
+0.30
-1.22
+0.38
40:46
+0.52
VIT-B-32
14.30
14:54
23.90
12.00
21.96
19.96
26.36
29.12
50,00
16.66
19:12
31.70
31.22
23.45
13.46
13.54
14:20
23.76
12:
12:10
20.04
20 56
25.00
26
30.58
$0.96
18
19:64
33.36
32:56
23.81
TDA
11.92
13.12
12:00
24.40
22.04
22:34
25.42
25.87
32.58
51.12
16.74
20.28
36.12
33.34
24.35
BCA
10.50
11:42
11:80
25.64
11.44
22.76
23:26
27.70
26.00 23.00
31.24
51,26
17.60
20.16
35.44
33.24
25.05
DMN-2S
14.00
14.14
13:36
26.04
12.50
23.50
23:36
25.04
26. 36
32.44
49.35
49.76
17,40
19:30
33.84
29.95
26.28
DPE
17:25
16:54
17:48
10
13:52
26.78
23:44
28.06
26 64
33,32
$1.70
18.66
22:
34.06
35, 33:42
25.09
BAT
19.08
19:70
19:30
26.36
19.34
25.08
25.42
25.56
26.16
35.46
50.08
17.10
27:04
36.70
36.14
27:58
BAT+BIUTAA
22.48
23.34
22 30
27.20
20.22
29.26
25.80
28.82
27.46
35.94
50.00
23.96
26. 26.96
36.78
36.30
29.05
+3.40
+2.64
42.90
+1.18
+0.84
+0.55
+11.38
+0..6
+1.62
+41.8
-1.70
+5.30
40.08
+1..8
40.16
+1.47
Caussian
JPEG
Gamsitzn
Elastic
JPEG
Ours
Airplaze
Astomnibie
Bard
Dog
Prag
Hore
1P1p
Figure 5: t-SNE plots of image features from BITTA and zero-shot VIT-B-16. The results indicate that our method produces more discriminative features that are more tightly clustered.
4.1 EXPERIMENTAL. SETUPS
Baselines. For performance comparison, we adopt some commonly used method. including TPT (Shu et al., 2022), TDA (Karmanov et al., 2024), BCA (Zhou et al., 2025). DMN-ZS (Zhang et al., 2024d), DPE (Zhang et al., 2024a) and BAT (Maharana et al., 2025). For compatibility experiments, we adopt TPT (Shu et al., 2022), DifITPT (Feng et al., 2023), CTPT (Yoon et al., 2024) and TPS (Sui et al., 2024). We provide more implementation details of each TTA method in Appendix A.
Implementation details. We test on CIFAR-10-C, CIFAR-100-C (Krizhevsky et al., 2009), and ImageNet-C (Deng et al., 2009) with a coruptiin level of 5. We assign template "a photo of <cls>' ' to the text encoder. The optimization for both vision and text encoder is based on AdamW (Loshchilov, 2017), with learning rates of 1e-3, 5e-4, and 5e-4 for CIFAR- -O-C, CIFAR- 100-C, and ImageNet-C, and batch sizes of 200, 200, and 64. More details can refer to the Appendix.
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of BITTA combined with different TTA methods over CIFAR-10-C using
ViT-B-16. A highlighted the improvement in green over the original method without BITTA.
Commenton
Frost
Mathod
Moan
TPT
19.91
44.91
58.76
72.25
43.53
70.03
24.27
75.48
78.51
73.21
85.06
70.84
57.23
52.29
61.27
63.84
TPT BITTA
42.37
47.13
68.89
73.62
46.92
71.67
15.72
76.45
79.70
73.71
86.25
71,83
50.10
54.94
62.06
65.37
A
-2.46
+2:23
+2.03
+1.37
+2.39
+1.64
+0.95
+0.97
+1.19
+1.00
+1.9
+0.19
+1.87
+2.65
+0.79
+1.33
DRTTPT
39.25
45.85
59.40
71:80
41.75
71.10
75.30
75.95
73.85
73.80
61.90
72.55
57.20
51.05
62.85
66.24
DIFTPT BITTA
41.91
48,50
62.40
73,00
46.35
73,05
76.75
76.85
90.10
74.90
85.95
71.90
68.15
54.85
63.75
66,03
A
+2.66
+2.85
+3.00
+1.30
+2.60
+1.95
+1.25
+0.90
+1.25
+1.00
+1,05
-0.65
+2.95
+3.90
+0.90
+1.79
CIFT
38.18
42.74
56.75
71.66
41.58
65.91
74.18
74.41
77.05
70.6
84.35
63.95
54.98
89.25
56.79
61.34
CTPT BITTA
39,40
44.10
58.31
72.86
43,04
69.46
74.67
74.77
78,05
71.088
85.15
64,05
55.66
50.15
59.12
62.61
A
+1.22
41.36
41.58
40.50
41.46
40.55
40.49
40.36
41.00
40.48
41.60
40.30
+0.68
+0.82
+0.33
+0:77
TPS
40.11
44.45
59.77
31.72
43.05
70.25
74.32
74.69
75.67
73.36
63.76
77.30
56.
52.85
62.59
64:28
TPS- BITTA
42,62
46.92
61.32
72.27
43.87
71.32
7482
74.97
77.82
72.87
84.39
77 30
50,82
87.26
06
65.34
+2.51
+2.44
41.39
40.55
+0.52
+1.07
40.50
+0.28
40.15
-0.49
+0.61
40.10
41.47
+4.41
-1.53
+1.96
Table 3: The classification accuracy of ViT-B-16 on cross-domain datasets. The experimental results show that the BITTA also demonstrates good compatibility on these different datasets.
Method
Flower102
DTD
Pets
Cars
UCF101
CalTech101
Food101
SUN397
Aircraft
EuroSAT
Mean Mean
TPT
68.98
47.75
87.79
66.87
68.04
94.16
84.67
65.
24.78
42.44
65.10
TPT-BITTA
87.33
91.85
86.50
66.09
TPS
71.28
50.45
87.41
69.04
70.75
95.13
85.18
68.45
26.30
43.80
66.78
TPS-RITTA
72.34
51.68
88.69
69.13
21.61
95.47
86.44
72,82
25.67
46.68
68.05
Table 4: Ablation study on prompt templates on VIT-B-16/VIT-B-22. Our method shows
Table 5: Generality on diverse architectures on ImageNet-C. Our method can boost zero- shot performance under different architectures.
great adaptability to different templates.
braac/ee-C
Varicas Beadiag Metaots de RosNeston
Gasozn
Iennabic
Bellccus
pouto et Kolate
24.78
42.56
31.35
MEMOIBITTA BITTA
33.63
14.53
34,07
31.70
blusey photo Kolayy
25.55
42.54
31.33
SAR BITTY
33:55
13:33
34.53
31.38
1a noloy phoco of <636s"
25.37
40.32
31.4
33.94
34.99
Pringe Teeghaeee MTTB-3
CIPAR:00C
CIARR (001
IrageNete
Vations Beadlss oo WilBae
Gasesn
Skat
Defoces
56 low contrtst phoro of coleer
69:38
17.31
MEMO/BITTA SAR
37.54
40:83
32:97
bauzey ghoto ot <elavy
37:15
$2.83
$2.24
$3.49
53.08
5a
09:33
SAR-BITTS
56.27
56.44
57.69
59.36
4.2
MAIN RESULTS
Comparison with different baselines. In Table 1, we present a performance comparison with several representative TTA methods to demonstrate the effectiveness of BITTA. The results show that our method consistently outperforms others across different visual backbones. In Figure 5, we present visualization results of t-SNE plots (Van der Maaten Hinton, 2008) which demonstrates that our method produces more tightly clustered category features, resulting in clearer distinctions between different categories. In Appendix H, we present detailed t-SNE results on VIT-B-I6/VIT-B-33
Compatible with different methods. As the unlearning mechanism is general design from the perspective of data utilization in TTA, BITTA can be easily integrated into previous TTA methods, leading to performance improvements. In Table 2, we compare the performance of several methods using the ViT-B- 16 backbone on the CIFAR- dataset when combined with BITTA, including TPT (Shu et al., 2022), CTPT (Yoon et al., 2024) and TPS (Sui et al., 2024). We fix their learning objective for low-entropy samples and introduce high-entropy sample selection module, implementing unlearning avoid overfitting on low-entropy samples. The results show that BITTA consistently helps these methods achieve better performance. More results are provided in Appendix F.2.
Generalization on different datasets. Cross-domain datasets are also commonly used in this field. To fully verify the effectiveness of our proposed method, we further supplemented experimental results on more cross-domain datasets as shown in Table 4.1, which confirm that BITTA has great generalization ability. More results on ImageNet variants are provided in Appendix F.9.
4.3 ABLATION AND FURTHER ANALYSIS
Effects of different prompt templates. Consistent with the prompt templates widely used in previous works, we use 'a photo of a <cls>' ' in our prior experiments. To examine the impact of different prompt templates on the experimental results, we replaced the prompt with various alternatives and reported the results in Table 4. As seen, the effects produced by different prompts are similar, suggesting that BITTA can flexibly adapt to various prompts and users can flexibly select the prompt in practical applications. Comparison with baseline methods can be found in Appendix F.4.
Generality on diverse architectures. To fully demonstrate the effectiveness of BITTA in other model architectures, such as CNNs and VIT, we present the effects of combining BITTA with two commonly used baseline (i.e., MEMO (Zhang et al., 2022) and SAR (Niu et al., 2023)) under different
8

---

Published as a conference paper at ICLR 2026
CIFAR-10-C
CIFAR-100-C
CIFAR-10-C
CIFAR-100-C
Mean Accuracy
*
M
*
VIT-B-16izero-shoo)
-TT-B-32(2ro--hhot
B. 16izero shot)
Update Steps'
Update Steps"
Update Steps"
Update Steps
Figure 6: Impact of varying update steps. Continuous update lead to the performance degradation. while the mean accuracy still significantly exceed zero-shot ViT-B-16 and ViT-B-32.
Pork
10.00
15.00
41:00
Figure 7: Ablation on dynamical selection module. Left: different low-entropy selection ratio. Optimal performance is achieved at different selection ratios for each dataset. Right: different high-entropy selection ratio. Fixed ratio is sufficient achieve optimal performance.
Aderie Su She
Abldi Shayees
Iere Sue
SPerTT
Tere Du Coreaty
After Macheshfed
Misceeshee
7 atb Evvrras anchiaa
battor
CarA 1EB can WIT
Hear EOTOy
Csii 5330 on 111-8-22
Caeck 108 can Winel
Gelencing &
stap
(a) Dift. Batch Size (b) Balancing Constant À
(c) Entropy Dynamics with BITTA
Figure 8: More ablation studies and further analysis. (a) Performance on various batch sizes. Our method performs well under various batch sizes on the benchmark datasets; (b) Performance with varying balancing constant X. Setting À to 1.2 is found to be optimal: (c) Entropy dynamics with BITTA. The results verify that the overfitting problems in misclassified samples can be alleviated.
architectures in Table 5. The experimental results clearly show that BITTA can significantly enhance the model performance regardless of the architecture. This outcome profoundly reveals that the overfitting problem widely exists in various architectures. More results can be find in Appendix F.8.
Impact of varying update steps. We perform multiple steps of parameter adaptation on a single batch. To evaluate the impact of different update steps overall performance, conduct ablation study by varying the number of update steps from to 4. Figure 6 demonstrates that continuous updates in a single batch cause loss of prior knowledge, which in turn reduces the accuracy on subsequent data and ultimately results in performance degradation. Therefore, setting the number of update steps to 1 is reasonable choice. More results on ImageNet-C can be found in Appendix F.3.
Ablation on dynamical selection. To demonstrate the necessity of this module, we show mean average accuracy across three datasets the left side of Figure 7. We gradually increased the selection ratio of low-entropy samples from 0.1 to 0.6. The results show that for each dataset, the optimal performance is achieved under different selection ratios, while our estimation module can ensure excellent performance across all datasets. Mcanwhile, on the right side, we also change the selection ratio of high-entropy samples from 0.1 to 0.3. found that setting the ratio to 0.1 is sufficient for all datasets to reach the optimal solution. Therefore, it is reasonable to design a dynamic selection ratio for low-entropy samples and a fixed selection ratio for high-entropy samples.
Performance on different batch size. In Figure 8(a), we show the average accuracy of our method using the ViT-B-16 backbone with smaller batch sizes. can be observed that even with a low batch size, our method still significantly outperforms the baseline method, demonstrating excellent adaptability, More results and discussion can be found in Appendix F.7.
Ablation study on X. We introduce À to balance the learning loss and the unlearning loss. As shown in Figure 8(b), we tested the performance of the model as À varied between 1.0 and 1.3. Empirically, setting À to 1.2 is found to be an appropriate choice. Although the results show the variance, it is worthy noting that the performance gap is smaller than 0.1 1 across different datasets, indicating a robustness of our methods regarding the X. More results can be find in Appendix F.6.

---

Published as a conference paper at ICLR 2026
Table 6: The comparison with other regularization techniques. The experimental results show that the unlearning module achieves better performance then other regularization techniques.
Counsiion
Blanic
Plaztes
JPECT
Method
Mexs
BAT
65.48
63.40
52.34
80.40
93.05
68.06
+wegght decay
61.14
80.49
92
61.57
09:15
BAT-BITTA
Table 7: The comparison of expected calibration error (ECE J). A highlighted the decline ratio in green over the original method without BITTA.
Mathed
24.35
BAT-RITTA
24.33
17.89
38.55
9.67
24.08
8.71
9.54
4.55
8.64
3.6
6.53
15.03
18.30
16.54
12.43
E3.719
$4.80%
12/05%
11.875
6,2873
17,09'
15995%
5,28%
23,89%
1800%
12:38'5
24,85%
11,9973
15.31/0
NAT-NTB--35
34.31
28.51
BAI-BITTA
23.47
31.51
26.58
11.29
23.00
11.55
18:57
9:53
8.66
18.53
4.79
7.64
14.89
23.02
19.36
15.33
16.37%
17:37%
17:41
CIRAK H
9U.1
17.82
32:17
15:11
25.66
13.74
13.06
12:67
15.35
17.15
36.95
21.31
30:30
77.47
25:03
BAT-BITIA
11.36
11.34
90335
STAF5S
50.25%
16.156
21.88%5
18.795
33.25%
23.385
38.25%
13.855
16:15
35.8775
BAT NIT--12-
34.80
22.12
15.83
23.76
15:55
13,45
13:12
15:45
13.00
2.12
15.00
21.21
35:82
18:52
18.64
$4.44%
25.36'5
33.68%
25,85%
15.41%
24.35%
33.08%
21.88%
28.64%
23.35%
31.82%
33.38%
18.38%
33.97%
18.43%
28.7599
RAT
BAT-BIITA
28.82
19.26
16,35
19A67
16:35
19.64
16.21
16.29
$4.11
9.15
34.06
22.35
15,45
15.47
17.47
25,895
33,65%
31.30%
30.811
43.08%
18.29%
15.08%
15.02%
20
28.55
$3.501
20.85%
27.386
HAT-BIITA
17.94
11.50
17.41
15.43
17.17
14.4
19.57
15.24
16.37
14.66
11.56
36.41
18.36
14.45
14.97
16.34
33.319
M6.3%
27 7000
26.859
18.54%
24.05%
155215
25.97%
35.725
48.05%
21
19.01%
19.049
27.6655
Comparison with other Regularization Techniques Apart from unlearning, we also try other regularization techniques such as weight decay (Fan et al., 2023; Chen et al., 2023) and distribution dissolve (Shen et al., 2024). As shown in Table 6, on CIFAR-106 10-C using ViT-B-16, although weight decay can achieve certain effect, it is unable to prevent the model from learning atypical features and fails to realize more reasonable data selection, thus its overall performance is weaker than that of our method. For distribution dissolve, although it effectively enables the model to unlearn high-entropy samples, it differs fundamentally from our algorithm. Our method directly and precisely guides the model confuse the classification of high-entropy images. In contrast, the distribution perturbation- based unlearning loss does not explicitly constrain the perturbation direction. This introduces the risk of the feature distribution incorrectly shifting from ambiguous regions (where the model fails to classify confidently) to discriminative regions (where the model can classify confidently). As result, while this scheme achieves comparable performance to ours on certain corruptions, its average performance still remains lower. More implementation details can be find in Appendix F.10.
Effects on mitigating overfitting. To demonstrate BITTA can effectively mitigate overfitting, we present entropy changes during optimization process using BITTA in Figure 8(c). It can be observed that BITTA can prevent the model from being overconfident in misclassified samples (i.e., the entropy of misclassified samples remains basically unchanged). This validates the superiority of BITTA. Additionally, Expected Calibration Error (ECE) is also an important metric that quantifies the calibration of model's predicted probabilities (Yoon et al., 2024; Farina et al., 2024). We show the changes of ECE after applying BITTA in Table 7. As can be seen, BITTA also bring stable reduction on Expected Calibration Error, which also demonstrate BITTA'S effects mitigating overfitting.
Additional Experiments and Discussion We also provide further analysis from different views in the Appendix, including the introduction of related works(in Appendix B), time costs of conducting TTA (in Appendix 1),, effects of different learnable modules (in Appendix F.5), performance on more cor- ruptions(in Appendix F.1 1). effects on non i.i.d. batches(in Appendix F.12), discussion on limitation and future work (in Appendix G.1). and visualization of prediction results (in Appendix H.5).
CONCLUSION
In this work, we propose BITTA, a new test-time adaptation framework focusing on the adapta- tion data. Through in-depth analysis, we identify that the previous selection criteria can induce unexpected overfitting, leading model becoming over-confident on wrongly classified samples. To better unleash the potential of adaptation, we introduce a bilateral information-awanl mechanism which simultaneously utilizes two parts of the test inputs encountered during inference. With the novel unlearning part, we mitigate the memorization atypical features and improve the model's adaptation performance. We hope this work can provide new insights for TTA researches.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
JWS, JNZ. and BH were supported by NSFC General Program No. 62376235, RIKEN Collaborative Research Fund, and HKBU CSD Departmental Incentive Scheme. MS was supported by JST ASPIRE Grant Number JPMJAP25BI I. JCY is supported by National Natural Science Foundation China (No. 62306178) and STCSM (No. 22DZ2229005).
ETHICS STATEMENT
We hereby confirm that all datasets utilized in this work are publicly accessible and employed in strict adherence their respective license agreements. This study involves no collection, use, or process- ing of human-related data (including personal information) or sensitive data (such as confidential, proprietary, or legally protected data), thereby eliminating any foreseeable risks pertaining to privacy infringement, security breaches, or fairness biases. The research is conducted exclusively for the purpose of advancing scientific knowledge in the field, with no financial, professional, or personal conflicts of interest declared by any author. Furthermore, all aspects of this work fully comply with the ethical guidelines outlined in the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of the experimental results, we summarize some critical factors that facilitate reproduction.
The datasets we used are all publicly accessible, which is introduced in Section 4. Following previous work, we test 5,000 images on ImageNet, and 10,000 images CIFAR- 10-C and 1IFAR-100-C, with all test images remaining consistent with those used in prior work.
Datasets.
Assumption. We set up experiments to Test-Time Adaptation (TTA) scenario, where a well-pretrained CLIIP model is available along with its full parameters. However, the ground-truth labels for the test images are not accessible, and there is no information transfer between different data streams. Developers can only optimize the model using the images available each specific moment.
Environment. All experiments are conducted on NVDIA Tesla V100-32GB GPUs with Python 3.9 and Py Torch 1.12.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, Advances in neural information processing systems, 35:23716-23777 2022.
Guohao Chen, Shuaicheng Niu, Deyu Chen, Shuhai Zhang, Changsheng Li, Yuanqing Li, and Mingkui Tan. Cross-device collaborative test-time adaptation. The Thirty-eighth Annual Conference Neural Information Processing Systems.
Ruizhe Chen, Jianfei Yang, Huimin Xiong, Jianhong Bai, Tianxiang Hu, Jin Hao, YANG FENG, Joey Tianyi Zhou, Jian Wu, and Zuozhu Liu. Fast model debias with machine unlearning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: // //openreview net/ forum? -ideiidd-BPPxxsseeooo
Can Cui, Yunsheng Ma, Xu Cao, Wengian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 958-979, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
11

---

Published as a conference paper at ICLR 2026
Zeshuai Deng, Zhuokun Chen, Shuaicheng Niu, Thomas Li, Bohan Zhuang, and Mingkui Tan. Efficient test-time adaptation for super-resolution with second-order degradation and reconstruction. Advances in Neural Information Processing Systems, 36:74671-74701, 2023.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Em- powering machine unlearning via gradient- based weight saliency in both image classification and generation. arXiv preprint arXiv:2310..2 12508, 2023.
Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, and Elisa Ricci. Frustratingly easy rest-time adaptation of vision-language models. arXiv preprint arXiv:2405. 010224405..3333 18330, 2024.
Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2704-272777 2714, 2023.
Jin Gao, Jialing Zhang, Xihui Liu, Trevor Darrell, Evan Shelhamer, and Dequan Wang. Back to the source: Diffusion-driven adaptation to test-time corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11786-11796, 2023.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(13)11156-11154, May 2021. doi: 10. 1609/aaai.v35i11 133.133.1111......1 URL htt.tttts///oo /oo/:////ss..... aaa www.aaa...oa.....aaa...a wwtppddwwpppppppppppeppppepppppe AAAAA//ta/aaaeeeeeaaaaaaaaa
view/ 0t///1377 17711.
Hao Guan and Mingxia Liu. Domain adaptation for medical image analysis: survey. IEEE Transactions on Biomedical Engineering, 19(3):113-11885 2021.
Iryna Hartsock and Ghulam Rasool. Vision-language models for medical report generation and visual question answering: A review. Frontiers in Artificial Intelligence, 7: 7:1430984, 2024.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint arXiv: 12261, 2019.
Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach to forgetting in deep generative models, 2023. URL https: https:://aaxi..rrrrrt/t// wwwpw/.wwg///////////000
arxiv.
Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. The IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-ingn image pre- training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888 12900. PMLR, 2022.
Xiang Li, Congcong Wen, Yuan Hu, Zhenghang Yuan, and Xiao Xiang Zhu. Vision-language models in remote sensing: Current progress and future trends. JEEE Geoscience and Remote Sensing Magazine, 2024.
Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribu- tion shifts. International Journal of Computer Vision, pp. 1-34, 2024a.
Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pp. 405-409, 2024b.
Hongbin Lin, Yifan Zhang, Zhen Qiu, Shuaicheng Niu, Chuang Gan, Yanxia Liu, and Mingkui Tan. Prototype-guided continual adaptation for incremmanemmttall unsupervised domain adaptation. In European Conference on Computer Vision, pp. 351-368. Springer, 2022.
12

---

Published as a conference paper at ICLR 2026
Hongbin Lin, Yifan Zhang, Shuaicheng Niu, Shuguang Cui, and Zhen Li. Monotta: Fully test-time adaptation for monocular 3d object detection. In European Conference on Computer Vision, pp. 96-114. Springer, 2025.
I Loshchilov. Decoupled weight decay regularization arXiv preprint arXiv:1711.05101, 2017.
Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. Advances in Neural Information Prucessing Systems, 36, 2024.
Sarthak Kumar Maharana, Baoming Zhang, Leonid Karlinsky, Rogerio Feris, and Yunhui Guo. Batclip: Bimodal online test-time adaptation for clip. 2025.
Seth Neel, Aaron Roth, and Saeed Sharifi-Malvaeerdi. Descent-to Gradient-based methods for machine unlearning, 2020. URL httpt:tt//// /ttpss///a/vvvv/ia org/ abaaabss///// 02923.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pp. 16888-16905. PMLR, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. arXiv preprint (rXxi:2302.12400 2023.
Alex Oesterling, Jiaqi Ma, Flavio P. Calmon, and Hima Lakkaraju. Fair machine unlearning: Data removal while mitigating disparities, 2024. URL https: /awxivivvvv.////a///aaaaaaaaaala http:///wwwwwwwwa/////////s//ase htrgg/abs/23000 147544
David Osowiechi, Mehrdad Noori, Gustavo Adolfo Vargas Hakim, Moslem Yazdanpanah, Ali Bahri, Milad Cheraghalikhani, Sahar Dastani. Farzad Beizaee, Ismail Ben Ayed, and Christian Desrosiers. Watt: Weight average test-time adaption of clip. arXiv preprini arXiv:2406. / 13875, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp- 9748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Re- member what you want to forget: Algorithms for machine unlearning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, volume 34, pp. 18075-18086. Curran Associates, Inc., 2021. URL http:::://pooceeedieeee neurips cc/paper, files/ pappr/2021/ file/ 927.66625543330e (tt.5aaf55.ss11333aaeeeeeeeeeee
pdf.
Shaofei Shen, Chenhao Zhang, Yawen Zhao, Alina Bialkowski, Weitong Tony Chen, and Miao Xu. Lanel-agnostic forgetting: A supervision-free unlearning in deep models. arXiv preprint 00X0::2404.00566, 2024.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei models. Advances
Xiao. Test-time prompt tuning for zero-shot generalization in in Neural Information Processing Systems, 35:14274-14220 2022.
Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long, Clipood: Generalizing clip to out-of dii-tribiiiiitsionn.. In International Conference on Machine Learning, pp- 31716-31733 PMLR, 2023.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15638-156500 2022.
13

---

Published as a conference paper at ICLR 2026
Manogna Sreenivas and Soma Biswas. Effectiveness of vision language models for open- open-world single image test time adaptation. arXiv preprint arXiv:2406.00481,, 2024.
Elaine Sui, Xiaohan Wang, and Serena Yeung-Levy. Just shift it: Test-time prototype shifting for zero-shot generalization with vision-language models, arXiv preprint arXiv:2403. /2952, 2024.
Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In 3/st USENIX Security Symposium (USENIX Security 22), pp. 4007-4022, Boston, MA, August 2022. USENIX Association. ISBN 978-1 939133-31-1. URL https://// /ww. www.usenix..ommm wttp://ooooooooeeeeeeeeeeeeeeeee /ttp//unnnnnnueeeeeeeeeeeee presentation/thun
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.1072 2020.
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang. and Qing Wang. Software testing with large language models: Survey, landscape, and vision. IEFE Transactions on Software Engineering, 2024.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312: 135-153, 2018.
Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark A. Hasegawa-Johnson, Yingzhen Li, and Chang D. Yoo. C-TPT: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. In The Twelfth International Conference on Learning Representations, 2024.
URL https:tpp/......... /tpp//pppeeeeeeeeeeeeee
net/ forum? i-= -idd:mddddj=eFO- jZZEHTBFOT.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint a1X14:2111.1143,, 2021.
Éloi Zablocki, Hédi Ben- Younes, Patrick Pérez, and Matthieu Cord. Explainability of deep vision- based autonomous driving systems: Review and challenges. International Journal of Computer Vision, 130(10):2455-2452, 2022.
Ce Zhang, Simon Stepputtis, Katia Sycara, and Yaqi Xie. Dual prototype evolving for test-time generalization of vision- language models. arXiv preprint arXiv:2410.12:1 2024a.
Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprint 1rXiv:2211.08332, 2023.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024b.
Jingyi Zhang, Jiaxing Huang, Xiaoqin Zhang, Ling Shao, and Shijian Lu. Historical test-time prompt tuning for vision foundation models. arXiv preprint arXiv:2410.20346, 2024c.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances neural information processing systems, 35:38629-38642, 2022.
Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision- language models. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, 2024d.
vision-language
Lihua Zhou, Mao Ye, Shuaifeng Li, Nianxin Li, Xiatian Zhu, Lei Deng, Hongbin Liu, and Zhen Lei. Bayesian test-time adaptation for vision-language models. In Proceedings of the Computer Vision 2025.
and Pattern Recognition Conference, pp.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
THE USE OF LARGE LANGUAGE MODELS
We use LLM as an assistant to help us polish our writing. We are grateful for the convenience that the development of LLMs has brought to research writing and call for the rational utilization of LLMs.
A DETAILS ABOUT CONSIDERED BASELINES AND METRICS
In this section, we provide details about the baseline and corresponding hyperparameters, as well as other related metries that are considered in our work.
TPT (Shu et al., 2022). For given test image, 63 augmented versions of the image generated based on random rotations, cropping, or flipping. Along with the original image, a total of 64 images fed into the CLIP visual encoder. At initialization, the prompt template <cls>' is used to generate text embeddings. The confidence threshold is set to 10%, and the top 10% of samples with the lowest entropy values are used to optimize the prompt template. The learning rate is set to 5 x 10-3
photo of
DifTTPT (Feng et al., 2023). Different from TPT, which utilizes conventional data augmentation techniques, e.g., random resized crops. expand the test data, DiffTPT use the pre-trained stable diffusion to generate data with richer visual appearance variation and selecting generated data with higher prediction fidelity. Following this work, utilize Stable Diffusion- V2 to generate the 63 augment views in the testing phase.
TDA (Karmanov al., 2024). Unlike conventional test-time adaptation (TTA) algorithms that adjust model parameters during the testing phase, TDA devises a training-free dynamic adapter, which constructs lightweight key-value cache where a small number of pseudo-labels are taken as values and the features of the corresponding test samples keys. Leveraging this key-value cache, TDA enables gradual adaptation to test data through progressive pseudo-label refinement without incurring any backpropagation operations, thus achieving efficient test-time adaptation in various real-world scenarios.
BCA (Zhou et al., 2025). BCA analyzes the prediction process via Bayes' theorem and identify likelihood and prior as its core determinants. They claim that existing methods only adapt class embeddings to tune likelihood, while neglecting the critical role of prior. Therefore, BCA leverages the posterior of incoming test samples to dynamically refine the prior of each class embedding, which enables the model to better accommodate distribution shifts and boost prediction accuracy.
DMN-ZS (Zhang et al., 2024d). DMN present a universal adaptation method. They design dual memory networks consisting of dynamic and static memory modules: the static memory module stores the knowledge of training data support training-free few-shot adaptation, while the dynamic memory module real-timely retains historical test sample features during inference, thus mining extra data information that is not included in the training set. This unique design not only improves the model's 's performance in few-shot scenarios but also ensures the model's 's applicability when training data unavailable. Moreover, the two memory modules adopt an identical flexible memory interaction mechanism, which can run in a training-free manner and be further optimized by integrating learnable projection layers. We utilize its zero-shot version (DMN-ZS) to conduct comparison experiment.
DPE (Zhang et al., 2024a). Following CLIP, multiple context prompt templates are used for prompt
ensembling. Specifically, For each class c, a total of S text descriptions, denoted as T. (i) The prototypes of these descriptions in the embedding space are calculated by Eq. (4). To further improve the quality of these prototypes over time, DPE updated them online through cumulative average with each individual sample in the test stream, as showing in Eq. (5), where t. = [ti, t2..t,ttt. ,tc).
t. 1 (.(:).)))
15

---

Published as a conference paper at ICLR 2026
(A:
t. - 1]
(5)
For visual prototypes, priority queue strategy designed to store the top-M image features for each class and symmetrically computed a set of that evolve over time. Using this priority queue, the class-specific visual prototype is obtained by Eq. (6), where S. donates the number of image features,
Ve = 3/2
1
Sc
S.
Based on these two types of prototypes, the final predictions are showing as follows, where A = aerp(-3(11 x)) is the affinity function, i is the temperature parameter. In experiments, the value of a is set to 6.0 and 3 to 3.0 on CIFAR-I0- CE-R--O-CAA and CIFAR-100-C, while on ImageNet-C, a is to 6.0 and 5.0.
exp (ff to + A(F. vo)) /t)
PΦy = velx) = E. exp ((f, to + v(f) /t)
In addition to the two types of prototypes and v., DPE also introduces learnable residual parameters i. and Ve. which are initialized to and are gradually used to update the class prototypes:
E. te + E | + +tel'
Ve
V2
vell
(8)
The optimization of L, and V. is based on entropy minimization and alignment loss which utilizes the contrastive InfoNCE loss to bring prototypes from the same class closer together while pushing prototypes from different classes further apart. The optimization objective is shown as Eq. (12).
Laug = 7 (PPPE (Xtes) )) = PDPE (y ye Xiest ) log PIPE (yr = Ve. X )
(9)
where DRE (Xost)
V (H (Prroto (An (X 7] Prrcro (An (Xust))
(10)
p.
aaiggn
3/
1
Σ
log Ee exp (t vor) En exp (tJ.v)
exp (t._ vo).vo)
log
exp (t_ (t..vo) Vo)
(11)
0
t." v" = -araaargninnnmm (Lang + ALalign).
(12)
BAT (Maharana et al., 2025). Existing TTA methods have severe limitations in adapting CLIP due to their unimodal nature. To address these limitations, BAT-CLIP design bimodal TTA method improve CLIP's robustness to common image corruptions. Specifically, BAT optimizes the LayerNorm layers of the bimodal encoder by minimizing the output entropy while enhancing the projection matching between visual and textual features and improving the inter-class separability between class prototypes. The specific loss function are as follows:
Lent = Σ p(l.)log(((()))
(13)
1 Σ Ppm 5
Eo sc:
(14)
Cap ΣΣ 111 + c|((1 ([[1-cos([er 01)).
(15)
IEC CEC
where p(l.) represents the likehood for class c, v. and &n represents visual and textual prototypes. The overall optimization objective is as follows, th and Φi represent the parameter of visual and text encoder's LayerNorm,
argmin (Lent Cpm Cep)
(16)
16

---

Published as a conference paper at ICLR 2026
CTPT (Yoon et al., 2024). Calibration is crucial aspect for quantifying prediction uncertainty. However, previous works have been mainly developed to improve accuracy, overlooking the im- portance of calibration. CTPT discovered that well-calibrated prompts exhibit more dispersed distribution of textual features. Therefore, it introduces Average Text Feature Dispersion (ATFD) assess the calibration degree of the model. A higher value of ATFD indicates more dispersed distribution of textual features, signifying better model calibration, while a lower value suggests poorer calibration. The formulation of ATFD shown follows:
N
ATFD (ppii (pwee....... (psww))
centroid U[p:oul l12
(17)
Ceentroid is the centroid of the text features computed by:
Ccentooid
(18)
By maximize ATFD during TTA process, CTPT can contribute better calibration.
TPS (Sui et al., 2024). Considering that TPT requires backpropagation through the text encoder to fine-tune the prompt, leads to high memory and time consumption. To reduce these costs, TPS attempts to perform backpropagation only in the feature space. Specifically, TPS fine-tunes the prompt directly the feature space by learning an offset vector S, for each class prototype P.. The formulation is follows:
Pcts Sc
P
(19)
||P. sell2
Similar to TPT, TPS selects low-entropy samples for training using confidence selection, and opti- mizes the offset vector se by minimizing the output entropy of the low-entropy samples. The loss function is as follows:
c=-Σile x p') logā(c Xo. P'),
(20)
CEC
where ş (c Xo) is computed as follows:
1 EE (
( Xo)
(21)
Maximum Softmax Probability (MSP). We maximum softmax probability to discriminate which class the sample belongs to. The score defined as follows,
SMSP(I: f) maxP c/æ; f) max softmar( (z)),
(22)
represents the given well-trained model, and c is one of the class Y {I.... ,.... The larger the softmax score, the higher the probability that the sample belongs class c, reflecting the model's confidence the sample.
17

---

Published as a conference paper at ICLR 2026
B RELATED WORK
Vision-Languaae Models (VLMs). VLMs have gained widespread attention for their ability to jointl process and understand both visual and textual information (Zhang et al., 2024b; Li et al., 2024; Hartsock & Rasool, 2024; Wang et al., 2024). CLIP (Radford et al., 2021) trains visual and language models simultaneously using contrastive learning, and its training on massive paired image-text datasets enables it to demonstrate strong zero-shot capabilities, making it suitable for wide range of cross-modal tasks. ALIGNLI et (2021), 1), on the other hand, leverages large-scale noisy data to broaden its applicability in cross-modal tasks and enhance its generalization ability. Subsequent models, such as DALLE- (Ramesh et al., 2021) and Flamingo (Alayrac et al., 2022), further enhanced task gencralization by incorporating more complex architectures and training methods (Yuan et al., 2021; Singh et al., 2022; Li et al., 2022).
Test-Time Adaptation. To mitigate the performance degradation caused by the distribution shift between the test and source domains, researchers have introduced test-time adaptation to ensure the reliability of models when deployed in -ittrf--distibtttinn scenarios (Wang et al., 2020; Zhang et al., 2022; Gao et al., 2023; Chen et al.). These methods have been widely applied across various machine learning tasks, including image classification (Lin et al., 2022), image super-resolution (Deng et al., 2023), and object detection (Lin et al., 2025). In recent years, with the rise of VLMs, an increasing number of studies have focused on test-time adaptation for VLMs (Shu et al., 2023; Sreenivas & Biswas, 2024; Zhang et al., 2024c). TPT (Shu et al., 2022) introduces Test-Time Prompt Tuning, which learns comminn-specinn prompts for each sample by minimizing output entropy. TDA (Karmanov al., 2024) constructs lightweight visual key-value cache store high-qualit visual features and pseudo-labod during testing, which are then used as class prototypes in the subsequent classification process and effectively reduces the computational overhead associated with backpropagation. DPE (Zhang et al., 2024a) optimizes bimodal class prototypes for both text and visuals better leverage multi-modal capabilities of VLMs. BAT (Maharana al., 2025) not only adjusts the encoder for more effective feature extraction but also enhances the alignment between text and image features. In our work, rather than exploring better optimization objectives, we pay attention to another fundamental view, i.e. the data quality. Our BITTA strategy can effectively avoid the memorization of atypical features.
Machine Unlearning. The core goal of machine unlearning is to adjust deep learning models to eliminate the influence of specific data points or classes. Its key application directions include defending against backdoor attacks (Graves et al., 2021), improving model faimess (Oesterling et al., 2024), and optimizing pre-training processes to enhance transfer learning performance (Sekhari et al., 2021). Various kinds of unlearning algorithm has been developed to achieve exact unlearning. (Neel et al., 2020; Sekhari et al., 2021) utilize methods hased on differential privacy (DP). (Thudi et al., 2022) conducted unlearning research based on additional datasets, while (Zhang et al., 2023; Heng Soh, 2023) proposed concept crasure strategy for diffusion models. Although existing unlearning methods cover a wide range, current mainstream unlearning techniques all require a target class as the object of unlearning. However, in this study, the object to be unlearned is features rather than classes, so there are significant difficulties in directly applying existing unlearning techniques. In the future, we aim to explore more sophisticated methods to achieve more precise unlearning of atypical features learned by the model.
18

---

Published as a conference paper at ICLR 2026
C PROOF OF THEOREM
C.1 PROOF OF THEOREM 3.1
First, we introduce the assumption about Theorem 3.1:
Assumption C.1 The generating function g is smooth invertible function with a smooth inverse everywhere.
Assumption C.2 The invariant variable takes values from finite set: C == {c+ E [K].
Now we prove Theorem 3.1 the method of proof by contradiction. We assume that Zigt experiences a huge distribution shift, making impossible to distinguish which class it belongs to, i.e., images of least two differnt categories from source domain can generate Ftyt after certain shift h or h':
Tigt g(ck= Serc + h) = g(ck', Sare' + h')
(23)
where
g(Cs, Save + h1) = glck; Sero) + So 1 (iee + ---10 b-hdd)hhh
(24)
g(ck, Sere h') g(ck, Saree) ) 11/ J Jotaur. +t +t-h'ddthh'
(25)
Jg is the Jacobian matrix of g. Therefore,
g(ck: Ssre) - g(Ck' Ssrer) = =f. Jatou.- + (-hhhdtthhh S Jalew.) + h')dt)h'
(26)
We define Dcc,, cwr is the minimal distance between e₁ and cR. i.e. V8i, 82 E source domain Sare
[19(ck, - g(cw', S )ll D(ck, C.r).
(27)
Therefore,
I1 Lo (stre +t --h)dtthh S (stov -t.eettthheeee +t)d')t) 2 D(ck,ck)
(28)
+ 11// Jelce.)(Sare + 1 (-)ddt)(ll + II( S Jolce + +1 (.)))d))/11 D(ck, (x)
(29)
J.([I + IMIII) D(ck, CK)
(30)
where J, is an upper bound the Jacobian spectrum norm. Therefore,
C()
max([|>|||||)))) (|1)) 2
(31)
2J
which violates the condition in Theorem 3.1 that D(c..er)
Therefore, conclude that class Cg be strictly distinguished from all the other classes CR when D(es.ow)
2..
2J.
From the above, it can be concluded that the classes strictly separable only when the invariant D(er.ew) Otherwise, if the shift becomes variable is under a moderate amount of shift, i.e., < 2.1g excessive, the distributions of different classes may overlap and become difficult to distinguish. This means it is theoretically difficult to distinguish high-entropy samples due to the distance from the source domain, therefore unlearning these atypical feature helps to avoid overfitting.
19

---

Published as a conference paper at ICLR 2026
C.2 PROOF OF THEOREM 3.2
First, we introduce the assumption about Theorem 3.2:
Assumption C.3 Since the calibration data and test data are derived from the same dataset, we assume that (X,, Yi) are exchangeable. The term "exchangeable" is defined as follows:
Definition C.1 Let ZI,...,ZZ. Zn E Z be random variables with a joint distribution. We say that the random vector (Zi,..., Zn) is exchangeable if. for every permutation Sp (Zr....,Z, (7)(1), Zm(m)), where d denotes equality in distribution, and Sn, is the set of all permulations on [m] = n1).
Then we introduce the following lemma:
Lemma C.1 Let Ui,..... Un+1 E R. Then for any t E [0, co+1 & quantiles(1i...., Un+1)
Un+1 S (u vn).
Therefore,
YA E [Vat1 - Q(1 (() Fnt1 Q(1 - oa] + Sp+1 & quantiles-aaii.. Sny Sn+1)
(32)
From this point on, then, to establish the coverage guarantee, we only need to show that the event Sn-1) holds with >= probability.
S,+1 quanttilea_a)SS-...
Based on the exchangeability assumption, since the prediction error S1...m, Sn+1 are obtained by applying the same function (X,, Y.), they are also exchangeable.
Therefore, we have
(1 a) (n + 1)
P(Sn+1 S quantiler- Sn, Sn+1))
aa,
(33)
n +
this completes the proof.
C.3 PROOF OF LEMMA C.1
First, if n the result holds trivially since quantile, Un+1) maxie(n+1] Un+1+ = +00. From this point on, then, we will assume S n+I to
x+1 while quantileu1+1/00000 avoid this trivial case.
(Ul,..., (1)
Let ((11) I ... < (nna) be the order statistics of Ui,... Uns and let ((++1;)) S ... S U(++m=+1) be the order statistics of Ui,..., Pnt1. Let = [t(n + 1)7 E [n], we see that
quantile;(e1..... un+1) ((++
(34)
and
(uantileq(++//) v,) U((k))
(35)
by definition. So, we need to verify that vnt1 S (1+ holds and only 0n+1 S ((ak))
First, by definition of the order statisties, must have ((++1:)) S ((na)) (i.e., the Ath smallest entry in the list cannot increase if we add new value to the list). Therefore,
Un+1 S U(++1sk)
Un+1 S U(nk))
(36)
To verify the converse, suppose that Wn+1 >> U(n+1:k)- In this case, we must have ((++1sk) ((cck)) again by definition of the order statistics, and so we have
Un+1 > U(n+1sk)
Unt1 U(n()))
(37)
which violates that Un41 S (((;k)) Therefore, we have Un+1 > 11++1:8) Un+1 > ((x)))
20

---

Published as a conference paper at ICLR 2026
D ALGORITHMIC REALIZATION
This section provides a detailed description the algorithm implementation of our proposed method. In general, we separate low-entropy and high-entropy samples to learn effective knowledge and unlearn irrelevant information, respectively. First, we predict the required sample selection ratio based on the class distribution of the dataset. Then, low-entropy samples are used to learn effective knowledge, while high-entropy samples are used to forget harmful information in order to prevent overfitting.
When estimating the sample ratio, we model the low-entropy samples using the following function: B), where 7/" represents the computed numerical threshold. the first batch of data is processed, we convert this numerical threshold into proportional threshold of to avoid the issue of threshold decay as the optimization process progresses, For high-entropy samples, we fix the selection ratio 7/ at 10% to simplify the process and avoid additional parameter adjustments.
q Max Maz(HH(aa))
After obtaining the selection ratios for low-entropy and high-entropy samples, fine-tune the model accordingly. For low-entropy samples, we minimize the entropy loss to reduce prediction uncertainty, enhance the consistency between image and text features, and improve class separability. For high- entroyy samples, we maximize the entropy loss to unlearn harmful information, thus preventing model overfitting.
Finally, the total loss obtained from the above process is used to update the LayerNorm layer. The optimization for both the vision encoder and the text encoder is based AdamW, with learning rates of le-3, 5e-4, and 5e-4 for CIFAR-1O-C, CIFAR-100-C; and ImageNet-C, respectively, and batch sizes of 200, 200, and 64.
Algorithm Our method
Input: well-trained CLIP model with image encoder E, and text encoder E,, corrupted data stream Xteat and its class number K, prompt template a photo of a <cls>'.
Output: Fine-tuned CLIP model.
for time step + 0 to T do
2: Input batch of image to Eg.
3: Imput text description based on prompt template to Eq.
4: // Est imate threshold
5:
if t== then
M
6:
Estimate selection ratio (T M
+ [H()))) < D.D.
7:
end if
8:
Select low-entropy samples and high-entropy samples.
9:
// Learning Loss and Unlearning Loss
10:
Calculate learning loss according to Eq. (3)
K
K
K K
(- Ep(s.
p(y: (og (1ow)) K
Σ v. Se
ΣΣ T| 5](1 sim( B,)))
i1j=1
11:
Calculate unlearning loss according to Eq. (2)
K
C
p(y. (nigh) log(p(u (es((((((nighh)))
==1
12: Calculate total loss (Creaming + XLunleaming).
Optimize LayerNorm layers of E. and E.
14: end for
21

---

Published as a conference paper at ICLR 2026
E MORE DETAILS ABOUT THRESHOLD ESTIMATE MODULE
E. SELECTION RATIO LOW-ENTROPY SAMPLES
In Section 3.3, discussed the design of the threshold estimate module. Here, we provide detailed explanation for the reasoning behind this design. During our preliminary exploration, we found that the optimal selection ratio for low-entropy samples varies across different datasets and corruption types. In Figure 9, we show the model's performance at various low-entropy sample selection ratios. The results indicate that the optimal selection ratio for low-entropy samples differs across datasets. In Table 8, we further investigate the optimal selection ratios corresponding to different types of damage within the same dataset. The results show that even within the same dataset, the selection ratios for different corruption types inconsistent, posing significant challenge for real-world applications. Without a unified selection ratio, the model may fail to adapt optimally when encountering unknown data in real world scenarios. Therefore, explore how standardize the selection ratio for low-entropy samples, aiming to enhance the model's adaptability and robustness.
CIFAR-10-C
CIFAR-10-C
CIFAR-100-C
CIFAR-100-C
75
ImageNet-C
422
68
ImageNet-C
38.1
29.4
74
31.9
88
29.2
73
318
29.0
42.0
71
41.9
65
57.8
41.6
41.7
418
31.7
64
37.7
28.6
62
-8.4
31.6
0.1
0.2
0.5
0.5
0.1
0.4
VIT-B-16
VIT-B-32
Figure 9: The model's performance at various low-entropy sample selection ratios. Optimal performance achieved at different selection ratios for each dataset.
Table 8: The optimal ratio of various corruption types. Even within the same dataset, the optimal
selection ratios for different corruption types are inconsistent.
V-T-B-16
Oanssin
Shot
Impulse
Defocs
Glass
Modion
Zoom
Show
Front
Fog
Brighness
Contratt
Blosic
Pixelate
JPBG
CIFAR-I0
0.8
0.8
0.7
0.6
CIFAR-100-4
0.3
0.3
0.3
0.6
0.5
0.6
0.5
0.6
0.6
0.5
0.5
0.5
0.4
0.6
0.6
ImageNes-C
0.1
0.1
0.1
0.2
0.1
0.3
0.4
Garesian
Shot
Impulse
Defosus
Glass
Maiion
Zoom
Snow
Frost
Fig
Brightness
Comrart
Elattic
Pixelate
JPEG
VIT-B-32
CIFAR-I0-C
0.5
0.5
0.6
0.6
0.4
0.6
0.6
0.5
0.6
0.6
0.6
0.6
0.5
0.6
0.6
CIFAR-100-C
ImageNet-C
0.3
0.2
0.2
0.2
10
0.0
0.1
0.3
0.1
0.1
0.3
0.1
0.1
0.3
0.3
0.2
0.1
0.1
0.3
0.1
0.2
0.1
0.3
0.1
To identify common characteristies across different selection ratios, we analyzed the entropy values of the samples at these optimal selection ratios, As shown in Figure 10, within the same dataset, the samples exhibit similar entropy values across different ratios. This leads us to hypothesize that the initial entropy value may reflect the optimal selection ratio. However, we also observed that the initial entropy values differ across datasets. By analyzing the initial data obtained during the adaptation process (i.e., the entropy values of the first batch), we noticed certain regularity. Specifically, the ratio of entropy correspending to the optimal selection ratio to the maximum entropy value in that batch exhibits decreasing trend, and this relationship closely resembles linear function. We extracted a subset of images from the training set to fit this pattern and derive the following proportional relationship:
22

---

Published as a conference paper at ICLR 2026
0.00038K 0.83,
(38)
Max Maz(H((z.))
where K is the total category numbers. Subsequently, considering that the entropy values of the low-entropy samples decrease over the adaptation process, we use 7f Σn TH((),) < 73 to convert the entropy threshold into ratio. This approach helps avoid the use of fixed numerical threshold, which could otherwise introduce a large amount of atypical features later on and exacerbate model overfitting.
CIFAR-10-C
CIFAR-100-C
Similar initial values
Similar initial values
gaussian
shot impluse defocus
Entropy Values
220-
10
glass
133
15-
mation
zoom
20
snow
frost
15
fog
brightness
030
contrast
elastic
pivelate
jpog
Time Step
Time Step
Figure 10:
The entropy values corresponding to different selection ratios. We found that the entropy values remain very close across different ratios.
E.2 SELECTION RATIO HIGH-ENTROPY SAMPLES
Similar to the selection ratio for low-entropy samples, we also explored the selection ratio for high- entropy samples. As shown Figure 11, unlike the selection ratio for low-entropy samples, we found that selecting fixed proportion of high-entropy samples is sufficient help the model achieve optimal performance. Therefore, we standardize the selection ratio for high-entropy samples to 0.1.
CIFAR-10-C
CIFAR-10-C
CIFAR-100-C
CIFAR-100-C
ImageNet-C
31.50
09.04
ImageNet-C
38.24 29.00
74.78
42.56
91.58
69.00
74.74
31.82
74.72
38.18
42,52
91.80
31.78
68.9 90
38.15
74.8
42.50
81.76
76.0
91.74
68.86
29.00
0.1
0.2
0.0
VIT-B-16
VIT-B-32
Figure 11: The model's performance at various high-entropy sample selection ratios. Unlike the selection ratio for low-entropy samples, optimal performance is achieved the same selection ratios for each dataset.
E.3 RELIABILITY OF DYNAMICALLY SELECTION MODULE
As mentioned above, we construct a dynamic selection module for low-entropy samples, which can adaptively adjust the sample selection ratio according to differences in data distributions. To verify the superiority of the fitted predictive function, we set a = 0.1, reselected approximately 2000 images from the training set of the ImageNet-C dataset the calibration set, and based on the ViT-B-16 backbone, calculated the error between the predicted selection ratio by the aforementioned predictive function and the groud-truth ratio in Table 8. Sort these errors, further solved the
23

---

Published as a conference paper at ICLR 2026
quantile described in Theorem 3.2 (the result is 0.028). Subsequently, conducted visual analysis of the matching between the prediction intervals and the true values on the test set, with the results shown in Figure 12. It can be observed that although the width of the prediction intervals is relatively narrow, almost all true values still fall within the prediction intervals. This result fully confirms the superiority of the predictive function fitted in this paper.
Prede coltrueeee
30708
18400
PODMA
Figure 12: Visualization of the predicted interval. Green crosses represent ground-truth values that are correctly within the interval, whilst the red crosses represent the values that outside of the interval. It can be seen that almost all ground-truth values are correctly within the interval, which demonstrates the superiority of the fitted prediction function.
24

---

Published as a conference paper at ICLR 2026
F ADDITIONAL EXPERIMENT RESULTS
In this section, we provide more experiment results from different perspectives to characterize our proposed algorithms.
F.1 TIME COSTS
We measured the average time costs taken by several different methods to test single image on the CIFAR-IO- -O-C, CIFAR- 100-, and ImageNet-C datasets, with the results shown Table As can be seen, our method demonstrates significant competitiveness in terms of time costs.
Table 9; Time costs of various methods in seconds.
ViT-B-16
CIFAR-100-C CIFAR-I0-C
ImageNet-C
TPT 0.0540
0.0612
0.2736
DPE
0.0498
0.0575
0.1002
BAT 0.0086
0,0096
0,0289
Ours 0.0088
0.0097
0.0292
ViT-B-32 CIFAR-IO-C
CIFAR-100-C
ImageNet-C
TPT 0.0525
0.0534
0.2586
DPE
0,0496
0.0562
0.0966
BAT
0.0032
0.0040
0.0238
Ours
0.0034
0.0041
0.0240
F.2 COMBINED WITH DIFFERENT METHODS
In the main text, we compared the performance of several methods combined with BITTA on the CIFAR-IO-C dataset, including TPT (Shu et al., 2022), CTPT (Yoon al., 2024), and TPS (Sui et al., 2024), all based on the ViT-B- 16 backbone. Here, we further present the results on the ImageNet- dataset using the same backbone. Consistent with the previous setup, fix the optimization objective for low-entropy samples and introduce high-entropy sample selection module to implement unlearning to avoid overfitting on low-entropy samples. The results in Table 10 demonstrate that BITTA consistently improves the performance of these methods, showcasing the robustness of our proposed approach and its applicability across different datasets.
Table 10: Comparison of BITTA combined with different TTA methods over ImageNet-C using ViT-B-16. A highlighted the improvement in green over the original method without BITTA.
Method
Coursion
Shatt
Impulke
Gians
Maiion
Contraus
Elatte
WEG
Mea
TPT
8.78
9.42
9.64
24-18
16,20
25.12
23.98
33.78
32.30
37.78
$5.56
19.10
14.26
35.88
34.84
25.41
TPT + BITTA
12.08
13.90
12.86
24.88
16.54
25.62
24.08
33.84
32.12
38.50
55.72
20.16
14.60
36,06
34.74
26.38
+4.45
+0.06
-0.15
+0.72
+1.06
+0.34
+0.15
-0.10
+0.97
BITTA
13.84
34.44
25.50
23.88
34.12
32.44
38,06
56.00
18.62
14.39
35.14
34.74
26.29
+3.06
+3.58
+3.14
-0.18
+0.10
+0.14
+0.28
+0.25
+0.12
20.08
-0.28
+0.00
+0.13
+0.08
+0.26
+0.76
TPS
9.52
10.18
9.22
23.76
15.80
21,00
23.10
33.82
31.98
38.25
$4.98
22.70
14:70
35.66
31.62
25.53
TPS BITTA
11.27
12:30
11,40
24.38
16.48
24.94
23.64
34.38
32.08
38.36
54.84
21.08
15:36
36.80
34.72
26.14
+0.66
25

---

Published as a conference paper at ICLR 2026
F.3 ABLATION STUDY ON UPDATE STEPS
We perform multiple steps of parameter adaptation on a single batch. To evaluate the impact of different update steps overall performance, conduct ablation study by varying the number of update steps from to 4. In Figure 13, we illustrate the average accuracy across 15 different corruption types the benchmark datasets using VIT-B-16 visual backbone. Continuous updates on a single batch can lead to over-fitting causing the loss of prior knowledge, which in turn reduces the accuracy on subsequent data and ultimately results performance degradation. This indicates that single-step update allows the model to strike good balance between retaining original prior knowledge and adapting to new data. The impact of the number of update steps on the VIT-B-32 visual backbone can be found in Figure 14. As can be seen, similar to the observations with the ViT- B-16 backbone, the model's performance gradually declines as the number of update steps increases. This phenomenon suggests that continuous updates lead to significant loss of prior knowledge of the model, which in turn affects its generalization ability and performance. On the other hand, when the number of update steps is lower, the model is better able retain its original prior knowledge, resulting in more stable performance. Therefore, based on the experimental results, we believe that reasonable choice, helps balance the retention of
setting the number of update steps prior knowledge and the improvement of model adaptability.
CIFAR-10-C
CIFAR-100-C
ImageNet-C
40
75
Mean Accuracy
*
42
VIT-B-16/zero-shot) www.w/..........................
28
12
26
VIT-B-16(Zero-shott
VIT-B-16(Zero-shot
22
25
20
Update Steps
Update Steps
Update Šteps
Figure 13: Ablation study on update steps across ViT-B-16. Continuous update lead to the performance degradation, while the mean accuracy still significantly exceed zero-shot ViT-B-16.
CIFAR-10-C
CIFAR-100-C
ImageNet-C
m
70
Mean Accuracy
30
29
6.-
VIT-B-32(zero
25
VIT-B-32(2ero-shot)
VIT-B-32(Zero-shot)
64
*
25
55
6 25
Update Šteps
Update Šteps
Update Steps
Figure 14: Ablation study on update steps across ViT-B-32. Similar to ViT-B-16 16, continuous update lead the performance degradation on ViT-B-32. Therefore, it is reasonable to set update steps to 1.
26

---

Published as a conference paper at ICLR 2026
F.4 ABLATION STUDY ON PROMPT TEMPLATE
In Section 4, we have demonstrated the impact of different prompt templates on model performance using the ViT-B-16 and ViT-B-32 backbone. To further validate the universality and adaptability of our method, this section provides detailed results on the comparison with baseline methods. The relevant experimental results are shown in Table 11 and Table 12. As indicated by the data in the table, our method outperforms baseline methods under various prompt templates, exhibiting strong adaptability and robustness different prompt templates. This suggests that regardless of the prompt template used, the model is able to effectively adjust and optimize its performance, showcasing its flexibility. Therefore, in practical applications, users can flexibly select and adjust the prompt templates based specific task requirements and scenario needs to achieve optimal model performance.
Table 11: Detailed results ablation study on prompt templates on ViT-B-16. Our method outperforms baseline methods across different prompt templates ViT-B-16.
Prompt Template
Method
CIFAR-IO-C
CIFAR-100-C
ImageNet-C
ViT-B-16
61.16
35.79
25.70
TPT
63.84
36.06
25.41
DPE
62,48
38.28
27.37
a photo of <els>/
BAT
73.34
41.15
31,06
Ours
74.78
42.56
31,88
+1.44
+1.41
+0.82
VIT-B-16 16
61.88
35.45
21.06
TPT
64.47
36.56
24.44
DPE
62.96
37.97
28.86
low centrast photo of <els>'
BAT
73.59
40.81
31.30
Ours
74.85
42.65
31.97
A
+1.26
+1.84
+0.67
ViT-B-
62.47
35.08
26.27
TPT
65.72
35.91
24.76
DPE
63.52
38.14
27.91
blurry photo of <els>' 'a noisy photo of <els>'
BAT
74.32
41.27
31.24
Ours
75.55
42.34
31,83
A
+1.23
+1.07
+0.59
VYT-B-16
63.02
35.18
25.80
TPT
67.21
36.98
25.64
DPE
63.76
38.43
31.21
BAT
74.09
30.98
75.37
42.22
31.64
+1.28
+2.61
+0.66
27

---

Published as a conference paper at ICLR 2026
Table 12:
Detailed results of ablation study on prompt templates on ViT-B-32. Similar the
results on ViT-B-16, Our method also shows great adaptability across different templates on ViT-B-32.
Prompt Template
Method
CIFAR-10-C
CIFAR-1000 100-C
ImageNet-C
VIT-B-32
59,01
31.78
23.45
TPT
63.46
31.55
25.41
DPE
61.15
34.58
25.69
'a photo of <cls>!
BAT
67.49
36.29
27.58
Ours
69.02
38.23
29.05
A
+1.53
+1.94
+1.47
ViT-B-32
60.76
32,12
23.95
TPT
65.03
31.78
25.96
DPE
62.77
34.73
25.91
low contrast photo of <cls>'
BAT
65.96
35.99
27.56
Ours
69.08
37.71
29.03
A
+3.12
+1.72
+1.47
VIT-B-32
$7.89
31.46
23.58
TPT
62.21
31.34
25.62
DPE
60.85
36.08
27.64
'a blurry photo of <els>'
66.66
35.74
26.70
Ours
69.11
37.15
28.46
+2.45
+1.41
+1.76
ViT-B-32
59.79
31.54
23.47
TPT
64.17
31.42
25.52
DPE
61.87
36.58
28.78
'a noisy photo of <els>/
BAT
66.10
35.97
26.84
Ours
69.27
37.80
28.57
+3.17
+1.83
+1.73
28

---

Published as a conference paper at ICLR 2026
F.5 EFFECTS OF DIFFERENT LEARNABLE MODULES
Recall that in our method, we optimize the LayerNorm parameters of vision and text encoders by introducing the learning loss and unlearning loss. In Figure 15, we illustrate the results of optimizing only the LayerNorm layers of a single modality encoder and optimizing the entire model. As can be seen, optimizing only the LayerNorm of the unimodal encoder leads decrease in accuracy. especially when only the text encoder is optimized. This is because the image feature distribution of corrupted data exhibits greater discrepancy from the source domain, making the regularization of image features more significant. At the same time, optimizing the entire model causes catastrophic forgetting of the model's 's prior knowledge, resulting in extremely poor performance. This highlights the correctness and necessity of optimizing the LayerNorm layers of both image and text encoders.
Ours
Ours
Vision Encoder
Vision Encoder
Mean Accuracy
Mean Accuracy
Text Encoder
Text Encoder
50
Entire Model
50
Entire Model
20
20
CIFAR 10.C
CIFAR: 100 C
ImageNet.
OFAR 10
CIFAR 100 C
ImageNet C
ViT-B-16
ViT-B-32
Figure 15: Ablation study on learnable modules. Optimizing the LayerNorm layers of both
encoders shows the best performance.
F.6 ABLATION STUDY ON A
In the main text, we show the ablation study on À ImageNet-C with ViT-B-16. this paragraph, we show more results on the other datasets and visual backbones in Figure 16. Similar to Figure 8(b), setting À to 1.2 found to be an appropriate choice and the performance gap still smaller than 0.1% across different datasets, indicating that our methods show great robustness regarding the À across various datasets and visual backbones.
CIFAR-10-C
CIFAR-10-C
CIFAR-100-C
CIFAR-100-C
ImageNet-C
ImageNet-C
74.78
38.24
31.88
68.02
74.78
38.18
74.74
91.86
68.00
38.16
74.72
38.14
38:12
74.70
91.84
68.98
1.2
13
1.0
1.1
1.2
ViT-B-16
ViT-B-32
Figure 16: Performance with varying balancing constant X. Setting À to 1.2 is found to be optimal on VIT-B-16 and ViT-B-32.
29

---

Published as a conference paper at ICLR 2026
F.7 ABLATION STUDY ON BATCH SIZE
In the main text, we conducted an in-depth investigation into the impact of batch size on performance. Here, show the average accuracy of our method using the ViT-B-16 backbone with smaller batch sizes. It can be observed that even with low batch size, our method still significantly outperforms the baseline method, demonstrating excellent adaptability. More results on VIT-B-32 visual backbone can be found in Figure 18. As can be seen, although there is some fluctuation in the model's adaptation performance under different batch sizes, the overall performance still significantly outperforms baseline methods. the experiments, we observed that larger batch sizes help the model balance stability and efficiency during the learning process, while smaller batch sizes may lead to instability in certain cases.
results on additional datasets and visual backbones. In Figure 17, we show
Based on these experimental results, have some impact on the model's adaptation performance, our model demonstrates good adaptability across different batch sizes. In other words, regardless of whether using small or large batch sizes, method consistently performs better than the traditional methods, proving its flexibility and stability in real-world applications. Therefore, batch size can be flexibly adjusted based specific application scenarios and computational resources to optimize model performance.
conclude that, although the choice of batch size does
CIFAR-10-C
CIFAR-100-C
ImageNet-C
BATT
BAT
BIAT
Accuracy
Accuracy
Mca
Batch Size
Batch Size
Batch Size
Figure 17: Ablation study on various batch sizes across ViT-B-16. Our method perform well under various batch sizes.
CIFAR-10-C BATT
CIFAR-100-C
ImageNet-C Ours. BAT
Ours BATT
Mesn Nccrrbyy
Bafcch Size
Batch Sire
Battch Size
Figure 18: Ablation study on batch sizes across ViT-B-32.
30

---

Published as a conference paper at ICLR 2026
F.8 GENERALITY ON DIVERSE ARCHITECTURES
In the main text, have presented some results when combining BITTA with different architectures (such as CNN and ViT) and different methods (such as MEMO and SAR). In this section, we further present the experimental results the ImageNet-C dataset. The data in Table 13 convincingly demonstrate that BITTA can be compatible with a variety architectures, fully showcasing its outstanding generalization ability. the same time, this result also further confirms the widespread existence of the overfitting problem in the research field of the Test Time Adaptation. The excellent performance of BITTA undoubtedly provides valuable and profound insights well as practical solutions for effectively addressing this problem.
Table 13: Comparison of BITTA combined with different TTA methods over ImageNet-C using ViT-B-16. A highlighted the improvement in green over the original method without BITTA.
ResNar101
Crnan Ganasion
Shott
Impuuiee
Defocoo
Mosino
Smow
Pron
Canmant
Elaste
Prozaate
Mean
MEMO
31.84
33.71
33.22
30.52
30.09
44.24
$2.00
19.75
64.20
58.52
69.12
33.45
57.68
60.18
54.76
45:55
MEMO BETTA
32.63
34,82
34.07
$1.70
32.11
45.54
$2.95
50.96
44.97
99.57
69.37
37.09
58.85
60.99
55.94
46.81
A
40.79
41.11
40.85
+1.18
42.02
41.30
40.93
+1.21
40.77
+1.05
+0.25
44.24
+1.17
+0.81
41.18
+1.26
SAR
33.45
33.33
34.52
31.38
32.58
65.19
52.34
50.07
45.25
59.46
39.18
58.63
60.78
55.35
46.78
SAR
33.92
33.94
34.90
55.51
47.20
+0.50
+1.32
+1.02
Goarswoo
Elatic
Preclse
26.88
33.21
37.74
40.83
28,69
34:21
5493
67.84
56.21
36.74
+1.11
+8.61
+1.1
+1.12
+1.26
+0.94
+1.00
+1.81
+1:00
+1.67
-0-17
+1.11
-0.71
+1.19
SAR
52.33
52.24
53.49
53.88
28.76
56.12
49.75
62.54
46.78
61:52
25:21
06-15
51.24
64.31
02.75
57.35
SAR + BITTA
56.27
56.44
57.69
58.26
55.31
61.45
56.35
64.39
62.30
72:56
77.49
60.17
62.52
70.3
65.58
60007
A
+3.44
44.20
44.20
+4.58
46.55
45.33
46.60
+1.85
415.57
46.74
14.28
42.74
411.28
+6.02
42.83
45.74
F.9 RESULTS ON IMAGENET VARIANTS
Considering that evaluating the robustness of VLMs against corrupted images of great significance. especially in safety-critical fields such autonomous driving, where images often affected by factors like blurriness and weather changes, we focus our research on the corruption benchmark in the main text. To fully verify the effectiveness of proposed method other datasets, add more experiments on ImageNet variants in Table 14. The experimental results show that the BITTA model also demonstrates good compatibility these different datasets.
Table 14: The classification accuracy of ViT-B-16 on ImageNet variants
Method
ImageNet
ImageNet-A
ImageNet-R
ImageNet-S
ImageNet- V2
Mean
TPT
68.98
54.77
77.06
47.94
63.45
62.44
TPT+BITTA
70.31
55.78
78.03
48.93
64.77
63.56
TPS TPS+BITTA
70.10
60.80
80.28
49.59
64.83
65.12
72.37
62.90
81.41
50.45
65.79
66.58
F.10 DETAILS ABOUT OTHER REGULARIZATION TECHNIQUES
For weight decay, we perform normalization on each batch according to the prediction entropy and assign weight between O and each image. For distribution dissolve, we replace the entropy- based unlearning loss in our original framework with distribution dissolution loss. Specifically, aim ensure that the feature distribution of high-entropy images extracted by the adapted CLIP model, denoted as (Xnigh), deviates from the original high-entropy image distribution (Xiihh) We re-run BITTA by updating the unlearning loss for high-entropy samples to: Luntearning -(Fe,, (Xnggh)
F 2
31

---

Published as a conference paper at ICLR 2026
F.11 RESULTS ON OTHER CORRUPTION TYPES
In addition to the above analysis, to further verify the effectiveness of BITTA across different corrup- tion types, conduct experiments extral corruption types ImageNet-C and ImageNet-C-Bat with ViT-B-16. The results indicate that BITTA also achieves universal performance improvements on these corruption types.
Table 15: The classification accuracy of ViT-B-16 on other corruption types
Dataset
ImageNet-C
Corruption
speckle noise
gaussian blur
spatter
saturate
BAT 33.60
22.64
37.78 38.54
49.52
BAT+BITTA
35.74
25.58
50.96
Dataset
ImageNet-C-Bal
Mean
Corruption
blue noise sample
brownish noise
caustic refraction
BAT
38.28
50.32
38.40 38.96
38.65
BAT+BITTA
39.96
51.22
40.14
F.12 RESULTS ON NON I.I.D BATCHES
For the robustness for non i.i.d. batches, we conduct the following experiments to demonstrate the effectiveness of our method. Using 10,000 test images from CIFAR-10 with batchsize of 200, we simulate a highly imbalanced class scenario by specifying that only classes 0)-4 appear in the first 25 epochs and only classes 5-9 appear in the last 25 epochs. The test results are shown in the table below. As can be seen, method still achieve stable performance improvement. Therefore, BITTA robust to non i.idd. batches.
Table 16: The classification accuracy of ViT-B-16 on non i.i batches.
Corruption
Gaussian
Shot
Impulse
Defocus
Glass
Motion
BAT BAT+BITTA
57.91
61.61
61.15
76.91
45.47
77.03
60.29
65.82
63.75
78.80
47.91
78.53
32

---

Published as a conference paper at ICLR 2026
G DISCUSSION
G.1 LIMITATION AND FUTURE WORK
In this paper, we have clearly identified the overfitting problem caused by the low-entropy sample selection strategy that has been widely adopted in previous related works. Meanwhile, we have conducted an in-depth discussion and revealed the potential advantages and possibilities of using high-entropy samples for regularization processing. However, it is undeniable that the research findings presented in this paper still have certain limitations.
Firstly, although the current implementation of the "unlearning" operation by using the sample- level method has been able to achieve relatively satisfactory results, there is still room for further exploration. Specifically, is it possible to further reduce the fine- fine-grained level to achieve a more accurate learning and unlearning process? This question points out a new exploration direction for future researches.
In addition, the dynamic sample selection strategy we proposed is mainly constructed based on empirical observations. In future researches, the focus can be placed providing more precise and reliable theoretical analyses for how to select an appropriate proportion of samples, so as to further improve and optimize this strategy.
G.2 BROADER IMPACT
As fundamental task in the field of computer vision, the robustness of image classification tasks in the face of various kinds of corruptions of paramount importance for the successful deployment of reliable deep learning systems in real-world scenarios. Especially in safety-critic autonomous driving, eтror predictions caused by noise are highly likely to lead to extremely serious consequences. In view of this, it is particularly necessary and urgent enhance the robustness of models on nut-of-distribuu
fields such as
data.
Our research highlights crucial yet long-overlooked issue in existing Test Time Adaptation methods: these methods typically rely solely on low-entropy samples to optimize the model. Such one-sided optimization approach is likely inadvertently cause the model to overfit.
To address this problem, the method we proposed ingeniously uses high-entropy samples to regularize the TTA process. Specifically, by leveraging the atypical features contained in the high-entropy part, it effectively eliminates the adverse effects of misclassified low-entropy samples. A series of comprehensive and in-depth experiments have fully verified the effectiveness and good compatibility of this method, strongly indicating that the sample selection strategy we designed is highly promising new method that can effectively prevent the model from overfitting. This achievement undoubtedly provides new perspective and profound insights for future research in the field of TTA, and is expected to drive further development and innovation in this field.
33

---

Published as a conference paper at ICLR 2026
H VISUALIZATION ANALYSIS
H.1 COMPARSION OF LOW-ENTROPY SELECTION CRITERIA AND BITTA
In Figure 1, we illustrate the difference between our bilateral information- aware test-time adaptation strategy. Here, we further provide a visual comparison between low- entropy selection strategy and BITTA. It can be seen that on different datasets, our method comprehensively leads the low-entropy selection strategy, which demonstrates the effectiveness of BITTA.
DIO WAY
coll bse
auDier
Figure 19: Performance comparison of low-entropy selection criteria and BITTA. It can be seen that BITTA effectively improve TTA performance across different datasets and model architectures.
H.2 VISUALIZATION OF CORRUPTED IMAGE
In Figure 20, we present visualization examples of 15 different types of corrupted data to illustrate the impact of various corruption types on image quality. Additionally, each corruption type includes 5 different severity levels. To more clearly demonstrate the effects of different levels of corruption on the images, we further present the effects caused by each severity level in Figure 21. From these figures, can be observed that, the highest level corruption, the distinguishability of the image significantly decreases, posing a great challenge for classification tasks. Therefore, the classification of corrupted images is not only an urgent issue that needs be addressed but also highly difficult research problem.
H.3 MORE ATYPICAL SAMPLES VISUALIZATION
In Figure 3(a), we presented the atypical samples from low-entropy part and high-entropy part. Here, further showcase more of these atypical samples. As can be seen from Figure 22, the model made extremely unreasonable judgments on these samples from both low-entropy part and high-entropy part, resulting in a complete mismatch between the image features and the pseudo-labels, This shows that the confidence selection strategy will cause the model to learn atypical features, and high-entrope samples becomes an ideal candidate to compensate for the atypical samples in the low-entropy part.
H.4 T-SNE VISCALIZATION
In addition the t-SNE visualization results presented in Section 4, further showcase the t-SNE plots for all 15 corrpption types in the CIFAR- 10-C dataset in Figure 23-24. A comparison with the zero-shot CL.P method reveals that the image features generated by method exhibit higher aggregation and clearer distinction between classes. This indicates that our approach significantly improves the zero-shot performance of CLIP when handling corrupted data.
H.S COMPARISON OF CLASSIFICATION RESULTS
In Figure 25-27, we visualize some examples in CIFAR-10-C, CIFAR-100-C and ImageNet-C, where we made correct predictions but not for zero-shot CLIP. This demonstrates the effectiveness of our method and shows that our method can well improve the classification accuracy of corrupted data.
34

---

Published as a conference paper at ICLR 2026
Gacesian
Shot
Impuise
Deforus
Glass
Clean
Fog
Brightnees
Costrast
Elastic
Pitelate
JPBG
Figure 20: Visualization of clean sample and corrupted versions under different types of corruption.
Savertty
Sevartty 2
Seventy 3
Sevenit
Severity
Clmn
Figure 21: Visualization of various severity levels, including Snow, Gaussian Noise, and Elastic Transform.
35

---

Published as a conference paper at ICLR 2026
Low Entropy
Atypical Festures
sliding door
perfume
coffeemaker
television
television
redwine
lasglasses
elephant
rocket
llama
baseball
rhinaceros bectle
earth star fungus
pear
plate
geyser
missile
paintbrush
Angora rabbit
snail
apple
aquarium
turtle
road
chimpanzee
chimpanzee
mouse
High Entropy
Atypical Features
computer mouse
#ymnasttc horizontal
eastern hog-nosed
computer keyboard
mountain
car mirror
electric gultar
Jigsaw puzzle
maze
rocket
television
boa constrictor
traffic light
spoonbill
folding chair
forest
wolf
moving van
military
couch
wolf
house
keyboard
television
television
Figure 22: More Visualization of atypical samples from low-entropy part and high-entropy part.
36

---

Published as a conference paper at ICLR 2026
Gaussian
Shot
Impuise
Delocus
Glass
Motion
Zoom
Snow
Frost
Pog
VIT-B-16
Brightness
Contrast
Elastic
Pixelate
JPEG
Airpane
Frog
Stip
Casssian
Shot
Impulse
Defocus
Glass
Motion
Zoom
Snow
Frost
Pog
Ours
Brightness
Contrast
Elastic
Pixelate
JPEG
Figure 23: The t-SNE plots of our method and Zero-Shot CLIP (VIT-B-16) for CIFAR-IO-C.
37

---

Published as a conference paper at ICLR 2026
Gaussian
Shot
Impuise
Delocus
Glass
Motion
Zoom
Snow
Frost
Pog
VIT-B-32
Brightness
Contrast
Elastic
Pixelate
JPEG
Airplane
Frog
Stip
Caussian
Shot
Impulse
Defocus
Glass
Motion
Zoom
Snow
Frost
Pog
Our
Brightness
Contrast
Elastic
Pixelate
JPEG
Figure 24: The t-SNE plots of our method and Zero-Shot CLIP (VIT-B-32) for CIFAR-IO-C.
38

---

Published as a conference paper at ICLR 2026
Top Prediction:
CLI Prediction:
automobile
automablle
horse
honse
680
dog
60g
cet
23
00
23.50
22.50
dog
bird
horee
brdd
automobiee
track
truck
automoblie
21.50
truck
bid
00g
00
dear
600
deer
bid
hrdd
60g
0eq
ship
22.00
23.00
frog
dog
trck
airplane
airplane
ship
sutomobile
21.50
23.00
ship
Figure 25: Comparison of classification results between zero-shot CLIP and our method on CIFAR- 10-C.
39

---

Published as a conference paper at ICLR 2026
Our Top Prediction:
CLIP Top Prediction:
b00l
bool
can
plate
plate
can
23.00
23.50
22.75
23.50
plate
rocket
mountain
mountain
rocket
cloud
akycraapa
skyscreper
cluud
23.50
24.30
24.30
23.50
23.75
24.15
24.20
skyscraper
whole
m9 k/
dluud
keyboard
mountain
21.50
21.75
22.00
22. 40
22.50
21.25
21.50
21.75
22.00
mountain
buy
9-1
011
man
30y
22.4
22.5
22.6
22.3
22.10
22.20
22.30
22.40
22.50
woman
chair
chair
Cilmpenzee
chimpangee
bicycle
motorcyc
matorcycle
bicyce
22.50
23.00
23.50
24.25
24.35
23.00
23.50
24.40
24.50
motorcycle
Figure 26: Comparison of classification results between zero-shot CLIP and our method on CIFAR- 100-C.
40

---

Published as a conference paper at ICLR 2026
Our Top Prediction:
CLIP Top Prediction:
grocery store
microwsve oven
refrigerator
vending machine
vending mechine
moving van
29
30.00
30.25
30.50
30.75
30 30.05
vending meachine
anghibious vericle
amphibious vehicle
go kart
go kart
lawn mower
lawn mower
sports
raco
27.00
28.00
23.60
29.70
26.00
27.00
28.00
29.43
29.50
race car
house
house
junce
chickadee
brambling
brambling
chickadee
brambling
Scotlish Terrier fer
Curly-costed Retriever
Stancard Foodle
Newfoundland dop:
Curty-coated
Giant Schnauzer
Bouvier Flandres
Bouvier des Flandres dog
Giant Schnauzer
31.00
31.00
32.00
31.00 31.25 31.50 31.86 31.88
Bouvier des Flandres dog
divider
divider
throne
throne
table lamp
tabl lamp
chitfonier
couch
25.00
26.26
26.30
25.00
26.20
26.35
couch
Figure 27: Comparison of classification results between zero-shot CLIP and method on ImageNet- C.
41


Reviews:
[{"review_id": "dSxbk5YjnL", "reviewer": "Reviewer_kaur", "summary": "The authors argue that previous methods have focused on improving the objective functions and have ignored the dataset perspective of test time adaptation. The standard paradigm revolves around picking up the most confident samples and applying TTA methods on these. This relies on an assumption that all low entropy samples are correct predictions. However, this can lead to memorization of atypical features as shown in figure 2 of their motivation where model becomes confident about its wrong predictions. Authors propose an interesting work around to this, they utilize high entropy samples and maximize the entropy on these samples. This leads to unlearning of such atypical features leading to higher performance across the board. Lastly, to improve the training process, authors propose that the optimal percentage for low entropy samples vary for each dataset.", "questions": "- Comparison with other regularization techniques. \n- visualization of attention maps over low confident misclassifications before TTA and high confident misclassifications after TTA.\n- You could have high entropy correctly classified samples. Does something like this exist and if yes, then does it impact your method negatively?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Paper introduce a novel data centric perspective to the test time adaptation. \n- paper is well motivated. \n- presentation is well done. \n- Results are promising.", "weaknesses": "It is unclear which features are atypical. could it also be because of high confident incorrect predictions being part of the training mix? maybe some sort of attention map visualizations would be nice here as well. \nHow does it compare to other regularization techniques such as weight decay.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mIOtRgqnmb", "reviewer": "Reviewer_c2Yw", "summary": "This paper addresses the problem in test-time adaptation of vision–language models whereby updating solely on low-entropy samples leads to inadvertent overfitting and sample-selection ratios that lack cross-domain robustness. The proposed method minimizes entropy for low-entropy samples while simultaneously strengthening image–text alignment and inter-class separability, and maximizes predictive entropy for high-entropy samples to suppress memorization of atypical features, thereby striking a balance between adaptation and robustness. The authors also provide theoretical support regarding the separability of hard samples and coverage guarantees for proportion prediction, and demonstrate consistent performance gains on CIFAR-10/100-C, ImageNet-C, and multiple cross-domain datasets, as well as in combination with methods such as TPT, CTPT, and BAT.", "questions": "1. How is the linear relationship between the dynamic low-entropy proportion and the number of classes fitted? Which data points are used, what is the goodness of fit, and are scatter plots with regression lines on CIFAR-10-C, CIFAR-100-C and ImageNet-C, together with a small-scale sensitivity check, included?\n2. Why is the high-entropy proportion fixed at 0.1? Is a brief sweep at 0.05, 0.10, 0.15, 0.20 on CIFAR-10-C and ImageNet-C (severity 5), reporting both Top-1 and ECE?\n3. In the component ablations, how much gain is attributed to low-entropy learning only, high-entropy unlearning only, and both together? Are the independent contributions of each component in the bilateral mechanism quantified?\n4. Does unlearning improve model uncertainty and risk control? Are ECE, NLL, and rejection AUROC reported based on existing outputs and compared with baselines?\n5. On which corruption types are the gains primarily concentrated? Are results broken down by distortion type provided, in addition to the main table, to clarify performance differences across corruptions?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Proposes a bilateral mechanism and uses dynamic proportion estimation to mitigate overfitting and sensitivity to sample selection.\n2. Provides an analysis of the separability of hard samples and coverage guarantees for proportion prediction, enhancing the method’s interpretability and robustness.\n3. Achieves consistent gains on CIFAR-10/100-C, ImageNet-C, and multiple cross-domain datasets.", "weaknesses": "The paper needs additional experiments to further demonstrate the effectiveness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mvs42rP0Yo", "reviewer": "Reviewer_obJ8", "summary": "This paper proposes BITTA, a method for test-time adaptation (TTA) that simultaneously performs learning on low-entropy (confident) samples and unlearning on high-entropy (uncertain) samples. Empirical evaluation is performed on several corruption and domain shift benchmarks, showing small but consistent improvements over prior TTA baselines.", "questions": "BITTA is a clean and well-written paper that presents a modest yet reasonable enhancement to entropy-based test-time adaptation. The idea of learning from confident samples and unlearning uncertain ones is conceptually sound and practically implementable. However, the contribution remains incremental, and the performance gains are minor. I recommend **rejection** at this stage, but I believe the idea has potential. If the authors further refine their method and demonstrate a larger improvement over existing baselines, the work could be strong enough for acceptance in a future submission.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. **Conceptually simple but reasonable intuition.** The idea of balancing learn and unlearn at test time is intuitive, and connects nicely to existing observations that confident-only updates lead to confirmation bias. \n\nS2. **Dynamic adaptation ratio.** Instead of fixing the proportion of confident vs. uncertain samples, BITTA adjusts it adaptively based on batch entropy values. This is a lightweight heuristic that makes the algorithm more flexible. \n\nS3. **Well-written and structured.** The methodology section is easy to follow, and the figures illustrating the bilateral update flow are clear.", "weaknesses": "W1. **Marginal improvement magnitude.** Most gains over strong baselines are within +0.3--1.0 percent points, often within the variance range reported in prior TTA studies. It would be great if the authors report the performance of average and variance of each experiment with multiple times. \n\nW2. **Comparison of previous unlearning methods.** The paper states that high-entropy samples trigger *unlearning* to mitigate the overconfidence. I believe that there are a number of research in terms of unlearning works, so it is important to compare them with the proposed method that authors proposed in this paper. Sorry for not referring several unlearning methods to compare due to lack of expertise about unlearning domain. \n\nW3. **No computational analysis.** BITTA claims to be lightweight, but no runtime or memory cost comparison is reported against other baselines. \n\nW4. **Incremental novelty.** I agree that the authors address an important problem in the TTA domain. However, the proposed method is conceptually simple and lacks substantial novelty. If the approach had demonstrated a larger performance gap over prior TTA methods, its impact could have been justified despite the simplicity. Unfortunately, the observed improvements are rather marginal (as described in W1), which limits the overall significance of the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yNcj08Hsda", "reviewer": "Reviewer_q3aD", "summary": "The article addresses the problem of test-time adaptation of a vision-language model given a stream of test data. In particular, the model selects samples with low entropy, assuming that the model's confidence reflects certainty in its predictions, and optimizes the model to further minimize entropy. Deviating from previous works, the approach selects samples of high entropy as well. For the latter, it is assumed that they have atypical features that need to be unlearned, something achieved by maximizing their entropy. The tradeoff between the two objectives aims to prevent overfitting and overconfidence while still learning from the stream of data. Experiments across domains and corruptions show the effectiveness of the approach (BITTA), outperforming recent competitors.", "questions": "Following from the weaknesses above:\n1. How does the method compare with other TTA approaches working online?\n2. Is the method robust to the batch composition?\n3. Is it possible to clarify the meaning of atypical features?\n4. Is BITTA reducing the overconfidence of the model/improving its calibration?\n5. How does BITTA relate to approaches for machine unlearning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of samples with high entropy/low confidence is sound and very interesting. It is interesting because it has not been explored by previous approaches, focusing mostly on learning from high-confidence samples. While it is unclear what type of atypical information is included in samples with high entropy, maximizing the entropy of the latter can still act as a regularizer for avoiding overconfidence, thus improving the results. \n\n2. The paper shows several analyses concerning multiple aspects of the approach, such as the impact of batch-size and update steps (Fig. 6, Fig. 8.a), hyperparameters (Fig. 8.b), selection module (Fig. 7), and changes in the entropy dynamic (Fig. 8.c). Moreover, experiments showcase the generality of the approach to other architectures/methods (e.g., Tab. 5 and Tab. 8). Overall, these analyses support the design choices and provide insights on the potential of the method, how the latter works, and the tradeoffs to keep in mind when applying it. \n\n3. The appendix contains several details regarding design choices (e.g., the threshold estimate module of Appendix E), and the supplementary material includes the code. All in all, these additions make the submission transparent and provide strong support for its reproducibility.", "weaknesses": "1. The setting is very similar to that of Episodic TTA (e.g., [a,b,c]), where the model is updated as the stream of target data becomes available. Some of these competitors are missing in the experimental results, and including them would make the comparisons more comprehensive. \n\n2. Related to the previous point, these methods test with a batch size of 1 (i.e., [a,c]), assuming no priors on the batch constitution. On the other hand, BITTA is very sensitive to the batch-size (e.g., results of Fig. 8.a and the adaptive threshold of 294-306). Open questions are whether the model would be i) effective for extremely low batch sizes and ii) robust to non i.i.d. batches (e.g., samples of the same class, as in [d,e]). \n\n3. While it is intuitive that maximizing the entropy of low confident examples can both prevent overfitting and reduce overconfidence, the fact that high entropy samples share atypical features with low entropy ones (65-70) is not intuitive and not clarified with the qualitative examples of Fig. 3.a and H.3. It would be helpful to either clarify the meaning of atypical features or provide evidence of these shared spurious factors, or down weigh the related statements. \n\n4. Lines 270-274 indicate that minimizing entropy on high-confidence samples leads to overconfidence and maximizing entropy on low-confidence ones reduces overfitting. This is also suggested by Fig. 2, Fig. 3.b, and Fig. 8.c at the level of entropy. To further analyze the phenomenon of overconfidence, it could be interesting to show the expected calibration error [f], an analysis reported by related TTA works exploring overconfidence and potential solutions (e.g., [g,h]). \n\n5. While maximizing entropy is a good unlearning strategy, it would have been interesting to explore other alternatives (e.g., [i]) to justify this design choice. Note that different unlearning choices could lead to different effects in terms of overfitting and overconfidence. Moreover, related works do not discuss how the paper relates to the machine unlearning literature and related approaches that used unlearning for downstream tasks (e.g., debiasing [j]).\n\n\n**References** ([a,b,f] already in the manuscript):\n\n[a] Karmanov, Adilbek, et al. \"Efficient test-time adaptation of vision-language models.\" CVPR 2024.\\\n[b] Zhang, Ce, et al. \"Dual prototype evolving for test-time generalization of vision-language models.\" NeurIPS 2024.\\\n[c] Zhou, Lihua, et al. \"Bayesian test-time adaptation for vision-language models.\" CVPR 2025.\\\n[d] Gong, Taesik, et al. \"Note: Robust continual test-time adaptation against temporal correlation.\" NeurIPS 2022.\\\n[e] Niu, Shuaicheng, et al. \"Towards stable test-time adaptation in dynamic wild world.\" ICLR 2023.\\\n[f] Guo, Chuan, et al. \"On calibration of modern neural networks.\" ICML, 2017.\\\n[g] Yoon, Hee Suk, et al. \"C-TPT: Calibrated test-time prompt tuning for vision-language models via text-feature dispersion.\" ICLR 2024. \\\n[h] Farina, Matteo, et al. \"Frustratingly easy test-time adaptation of vision-language models.\" NeurIPS 2024.\\\n[i] Fan, Chongyu, et al. \"Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.\" ICLR 2024.\\\n[j] Chen, Ruizhe, et al. \"Fast model debias with machine unlearning.\" NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4SEinTlpMP", "reviewer": "Reviewer_uPAE", "summary": "The paper proposes BITTA, a bilateral information-aware TTA framework for VLMs (e.g., CLIP): low-entropy samples are used for learning (entropy minimization) and high-entropy samples for unlearning (entropy maximization), with a heuristic to dynamically set the low-entropy selection ratio. Experiments on CIFAR-10/100-C and ImageNet-C show consistent but modest gains and some compatibility with existing TTA methods.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Clear identification of a failure mode of fixed low-entropy selection (overconfidence on errors).\n2. Simple, general plug-in that works with multiple TTA baselines and backbones.\n3. Sensible diagnostics (entropy dynamics, t-SNE) and reasonable ablations (batch size, steps, λ).\n3. Low computational overhead; easy to implement.", "weaknesses": "1. Technical novelty is limited; the idea (minimize on confident, maximize on uncertain) is incremental, and the theory is high-level with strong assumptions.\n2. ImageNet generalization is under-evaluated: key variants (e.g., ImageNet-R/A/V2/Sketch) are not systematically included in the main results.\n3. Missing important baselines (e.g., DiffTPT [1], DMN-ZS [2]), weakening the empirical case.\n4. Gains are often modest with occasional regressions; clearer analysis of when it helps/hurts is needed.\n\n[1] Diverse data augmentation with diffusions for effective test-time prompt tuning\n[2] Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KrVJPQ8UZ0", "reviewer": "Reviewer_y1BL", "summary": "The paper tackles a known failure mode in VLM TTA: selecting only low-entropy samples for entropy minimization (EM) can overfit \"atypical features\" and amplify overconfidence on misclassified cases. The paper argues that the high-entropy samples which the current model is unconfident can serve as a potential candidate for regularization. Thus, BITTA proposes a bilateral strategy: (i) learn with low-entropy samples using a standard TTA objective, and (ii) unlearn with a small set of high-entropy samples by maximizing their predictive entropy. The intent is to curb memorization of atypical features while preserving core representations. BITTA is used as plug-in to multiple TTA algorithms, yielding gains on multiple benchmarks.", "questions": "- Regarding \"atypical feature\" assumption, what are \"atypical features\"? Would it be possible to provide an operational definition of “atypical features”?\n\n- Would it be possible to quantify their prevalence/strength of \"atypical features\" by entropy bins (low/medium/high) to substantiate the \"atypical\" feature assumption?\n\n- Would it be possible to show that the unlearning process for high-entropy samples can reduce these atypical signals—especially for misclassified low-entropy cases?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- Clear diagnosis of a common TTA pitfall (confidence selection → overfitting to atypical cues).\n\n- Simple, compatible design: bilateral learning + unlearning that can wrap around existing TTA learners.\n\n- Dynamic selection ratio that reflects dataset/noise distribution rather than a fixed value.", "weaknesses": "- What the method assumes. The paper assumes two things:\n(1) Low-entropy and high-entropy samples both carry spurious (atypical) cues; and\n(2) those cues are stronger or more frequent in the high-entropy group.\nThis is why the method adds an “unlearning” branch on high-entropy samples.\n\n- What is missing. The paper does not make this assumption clear; it does not define what counts as an “atypical feature,” and never measures how much atypicality exists in each entropy group (low / medium / high). \n\n- Why this matters. Without a clear definition and measurement regarding “atypical feature,” we cannot tell whether the gains of BITTA come from truly removing reliance on \"atypical\" features or from a generic regularization effect. That weakens the main motivation for the bilateral design and the authors' claims.\n\n - What evidence would resolve this. Provide a formal definition an operational test for “atypical features”, and report their level by entropy bin. Then show that the unlearning process for high-entropy samples reduces these atypical signals—especially for misclassified low-entropy cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)