Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Authors: Zixuan Hu
Yichun Hu
Xiaotong Li
SHIXIANG TANG
LINGYU DUAN
Venue: ICML 2025 poster (2025)
Keywords: Test-Time Adaptation
Transfer Learning
Primary area: applications->computer_vision

Abstract: Wild Test-Time Adaptation (WTTA) is proposed to adapt a source model to unseen domains under extreme data scarcity and multiple shifts. Previous approaches mainly focused on sample selection strategies, while overlooking the fundamental problem on underlying optimization. Initially, we critically analyze the widely-adopted entropy minimization framework in WTTA and uncover its significant limitations in noisy optimization dynamics that substantially hinder adaptation efficiency. Through our analysis, we identify region confidence as a superior alternative to traditional entropy, however, its direct optimization remains computationally prohibitive for real-time applications. In this paper, we introduce a novel region-integrated method **ReCAP** that bypasses the lengthy process. Specifically, we propose a probabilistic region modeling scheme that flexibly captures semantic changes in embedding space. Subsequently, we develop a finite-to-infinite asymptotic approximation that transforms the intractable region confidence into a tractable and upper-bounded proxy. These innovations significantly unlock the overlooked potential dynamics in local region in a concise solution. Our extensive experiments demonstrate the consistent superiority of ReCAP over existing methods across various datasets and wild scenarios. The source code will be available at https://github.com/hzcar/ReCAP.

Beyond Entropy: Region Confidence Proxy for Wild Test- Time Adaptation
Zixuan Hu 12 Yichun Hu'
Xiaotong Li'
Shixiang Tang²
Ling- Yu Duan 1
Abstract
(a)
Butch
Batch
Cluss
Wild Test-Time Adaptation (WTTA) is proposed to adapt source model to unseen domains under extreme data scarcity and multiple shifts. Pre- vious approaches mainly focused sample se- lection strategies, while overlooking the funda- mental problem on underlying optimization. Ini- tially, we critically analyze the widely-adopted entropy minimization framework in WTTA and uncover its significant limitations noisy opti- mization dynamics that substantially hinder adap- tation efficiency. Through our analysis, we iden- tify region confidence as a superior alternative to traditional entropy. however, its direct optimiza- tion remains computationally prohibitive for real- time applications. In this paper, we introduce novel region-integrated method ReCAP that bypasses the lengthy process. Specifically, we propose a probabilistic region modeling scheme that flexibly captures semantic changes in embed- ding space. Subsequently, we develop a finite- to-infinite asymptotic approximation that trans- forms the intractable region confidence into a tractable and upper-bounded proxy. These inno- vations significantly unlock the overlooked po- tential dynamics in local region in concise so- lution. Our extensive experiments demonstrate the consistent superiority of ReCAP over exist- ing methods across various datasets and wild scenarios. The source code will be available at https://githin www/hocaaeecccccoooooooo ReCAP.
Batchh 1
Battch 2
Batch
Batcch 2
Batch
Wild
Multiple Concurrent
Extreme Data Scarcity
(b)
+
Serpe Wiighernn Semler g Pretene
Amuo
Mill
Adlittation Process Jay Tare)]
Figure I. (a) Illustration Mild (Wang et al., 2020) and Wild (Niu et al., 2023) TTA settings. (b) Comparison of the adaptation process between mild and wild scenes on the Zoom domain of ImageNet-C dataset (Hendrycks & Dietterich, 2019). Different colors of points represent different predicted classes of samples in the local region. The results highlight that entropy minimization in the wild scenario causes significant local prediction instability.
distribution shifts between training and testing data (Ben- David et al., 2010; Koh et al., 2021; Hu et al., 2024). To tackle this issue, Test-Time Adaptation (TTA) (Iwasawa & Matsuo, 2021; Alfarra et al., 2024; Liang et al., 2024) has emerged as a critical paradigm, enabling source models to adapt to target distributions through online updates. Its dom- inant approach involves optimizing the model to minimize prediction entropy. thereby enhancing the model's global confidence in the target domain.
Wang al., 2022) have achieved promising results under mild condi- tions, they show significant performance drops in wild sce- narios involving extreme data scarcity and multiple concur- rent shifts (Niu et al., 2023), as shown in Fig. 1(a). To enable effective adaptation under these wild settings, recent works focus on developing selection criteria to leverage reli- able samples only for entropy minimization. For example, SAR (Niu et al., 2023) excluded samples with high entropy and, DeYO (Lee et al., 2024) filtered out samples with sen- sitive prediction changes under image transformation.
While TTA methods (Zhou & Levine, 2021;
1. Introduction
Deep neural networks have exhibited remarkable success across various visual tasks (Girshick, 2015; He et al., 2016). However, their performance is often compromised by the
'Corresponding author 'School of Computer Science, Peking University, Beijing, China *Peng Cheng Laboratory, Shenzhen, China 'The Chinese University Hong Kong, Hongkong, China. Correspondence to: Ling- Yu Duan <lingyu @ wkw.edu.cn..
Orthogonal to sample selection, this paper delves into a fundamental yet overlooked challenge: the noisy optimiza- tion dynamics introduced by typical entropy minimization.
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
In wild scenes, we observe a notable instability where the semantically similar samples within the local scope demon- strate a hard-to-compromise prediction discrepancy in wild scenes, as shown in Fig. 1(b). Such inconsistency leads the underlying optimization dynamics for these samples to become essentially conflicting. When entropy minimization solely based on the individual sample, this narrow atten- tion inevitably amplifies noisy dynamics, undermining both local consisteney and overall adaptation efficiency.
propose superior alternative as region confidence, a novel training objective that leverages local knowledge to mitigate noisy conflicts. 2) To ensure real-time processing capabil- ity, we propose ReCAP, a novel training framework that incorporates two key components: a probabilistic modeling mechanism to flexibly capture variations in local region, and a finite-to-infinite asymptotic analysis to provide an efficient proxy for optimizing the intractable terms. 3) We demonstrate that ReCAP significantly outperforms existing WTTA methods through extensive experiments. Notably, ReCAP can seamlessly integrate with the orthogonal sample selection approaches in negligible computational overhead, showcasing a comprehensive framework for WTTA.
Building on the above observations, it is essential to mini- mize the bias in the optimization direction, as well as the variance of unstable local predictions. Therefore, we pro- pose a novel TTA strategy to enhance region confidence, which reflects the model's s prediction certainty and consis- tency across the local region, rather than solely relying on biased individual predictions. We take two key statistical measures into consideration of the objective design: the bias ferm that quantifies the global entropy and the variance term that captures the prediction divergence within the region. These two terms work together to rectify overall optimiza- tion dynamies and reduce prediction disparity, promoting consistent adaptation across the entire region.
2. Related Work
We revisit the TTA methods for further analysis and put other related areas into the Appendix E due to space limits.
Test-Time Adaptation aims to enhance the performance on out-of -itt-----istiittionn samples during inference. Depending on whether the training process is altered, TTA methods can be mainly divided into two groups: 1) Dest-Time Training (TTT) (Sun et al., 2020; Bartler al., 2022; Hakim et al., 2023; Liu et al., 2024) jointly optimizes the model on train- ing data with both supervised and self-supervised losses, and then conducts self-supervised training at test time. 2) Fully Test-Time Adaptation (Fully TTA) (Boudiaf et al., 2022; Hong et al., 2023; Zhao et al., 2023; Press et al., 2024; Hu et al., 2025) refrains from altering the training process and focuses solely on adapting the model during testing. In this paper, we foeus Fully TTA, as it is more gener- ally applicable than TTT, allowing adaptation of arbitrary pre-trained models without access to training data.
Despite the advantages of region confidence, the uncertain region scope and highly complex computations make it impractical for real-time testing. To overcome this, we in- troduce a new training framework, "Region Confidence Adaptive Proxy (ReCAP)", which incorporates a probabilis- tic region modeling mechanism and a highly efficient region optimization proxy. Specifically, ReCAP introduces a prob- abilistic representation to describe local regions as multi- variate gaussian distribution, identifying a suitable region in feature space. Building upon this foundation, we propose finite-to-infinite optimization proxy. Initially, we conduct a quantitative analysis of region confidence statistics under finite distribution sampling. Subsequently, we develop an asymptotic approximation convert the intractable bias and variance terms into a concise, upper-bounded proxy. These upper bounds seamlessly integrate the extensive op- timization dynamics of infinite local samples in a straight- forward manner. As a result, our method establishes an efficient proxy for optimizing region confidence, replacing entropy-based approaches to unlock significantly enhanced adaptation efficiency.
Due to the broad applicability of TTA (Shin et al., 2022; Lin et al., 2023; Gao et al., 2024; Karmanov et al., 2024; Wang et al., 2024), a variety of methods have been developed, For instance, some methods adjust the affine coefficients of Batch Normalization layers to adapt to the target domain (Schneider et al., 2020; Hu et al., 2021; Lim et al., 2022). Others refine the prediction to provide a more robust train- ing process (Zhang et al., 2022; Chen et al., 2022). Since Tent (Wang et al., 2020) introduces entropy minimization to enhance model confidence and reduce error rates, numer- ous works follow this practice of entropy-based training, Building upon Tent, SAR (Niu et al., 2023) and De (Lee et al., 2024) propose selection strategies for Wild TTA scenes, which exclude harmful samples to enhance the accu- racy of adaptation directions. In contrast to these selection approaches, this paper introduces a novel strategy that re- places entropy-based based training with a framework designed to encourage and integrate consistent optimization dynamics, significantly enhancing adaptation efficiency.
We evaluate the effectiveness and generalizability of our method through experiments on both ResNet (He et al., 2016) and VIT (Dosovitskiy et al., 2020), achieving state- of-the-art results on ImageNet-C (Hendrycks & Dietterich, 2019) with average gains of +2.0%, +1.1%,4 +1.9% on 15 corruption shifts in three wild scenarios.
Contributions. 1) We analyze and verify the limitation of widely adopted entropy minimization in introducing con- flicting dynamics in WTTA scenarios. To address this, we
2

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
(a)
Distribution of Inconsistency Probability
(b) Entropy vs. KL Divergence
[c) Optimization on Different Areas
Number of Sample
Mild Wild
Entropy of Sample
Mild Wild
40-
Accuracy (Avg)
6k-
4k-
22
os
2
EM on Ares2
2k
Areal
Area 1
Area 2
ReCAP Anest &J
0.15
1000
1500
Local Imconsiteeny(Probablltyy
Local Inconsistency(KL Divergence)
Online Batch
Figure 2. Local consistency during the entropy minimization process under mild and wild (imbalanced label shift) scenarios. Consistency is measured by prediction discrepancies between each sample and its 256 neighboring samples. (a) shows the probability of inconsistent predictions neighbors. (b) records the entropy and average KL Divergence between prediction probabilities samples and their neighbors. (c) investigates adaptation performance when using samples with varying levels of local consistency. All experiments are conducted on ImageNet-C of Gaussian noise with ResNe150. 'EM' denotes entropy minimization and 'ReCAP' denotes our method.
3. Local Inconsistency: A Barrier to Efficient Adaptation
conflicting dynamics, introducing noise that hinders adap- tation in the affected region. Such instability often stems from blurry decision boundaries and is prevalent in real- world deployments under domain shifts (Arani et al., 2022). Hence, is crucial to evaluate the reliability of entropy mini- mization in preserving local consistency under wild scenes.
3.1. Preliminaries for Wild Test-Time Adaptation
In Test-Time Adaptation (TTA), we have model Fo that has been pre-trained on the source domain Ds = (Xtrain, Yerain J and need to evaluate it on the target do- main Dj = (Xtest, Yreat)- Here, denotes the model parameters, and X, denote samples and labels with the distribution shift P(Xtrain, (trrain A P(Xtest, Ytest).
To assess local consistency, we record the prediction proba- bilities of test samples and their neighbors (sampled from the local region in feature space). From Fig. 2(a), the incon- sistent probability converts from an unimodal distribution near zero to a fat-tailed distribution in wild scenes, reflect- ing high risk of misaligned prediction within local space. Fig. 2(b) further reveals that while the entropy value is opti- mized to a similar level in both scenarios, the wild setting exhibits notably larger prediction discrepancies. These find- ings demonstrate that conventional entropy minimization can undermine local consistency.
Unlike mild scenes in (Wang al., 2020), Wild TTA tackles more complex environments involving extreme data scarcity and multiple concurrent shifts, including three practical sce- narios: 1) Limited data stream, where batch size is restricted to 1. 2) Mixed testing domain, where target domain consists of k different sub-domains: DD((xeee = E II; DD with 11, being mixing coefficient. 3) Imbalanced label shift, where the test label distribution is imbalanced and shifts over time according to a function (v).
Additionally, we evaluate the impact of using samples with varying levels of local consistency during adaptation, as shown in Fig. 2(c). Remarkably, entropy minimization us- ing samples with low entropy and low consistency (Area 2) still carries performance collapse. Conversely, training with high consistency samples (Area 1) achieves compa- rable adaptation performance compared to joint training on Areas 1&2, showcasing superior adaptation efficiency. These results suggest that entropy minimization, even when combined with advanced selection, still hinders adaptation efficiency it fails to ensure prediction consistency.
To address these challenges, most existing methods (Niu et al., 2022; 2023; Lee et al., 2024) design various selection indicators to identify reliable samples and update 0 through minimizing the entropy loss Lent:
(x) (1) (o())
Σ po(x); log po
where C is the number of classes and pe(x)= = Fx) (1) (po (z)......PP(())))) E RC is prediction probability on x.
=
3.2. Exploring Entropy Minimization via Local Consistency
3.3. From Sample Confidence to Region Confidence
Entropy minimization promotes the prediction probability to converge toward the dominant class, boosting confidence on unlabeled data. Its effectiveness heavily relies on the lo- cal consistency (nearby points share the similar prediction) to extend sample-wise confidence to regional scale (Zhou et al., 2003; Wei et al., 2020). Intuitively, when predictions within a local space are consistent, optimization dynamics for individual points align with the overall direction, magni- fying the local effects. Conversely, inconsistencies create
Building on the above findings, it is essential to address the bias between the optimization direction and regional objective, while also reducing the variance of inconsistent prediction probabilities within the local region. To this end, we introduce a novel objective called region confidence to replace vanilla entropy. This objective optimizes both region-wise confidence and stability simultaneously, thereby improving global optimization efficiency. The mathematical definition is as follows:
3

---

Beyond Entropy: Region Confidence Proxy for Wild Test- Teme Adaptation
Probabilistic Region Modeling
Region Confidence
Bias Term
Lowe (F)dEE
Prosddpp
Port adopt
Variance Term
Original Prediction
Entropy Misimization
p(a)
pira
Inputxx
Encoder
p
From Finite Sampling to Infinite Approximation
&)
p
bound
upper bound
Selection
Overall Optimization Procedure
Las (x) dem (x)
Car
a(x)
A (2) + 2mm
Efficient Proky Region Confidence
Figure 3. Overview of our ReCAP. ReCAP performs probabilistic modeling to determine local regions in the latent space (Sec. 4.1). We further derives two closed-form upper bounds for the intractable bias and variance terms via a finitettoinnfmite asymptotic approximation, offering an efficient proxy for optimizing Region Confidence without the lengthy sampling process (Sec. 4.2 & 4.3).
Definition 3.1. (Region Confidence) Let consider local region Ω of sample The region confidence of x on Ω is defined as integrals of entropy loss on Ω (Bias term) plus the Kullback-Leibler divergence between prediction probabilities of x and those of samples in St (Variance term):
et al., 2023; Yu et al., 2024a), we model local regions in feature space to flexibly capture rich semantic information. To avoid class mixing, we further select the latent space extracted by the backbone, which ensures optimal class separability in the model. Hence, we explore region con- fidence within this latent space, and po(x) in Eq. can be replaced by the probability of its corresponding feature 2:
CRc(+) = 1 Cau (z)dz+ +N / Q
DKL (po( po (2)) dž,
Ca
Bias term
Variance term
po(2); (softmax (Az + b)); ΣE e2y-24hy
(3)
(2)
where ( = -i Pi log P: denotes the Kullback- Leibler divergence and X denotes a trade-off coefficient.
where the subscript (-); denotes the i-th class. A gCxd and b E R denote the linear and bias coefficients of the affine layer in the classifier, respectively.
These two terms have distinct but complementary effects. The bias term integrates entropy loss over infinite samples in the local region, enabling an optimization that aligns with the overall training dynamics. The variance term penalizes large discrepancies, enhancing local consistency and reduc- ing the dispersion of dynamics. By combining two terms, region confidence promotes consistent and confident predic- tions within the region, harnessing the potential dynamics embedded in the local space to boost adaptation efficiency.
Rather than treating the local region as a static range, we model as probabilistic representation following a multi- variate Gaussian distribution. Specifically, the center of this distribution corresponds to each feature, while the variance, which defines the scope, is estimated using small set of in in-distribution data. The region is determined follows:
(4) where Ω (zt) is the local region of the t-th test batch 2; and Σ is the diagonal matrix variance on small set of source data, egg., 500 samples are enough for ImageNet-C dataset. a hyper- param--rr to control the scope.
Ω (zx) == N (ZI, T Σ),- = Diag (Vards (=)),
4. Region Confidence Proxy
Despite the advantages of region confidence, there still two considerable challenges to optimize it: 1) Uncertain region scope. The static regions fix the sample location, making it difficult to determine an appropriate local scope. 2) Heavy computational burden. Both terms in Eq. are intractable, relying on extensive sampling and additional model forwards for measurement. To tackle these issues, propose new TTA method called "Region Confidence Adaptive Proxy (ReCAP)", which incorporates probabilis- tic region modeling mechanism (Sec. 4. and an efficient proxy for optimizing region confidence (Sec. 4.2).
4.2. Efficient Metric of Region Confidence
Based on the local region defined above, the computation of two terms depends on an infinite number of potential samples from the distribution and requires extensive sam- pling to capture the statistical information. However, the computational overhead and memory usage increase almost linearly with the number of sampling, making it impractical for real-time requirements in TTA testing. To address this issue, we develop lnite-to-infinite approximation for the two terms, leading to a highly efficient implementation.
4.1. Probabilistic Region Modeling
Since different directions in feature space can imply poten- tials of valuable semantic changes (Bengio et al., 2013; Li
Taking the bias term as an example, we first consider the case of finite sampling:
4

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Lemma 4.1. (Bias Term under Finite Sampling) Given N
distribution (u, E). The expectation of Kullback-Leibler divergence between the output probability over this distri- bution and the probability at its center has upper bound:
features 21,...;22 NN drawn from the local region and their corresponding probabilities: po(z1),- ......... ...
: p=(=x). The bias
term on these features satisfies the following inequality:
EznN(Ex KL (po(=)]lpo())
N
Dt= po(za))
N
(po())) &
N
Σ
log Po (zz);
e
C
cavsth
log
(8)
j=1 kk=1 cax
2k== cax
where po(z;) is defined in Eq. 3.
(u.-a)) D a(a)-a))'
CRI,
In the following, we consider the case where N grows to infinity and find that Σ po(ss), in Eq. 10 gradually con-
where the upper bound Lui is called Regional Instability.
N verges to the expectation of the prediction in the local region.
This proposition also provides a theoretical result that cap- tures local information without the need for sampling. By combining two upper bounds, we ultimately present effi- cient proxy of region confidence in Eq.2.
For the remaining summation term Σ logpo(2) Po(2)); i. it ac- tually corresponds to the negative log-likelihood and can be scaled using the following lemma:
Lemma 4.2. (Upper Bound of Negative Log-Likelihood Given a feature 2 and its local region Qt which follows a Gaussian distribution N(,; E). The expected value of the logarithm of the predicted probability for the i-th class sat- isfies the following inequality:
4.3. Overall Procedure of ReCAP
Following common practices (Niu et al., 2022; Lee et al., 2024), the loss function requires filtering and weighting based on reliability. Unlike traditional entropy-based crite- ria, our analysis in Sec. 3.2 shows that regional confidence can also serve measure of reliability from orthogonal perspective. Building on this insight, we leverage Regional Entropy LRE to identify reliable samples and enhance their optimization dynamics during adaptation. Formally, the overall procedure is as follows:
-Bn-N(p.)) Pe))
< log
la aa)= bi))+(/ (()
where the superscript (.) denotes the transpose operation. Through the above two lemmas, we can further derive closed-form upper bound for the bias term via asymptotic approximation. Refer to the Appendix A for missing proofs and detailed explanations.
min a(z) (((()))(((CAAA [[Cnn(s)<TRR) (CRE(T) + (CRE(E)
ACR((x)),
where a(x) exp(Lne(() Lo)
Proposition 4.3. (Efficient Metric of Bias Term) Given a feature= 2 and its local region St which follows a Gaussian distribution N N(u, E). The expectation of entropy loss over the entire distribution has a closed-form upper bound:
where azx) and <T) denotes the weighting and selection term. Lo, THE and X are hyper-parameters.
5. Experiments
Eg [eent] BENN(+,E)
Σ (3); log (3))s
5.1. Experimental Setup
For a fair comparison, we follow the identical network ar- chitectures, optimizer, and batch sizes as the Wild TTA benchmark proposed in (Niu et al., 2023).
car 2a, Eas
log
j=1 kk= e but tox Σ az
aala-aa)) (a: a()a as)*
LRE.
Datasets. We conduct our experiments on three datasets to evaluate the robustness and generalization capability of our method under diverse distribution shifts: 1) ImageNet-- C (Hendrycks & Dietterich, 2019), a large-scale dataset categorized into 15 common corruption types and 5 sever- ity levels for each type. 2) ImageNet-R (Hendrycks et al., 2021) and 3) VisDA-2021 (Bashkirova et al., 2022), two datasets which encompass diverse domain shifts due vary- ing styles and textures (e.g., sketch, cartoon), compared to ImageNet-C to assess the efficacy for more challenging wild test scenarios in the Appendix B.
where the upper bound LRE is called Regional Entropy.
Notably, Proposition 4.3 offers an easy-to-compute metric without any additional sampling or model forward passes. Instead of minimizing the exact bias term in Eq. 2, we can optimize its upper bound to implicitly enhance overall sample confidence within the region with minimal cost.
Meanwhile, we apply a similar mathematical framework to derive a closed-fon foosed-formma upper bound for the variance term.
Propusition 4.4. (Efficient Metric of Variance Term) Given a feature and its local region Q which follows Goussian
Compared Methods. We compare our ReCAP with the following state-of-the- methods: DDA (Gao et al., 2022)
5

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Table 1. Comparisons with state-of-the-an methods ImageNet-C (severity level 5) under Limited Batch Size regarding Accuracy
(%) We report mean performance over independent runs. The best results are bold and the second- best are in underline.
Noise
Blur
Weather
Digital
Method
Average
Gauss,
Shot
Impul.
Defoc
Glass
Motion
Zoom
Snow
Frost
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
ResNet50
18.0
19.8
17.9
19.8
11.4
21.4
24.9
40.4
47.3
33.6
69.3
36.3
18.6
28.4
52.3
30.6
MЕMO
18.5
20.5
18.4
17.1
21.8
26.9
40.4
47.0
69.5
36.5
19.2
32.1
53.3
31.2
DDA
42.4
43.3
42.3
16.6
19.6
21.9
26.0
35.7
40.1
13.7
61.2
25.2
37.5
46.6
54.1
35.1
Tent
2.5
2.9
2.5
13.5
3.6
18.6
17.6
15.3
23.0
1.4
70.4
42.2
6.2
49.2
53.8
21.5
EATA
24.9
28.0
25.8
18.3
17.0
31.2
29.8
42.5
44.1
41.3
70.9
44.2
27.6
46.8
55.4
36.5
SAR
25.5
28.0
24.9
18.7
16.3
28.6
31.4
46.2
44.9
33.4
72.8
44.3
15.3
47.1
56.1
35.6
DeYO
41.2
44.3
42.5
22.4
24.7
41.8
21.9
54.8
51.6
21.9
73.1
53.2
59.8
44.1
ReCAP (Ours)
42.5
44.4
42.9
19.4
25.0
42.2
44.0
49.7
52.4
57.5
72.9
53.6
29.5
60.4
60.0
46.4
ReCAP+SAR
41.7
44.5
40.6
24.8
25.8
44.0
47.0
56.2
53.0
52.8
73.4
54.6
61.7
60.7
48.6
ReCAP=DeYO
42.5
44.8
42.8
25,9
27.2
43.7
44.9
55.8
52.8
51.9
73.5
54.8
50.9
61.55
60.7
48.9
Vitbase
9.5
6.7
8.2
29.0
23.4
33.9
27.1
15.9
26.5
47.2
54.7
44.1
30.5
44.5
47.8
29.9
MЕMO
21.6
17.3
20.6
37.1
29.6
40.4
34.4
24.9
34.7
55.1
64.8
54.9
37.4
55.4
57.6
39.1
DDA
41.3
41.1
40.7
24.4
27.2
30.6
26.9
18.3
27.5
34.6
50.1
32.4
42.3
52.2
52.6
36.1
Tent
42.2
1.0
43.3
52.4
48.2
55.5
50.5
16.5
16.9
66.4
74,9
64.7
51.6
67.0
64.3
47.7
EATA
30.1
24.6
34.2
44.3
48.4
42.4
38.1
46.0
60.7
65.8
61.2
57.8
46.6
SAR
42.7
39.5
41.9
54.6
51,2
58.3
54.4
60.2
54.7
70.3
75.9
66.8
58.4
69.5
66.3
57.6
DeYO
53.4
50.4
55.0
58.7
59.5
64.5
52.5
68.1
78.3
67.9
73.8
ReCAP (Ours)
53.5
56.7
56.9
59.2
60.5
65.3
64.0
69.6
67.2
74.1
78.4
64.6
70.2
74.4
71.5
65.7
ReCAP+SAR
55.3
56.2
56.5
60.0
61.2
66.5
65.2
69.8
68.0
74.5
78.6
68.4
71.0
74.9
71.6
66.5
ReCO
54.4
55.3
55.5
59.8
61.1
65.2
64.9
69.3
67.9
73.9
78.4
67.1
70.7
74.4
71.0
65.9
performs diffusion-basee adaptation to map the input image back to the source domain. MEMO (Zhang et al., 2022) minimizes marginal entropy across augmented variants of test samples. EATA, SAR, and Ye are selection-band methods with distinct designs. EATA (Niu al., 2022) com- bines entropy- -based selection with weighting mechanism. SAR (Niu et al., 2023) minimizes entropy using sharpness- aware optimization. De (Lee et al., 2024) employs dual filtering based on disentangled factors.
Implementation Details. Following the common setting in TTA (Niu et al., 2023; Lee et al., 2024), we conduct ex- periments on ResNe150-GN (Wu & He, 2018) and ViTe LN (Dosovitskiy al., 2020) obtained from cimm (Wight- man, 2019). For the optimizer, we use SGD, batch size of 64 (except for batch size=1), with momentum of 0.9, and learning rate of 0.00025/0.001 for ResNet/ViT. For ReCAP, Lo and TRE in Eq. 9 is set to 0.7/1.0 1.0 x InC and 0.8/1.0 1.0 x InC (C' is the number of classes) for ResNet/ViT. The syper-parameter by default. For trainable parameters, according to common practices (Wang et al., 2020), we adapt the affine parameters of normalization layers.
Eq. 4 1.2 and in Eq. 9 0.5
Table Comparisons with SOTA methods ImageNet-CG (sever-
ity level 5, 4) under Mixed Testing Domain.
Methood
Level=5
Level=4
Average
ResNet50
30.6
42.7
36.7
MEMO
31.2
43.0
37.1
5.2. Evaluation Results
DDA
35.1
43.6
39.4
Evaluation on Data Scarcity. To evaluate the effectiveness of our method under severe data scarcity, we compare our ReCAP with prior approaches in challenging scenarios with limited data streams, i.e., batch size As shown in Tab. 1, ReCAP significantly improves adaptation performance, emerging as the only method to consistently outperform the source model across all corruption types. Notably. ReCAP achieves superior results in 25 out of 30 cases, substantially surpassing the previous SOTA method by +2.3% and +1.6% on ResNet and ViT evaluations, respectively. These results underscore the robustness of our method in the face of data limitations and validate its effectiveness in enhancing adap- tation efficiency for resource- con.trccic..ttrieedd..o.. low-data training.
Tent
13.4
20.6
17.0
EATA
38.1
47.7
42.9
SAR
38.3
48.6
43.5
DeYO
38.6
50.2
44.4
ReCAP (Ours)
41.5
51.2
46.4
ReCAP+SAR
42.1
51.9
47.0
ReCAP+DeYO
42.2
52.4
47.3
VitBase
29.9
42.9
36.4
MЕMO
39.1
51.3
45.2
DDA
36.1
45.1
40.6
Tent
16.5
64.3
40.4
EATA
55.7
63.7
59.7
SAR
57.1
64,9
61.0
DeYO
59,4
66.8
63.1
ReCAP (Ours)
59.4
67.1
63.3
Evaluation on Multiple Concurrent Shifts. To evaluate the ability of our ReCAP to handle complex distribution shifts, we compare various methods under mixed testing
ReCAP+SAR
60.0
67.2
63.6
ReCAP+DeYO
59.8
67.0
63.4
6

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Table 3. Comparisons with state-of-the-ee methods on ImageNet-C (severity level 5) under Imbalanced Label Shift.
Noise
Blur
Weather Frost
Digital
Model+-Method
Average
Gauss,
Shot
Impul.
Defoc
Glass
Motion
Zoom
Snow
Fog
Brit.
Contr.
Elastic
Pixel
JPEG
ResNer50
17.9
19.9
17.9
19.7
11.3
21.3
24.9
40.4
47.4
33.6
69.2
36.3
18.7
28.4
52.2
30.6
MEMO
18.4
20.6
18.4
17.1
12.7
21.8
26.9
40.7
46.9
34.8
69.6
36.4
19.2
32.2
53.4
31.3
DDA
42.5
43.4
42.3
16.5
19.4
21.9
26.1
35.8
40.2
13.7
61.3
25.2
37.3
46.9
54.3
35.1
Tent
2.6
3.3
2.7
13.9
7.9
19.5
28.7
16.5
21.9
1.8
70.5
42.2
6.6
49.4
53.7
22.8
EATA
27.2
28.5
28.4
15.1
16.7
24.6
25.5
32.5
32.2
40.0
66.5
33.2
24.1
42.2
38.6
31.7
SAR
34.0
36.7
36.2
21.8
20,9
33.2
32.4
38.7
45.6
50.6
72.9
46.8
14.3
52.2
56.8
39.5
De
41.7
44.0
42.5
23.4
23,9
41.3
13.0
53.9
52.2
38.6
73.1
52.3
46.8
59.3
59.1
44,3
ReCAP (Ours)
42.0
44.1
42.7
19.8
24.3
39.7
40.2
46.0
52.2
57.3
73.1
52.4
33.7
59.4
59.5
45.8
ReCAP+SAR
42.2
44.4
42.7
24.1
24.3
41.6
43.8
51.6
52.4
56.8
73.1
52.9
38.9
60.2
59.4
47.2
ReCAP-DeYO
42.5
44.5
43.3
25.8
26.7
43.3
39.1
54.2
53.2
59.3
73.4
53.8
49.2
61.4
60.3
48.7
Vitbuse
9.4
6.7
8.3
29.1
23.4
34.0
27.0
15.8
26.3
47.4
54.7
43,9
30.5
44.5
47.6
29.9
MEMO
21.6
17.4
20.6
37.1
29.6
40.6
34.4
25.0
34.8
55.2
65.0
54.9
37.4
55.5
57.7
39.1
DDA
41.3
41.3
40.6
24.6
27.4
30.7
26.9
18.2
27.7
34.8
50.0
32.3
42.2
52.5
52.7
36,2
Tent
32.7
1.4
34.6
54.4
52.3
58.2
52.2
7.7
12.0
69.3
76.1
56.7
69.4
66.4
47.3
EATA
35.8
34.8
36,8
45.1
47.3
49.3
47.8
56.6
55.5
62.1
72.3
21.6
56.0
64.6
63.7
50,0
SAR
48.2
48.7
49.0
55.4
54.5
59.2
54.3
55.8
54.5
70.0
76.9
66.1
62.2
70.2
66.5
59.4
DeYO
53.0
34.4
48.8
57.6
58.5
63.3
35.4
67.4
66.0
73.0
77.7
66.6
68.1
73.1
69.8
60.8
ReCAP (Ours)
53.1
38.5
49.6
57.3
59.0
63.8
60.7
67.8
66.3
72.9
77.7
66.8
68.2
73.0
70.0
63.0
ReCAP+SAR
53.2
42.7
50.9
58.0
58.9
63.8
60.9
67.6
66.1
73.2
77.6
67.6
68.3
73.1
69.8
63.4
ReCAP-DeYO
53.1
53.9
54.1
57.4
58.8
63.6
60.6
67.8
66.4
72.9
77.7
67.1
68.3
73.1
69.8
64.3
domain and imbalanced label shift. As shown in Tab. 2 & 3, Tent and MEMO struggle with multiple concurrent shifts, even performing worse than the no-adapt model. While se- lection methods like SAR and De perform competitively long-term adaptation scenarios, the limitations of entropy- based approaches still hinder their adaptation efficiency. In contrast, ReCAP achieves SOTA performance across nearly all corruption types. For label shifts, ReCAP showcases significant superiority over other methods, with an average gain of + +1.5% and +2.2% in ResNet and VIT testing, re- spectively. These results validate the stable improvements offered by ReCAP under multiple concurrent corruptions.
Specifically, our method consistently improves SAR with an average gain of +11.0% +3.1 %. and +5.9% % across the three wild scenes. Similarly, De achieves improvements of +3.3%, +1.6%, and +4.0% when integrated with our method. These significant gains validate the effectiveness of our proposed region confidence optimization strategy, which can seamlessly integrate with various methods to boost adaptation performance.
5.3. Running Time Comparison
We measure the rumning time required for the adaptation of various methods under ImageNet--C. As shown in Tab. 4, Tent achieves the fastest speed (1 10s) as it only performs en- tropy optimization on samples without additional operations. DDA requires significantly more time since it needs to use the source data to train additional networks. The sharpness- aware optimization in SAR and the data augmentation in Yo require additional model forward or backward passes, resulting in more time cost. Our ReCAP achieves signifi- cantly superior performance with the second-best time cost (just +5% compared to Tent), highlighting its efficiency.
ReCAP can boost entropy-based methods. To investigate the complementarity of our approach with prior entropy- based methods, we test its integration with the latest SOTA SAR and De across all three wild scenarios. Through replacing entropy minimization with region confidence op- timization proxy, the performance combined with our ap- proach shows obvious gains in many downstream scenes.
Table Efficiency comparison of various methods. We assess TTA approaches for processing 50,000 images in Gaussian corruption type, using single Nvidia RTX 4090 GPU.
6. Ablation Study and Visualization
Method
Forward
Backward
Other computation
Time
In this section, without loss of generality, we conduct ab- lation studies and visualizations on ResNet in label shift scenes for the sake of brevity. Focusing two pivotal components of ReCAP, Probabilistic region modeling and Efficient metric of region confidence, we perform various experiments to analyze their impacts. Refer to the Appendix C for more experiments and analysis.
No-adapt
50,000
N/A
N/A
84s
DDA
Diffusion model
13,277s
Tent
50,000
50,000
N/A
110s
EATA
50,000
19,608
regularizer
118s
SAR
66,418
30,488
Model updates
164s
DeYO
82,843
24,714
Data augmentation
144s
ReCAP
50,000
19.512
Eq. 9
116s
7

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Peciepartee Inckrpmn
40
SOTA
0.12
KL Divergence
44
80
0.10
0.08
DeYo
0.06
ReCAP
Figure 4. (a) Performance with varying strengths À of the variance
term. (b) Performance with different ranges of the local region.
400
800 1200 1600
DeYO
Ours
Figure 6. Visualization of prediction results and KL Divergence of prediction probabilities in local region. Different colors points represent different predicted classes of samples within the region.
6.3. Prediction Consistency Comparison
To analyze the effect local consistency, we sample 128 points within the local distribution of each sample to assess fluctuations in prediction probabilities during the adapta- tion process. As shown in Fig. 6, compared to Yo, our approach consistently exhibits significantly lower KL Di- vergence values throughout the entire optimization process. Furthermore, we visualize the prediction results during the training process. The visualization reveals that the prior method fails to ensure consistent predictions, whereas method progressively enhances consistency, effectively re- ducing the dispersion in training dynamics.
Figure The t-SNE visualization of feature space Snow corrup- tion type after model adaptation.
6.1. Tlyper-parameter Robustness
There are two important syper-parametert in our method: the coefficient A. which determines the trade-off of bias term and variance term, and the coefficient which controls the scope of the local region. We conduct ablation experiments on these two key coefficients independently:
7. Conclusion
As shown in Fig. 4(a), the different strengths of variance term yields stable performance gains. However, when À exceeds the optimal range, model updates may prioritize consistency over confidence optimization, leading worse performance. To balance the effects of terms in region confidence, we set À to 0.5 by default. In Fig, 4(b), perfor- mance improves as the region scale increases, peaking at T == 1.2. However, when exceeds reasonable range (c.g., 2.5), the local region may suffer from category mixing, lead- ing to performance degradation. Therefore, we set T to 1.2 by default. Within the range of 7 E [0.5, 1.5], our method consistently outperforms prior SOTA methods, demonstrat- ing the robustness of the region scale in approach.
In this paper, we propose ReCAP, a novel method designed to capture consistent optimization dynamics that are often disrupted by entropy minimization, offering an effective solution for leveraging the region-integrated training. We construct novel training objective to replace vanilla en- tropy and further develop an asymptotic analysis framework to derive a more practical and flexible proxy for efficient training in TTA. We demonstrate its consistent effectiveness across a wide range of shift types, challenging wild scenar- ios, and diverse model architectures. We hope that our work will inspire future research move beyond the focus on indi- vidual data points, exploring more effective ways to leverage regional knowledge for robust and efficient adaptation.
6.2. Class Separability after Adaptation
Limitations. First, our evaluations focus on classification benchmarks. While the impact of Wild TTA in other tasks remains underexplored in existing research, it's crucial for understanding the capability boundary of methods. In future work, we plan to establish more comprehensive benchmark across various tasks for broader evaluation. Second, the local region we design domain-wiss í.e. the same shape to all samples within a domain. However, given the differences in class cluster boundaries or the distance to boundaries, the region should ideally vary at the class-wise or sample-wise level. In future work, we will explore more fine- grained region modeling mechanisms to address this limitation.
To analyze the effects of WTTA methods on feature repre- sentations, we visualize the feature representation differ- ent categories after model adaptation using t-SNE (Van der Maaten & Hinton, 2008) in Fig. 5. Compared to the latest SOTA DeYO, our ReCAP exhibits more compact intra-class features and more distinct inter-class separability (e.g., class I in orange and class 7 in grey). Since our method facilitates a more efficient transfer process, the detrimental effects of distribution shifts on the clustering properties the feature space can be quickly alleviated, yielding representations with clear classification boundaries in testing data.
8

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Acknowledgements
time adaption. In International Conference on Artificial Intelligence and Statistics, pp. 3080-3090. PMLR, 2022.
This work was supported by the Program of Beijing Mu- nicipal Science and Technology Commission Foundation (No.Z24 (00000000032 in part by the National Natural Science Foundation of China under Grant 62088102, in part by Joint Lab of Future Urban Infrastructure sponsored by Fuzhou Chengiou New Infrastructure Group and Boyun Vision Co. Ltd, and in part by the PKU-NTU Joint Research Institute (JRI) sponsored by a donation from the Ng Teng Fong Charitable Foundation.
Bashkirova, D., Hendrycks, D., Kim, D., Liao, H., Mishra, S., Rajagopalan, C., Saenko, K., Saito, K., Tayyab, B. U., Teterwak, P., et al. Visda-2021 competition: Universal domain adaptation to improve performance on out-of- distribution data. In NeurIPS 2021 Competitions and Demonstrations Truck, pp. 66-79. PMLR, 2022.
S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
Impact Statement
This paper presents work whose goal is to advance the field of Test-Time Adaptation. Its societal impact lies primar- ily in its potential to expand the applicability of machine learning models in real-world complex scenarios. By im- proving the stability and efficiency model adaptation, our work facilitates model deployment on edge devices where storage and computational power are limited. Ethically, it promotes more sustainable machine learning practices by re- ducing computational overhead, which in turn lowers energy consumption. This aligns with broader environmental sus- tainability goals and supports the development of more effi- cient, eco-friendly AI technologies for various industries.
Bengio, Y., Mesnil, G., Dauphin, Y., and Rifai, Better mixing via deep representations. In Proceedings of the 30th International Conference Machine Learning, PP. 552-560. PMLR, 17-19 Jun 2013.
Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., and Raffel, C. A. Mixmatch: A holistic ap- proach to semi-supervised learning. Advances in neural information processing systems, 32, 2019.
Boudiaf, M., Mueller, R., Ben Ayed, I., and Bertinetto, L. farameter-free online test-time adaptation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8344 8353, 2022.
References
Alfarra, M., Itani, H., Pardo, A., shyma yaser alhuwaider, Ramazanova, M., Perez, J. C., zhipeng cai, Müller, M., and Ghanem, B. Evaluation of test-time adapta- tion under computational time constraints. In Forty- first International Conference on Machine Learning, 2024. URL https:tt////. http://ppprreeeeeeeeeeeeeeeeeeee wttppeerweeeeeeeeeeeeeeeeeeeeeee 1d=6FtAXU4ean.
Chen, D., Wang, D., Darrell, T., and Ebrahimi, S. Con- trastive test-time adaptation.
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 295-305, 2022.
Deng. J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Arani, E., Sarfraz, F., and Zonooz, B. Learning fast, learning slow: A general continual learning method based on com- plementary learning system. In International Conference on Learning Representations, 2022.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D.,, Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
Araslanov, N. and Roth, S. Self-supervised augmentation consistency for adapting semantic segmentation. In Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 173384-53940 2021.
Gao, J., Zhang, J., Liu, X., Darrell, T., Shelhamer, E., and Wang, D. Back to the source: Diffusion-Ariven test-time adaptation. arXiv preprint arXiv:2207.03442, 2022.
Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gut- freund, D., Tenenbaum, J., and Katz, Objectnet: A large-scale bias-controlled dataset for pushing the lim- its of object recognition models. Advances in neural information processing systems, 32, 2019.
Gao, J., Yao, X., and Xu, C. Fast-slow test-time adaptation for online vision-an and languaee navigation. In Forty-first International Conference on Machine Learning, 2024.
Girshick, R. Fast r-cnn. In Proceedings of the IEEE inter- national conference on computer vision, pp. 2015.
Bartler, A., Bühler, A., Wiewel, F., Döbler, M., and Yang, B. M13: Meta test-time training for self-supervised test-

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Hakim, G. A. V., Osowiechi, D., Noori, M., Cheragha- likhani, M., Bahri, A., Ben Ayed, I., and Desrosiers, C. Clust3: Information invariant test-time training. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6136-6145, 2023.
R. L.., Gao, I., al. Wilds: A benchmark of in-the- wild distribution shifts. In International conference on machine learning, pp. 5637-5664. PMLR, 2021.
Lee, J., Jung. D., Lee, S., Park, J., Shin, J., Hwang, U., and Yoon, S. Entropy is not enough for test-time adap- tation: From the perspective of disentangled factors. In The Twelfth International Conference on Learning Repre- sentations, 2024.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
Li, B., Wang. Y., Che, T., Zhang, S., Zhao, S., Xu, P., Zhou, W., Bengio, Y., and Keutzer, K. Rethinking distribu- tional matching based domain adaptation. arXiv preprint arXiv:2006. /...6..3.33 13352, 2020.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturba- tions. In International Conference on Learning Represen- tations, 2019. URL https: brtps://openreview..co forum? d=HJz CqVm.
Li, X., Hu, Z., Liu, J., Ge, Y., Dai, Y., and Duan, L.-Y. Modeling uncertain feature representation for domain generalization. 2023.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of dittr--iisttiiution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021.
Liang, J., He, R., and Tan, T. A comprehensive survey on test-time adaptation under distribution shifts. Interna- tional Journal of Computer Vision, pp. 1-34, 2024.
Lim, H., Kim, B., Choo, J., and Choi, S. Tin: A domain- shift aware batch normalization in test-time adaptation. In The Eleventh International Conference on Learning Represenaatinss, 2022.
Hong, J., Lyu, I.., Zhou, J., and Spranger, M. Mecta: Memory-economee continual test-time model adaptation. In 2023 International Conference on Learning Represen- tutions, 2023.
Lin, W., Mirza, M. J., Kozinski, M., Possegger, H., Kuehne, H., and Bischof, H. Video test-time adaptation for action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22952- 22961, 2023.
Hu, X., Uzunbas, G., Chen, S., Wang, R., Shah, A., Neva- tia, R., and Lim, S.-N. Mixnorm: Test-time adaptation through online normalization estimation. arXiv preprint 21 2021.
Liu, W., Shen, X., Hi, Liu, B., Pun, C.-M., and Cun, X. Depth-aware test-time training for zero- shot video object segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 09218-19277, 2024.
Hu, Z., Li, X., Tang, S., Liu, J., Hu, Y., and Duan, L.-Y. Lead: Exploring logit space evolution for model selec- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28664- 28673, 2024.
Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., and Tan, M. Efficient test-time model adaptation with- out forgetting. International conference on machine learning, pp. 16908. PMLR, 2022.
Hu, Z., Hu, Y., and Duan, L.-Y. Seva: Leveraging single- step ensemble of vicinal augmentations for test-time adap- tation. arXiv preprint arXiv:2505.0m 04087, 2025.
Iwasawa, Y. and Matsuo, Y. Test-time classifier adjustment module for agnel-agnostic domain generalization. Ad- vances in Neural Information Processing Systems, 34: 2427 2240, 2021.
Niu, S., Wu, J., Zhang, Y., Wen, Z., Chen, Y., Zhao, P., and Tan, M. Towards stable test-time adaptation in dynamic wild world. In Internetional Conference on Learning Representations, 2023.
Karmanov, A., Guan, D., Lu, S., Saddik, A., and Xing, E. Efficient test-time adaptation of vasion-language models. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 14162-14171, 2024.
Prabhu, V., Khare, S., Kartik, D.., and Hoffman, J. Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips,
Press, O., Schneider, S., Kümmerer, M., and Bethge, M. Rdumb: A simple approach that questions our progress
10

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
in continual test-time adaptation. Advances in Neural Information Processing Systems, 36, 2024.
Xie, Q.. Dai, Z., Hovy, E., Luong, T., and Le, Q. Un- supervised data augmentation for consistency training. Advances in neural information processing systems, 33: 6256 6268, 2020.
Sajjadi, M., Javanmardi, M., and Tasdizen, T. Regulariza- tion with stochastic transformations and perturbations for deep cemi-supervised learning. Advances in neural information processing systems, 29, 2016.
Yu, Y., Wang, Y., Xia, S., Yang. W., Lu, S., Tan, Y.P.., and Kot, A. Purify unlearnalee examples via rate-constrained variational autoencoders. In International Conference PMLR, 2024a.
Schneider, S., Rusak, E., Eck, L., Bringmann, O., Bren- del, W., and Bethge, M. Improving robustness against common corruptions by covariate shift adaptation. Ad- vances in neural information processing systems, 33: 11539-11551, 2020.
Machine Learning, pp.
Yu, Y., Wang, Y., Yang, W., Guo, L., Lu, S., Duan, L.- Y., Tan, Y.-P., and Kot, A. C. Robust and transferable backdoor attacks against deep image compression with selective frequency prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024b.
Shin, I., Tsai, Y.-H., Zhuang, B., Schulter, S., Liu, B., Garg, S., Kweon, I. S., and Yoon, K.-J. Mm-tta: multi-modal test-time adaptation for 3d semantic segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 16928-16937, 2022.
Zhang, M., Levine, S., and Finn, C. Memo: Test time
robustness via adaptation and augmentation. Advances in neural information processing systems, 35:38629-38666 2022.
Sun, Y., Wang, X., Liu, Z.., Miller, J., Efros, A., and Hardt, M. Test-time training with supersiun for generaliza- tion under distribution shifts. In International conference on machine learning, pp.
Zhao, B., Chen, C., and Xia, S.-T. DELTA: DEGRADATION-FRES FULLY TEST-TIME ADAP- TATION. In The Eleventh International Conference on Learning Representations, 2023.
PMLR, 2020.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
Zhou, A. and Levine, S. Bayesian adaptation for covariate shift. Advances in neural information processing systems, 34:914-927, 2021.
Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimiza- tion. In International Conference on Learning Represen- tations, 2020.
Zhou, D.,, Bousquet, O., Lal, T., Weston, J., and Schölkopf, Learning with local and global consistency. Advances in neural information processing systems, 16, 2003.
Wang, Q. Fink, O., Van Gool, L., and Dai, D. Continual test- time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7201-7211. 2022.
Wang, Y., Cheraghian, A., Hayder, Z., Hong, J., Ramas- inghe, S., Rahman, S., Ahmedt-Aristizabal, D., Li, X., Petersson, L., and Harandi, M. Backpropagation- free net- work for 3d test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23221- 2024.
Wei, C., Shen, K., Chen, Y., and Ma, T. Theoretical analysis of self-training with deep networks on unlabeled data. arXiv preprint arXiv:2010.03622. 2020.
Wightman, R. Pytorch image models. https::/gitti:i com/ rww www.ightaan////mm hatpaaanyyytrcchh-hhe http://ttrciimmemmmmmmm. image-models
2019.
Wu, Y. and He, K. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Appendix
The structure of Appendix is as follows:
Appendix contains missing proofs in the main manuscript.
Appendix presents additional experimental results on supplementary datasets.
Appendix C provides further ablation studies and visualizations.
Appendix D details the datasets and the methods used for comparison.
Appendix E expands on related work in relevant fields.
A. Theoretical Proof
Below, we will provide detailed proofs of the theoretical results presented the methodology Sec. 4.2.
Notation. First, we recall the notation that we used in the main paper as well this appendix: C' denotes the number of classes. Fy denotes the model and & denotes the model parameters. denotes testing sample and Rd denotes its corresponding feature. E denotes the operation of expectation. (N,, Σ) denotes the Gaussian distribution with mean of and covariance of Σ. The subscript (.), denotes the i-th dimension, corresponding to the i-th class. A E Cxxd and b E R denote the linear and bias coefficients of the affine layer in the classifier, with a; and b; being their i-th dimensions, c
respectively. (=); (softmax (As b)); 2 eta denotes the predicted probability of sample æ belonging
i-th class. (xne(x)= (2) log (o --Ei (=(;;; po)); denotes the prediction entropy of the sample æ and its corresponding feature
A.1. Two Lemma Inequalities
Subsequently, we provide the proof of two important inequalities that we need derive the conclusion of Proposition 4.3 & 4.4 in the main paper.
Lemma A.1. (Bias Term under Finite Sampling) Given N features .1,-..,22 drawn from the local region and their corresponding probabilities: pe Pe(zN)) The bias term on these features satisfies the following inequality:
Σ Ek-noo po(Z));
(poi(p((z))) &
log (z)),
(10)
N
Proof. We begin by examining the difference between the left-hand side (LHS) and the right-hand side (RHS) of the inequality. By merging identical logarithmic terms, we can reformulate the expression into multiple summations, which we then simplify using the commutative property of summation:
RHS LHS
po(=))s N
Σ po(zz))) logpo(#));
(11)
(po(2.); (=*)s)s) log)o(=:))-
Since C dimensions of the probability in Eq. are independent of each other, we can treat each dimension separately, Therefore, suffices to prove the following inequality for each dimension:
N
N
1
po (2s), - Po (zx), log po (zi), > 0.
(12)
N
1=1
Km
12

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Applying Fubini's theorem allows to interchange the order of summation in Eq. 12. We also interchange the indices and I to obtain the following identity:
N
N
N =
po (zi), pe (zx);) logpo (zi)g
(po (zi)g - pe (zx);) log pe (zi)s
(13)
N
(po (Zk)) po (zi), log po (zx);-
We notice that the results in the first and third lines in Eq. 13 differ slightly, with the only variation being in the logarithm of the probability. Therefore, replace the original expression with the average of these two terms and combine the common terms into the form of a product of two differences:
pp (ze); - pe (zk); log pe (zt)
N
N (po (zx); - -po (zi),) (og) (zx)z)
po (zi), -po (zR);) logoo (zi)g +
(14)
R=1
po (Zi), pe (z+), (log pa (zi); log po (=x)
2N
Since po (zi), - po ( and log po (zi) log (zx); have the same sign, the product of these two terms is non-negative:
(po (zi), - po (z*),) (log po (zi), - log (zx),) >
(15)
Combining Eq. 14 and Eq. 15, we conclude that the inequality in Eq. 12 holds. Summing over all C' dimensions yields the desired result:
N
Ek=1 (o(=))s
(eme(p)(2))) <
Σ log (zz),
(16)
N
Lemma A.2. (Upper Bound of Negative Log-Likelihood) Given a feature 2 and its local region 12 which follows a Gaussian distribution N(p, 2). The expected value of the logarithm of the predicted probability for the i-th class satisfies the following inequality:
- -E log (o)) S log Σ
aa)t bb)+ (aloo-a,)De (a)
(17)
J=l
where the superscript (T) denotes the transpose operation.
Proof. First, we transform the left-hand side (LHS) of the inequality to eliminate the fraction, which complicates scaling. rewrite it as follows:
LHS E2)N((,))) log (b)
(18)
1=1
Since the logarithm function is concave (i.e., log(x)) 3 <0), we can apply Jensen's inequality and the additivity of expectations derive the following result:
C
LHS log E Ee,N(H.))
Σ
clay -ay +()b- log EenNia aa) (b,
Σ
(19)
1=1
1=1
13

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Through leveraging the moment property of the Gaussian distribution Ex Ex~V(u.o3)er eH+11/20 and noting that a N (a: μ + bi, ). we can directly compute the expectation in Eq. 19:
log
E N(n aa) 24(s ha)) log Σ
(a (b)
-a;)"
(20)
j=1
Combining Eq. 19 and Eq.20, we obtain the inequality that needs to be proved:
eapztbb
Σ ( ela--ai)att0 (600016---))))) (asl,alaa)ae
-E log
S log
(21)
2j=1 edg
A.2. Closed-form Upper Bound
Finally, we utilize the two inequalities derived above to obtain the crucial results in this paper, Regional Entropy, which provides closed-form upper bound for the expectation of the entropy loss over the local distribution, and Regional Instability, which offers a closed-form upper bound for the expectation of the KL divergence between the prediction probability distribution over the distribution and the original prediction at its center.
Proposition A.3. (Efficient Metric of Bias Term) Given a feature z and its local region St which follows a Gaussian distribution (M, E). The expectation of entropy loss over the entire distribution has a closed-form upper bound:
En[Cent] BE-N(( Σ po (2); log po
(22)
- Ea
log
& LRE.
Ekel ean 2ax E az
where the upper bound LR is called Regional Entropy.
Proof. First, by the definition of expectation, we can estimate the expectation of entropy using infinite number of sampling 21, .2..... _ iidd N(2, E):
E [[ent
lim N No+
po (ãa), log po (ži)g-
(23)
i=1 j=1
Using Lemma A.1, we can bound the values of po (2:), i = 1,2, 1.,....,NN . in Eq. 23, by their mean. This gives us the following inequality:
N
C Σn k= (o(z))s
E[ & lim No
log pa (ž4),
(24)
N
N
Next, we use the expectation to approximate polsx); as N approaches infinity. To ensure the integrability, we first take the expectation and then apply the softmax operation. This can be derived from Eq. 20 as follows:
E
-. b ta,
Pa (2);
(25)
EE~N(E,>) Ein edg
e +
Dj=1
Through the definition of the limit, have that for any < 0, there exists a positive integer No such that for any N > No. the following inequality holds:
Σn po(zk)s
po((2) Se.
(26)
N
14

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Combining Eq. 24 and Eq. 26, and substituting the specifie value of po(2)) from Eq. 25, have:
N
[lCent] <
lim N
po(z); + logpo (Ei),
1=
(27)
e + j EoJ
lim N-+00
t log po (Ei)
k= c bu for
j=1
Through the discrete form of Fubini's theorem, we can exchange the order of summation and extract terms that are independent of the limit calculation:
ebg-sthy+ Eo,
lim N-++00
+e
log po (Ez),
kmm eau.stbut
Dan
(28)
e e b Eay
1
+
lim
log (2))
Lk=1 c+ 2 Lay
N-0400
N
j=1
1m1
Through the definition the expectation and Lemma. A.2, we have:
log pa (z)) -E,,)(E,S) log)o(E))
(29)
log
Combining Eq. 27, Eq. 28 and Eq. 29, we have:
Ea arztb f Eas
[[Cend]
+ e) log Σ (a-aa))2 1=1
k= e a by jan 2.
(30)
0 (b)+bb))+ (aaa
for Ve 0.
Taking e - 0 in Eq. 30, we obtain the inequality we need to prove in Eq, 22.
Proposition A.4. (Efficient Metric of Variance Term) Given a feature z and its local region 12 which follows a Gaussian distribution (u, E). The expectation of Kullback-Leible divergence between the output probability over this distribution and the probability at its center has a upper bound:
C
C
t
KL (LL
(o)))
<
log
eala--ou a,) E(a=-as) & CRI, (31)
eat
ea
3=1
1k=1
1=
where the upper bound Lii called Regional Instability.
Proof. We first transform the left-hand side of the inequality into the following form:
Po(2), log pa (2); pe (3);
LHS = E
Σ
i=
(32)
--E po(2);log)
Po (o());
Σ1=1 e4,-
Since p=(2); independent of the expectation operation, by applying Lemma A.2, we have:
LHS < Σ (z()); log po(2),
Σ
lay -0))-++)++00. b.) (a a; D,) a0;)
(33)
t=1
F=1
15

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Substituting the definition of po(2), we have:
po(z);
1=1
gor
((0--a))+((-bbb))++
(34)
0
ea
k=
edgzzbb
e
kk=
Combining Eq. 23 and Eq. 24, we obtain the inequality we need to prove in Eq. 31.
B. Further Experiments
In this section, we broaden the scope of our investigation to evaluate the performance of our method across variety of complex and diverse scenarios. To this end, we conduct experiments on Wild TTA scenarios using two challenging datasets: ImageNet-R (Hendrycks et al., 2021) and VisDA-2021 (Bashkirova et al., 2022). Both datasets present an array of distribution shifts and variations in data styles that extend beyond the typical corruptions found in ImageNet-C (Hendrycks Dietterich, 2019), thereby providing a more comprehensive evaluation framework. By applying our method these datasets, we examine its robustness under mixed testing domain scenarios, incorporating the cases with label shifts batch size restricted
B.1. Wild Scenes on ImageNet-R and VisDA-2021
We conduct additional experiments on WTTA scenarios using the ImageNet-R (Hendrycks al., 2021) and VisDA-2021 (Bashkirova al., 2022) datasets with ResNet and ViT architectures. These datasets are characterized by diverse distribution
Limited Batch Size =
Imbalanced Label Shift
Method
Average
ResNet
VitBase
Avg
ResNet
VitBase
Avg
No-Adapt Model
40.8
43.1
42.0
40.8
43.1
42.0
42.0
Tent (Wang et al., 2020)
43.2
43.8
43.5
42.4
46.8
44.6
44.1
EATA (Niu et al., 2022)
44.1
52.5
48.3
42.1
50.5
46.3
47.3
SAR (Niu et al., 2023)
46.7
55.5
51.1
44.3
54.4
49.4
50.2
DeYO (Lee et al., 2024)
48.1
59.2
53.7
46.7
58.5
52.6
53.1
ReCAP (Ours)
51.5
61.1
56.3
49.640.2
60.4
55.0
55.7
Table Comparisons with state-of-th th-the-art methods ImageNet-R under Limited Batch Size 1 & Imbalanced Label Shift regarding Accuracy (%). We report mean&sstd over independent runs. The best results are in hold.
Limited Batch Size =
Imbalanced Label Shift
Model+MMthod
Average
ResNet
VitBase
Avg
ResNet
VitBase
Avg
No-Adapt Model
43.5
44.3
43.9
43.5
44.3
43.9
43.9
Tent (Wang et al., 2020)
43.9
50.6
47.3
43.7
50.1
46.9
47.1
EATA (Niu et al., 2022)
44.2
49.5
46.9
43.5
51.6
47.6
47.2
SAR (Niu et al., 2023)
45.2
52.8
49.0
44.7
53.9
49.3
49.2
DeYO (Lee et al., 2024)
45.8
58.5
52.2
45.2
57.1
51.2
51.7
ReCAP (Ours)
48.0
59.2
53.610.6
47.7
58.5
53.1
53.4
Table Comparisons with state-of-the-ar art methods VisDA-2021 under Limited Batch Size Imbalanced Label Shift regarding Accuracy (%). We report msd over independent runs. The best results are in bold.
16

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
shifts, including variations in data styles that extend beyond mere corruption. Consequently, for these two datasets, consistently consider the mixed testing domain scenarios and incorporate cases with label shifts or batch size 1. This rigorous testing environment ensures comprehensive assessment of model robustness under real-world conditions. All evaluations are performed using the same implementation details as outlined in the main paper.
Tab. presents the results on ImageNet-R for batch size = I and imbalanced label distribution shift scenarios. Consistent with the findings the main paper for ImageNet-C, ReCAP demonstrates superior performance across various scenarios and architectures on the ImageNet-R benchmark. We also compare our ReCAP method with previous sate--of-theaane approaches on the VisDA-2021 dataset. The results in Tab. align with those observed on ImageNet-C and ImageNet-R, where ReCAP similarly exhibits the best performance across all Wild settings VisDA-2021.
We further investigate the performance of our method under different distribution shifts. As discussed in the main paper, our ReCAP approach provides an efficient proxy to optimize region confidence, effectively reducing inconsistent predictions and enhancing global optimization efficiency. Compared to other sample selection WTTA methods, ReCAP consistently improves performance across various architectures and scenarios, achieving average performance gains of +2.6% and +1.7% ImageNet-R and VisDA-2021, respectively. The experimental results further validate the generalizability of our method across different types of shifts, providing more comprehensive understanding of its effectiveness.
C. Additional Ablation Study and Visualization
In Section 6 of the main paper, we provided a comprehensive validation of the hyperparameter robustness of À and т, along with visualizations that illustrate the effects of ReCAP on class separability and local consistency during the adaptation process. In this section, we extend our analysis by further investigating the sensitivity of key parameters and the evolution of the model adaptation, providing additional insights into the effectiveness and robustness of our method.
C.1. Sensitivity of TRE in ReCAP
The hyperparameter TRE plays a crucial role in determining the sample selection criterion within the ReCAP framework. To understand its impact on performance, leads to the inclusion of more samples in the training process, which results in improved performance. The performance peaks 0.8/1.0 In(C) for ResNet/ViT, respectively, indicating optimal balance between sample inclusion and computational efficiency. However, when THE exceeds this optimal range, the sample selection mechanism becomes too permissive, allowing for the inclusion of noisy or detrimental samples, which ultimately degrades performance. Despite this,
evaluate ReCAP under varying TRE values. As shown Fig. 7, increasing TRE
45
Prior SOTA
63
Accuracy990
61.
60
39
ReCAP(RasNect)
ReCAP(VIT)
(100))))))))
(A
Figure 7. Performance under different selection boundary TRE for ResNet and ViT on ImageNet-C under label shifts.
Table Effects of components in ReCAP. For fair comparison, '+ Vanilla Entropy' entropy-based selection and weighting.
Component
Corruption Category
Average
Vanilla Entropy
Cre in Eq. 7
Cei in Eq.
Noise
Blur
Weather
Digital
24.9
19.6
41.5
37.8
31.4
41.9
29.9
56.4
44.5
43.3
41.3
26.8
55.0
47.3
42.7
42,9
31.0
57.1
$1.2
45.8
17

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Iter=0
Iter=400
Iter=800
Iter=1200
1ter=1563
DeYO
ReCAP
Figure& 8. The evolution of feature space under DeYO and ReCAP methods. The visualizations are conducted ImageNet-C under labl shift scenario with ResNet50.
ReCAP maintains a consistent performance advantage over prior state-of-the-ae methods across a wide range of THE values, showcasing its robustness to variations in the sample selection boundary.
C.2. Effectiveness of Components in ReCAP
We investigate the impact of individual components within the ReCAP framework by comparing the full ReCAP method with variations that omit key parts of the approach. Specifically, we compare ReCAP to a vanilla entropy minimization strategy and systematically add back components to assess their contribution to performance. The results, as shown Tab, reveal that the region confidence achieves its best effect only when both components gains of +2.5% and +3.1% 1%, respectively. This demonstrates the complementary nature of these components in enhancing adaptation performance. Overall, the full ReCAP method consistently delivers the best performance, further validating the compatible effects of its key components in improving adaptation across various scenarios.
included, with performance
C.3. Evolution Process of Model Adaptation
To further validate the effectiveness of ReCAP in improving adaptation efficiency, we visualize the evolution of the model's feature space using t-SNE (Van der Maaten & Hinton, 2008). Fig. 8 illustrates the adaptation process for both ReCAP and the latest SOTA method, YO. Notably, ReCAP demonstrates superior adaptation efficiency by achieving better class separability throughout the adaptation process. At Iteration 800, ReCAP exhibits distinct, well-separated class clusters, even outperforming DeYO's final state (at Iteration 1563) in terms of class boundaries. This early emergence of clear class separability highlights the efficiency of our method in accelerating the adaptation process, ensuring that ReCAP achieves more structured and organized feature space compared to De These visualizations not only reinforce the advantages of ReCAP in enhancing adaptation efficiency but also provide strong evidence of its effectiveness in real-world scenarios where quick and robust adaptation is critical.
D. More Implementation Details
D.1. Baseline Methods
We compare ReCAP with the following SOTA methods: MEMO (Zhang al., 2022) enhances prediction consistency by leveraging multiple augmented copies of input samples, ensuring stable model outputs despite data variations. Tent (Wang et al., 2020) reduces the entropy of test samples to guide model updates, driving the model to make more confident predictions. to minimize entropy specifically for the selected samples. SAR (Niu et al., 2023) introduces sharpness awareness with entropy-based selection into the entropy minimization process, ensuring more stable adaptation in challenging wild scenarios. DeYO (Lee et al., 2024) prioritizes samples with dominant shape information and applies a dual selection criterion to identify more reliable samples for adaptation.
(Niu et al., 2022) combines sample selection based entropy with weighted adjustments
18

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
D.2. More Details on Dataset
In this paper, we primarily evaluate the out-of-distribution (OOD) generalization ability of all methods using a widely adopted benchmark: ImageNet-C (Hendrycks & Dietterich, 2019). ImageNet-C is derived by applying series of corruptions to the original ImageNet (Deng al., 2009) test set, making it large-scale benchmark for assessing model robustness under real-world distribution shifts. The dataset consists of 15 distinct types of corruptions, including Gaussian noise, shot noise, impulse noise, defocus blur, glass blur, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic transformation, pixelation, and JPEG compression. Each comuption type is further categorized into five severity levels, with higher severity indicating more extreme perturbations and greater distribution shifts. These corruptions simulate real-world degradations that can occur in diverse environmental conditions, making ImageNet-C an essential tool for evaluating the resilience of models in challenging, real-world scenarios. As illustrated in Fig. 9, these corruptions span broad spectrum, challenging the model to adapt varied distortions of input images.
Additionally, we conduct experiments on two other challenging benchmarks, ImageNet-R (Hendrycks et al.. 2021) and VisDA-2021 (Bashkinova et al., 2022), to further validate the robustness and adaptability our method across different types of distribution shifts. ImageNet-R consists of 30,000 images representing artistic renditions of 200 classes from ImageNet, with each image showcasing various creative transformations, such as paintings, drawings, and sculptures, sourced from platforms like Flickr and curated through Amazon MTurk annotators. These artistic variations introduce unique challenges in terms of visual style, texture, and color distribution, which notably different from the original ImageNet images. As shown in Fig. 10, these renditions demand the model to generalize beyond typical object recognition tasks and adapt to complex, pon-poocorcallistic representations.
Vi2D,- on the other hand, is a more diverse dataset that encompasses a broader range of domain shifts. It includes images from multiple sources such as ImageNet-O/Rff and ObjectNet (Barbu et al., 2019). The domain shifts in VisDA-2021 involve variety of challenges, such as changes in artistic visual styles, textures, viewpoints, and corruptions. This diversity in shifts ensures comprehensive evaluation of model performance under real-world conditions with large variations in object appearance and environmental factors.
Gaussian Noise
Shot Noise
Impulse Noise
Defocus Blur Frosted Glass Blur
Motion Blur
Zoom Blur
Snow
Frost
Fog
Brightness
Contrast
Elastic
Pixelate
JPEG
Figure Visualizations of different corruption types ImageNet corruption benchmark, which are taken from the original paper of ImageNet-C (Hendrycks & Dictterich, 2019).
19

---

Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation
Painting
Sculpture
Embroidery
Origami
Cartoon
Toy
Figure 10. Visualizations of different style shift types in ImageNet-R benchmark, which are taken from the original paper of ImageNet-R (Hendryeks al., 2021).
E. Related Work
E.1. Consistency Learning
Consistency learning is key paradigm in semi-supervised learning (Berthelot et al., 2019), domain adaptation (Li et al., 2020; Araslanov Roth, 2021), which enforces the model to produce stable and consistent predictions under different perturbations of input data. can broadly categorized into two main approaches. First, consistency can be used as an effective criterion for identifying reliable samples (Prabhu al., 2021; Yu al., 2024b). This approach is based on the understanding that consistency under image transformations serves as a dependable indicator of model errors (Wei al., 2020). For instance, methods like Ye (Lee et al., 2024) select samples by evaluating the variation in pseudo-label probabilities under different augmentations, using this as a selection indicator.
Second, consistency learning can act as a regularization technique by introducing data augmentation (Sajjadi et al., 2016). By requiring the model to maintain consistent predictions across different augmentation variants the same data, this approach enhances the model's 's robustness (Zhang et al., 2022; Xie al., 2020). This technique has been widely used in semi-supervised learning and unsupervised domain adaptation. Unlike traditional regularization methods that rely on introducing augmented variants of the original samples, the proposed efficient proxy of Region Confidence in this work enhances local consistency directly from the features of the original samples. This climinates the need for a lengthy process to obtain augmented variants, significantly improving the efficiency of optimizing consistency.
20


Reviews:
[{"review_id": "4bTG1jZL7M", "reviewer": "Reviewer_6Q1i", "summary": "This paper proposes a region modification based mechanism, called “Region Confidence Adaptive Proxy (ReCAP), to address the problem of will test-time adaptation (WTTA). Further, it develops a finite-to-infinite asymptotic approximation, which is a tractable upper bound to the intractable region confidence.\nExperimental results show improved performance of ReCAP compared to other approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for carefully reviewing our paper and offering a positive assessment. We appreciate your recognition of the contributions made by our work, particularly the idea of the tractable bound on the intractable region confidence and the theoretical results. \n>Q1: Some of the Empirical Gains are marginal. For example, VitBase in mixed testing domain.\n\nA1: Thank you for your feedback. In highly competitive TTA scenarios, performance gains tend to approach saturation in some cases. However, larger domain gaps, such as more severe corruption or mixed style shifts, still present significant challenges in terms of adaptation efficiency and robustness for TTA methods.\n\nTo access the empirical gains under more severe shifts, we increase the severity level from 4 & 5 to levels 6 & 7 (see Tab. 1), and our method achieves significant improvements of **+5.7** and **+5.2** over DeYO. Furthermore, under complex style shifts, ReCAP achieves average gains of **+2.6** on ImageNet-R and **+1.7** on VisDA (Appendix B.1). Overall, our method consistently outperforms prior methods across 3 datasets, 3 wild settings, and 2 base models, achieving gains of >+1.5 in the majority of scenarios.\n\nTable 1: Comparisons on ImageNet-C (severity level 6, 7) using VitBase under Mixed Testing Domain.\n|Method|Level 6|Level 7|\n|-|-|-|\n|Source|18.87|12.80|\n|TENT|2.45|0.99|\n|EATA|30.58|16.32|\n|SAR|32.11|17.74|\n|DeYO|29.54|16.08|\n|ReCAP|**35.27**|**21.32**|\n>Q2: Limited applicability of bs=1 setting. Can we not accumulate more examples to increase the batch size?  \n\nA2: We appreciate your comment. While bs=1 may seem contrived, some real-world applications (e.g., edge computing) face hardware constraints that necessitate the use of small mini-batches. Following your comment, we evaluate the effect of accumulating examples with varying sizes (See Tab. 2). However, small batch sizes still present a crucial bottleneck, hindering adaptation performance. This underscores the importance of developing robust TTA solutions tailored to such restrictive conditions.\n\nTable 2: Accuracy of Tent on ImageNet-C across different accumulated batch sizes, evaluated on ResNet50.\n||no-adapt|bs=1|bs=4|bs=16|bs=32|bs=64|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Tent (%)|30.6|21.5|23.5|25.9|28.6|33.9|\n>Q3: Line 235-236: It should be Eq. 5 in place of Eq. 10.\n\nA3: Thank you for pointing out this typo, and we will correct it.\n>Q4: Additional experiments in the CTTA setting will enhance the contribution of this paper.\n\nA4: Thank you for your constructive suggestion. Following your advice, we evaluate our method in CTTA scenarios for classification (Tab. 3 in this response) and semantic segmentation (Tab. 1 in response to Reviewer ynrW). While ReCAP is not designed for CTTA setup, it shows competitive performance and outperforms several strong baselines. These additional evaluations further highlight the broad applicability of our method across mild, continual, and wild settings.\n\nTable 3: Error rate (%) in CTTA scenario (CIFAR100C) [1], evaluated on ResNeXt-29.\n|Method|Gaus.|Shot|Impu.|Defo.|Glas.|Moti.|Zoom|Snow|Fros.|Fog|Brig.|Cont.|Elas.|Pixe.|Jpeg|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Source |73.0|68.0|39.4|29.3|54.1|30.8|28.8|39.5|45.8|50.3|29.5|55.1|37.2|74.7|41.2|46.4|\n|TENT|37.2|35.8|41.7|37.9|51.2|48.3|48.5|58.4|63.7|71.1|70.4|82.3|88.0|88.5|90.4|60.9|\n|CoTTA|40.1|37.7|39.7|26.9|38.0|27.9|26.4|32.8|31.8|40.3|24.7|26.9|32.5|28.3|33.5|32.5|\n|SAR|39.7|34.3|36.5|26.4|37.4|28.6|26.1|32.7|31.4|36.6|26.1|29.6| 33.0|29.8|38.1|32.4|\n|EcoTTA|39.1|35.7|37.5|26.2|37.7|28.3|26.3|32.2|31.0|36.9|25.9|27.4|32.7|28.4|34.7|32.0|\n|DeYO|39.0|34.1|36.3|26.7|37.2|28.4|26.2|32.4|31.6|36.2|25.5|26.8|32.2|30.1|38.3|32.1|\n|ReCAP|38.8|33.5|36.5|26.5|37.9|28.2|26.4|31.1|29.6|34.0|25.8|27.7|32.0|28.2|38.1|31.4|\n>Q5: How is the hyperparameter L_0 in equation 9 tuned for the experiments?\n\nA5: For hyperparameter L_0, we perform a grid search over a range of values on a small validation set, which comprises 10% of the Gaussian-type data from ImageNet-C. Additionally, we conduct a sensitivity analysis (see Appendix C.1) to confirm the robustness of our method to variations in L_0. Further details on this tuning process will be included in the revised version.\n>Q6: Is the accuracy in Figure 4 measured on the test set itself, on which the final performance is reported? Whether there is a validation split for tuning?\n\nA6: For the sensitivity analysis in Figure 4, we measure accuracy on the entire test set to validate the robustness of our method. For hyperparameter tuning, we use a small validation split (the same set used for L_0). The hyperparameters selected through this process are then validated and shown to be robust in Figure 4. We will provide a clearer explanation of this procedure in the revised version to avoid any confusion.\n>References  \n[1] Wang, Qin, et al. \"Continual test-time domain adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The theoretical claims in the main paper have been checked. The details of the proofs in Appendix have not been thoroughly verified.", "experimental_designs_or_analyses": "The experiments follow the existing WTTA line of work.", "relation_to_broader_scientific_literature": "The problem of WTTA is challenging. However, with the advent of more realistic continual test-time adaptation [1] approaches, the real world applicability of WTTA seems limited compared to recent progress.\n\nReferences:\n1. Wang, Qin, et al. \"Continual test-time domain adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths**\n* Proposed a tractable upper bound to the intractable region confidence.\n* The theoretical results are interesting.\n* Most of the experimental results show improvements.\n\n**Weaknesses**\n* Some of the empirical gains are marginal; for example, in Table 2, VitBase, DeYO -> ReCAP gain is =< 0.5.\n* Limited real-world applicability of WTTA, with batch size = 1 setting.", "other_comments_or_suggestions": "1. Line 235-236: It should be Eq. 5 in place of Eq. 10.\n2. The recent focus in the area of test-time adaptation has shifted towards continual test-time adaptation (CTTA), so experiments in the CTTA setting will enhance the contribution of this paper.", "questions_for_authors": "1. How is the hyperparameter L_0 in equation 9 tuned for the experiments?\n2. Is the accuracy in Figure 4 measured on the test set itself, on which the final performance is reported? Whether there is a validation split for tuning?\n3. In real-world applications, isn't the setting, such as batch size = 1, too contrived, though it is challenging? Can we not accumulate more examples before updating, effectively increasing the batch size?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pvxkb6cnwL", "reviewer": "Reviewer_7x3y", "summary": "This paper proposes a new method, ReCAP, a novel approach to addressing the main limitation of TTA in entropy minimization. The key idea of this work is that EM heavily relies on local consistency, and when this consistency is disrupted, model performance degrades. To resolve this issue, instead of optimizing the confidence of individual samples, ReCAP optimizes region-based confidence using bias and variance terms through Region Confidence Optimization. Furthermore, to enable low-cost computation and accuracy, the method employs approximation theories (e.g., Finite-to-Infinite Approximation). When applied to low-data settings (batch size = 1), the proposed method demonstrated a +3.5% improvement in performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed review and positive feedback on our contributions, including meaningful findings, novel region-based confidence optimization, and comprehensive evaluation. Building on your comments, we provide additional explanations and experiments to further demonstrate ReCAP's effectiveness and efficiency.\n>Q1: Generalization ability in more complex tasks.\n\nA1: Thank you for raising this important point. While our current experiments focus on classification, the core idea of region-based confidence optimization is inherently versatile. Additional evaluations on segmentation (Tab. 1 in response to Reviewer ynrW) and object detection (Tab. 1 in this response) show consistent improvements of ReCAP over entropy-based methods, indicating that ReCAP can be integrated into diverse model architectures and effectively extended to various complex tasks.\n\nTable 1: Comparisons of detection performance on KITTI-C benchmark in [1] with MonoFlex, regarding AP.\n|Method|Gauss.|Shot|Impul.|Defoc.|Glass|Motion|Snow|Frost|Fog|Brit.|Contr.|Pixel|Sat.|Avg|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Source|4.2|7.5|5.6|2.6|3.8|10.9|15.6|10.5|7.5|24.8|7.1|29.1|31.9|12.4|\n|TENT|16.0|25.1|23.8|21.7|11.6|27.1|26.9|26.9|30.5|35.8|33.7|41.1|35.2|27.3|\n|EATA|16.8|25.9|24.7|22.1|13.6|27.5|27.7|27.4|30.7|35.6|33.9|41.0|35.6|27.9|\n|DeYO|19.2|26.1|24.7|23.2|15.6|28.5|28.5|29.3|30.8|35.1|34.2|40.8|36.2|28.6|\n|MonoTTA (latest SOTA)|21.3|28.2|26.2|25.8|19.4|31.8|29.3|30.2|32.1|36.1|36.5|41.2|37.4|30.4|\n|ReCAP|21.3|29.3|26.3|26.7|20.1|31.1|32.2|32.6|31.7|36.7|36.1|41.3|37.5|31.0|\n>Q2: How does ReCAP handle highly dynamic shifts where local consistency may be entirely lost?\n\nA2: Thank you for your insightful question. Our finite-to-infinite approximation is derived without assuming any consistency condition, ensuring its applicability even when consistency is entirely lost. Based on this foundation, ReCAP employs region-confidence optimization to enhance local consistency, which is crucial for robust adaptation.\n\nMoreover, we evaluate ReCAP in a highly dynamic setting where the data stream undergoes rapid transitions across different domains, including style, corruption, and label shifts (see Appendix B.1). The results demonstrate that ReCAP exhibits strong robustness and achieves SOTA performance, validating its capability to address highly dynamic scenes.\n>Q3: Quantitative comparison of computational cost reduction.\n\nA: ReCAP reduces the computational cost via feature-level region modeling, eliminating the overhead of image-level region modeling and augmentation. Furthermore, its finite-to-infinite approximation serves as an efficient proxy, removing the need for costly sampling. As shown in Tab. 2, these designs achieve significant runtime reduction. We will incorporate this quantitative comparison into the revised version to enhance clarity on the efficiency of ReCAP.\n\nTable 2: Running time on 50,000 images.\n|Region Type|Time (s)|\n|-|:-:|\n|Image-level region (16 augmentation)|1798|\n|Feature-level region (w/o proxy, 16 sampling)|163|\n|Feature-level region (w/ proxy)|116|\n>Q4: Clarification on ablation study settings.\n\nA4: In our analysis of the effect of λ in Section 6.1, we fixed τ at 1.2, which aligns with the default value used across all experiments. We will explicitly state it in the revised version.\n>Q5: Variance + Bias vs. Mutual Information.\n\nA5: Our variance + bias function serves a fundamentally different role from mutual information (MI) in several key aspects:\n1. Different Objects: MI is defined between two random variables, measuring their shared information, whereas our variance + bias function is computed over a local region surrounding a single sample x, capturing localized prediction stability.\n2. Different Purposes: MI primarily quantifies mutual dependence, while our function serves as prediction confidence and consistency measure within a local feature region, making it more aligned with adaptation objectives.\n3. Different Optimization Effects: MI encourages statistical association but does not address prediction probability discrepancies. In contrast, our function directly optimizes both prediction uncertainty (bias) and local discrepancy (variance), enhancing robustness under domain shifts.\n\n>Q6: Sensitivity of RaCAP.\n\nA6: We have extensively evaluated ReCAP across diverse datasets (ImageNet-C, VisDA, ImageNet-R), tasks (classification, segmentation, detection), TTA settings (wild, mild, and continual), and hyperparameter configurations. Our results consistently show its robustness and reliability across these various conditions.\n\nAdditionally, we notice that the question might miss a word (e.g., choice of L_0 when adapting). If we have misunderstood your concern, please clarify, and we would be happy to provide further insights.\n>References  \n[1] Lin, Hongbin, et al. \"Monotta: Fully test-time adaptation for monocular object detection.\" European Conference on Computer Vision, 2024.", "claims_and_evidence": "The paper experimentally demonstrates that entropy minimization leads to performance degradation when local consistency is disrupted. The results show that even when entropy values are similar, prediction differences can be significant in domain-shift environments. Furthermore, the proposed RCO method improves the stability of TTA, and ReCAP outperforms traditional entropy-based methods such as Tent and MEMO, proving to be particularly effective in domain shift scenarios. The study also validates that the Bias Term and Variance Term play a crucial role in maintaining prediction consistency through mathematical formulations and empirical analysis. Additionally, the paper demonstrates that ReCAP achieves higher performance than Tent with only a 5% increase in computational cost.", "methods_and_evaluation_criteria": "The study employs widely used datasets in TTA research, including ImageNet-C, ImageNet-R, and VisDA-2021, to evaluate performance. Comparisons are made with state-of-the-art TTA techniques such as Tent, MEMO, DDA, EATA, SAR, and DeYO. The evaluation focuses on improvements in accuracy under domain shifts, robustness in low-data scenarios, and performance across mixed-domain tests. The inclusion of an ablation study analyzing key hyperparameters such as region size and bias-variance tradeoff strengthens the validity of the evaluation framework, aligning well with the study’s research objectives.", "theoretical_claims": "The paper theoretically supports its approach by introducing an optimization framework leveraging Bias and Variance Terms to balance confidence estimation and prediction consistency. Additionally, it proposes a Finite-to-Infinite Approximation method to reduce computational cost while effectively approximating regional confidence. The mathematical derivations appear valid, and the experimental results substantiate the proposed theoretical foundation.", "experimental_designs_or_analyses": "The experimental design appears relatively reliable, demonstrating that ReCAP maintains high performance even in low-data settings and remains stable across various corruption types and domain shift scenarios. The study also presents t-SNE visualizations, confirming that ReCAP enhances class separability. Overall, the experiments are appropriately designed to support the paper’s claims.", "relation_to_broader_scientific_literature": "This work builds upon prior entropy minimization-based TTA research, such as Tent, MEMO, and EATA, extending the optimization approach from sample-level to region-level confidence estimation. Additionally, it is relevant to domain adaptation research, distinguishing itself by focusing on maintaining local consistency as a key factor in adaptation performance.", "essential_references_not_discussed": "This paper leverages the appendix to cite all relevant studies comprehensively.", "other_strengths_and_weaknesses": "Strength\n- While entropy minimization has been used in TTA, its accuracy gains have been limited. This paper provides a meaningful finding by identifying its limitations and proposing an effective solution.\n- The paper effectively explains why local consistency is critical in TTA and thoroughly discusses the limitations of existing methods, making a strong case for the necessity of ReCAP.\n- Instead of optimizing confidence at the sample level, the paper introduces region-based confidence optimization, which is a more robust and reliable strategy for TTA.\n- The paper rigorously evaluates ReCAP across various datasets and settings, including different domain shifts, data scarcity scenarios, and mixed-domain testing. This strengthens the credibility of the proposed method and demonstrates its robustness in real-world applications.\n- Unlike computationally intensive methods like DDA, ReCAP maintains a lightweight adaptation process while still improving accuracy.\n- ReCAP not only outperforms baseline methods but also enhances other approaches such as SAR and DeYO, demonstrating its adaptability and versatility\n\nWeakness\n- While the method is effective for classification, it is unclear how well it would generalize to more complex tasks like object detection, segmentation, or NLP.\n- The paper assumes that the finite-to-infinite approximation holds consistently, but in scenarios where domain shifts occur rapidly, there is a possibility that this assumption might not always hold. Investigating its robustness in highly dynamic environments could provide further insights.\n- While the paper discusses applying features to reduce computational cost, providing a quantitative comparison of the actual reduction in computation would strengthen the analysis.", "other_comments_or_suggestions": "- It would be helpful if the paper clarified what value of 𝜏  was fixed when conducting experiments on the effect of 𝜆 in Section 6.1.\n- The t-SNE plots effectively illustrate the improvements in feature space adaptation, making it easier to understand the impact of ReCAP on prediction consistency and clustering quality.", "questions_for_authors": "- Does the variance + bias function serve the same role as the traditionally used mutual information?\n- How does ReCAP handle rapid domain shifts where local consistency may be entirely lost?\n- Could ReCAP be extended to structured prediction tasks like segmentation?\n- How sensitive is ReCAP to the choice of when adapting to new domains?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CZbIfatrpS", "reviewer": "Reviewer_eZDc", "summary": "This paper introduces a new Test-Time Adaptation Method (TTA) to combat domain shifts appearing at test time in extreme scenarios. In particular, it proposes ReCAP, a method that optimizes two terms: a bias term resembling a regional entropy around a given test data, and a variance term to enhance the consistency of the prediction of the model under neighboring features. Experiments are carried out on standard TTA benchmarks yielding consistent performance gain.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your positive comments and constructive suggestions on improving our paper. We will address your questions below.\n>Q1: I checked the ablation experiments and did not find the one ablating the impact of $\\mathcal{L}_0$.\n\nA1: Due to space constraints, we provide the ablation study on $\\mathcal{L}_0$ in Appendix C.1. As shown in Appendix Fig. 7, ReCAP consistently enhances performance across a broad range of $\\mathcal{L}_0$ values, demonstrating its robustness to different selection boundaries. This result confirms that ReCAP does not rely on precise tuning of $\\mathcal{L}_0$ and remains effective across varying settings. To improve accessibility, we will incorporate this ablation study into the main paper.\n>Q2: When ReCAP is combined with SAR, is the data selection mechanism of SAR employed or the proposed one?\n\nA2: When integrating ReCAP with SAR, we replace the original entropy selection with our proposed strategy, allowing for a direct evaluation of ReCAP's effectiveness. Likewise, when combining ReCAP with DeYO, we follow the same replacement strategy. We will clarify this in the revised version to eliminate any ambiguity.\n>Q3: Efficiency comparison with EATA and ETA.\n\nA3: Thank you for raising this point. While EATA performs fewer backward passes on test samples, it requires additional computation for Fisher regularization on extra source samples, resulting in a higher runtime compared to ReCAP.\n\nFor comparison with ETA, we provide additional evaluations in Tab. 1. Although ETA offers a slight runtime improvement, it struggles to adapt to dynamic shifts in wild TTA scenarios. In contrast, ReCAP effectively balances efficiency and performance, achieving superior accuracy with marginal additional computation cost.\n\nTable 1: Running time for 50,000 images and accuracy on ImageNet-C under label shifts using ResNet.\n|Method|Time (s)|Accuracy (%)|\n|-|:-:|:-:|\n|Tent|110|22.8|\n|ETA|112|26.2|\n|EATA|118|31.7|\n|ReCAP|116|47.2|\n>Q4: It is important, given the efficiency of ReCAP, to show the performance gain under computational budgeted evaluation.\n\nA4: Thank you for your valuable suggestion. We agree that this evaluation is essential and realistic for assessing TTA methods. As shown in Tab. 2, ReCAP benefits from the computational efficiency of the upper-bound proxy, resulting in minimal performance degradation while achieving more significant gains under strict time constraints. This demonstrates ReCAP's ability to provide efficient adaptation under time limitations, making it well-suited for real-world deployments with computational budgets.\n\nTable 2: Error rate on ImageNet-C under computational time constraints.\n|Method|Realistic|Gaus.|Shot|Impu.|Defo.|Glas.|Moti.|Zoom|Snow|Fros.|Fog|Brig.|Cont.|Elas.|Pixe.|Jpeg|Avg.|\n|-|:-:|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|EATA|✗|65.5|62.4|63.5|66.6|67.2|52.0|47.3|48.2|54.1|39.9|32.1|55.0|42.3|39.2|44.8|52.0|\n|EATA|✓|69.3|67.1|69.2|71.1|71.7|57.5|49.9|51.9|57.4|42.4|32.6|60.7|45.1|41.4|47.4|55.6(+3.6)|\n|SAR|✗|69.5|69.7|69.0|71.2|71.7|58.1|50.5|52.9|57.9|42.7|32.7|62.9|45.5|41.6|47.8|56.2|\n|SAR|✓|79.4|78.5|78.1|79.9|79.3|67.5|56.1|60.5|63.1|47.4|34.0|75.3|51.7|46.6|53.8|63.4(+7.2)|\n|DeYO|✗|64.1|61.4|62.1|66.0|66.2|51.7|47.4|47.5|54.0|39.8|31.9|54.0|41.9|38.7|44.3|51.4|\n|DeYO|✓|69.7|67.6|68.2|73.2|72.2|59.0|50.8|52.8|58.1|42.7|32.5|62.9|45.5|41.5|48.1|56.3(+4.9)|\n|Ours|✗|64.1|60.4|62.1|67.0|67.2|50.6|47.2|45.8|51.7|38.2|32.2|53.5|41.8|38.4|43.9|50.9|\n|Ours|✓|68.2|65.2|67.1|70.7|71.0|55.7|49.8|50.0|53.8|40.6|32.7|52.9|44.9|40.6|46.7|54.0(+3.1)|\n>Q5: One extra [Optional] experiment is to extend the evaluation to the Practical TTA setting which is closely related to the wild TTA setting.\n\nA5: Thank you for this insightful suggestion. We agree that the PTTA setting (Continual + Label Shifts) is closely related to the wild TTA setting. As shown in Tab. 3, despite ReCAP not incorporating any additional design specifically for continual adaptation, it still outperforms entropy-based methods and specific-design RoTTA in PTTA setup. This result further validates the effectiveness and robustness of ReCAP across diverse test-time conditions.\n\nTable 3: Accuracy on ImageNet-C under the PTTA setup, evaluated on ResNet50.\n|Method|Gaus.|Shot|Impu.|Defo.|Glas.|Moti.|Zoom|Snow|Fros.|Fog|Brig.|Cont.|Elas.|Pixe.|Jpeg|Avg.|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Source|17.9|19.9|17.9|19.7|11.3|21.3|24.9|40.4|47.4|33.6|69.2|36.3|18.7|28.4|52.2|30.6|\n|Tent|13.7|0.9|0.2|3.0|0.4|0.3|0.4|0.6|0.2|0.2|1.7|0.4|0.1|0.2|1.2|1.6|\n|SAR|32.0|14.0|17.7|16.7|12.6|1.1|16.5|44.5|42.4|11.1|7.7|46.6|8.6|0.6|38.6|20.7|\n|DeYO|40.7|44.1|41.3|17.7|22.1|41.3|16.5|41.2|50.5|30.9|73.2|51.4|42.4|56.5|58.2|41.9|\n|RoTTA|40.2|41.2|40.8|20.7|20.3|40.2|33.2|45.2|51.1|52.1|70.2|50.1|40.1|52.1|57.1|43.6|\n|ReCAP|42.2|44.3|42.1|18.9|23.9|42.1|28.7|44.7|51.6|52.5|71.2|52.2|41.5|57.9|58.3|44.8|", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I skimmed through the proofs in the Appendices and they seem correct", "experimental_designs_or_analyses": "I checked the experimental sections in the paper and they all seem relevant, consistent with earlier work, and providing supportive results.", "relation_to_broader_scientific_literature": "This paper does a good job in linking their main contributions to earlier works. They further show experimentally how they can combine their proposed method with previous state-of-art showing further performance gain.", "essential_references_not_discussed": "I think the paper did a good job relating itself to other related works.", "other_strengths_and_weaknesses": "While I am generally very positive about this paper, the following experiments I think are missing and would strengthen the paper.\n\n1) Ablating $\\mathcal L_0$: I checked the ablation experiments and did not find the one ablating the impact of $\\mathcal L_0$. Further, when the proposed method is combined with SAR, is the data point selection mechanism of SAR employed or the proposed one?\n\n2) In the efficiency comparison in Table 4: The proposed ReCAP computes more backward passes than EATA, however it is still more efficient in runtime. This seems a bit contradictory and deserves more discussion along with comparison against the more efficient variant of EATA (i.e. ETA). It is also important, given the efficiency of ReCAP, to show the performance gain under computational budgeted evaluation [A].  \n\n3) One extra [Optional] experiment is to extend the evaluation to the Practical TTA setting [B] which is closely related to the wild TTA setting.\n\n[A] Evaluation of test-time adaptation under computational time constraints, ICML 2024\n\n[B] Robust test-time adaptation in dynamic scenarios, CVPR 2023", "other_comments_or_suggestions": "Please refer to the \"Other Strengths and Weaknesses\" Section.", "questions_for_authors": "Please refer to the previous sections", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oPIlwY0VFQ", "reviewer": "Reviewer_ynrW", "summary": "This paper introduces ReCAP, a novel TTA method based on local inconsistency of predictions.\nBased on the finding that the local inconsistency increases and adaptation becomes difficult under wild distribution shifts, the region confidence is proposed as an alternative to entropy, a common objective in TTA.\nIts finite-sample approximation is also derived to overcome the computational intractability of the original region confidence.\nExperimental results show that ReCAP had higher accuracy on corrupted test data under wild settings (online, mixed shifts, and imbalanced labels).\n\n## update after rebuttal\nI appreciate the author's rebuttal and additional experiments. My concerns have been addressed. I have updated my score to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and providing valuable feedback. We would like to answer your questions below.\n>Q1: Experimenting on continual TTA settings performed in recent TTA studies (e.g., EATA) would strengthen the efficacy of ReCAP in wild TTA settings.\n\nA1: Thank you for your constructive suggestion to evaluate CTTA settings. We agree that such experiments would further strengthen the efficacy of our ReCAP. We conduct extensive experiments on CTTA for both classification and segmentation tasks. \n\nFor classification, CTTA setup (Tab. 3 in response to Reviewer 6Q1i) and additional PTTA (CTTA + label shift)  setup (Tab. 3 in response to Reviewer eZDc) demonstrate that ReCAP consistently outperforms prior methods. For segmentation, results in Tab. 1 (this response) further confirm that ReCAP maintains robust adaptation performance in continual scenarios. These results underscore the broad applicability of ReCAP across continual and wild TTA settings.\n\nTable 1: Semantic segmentation results (mIoU) on the Cityscapes-to-ACDC CTTA setup based on the Segformer-B5 architecture.\n|Condition|Fog|Night|Rain|Snow|Fog|Night|Rain|Snow|Fog|Night|Rain|Snow|Fog|Night|Rain|Snow|Avg|\n|-|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Source|69.1|40.3|59.7|57.8|69.1|40.3|59.7|57.8|69.1|40.3|59.7|57.8|69.1|40.3|59.7|57.8|56.7|\n|TENT|69|40.2|60.1|57.3|66.5|36.3|58.7|54|64.2|32.8|55.3|50.9|61.8|29.8|51.9|47.8|52.3|\n|EATA|69.1|40.5|59.8|58.1|69.3|41.8|60.1|58.6|68.8|42.5|59.4|57.9|67.9|42.8|57.7|56.3|57.0|\n|CoTTA|70.9|41.1|62.4|59.7|70.8|40.6|62.7|59.7|70.8|40.5|62.6|59.7|70.8|40.5|62.7|59.7|58.4|\n|SAR|62.2|37.7|55.5|53.0|64.6|39.3|56.8|53.9|65.7|39.0|58.1|55.0|66.1|38.0|59.1|55.3|53.7|\n|Ours|72.7|43.8|63.9|61.1|71.9|42.2|64.1|60.1|71.0|40.5|63.5|58.8|70.3|39.3|62.8|57.2|59.0|\n>Q2: Sampling Number $N$ Sensitivity. \n\nA2: We appreciate your question and apologize for any confusion. Our method does not need any sampling due to the finite-to-infinite approximation in Propositions 4.3 and 4.4. Therefore, our method is entirely unaffected by the value of $N$. We will provide additional clarifications in the revised version to enhance clarity.\n>Q3: Comparing ReCAP with a simple Monte Carlo approximation of the original region confidence in Eq. (2) would make the proposed method more convincing.\n\nA3: We sincerely appreciate your valuable suggestion. We conduct a comprehensive comparison with the Monte Carlo (MC) approximation using different sampling numbers. As shown in Tab. 2, while MC provides a direct estimate of region confidence,  its accuracy is highly sensitive to the number of samples, leading to increased variance and a computational cost that scales linearly with the sample size.\n\nIn contrast, ReCAP achieves significantly higher accuracy with lower variance during adaptation, demonstrating its superior stability and efficiency. These results further reinforce the motivation behind our finite-to-infinite approximation. We will incorporate this comparison into the revised version.\n\nTable 2: Comparison between the MC approximation and our finite-to-infinite approximation under 3 independent runs.\n| |ReCAP|MC (4)|MC (16)|MC (64)|MC (128)|\n|-|:---:|:---:|:---:|:---:|:---:|\n|Sampling number|NA|4|16|64|128|\n|Average Accuracy|42.2|31.7|34.8|38.7|40.7|\n|Standard deviation|0.1|9.8|3.7|1.4|0.3|\n|Running Time(s)|116|125|163|278|454|\n>Q4: Ablation on the sample weighting and selection in Eq. (9) would be helpful.\n\nA4: We appreciate the reviewer's suggestion and have conducted an additional ablation study to analyze the impact of different sample selection and weighting strategies. As shown in Tab. 3, the results lead to the following key observations:\n1. Our region-based confidence optimization consistently enhances performance, surpassing the previous SOTA even when combined with the simplest entropy-based selection and weighting strategies.\n2. The combination of our proposed selection and weighting achieves the best overall accuracy, further validating the effectiveness of our design.\n\nTable 3: Ablation study on selection and weighting strategies.\n|w/o selection|Entropy selection|Our selection|w/o weighting|Entropy weighting|Our weighting|ReCAP Accuracy|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|✓|||✓|||36.0|\n||✓||✓|||38.9|\n|||✓|✓|||43.2|\n||✓|||✓||44.9|\n|||✓||✓||45.7|\n|||✓|||✓|47.2|", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The motivation for using the proposed approximation of the region confidence in Eqs. (6) and (7) is unclear. One can use a simple Monte Carlo approximation. Providing some evidence that the proposed approximation is more efficient than Monte Carlo would be convincing.", "theoretical_claims": "I have checked the derivation of the region confidence.", "experimental_designs_or_analyses": "- Experimenting on continual TTA settings performed in recent TTA studies (e.g., EATA) would strengthen the efficacy of ReCAP in wild TTA settings.\n- How was the sampling number from the region $N$ set? Examining the sensitivity of $N$ would be helpful.\n- Comparing ReCAP with a simple Monte Carlo approximation of the original region confidence in Eq. (2) would make the proposed method more convincing.\n- Ablation on the sample weighting and selection in Eq. (9) would be helpful.", "relation_to_broader_scientific_literature": "The region confidence expands the commonly used sample-wise entropy.\nIt can improve existing entropy-based TTA methods.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Test-time adaptation aims to reduce generalization error by updating on shifted data. This work analyzes the "mild" (= single shift) and wild (= multiple shift, and small batches) settings for test-time adaptation and finds that local inconsistency is a challenge. The proposed method, ReCap (Region Confidence with Adaptive Proxy), identifies more local regions in which to optimize instead of simply minimizing entropy or some thresholding of entropy as in prior work. Experiments show improvement on the common benchmark of ImageNet-C, in comparable settings to prior work including with batch size one and with imbalanced classes, and with comparison to recent and strong methods. Furthmore ReCAP can be combined with base methods, like SAR and DeYO, and shows further improvement. This is a clear accept given the contributions and the reviewer consensus for acceptance (4/4 agree with scores of 4, 4, 4, 3). The authors provided a rebuttal, all reviewers acknowledged it, and eZDc + 7x3y + ynrW raise 3 to 4 and 6Q1i maintains 3, so the consensus is clear, and the area chair agrees with acceptance.
Decision: Accept (poster)