Adaptive Energy Alignment for Accelerating Test-Time Adaptation
Authors: Wonjeong Choi
Do-Yeon Kim
Jungwuk Park
Jungmoon Lee
Younghyun Park
Dong-Jun Han
Jaekyun Moon
Venue: ICLR 2025 Poster (2025)
Keywords: Test-Time Adaptation (TTA)
Out-of-domain (OOD)
Energy-based Model (EBM)
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose an adaptive energy alignment (AEA) that accelerates the model adaptation in TTA.

Abstract: In response to the increasing demand for tackling out-of-domain (OOD) scenarios, test-time adaptation (TTA) has garnered significant research attention in recent years. To adapt a source pre-trained model to target samples without getting access to their labels, existing approaches have typically employed entropy minimization (EM) loss as a primary objective function. In this paper, we propose an adaptive energy alignment (AEA) solution that achieves fast online TTA. We start from the re-interpretation of the EM loss by decomposing it into two energy-based terms with conflicting roles, showing that the EM loss can potentially hinder the assertive model adaptation. Our AEA addresses this challenge by strategically reducing the energy gap between the source and target domains during TTA, aiming to  effectively align the target domain with the source domains and thus to accelerate adaptation. We specifically propose two novel strategies, each contributing a necessary component for TTA: (i) aligning the energy level of each target sample with the energy zone of the source domain that the pre-trained model is already familiar with, and (ii) precisely guiding the direction of the energy alignment by matching the class-wise correlations between the source and target domains. Our approach demonstrates its effectiveness on various domain shift datasets including CIFAR10-C, CIFAR100-C, and TinyImageNet-C.

Published as a conference paper at ICLR 2025
ADAPTIVE ENERGY ALIGNMENT FOR ACCELERATING TEST TIME ADAPTATION
Wonjeong Choi', Do-Yeon Kim', Jungwuk Park', Jungmoon Lee', Younghyun Park?,
Dong-Jun Han", Jaekyun Moon
Korea Advanced Institute of Science and Technology (KAIST),
2 Agency for Defense Development (ADD), 3 Yonsei University
jsnjd55577 dy kim, savertm, undanseh) (eannsehekkiist...m acc.st.cc.u .o. dn foffff3699add..oo re. kr; "aeeerddhhhooo djheyonsei ac. kr; jmoon@kaist.eduu
ABSTRACT
In response the increasing demand for tackling out-of-domain (OOD) scenarios, test-time adaptation (TTA) has garnered significant research attention in recent years. To adapt a source pre-trained model to target samples without getting access to their labels, existing approaches have typically employed entropy minimization (EM) loss as a primary objective function. this paper, we propose an adaptive energy alignment (AEA) solution that achieves fast online TTA. We start from the re-interpretation of the EM loss by decomposing it into two energy-based terms with conflicting roles, showing that the EM loss can potentially hinder the assertive model adaptation. Our AEA addresses this challenge by strategically reducing the energy gap between the source and target domains during TTA, aiming effectively align the target domain with the source domains and thus to accelerate adaptation. We specifically propose
novel strategies, each contributing a necessary component for TTA: (i) aligning the energy level of each target sample with the energy zone of the source domain that the pre-trained model is already familiar with, and (ii) precisely guiding the direction of the energy alignment by matching the class wise correlations between the source and target domains. Our approach demonstrates its effectiveness on various domain shift datasets including CIFARIO-C, CIFARI00-C, and TinylmageNet-C.
INTRODUCTION
Despite the huge success of deep neural networks (DNNs) in various fields (Mikolov et al. 2013
Krizhevsky et al. 2017). the randomness and dynamic nature of test samples encountered during inference still pose limitations to the application of DNNs. Particularly, the performance of DNNs significantly degrades in out-of-domain (OOD) scenarios (Hendrycks & Gimpel| 2017), where domain distributions of train data and test data different. This remains substantial challenge from the perspectives of robustness and practicality of DNNs.
As promising direction to address the OOD problem, test-time adaptation (TTA) has been actively studied in recent years. TTA is a paradigm that involves adapting pre-trained model (pre-trained on source domains) using unlabeled OOD samples (from the target domain) during test-time. Source domains are typically not available during TTA due to memory and privacy issues (Wang et al. 2023 Liang et al. 2023). This assumption of the source-free property has played a pivotal role in the growing interest in TTA.
The main focus of TTA to adapt the pre-trained model on the given target samples to improve the performance at test-time. To tackle real-world scenarios where target samples arrive sequentially in batch-wise manner, online TTA settings have become increasingly important recent years (Wang et al. 2023). For instance, in autonomous systems (e.g., self-driving vehicles) with dynamically evolving environments (c.g., weather, lighting), the model needs to continuously make predictions on OOD samples in online manner. In such scenarios, the key is to sequentially adapt the model on streaming target samples to accumulate knowledge of all past data during test-time, so that the model make more reliable predictions for future target samples. However, while many TTA methods

---

Published as a conference paper at ICLR 2025
Source Garget
clss Class
target
Classiff cnssiiication (%6
4.54
class
Energy
Endrgy gap
-16
-10
-co-o 200 2040
-to 204060
-20
Caurioatin CTD
-60 -40
20
60 40
20
Batch Index
(a) No adaptation
(b) AEA (ours)
(c) Cls. error & Energy gap
Figure I: (a-b) Sample-wise logit (z, y-axes) and energy (z-axis) distribution in the early stage of adaptation batches. (a) Without adaptation, target samples (CIFARIO-C) are mixed up the high-energy region, while source samples (CIFARIO) have low energies and are clearly separated. (b) Our AEA successfully reduces the energy gap A by accurately guiding the direction of energy alignment. As result, (c) our approach can accelerate test-time model adaptation and achieve remarkable performance even in few adaptation batches.
have been proposed, they still exhibit limited performance in online adaptation settings. Particularly, they often require large number of adaptation batches for the model to achieve satisfactory performance, leading to suboptimal early-stage performance (e.g., on the first few batches). Our goal to thoroughly investigate the rationale behind this phenomenon and address this issue.
Motivation. To overcome the aforementioned challenges, we propose a new TTA method by leveraging the idea of mnergy-based models (EBMs) (LeCun et al. (006),, which have proven to be effective in addressing distribution shift problems (Liu et al.) 2020; Xie et al. 2022) Xiao et al. 2023) Herath et al. 2023). Their findings suggest that energy scores can serve expressive indicators for detecting OOD samples. We start with Fig. Tal where we visualize sample- wise logits and energy distributions of source/target samples in our TTA scenario. From Fig. Ia we see that a source pre-trained model yields higher energies for the domain-shifted target samples and lower energies for the source samples, resulting in a notable energy gap. Our objective is to reduce this energy gap, aiming to alleviate domain disparity by aligning the feature representation of the target domain with that of the source domain and achieve performance improvements.
However, one key aspect that needs to be considered during the energy gap reduction is the energy alignment direction: order to mitigate the domain discrepancy in a class-wise manner (in Fig. Ta each target sample should be aligned in distinct direction based on its class information. Addressing these energy gap and alignment issues is crucial in many TTA scenarios due to the significant domain disparity between the source and target datasets, particularly in the early stage of batch arrival. Although Yuan et al. (2044) recently introduced an energy-bast based TTA method, it does not consider the directional aspects of energy alignment, which limits its adaptation performance. To the best of our knowledge, these aspects have been largely overlooked in existing TTA research and remain unresolved. We thus aim fill this gap by answering the following research question:
Under the TTA setting where no knowledge of the source domain is available, how can we effectively align the energies between source and domain-shiftedd target samples while considering their class information, to accelerate performance enhancements?
Key idea. In this paper, we propose adaptive energy alignment (AEA), a new energy-based TTA method that achieves fast online adaptation. We first show that the conventional entropy minimization (EM) loss can be decomposed into two energy-based terms with conflicting roles. This hinders the energy gap reduction when the EM loss is used alone, resulting in limited performance not only in the early-stage of batch arrivals but also throughout the entire online batches (in Fig. [C and Sec. 3.2). To address this, we introduce an energy alignment scheme that strategically reduces the energy gap between the source and target domains during TTA, based on the following ideas: (i) aligning the overall energy levels (i.e., magnitude) of target samples with those of source domain, which the pre-trained model is already familiar with, and (ii) precisely guiding the direction of energy alignment by considering the structural relations between different classes, facet that has been less configured in previous works. As depicted in Fig. [b] Ic by leveraging rell-trained knowledge of source domain in the tre-trained model, our approach robustly reduces the energy gap within few
2

---

Published as a conference paper at ICLR 2025
adaptation batches (i.e., fast adaptation) while maintaining class wise correlations between the source and target domain.
Summary of contributions. Overall, our AEA is equipped with advanced optimization losses with two distinct roles for energy alignment during TTA. We summarize our contributions as follows:
Based the energy-based re- re-interpretation of EM loss, propose a source-free energy align- ment (SFEA) loss for fast TTA. The key idea is to strategically align the overall energy magnitudes between source and target domains, aiming effectively reduce the energy gap during adaptation.
To guide the energy alignment direction of each target sample in the logit space, we introduce a logit cosine similarity (LCS) loss as key component. By taking advantage of a well-trained classifier's 's weight matrix, our LCS loss ensures that the class-wise correlation of the target domain aligns well with that of the source domain during energy alignment.
Consequently, our AEA is able to reduce the energy gap between target and source samples (SFEA loss) while considering each test sample's energy alignment direction (LCS loss). Since ABA only modifies the loss function, it introduces merely negligible extra delay/resources, as demonstrated in Sec. B.7 Our approach demonstrates remarkable performance across various domain shift datasets, including CIFARIO-C, CIFARI0O-C, and Tiny Ientmm
2 RELATED WORKS
time adaptation (TTA). To tackle the TTA problem, several works (Liang et al. 2020 Wang al.) 2022) have employed pseudo-labeling techniques, where the most probable labels for unlabeled target samples are used for supervised model adaptation. [Snn et al. (2020) have incorporated self-supervised learning schemes for pre-training the model. Afterwards, these auxiliary tasks are leveraged for model adaptation at test-time. Also, normalization calibration methods (Schneider et al. 2020; Wang et al. 2021 Niu et al.) (023) have been suggested to adapt normalization layers (e.g., batch/instance/groon group normalization) which encode distribution-specific knowledge. Other methods such as consistency regularization within augmentations (Zhang et al. 2022) and class prototype-based approaches (Iwasawa & Matsuo 2021) have been also proposed. Regarding TTA settings, online test-time adaptation (OTTA) (Wang et al., (023) has received attention motivated by many practical scenarios in which target samples arrive sequentially in a batch-wis wise manner. The goal is to continuously adapt the model to the target samples in an online fashion.
As fundamental line of works, researchers have typically employed entropy minimization (EM) (Liang et al.] [2020) Wang al.) [02T]] Niu et al., [022] ZZhng et al., [022] NNu al.] (023) as a primary objective function to adapt the model using un unlabeled target samples. While the EM Joss is effective TTA, our energy-based re-interpretation of the EM loss (in Sec. 3.2) reveals its potential hindrance to the model's assertive adaptation. Specifically in OTTA scenarios, existing works exhibit sub-optimal carly-stage performance, resulting from the insufficient adaptation to the target domain. In contrast, our AEA improves both the early-stage and overall performance in OTTA by explicitly reducing the energy gap between the source and target domains.
Enargy-based models (EBMs). EBMs (LeCun al. 2006) adopted score-based learning approach, allowing them to model various types of distributions in a flexible way. Such effectiveness in modeling distribution has enabled the utilization of EBMs in complex tasks such image generation (Du & Mordatch 2019 Du et al., 2020) and generative classifiers (Larochelle & Bengio 2008 Yang & Ji 2021) in a high- nigh-dimensional space. Also, some researchers have realized that modern discriminate
models can be interpreted as EBMs as well (LeCun et al. 2006 Grathwohl et al.) 2020 Liu et al.
2020). EBMs have been also utilized to address various distribution shift problems such as out-of
distribution detection (Liu et al. 2020). anomaly detection (Du et al.] 2022), domain adaptation
(Zou et al. 2021 Xie et al. 2022; Herath et al., 2023) and domain generalization (Du et al., 2022
Xiao et al. 2023). Recently, Yuan et al. (2024) have suggested an energy- TTA approach that decreases the target energies within the model's 's distribution to improve model generalizability.
However, it focuses solely on reducing the overall energy level without considering the energy alignment direction and requires multiple iterations to generate negative samples during adaptation. This potentially limits its performance and practicality, as demonstrated in Sec. and Sec. B.7
3

---

Published as a conference paper at ICLR 2025
To address the shortcomings of existing works, we propose new AEA scheme that not only aligns energy levels (i.e., magnitudes) between source/target domains but also considers energy alignment direction to further improve adaptation performance. Drawing insight from revisiting the EM loss, our scheme introduces two key loss functions that accelerate TTA in a challenging source-free TTA setup where the source domains are not available at test-time. Our approach introduces negligible computational costs during adaptation, enhancing its practicality in a wide range of applications.
3 AEA: ENERGY-LEVER ALIGNMENT FOR TTA
3.1 PROBLEM SETUP AND PRELIMINARY
Settings. Let Dg {Ks. V/s} be the source data sampled from the source domain distribution pS v), and Dr
(XTTJT) be the target data from the target domain distribution P (a, y). where ps (x, y) o pz (x, y) but they have common label sets. Each pair (I, y1) E X' x of samples I, and labels y; from the source/target domains follow the joint distribution ps (x, P2xx, IP"(x: y).
respectively. Given a discriminate model f pre-trained on the source data Ds. TTA aims to adapt the model to the target domain pd (ar, y). . In online settings, is required to perform adaptation (i) without access to source data and (ii) using unlabeled target samples (or batches), which are incoming in sequential manner. Specifically, for current test time step i, the model fe adapts to mini-batch B, = { consisting of domain-shifted, unlabeled target samples 2, ~ PT((x) with the size of BB. To make predictions for the next batch ((++1)- the model needs to persistently adapt in an online manner by accumulating knowledge from past batches (i.e., B1, B2...., B,).
Energy-based models (EBMs). By defining an energy function Eg(x) IR + R that maps input x E RD to non poobbbbbaiii scalar called the energy, one can design energy-based models (EBMs) (LeCun al., (2066) with neural networks parameterized by 0. Following the energy- based interpretation discriminate models (Grathwohl et al. 2020; Liu et al. 2020), we consider neural network So (a)) modeling categorical distribution with softmax function (y(y)x) exp(fe(=)iv)) where [z)) denotes the network output (i.c., logit) for input x and class y/.
Ey exp(fo(=)l"")
Also, we can define the energy function Ex(x) (also known as the free energy) with a log partition function as
Ea(x) = = Σ exp(fo(x)(1)).
Detailed derivations can be found in Sec. A.1 The energy score Eg (x) is well-known to serve as a representative indicator for distinguishing between in-distribution (ID) and out-of-distribution (OOD) samples (Liu et al. 2020 Xie et al. 2022, Du et al. 2022 Herath et al. 2023). Since most modern discriminate models have employed negative lig-liilihood (NLL) as a supervised loss, this turns out to push down the energy for ID samples, while those for OOD samples remain relatively high.
3.2 OBSERVATIONS AND MOTIVATIONS
As shown in Fig. la an energy disparity between the source and the target domains is also observed in TTA. Even in online TTA where the model is sequentially adapted, this occurrence still becomes pronounced in the early-stage of batch arrival since the model is less adapted to the target domain with few batches. Recently, several works (Xie al., 2022: Xiao al. 2023 Herath al. 2023) have discovered that reducing the energy gap enables rapid and effective adaptation to the target domains, serving as a key factor in mitigating domain gaps. However, in TTA, it is challenging to reduce the energy gap by using only small number of unlabeled target samples without any source data at test time. Furthermore, we notice that the conventional loss function, entropy minimization (EM), has limitation in sufficiently aligning the energy, resulting in unsatisfactory performance. We further discuss this issue in the following section.
Revisiting entropy minimization. We provide an energy-based re-interpretation of entropy min- imization (EM) loss to substantiate the validity of our core idea, the energy alignment. The EM loss, denoted by LEM (x; 0). has been widely utilized in TTA works (Liang et al.) 2020 Wang al. 2021 Niu al. 2022 Zhang et al. 2022 Niu et al. 2023), where the expected Shan- non entropy H( b) for sample x from a target domain p' is employed as a loss function, i.e.,
LEM (x: 0) = H(x; 0) Σソ= Po (35 (:)) Po (z). Also, this can be decomposed into two
4

---

Published as a conference paper at ICLR 2025
contrastive energy- based terms from definition of softmax function and Equ. ① as
K
K
H(z;0)
Σ pa (y)()))))
log (fo(r)lus] ((o(vslo))o((t))vv)
fo(t)lbu))
(2a)
K
Σ
pa (o(y) Eu( uj) Ea(x),
(2b)
where Eo (x. m3) = (2)) and K is the number of classes. As the last term in Eqn. 2ah does not depend on j. the sum of the outer probabilities is equal to 1, resulting in factoring out the energy term (x)).. . In Eqn. (2b). the first term encourages minimizing the energy Eg (z, yy) with proportional to the predictive probability for the corresponding class yj- This makes confident predictions even more confident by increasing the logit of each class in proportion to its confidence. In contrast, the second term, the free energy Eg Ex)),, aims increase the ensemble of energies, which can be considered as penalization term that reduces the overall scales of the logits for all classes.
Limitation of EM loss. Unlike the typical super- vised NLL loss where the predictive probability (i.e., Po (yVyaa)) in Eqns. (2) is allocated to the ground truth class, the E. loss reduces the en- ergy for each class in dispersed manner. Also, the second term even tends to increase the over- all energy. This results in insufficient reduction of the energy gap between the source and tar- get domains, potentially hindering the assertive model adaptation in TTA. To demonstrate these effects, in Fig. [ we compare the energy gap and accuracy of several methods. The results reveal that the EM loss, which lacks the ability to sufficiently reduce the energy gap, exhibits limited accuracy, especially in the early stage of batch arrivals. On the other hand, by using our proposed scheme (denoted as Ours) designed to strategically reduce the energy gap, the model adapts to the target domain more quickly, resulting in improved performance throughout all stages and getting close to the Oracle performance. Based our conviction that energy alignment is key factor in accelerating adaptation and improving performance, in the following sections, we present our AEA, new TTA method that strategically reduces the energy gap while precisely guiding the direction of energy alignment by maintaining class-wise correlations between the source and target domains. A high-level description of our idea can be found in Fig.
BrAceet
Oreeee
Durs
4.3
3.00
325
150
100
125 150
Figure 2: (left) Energy gap batch index: (right) Ac- curacy vs batch index. We carry out the online TTA on CIFAR1O-C (Gaussian noise) with four different meth- ods: (1) BnAdapr: Only BN parameters are updated, (2) Oracle: Energy alignment assuming knowledge of the source domain, (3) EM loss, (4) Ours: combination
EM loss and our SFEA loss explained Sec.
3.3 SOURCE-FREE ENERGY ALIGNMENT (SFEA) LOSS
As aforementioned, there exists an energy disparity between the source and the target domains, and conventional loss function (i.e., EM loss) has shown its limitation in reducing this gap. To accomplish this, we propose source-free energy alignment (SFEA) loss, which explicitly minimizes target energy, partially offsetting the free-energy maximization term (i.e., Eg (z)) of the EM loss in Eqn. (2b). This allows to achieve better energy gap reduction compared to using the EM loss alone. If the gap is sufficiently reduced, we can encourage the model learn representations of target samples to become similar to those of the source samples, promoting domain-invarian representation learning.
Formulation of SFEA loss. To make the energy of the target samples closer to the energy of the source domain, we first estimate the energy of the source domain. One of the challenging aspects here is that we need to estimate it without access to any source information (c.g., data samples/statistics) To address this, we employ source-like sample selection scheme, given that reliable target samples can be utilized to represent the source domain distribution to some extent (Ma et al., 2021 Du et al. 2023). Specifically, for each adaptation batch B, we construct a source-like like batch B, a set of indices of samples with low energy scores less than the threshold 8n defined for each batch as follows:
5

---

Published as a conference paper at ICLR 2025
Source Target
Class A
Sourne Target
Class A
Source
Class Class 8
Class B
Class B
Target
Erergy
Energy
Erergy
Energy
Logit
Loght
(a) Before adaptation
(b) with SFEA
(c) with SFEA LCS
Figure 3: High- description of our AEA. The z-axis represents the logit (projected), while the y-axis represents the energy. (a) Before adaptation, there exists the energy gap between the source and target domains, and the boundary of the two classes is fairly unclear in the target domain. (b) Our AEA sufficiently reduces the energy gap through the source-free energy alignment (SFEA) loss in Sec. 3.3 (c) Also, our logit cosine similarity (LCS) loss in Sec. [.4] adaptively guides the direction of energy alignment in the logit space by maintaining the class-wise correlations in the target domain to those in the source domain.
É.
Σ Eg(a,), EEB
B= -i: E(xi) S dB},
B
where É. the approximated energy the source domain and ón is the threshold for sample selection, corresponding to the bottom a% of target samples within each batch. Subsequently, we consider the averaged energy of B as the approximated energy Ég for the source domain. In light of this approximated energy E,, introduce source-free energy alignment (SFEA) loss CSFEA which can penalize the target samples with higher energy (x) than F, as follows:
(a 0) = log(1 + exp(Ex(x) E.)).
Notably, in Eqn. (4), employ the softplus function (i.e., log(1 + 2)) Glorot et al. 2011 Dugas et al. (000), a differentiable and smoothed ReLU function (Agarap) 2018) as our objective. The rationale behind this choice lies in the assumption that the true energy of the source domain is likely be slightly lower than our approximation E,, given that we approximate it using source-like target samples. Therefore, we aim to impose a small penalty even on target samples whose estimated energy (x( is less than or equal to É.. By doing so, our loss ultimately adapts the model to make the energy for each target sample closer to the approximated energy for the source domain. Although our scheme minimizes the target energy towards adaptively changing goal (i.e., the approximated source energy) due to the infeasibility of accessing the source domain, our method turns out to align the energy of source and target domains adaptation progresses, as shown in Fig. [I and Fig. 2 In Sec. we also demonstrate that our SFEA loss effectively reduces the energy gap and achieves attainable TTA performance.
Key advantages of SFEA loss. Our LSFEA offers the flexibility to gradually adjust the degree of penalization throughout the online batch arrivals, considering the evolving domain gap between the source domain and the target samples. In other words, for the first batches where the domain gap is particularly pronounced, the energy gap (i.c., Es(x) - É.) is also more significant, resulting a stronger penalization effect. This effect steadily decreases as adaptation proceeds, until the energy gap between the source and target domains is sufficiently reduced. As result, the model can be updated more assertively the initial steps, accelerating adaptation and leading to significant improvement even in early-stage performance, as confirmed in Fig. 2 Moreover, compared to Yuan ct al. (2024), which requires multiple iterations to decrease target energies, our SFEA loss efficiently reduces the energy gap with simple computations, requiring negligible additional costs (i.e, time delay and memory usage) as demonstrated in Sec. Sec. In the following subsection, we further take into account the directional aspects of energy alignment achieve more robust adaptation.
3.4 LOGIT COSINE SIMILARITY (LCS) LOss
Another key aspect consider in AEA is the energy alignment direction of each target sample. This is motivated by the separated decision boundaries among classes in the target domain (see Fig. and Fig. 3b). Basically, the energy function in Eqn. [1 is estimated for all possible classes, indicating

---

Published as a conference paper at ICLR 2025
that CSFEA reduces the energy gap in terms of overall magnitude. In this subsection, we propose our second key component that supplements LSFEA EA by considering the energy alignment direction, depending each target sample's 's class information. To accomplish this, we propose a logit cosine similarity (L.CS) loss, which aims to accurately guide the direction of energy alignment. Our LCS can maintain class-wise correlations of target samples to those of the source domain in a logit space. We have confirmed that energy alignment in a direction that preserves class-wise correlation yields better performance, allowing target samples to be clearly distinguishable in the logit space (as in Fig. (c)..
Key idea of LCS loss. Specifically, our LCS loss aims to further align the logits of each target sample with those of source sample for the same class k. i... W Zug is aligned towards with W 2 where W is model's 's classifier weight and Zug and Eure denote the output features right before the classifier for the target and source samples, respectively. This process effectively guides the direction of energy alignment in the logit space. However, the feature of source sample Auc is not available at the time of TTA. To get around this, we bring up our idea from previous works (Liang et al. 2020 [wasawa] & Matsuo 2021; Jang et al. (223) that utilize the k-th column vector of W (denoted by W&) as an anchor point to make sample-wise pseudo- labels. In contrast to them, we directly employ Ws. which corresponds to the classifier's weight vector for class k, as a surrogate source feature Esre to compose our loss function as described in Fig. 4 Since the source domain feature of class k is likely to be aligned with W& to maximize the k-th logit during pre-training on the source domain, W₂ may serve as good representative feature for Zace of class k. By taking advantage of this well-trained classifier's weight matrix, our AEA is able to effectively reduce the energy gap while maintaining fass-wiss correlations between the source and target domains.
fo(x)
To(x)
Feature Extractor
cas
T,
Target Sample
Classifiur (199
WTW
X
W;
Wa
Tr
T
Tx
Surrogate
Logits (7.)
of Aur
Figure 4: Description of the logit cosine similarity
(LCS) loss calculated from the classifier's weight W.
Formulation of LCS loss. We define function S(x; 0) to measure the similarity of two logits as
S 0) = Cooo (1) where 7 WW
(5)
where Teas [] denotes the cosine similarity operation. Here, (x( represents the logits for a given target sample i, while fg is virtual logit computed as WTW, g. as illustrated in Fig. [4 The label corresponds to the most confident class for target sample x. We note that the cosine similarity is used for distance metric in that, unlike the feature representation, the weight does not inherently contain magnitude-related information; thus, the estimated virtual logit has its meaning in terms of direction rather than magnitude. Now we can formulate our LCS loss as follows:
(LCs(E;)) 0) = w(x) (=);))) I[C(z;;) 0) 2 Co}.
(6)
where T(.} is an indicator function, C(r; 0) is the predictive confidence of sample x, and Co is pre-defined confidence threshold. In addition, w(x) is sample-wise weight, which is defined as w(x) exp (C(x: 0) Co). The weight w(x) potentially encourages confident samples to have stronger effect on LCS loss. Ultimately, the LCS loss tends to align the class-wise correlation of the target domain with that of the source domain by bringing the logit fo(x)) each target sample closer to the virtual logit Jg. Consequentially, our approach robustly guides the direction of energy alignment while maintaining class-wise correlations, thereby providing additional performance improvements.
3.5
OVERALL LOss OF AEA
The overall AEA loss defined as combination of LSFEA, LLCS with their respective coefficients, along with the standard unsupervised EM loss LEM as follows:
Liotal = LEM + d1 LSFEA + X2 LLCs.
(7)
Incorporating our proposed losses (CSFEA, LLCS) consistently enhances performance and acceler- ates adaptation across multiple benchmark datasets (in Sec. 4. Since our method simply introduces the extra loss functions on top of the EM loss, the additional computational cost becomes negligible (in Sec. B.7). As result, AEA can be successfully applied to wide range of real-world scenarios, including resource-consee constrained applications. In the following section, we validate its effectiveness and versatility on diverse domain shift datasets.
7

---

Published as a conference paper at ICLR 2025
Table 1: Classification emrors (%, J) CIFARIO-C with the highest severity level 5 for ResNet-26.
Noise Shot
Blur
Weather Frost
Digital
Method
Gauss.
Impul.
Defoc.
Class
Motion
Zoom
Snow
Brit.
Contr.
Elastic
Pixel
JPb
No adaptation
73.1
66.8
69.4
40.2
51.9
37.9
38.1
25.5
39.3
39.4
10.1
55.2
26.7
61.5
31.0
44,4
TENT
32.7
30.2
39.1
15.7
35.9
18.2
16.1
21.8
23.1
19.4
12.7
16.0
26.1
22.3
30.4
24.0
BN adaptation
46.1
25.5
17.6
28.9
26.6
35.5
SHOT
29.5
26.5
35.4
14.6
13.7
17.0
14.6
19.8
22.2
17.8
11.8
16.2
24.6
20.4
26.9
22.1
T3A
65.3
65.5
37.0
47.7
24.8
36.6
10.1
50.0
25.1
51.8
29,8
40.4
TTT
24.9
23.0
30.2
13.5
34.6
20.4
15.9
19.3
17.9
14.1
9.4
26.4
23.7
16.0
23.8
20.9
NOTE
48.8
42.5
47.5
25.0
40.9
24.4
23.7
20.3
23.5
22.8
9.1
31.6
24.7
41.9
29.2
30.4
Conjugate PL
32.7
35.8
16.1
26.1
22.3
30.4
EATA
38.7
36.2
46.0
17.1
40.8
19.8
17.9
24.9
25.4
20.9
14,1
17.6
28.7
26.4
35.3
27.3
SAR
36.3
23.2
16.5
26.2
22.8
30.6
TEA
27.7
25.5
34.2
15.3
34,8
18.0
15.9
20.0
20.5
17.7
12.4
16.4
25.7
19.2
26.3
22.0
AEA (ours)
24.7
22.7
31.9
13.8
30.8
16.2
13.8
17.5
17.5
15.4
10.8
13.7
23.3
17.4
23.8
19.5
Table 2: Classification emors (%, J) on CIFAR100-C with the highest severity level 5 for ResNet-26.
Noise Shot
Blur Motion
Weather Frost
Digital
Method
Causs.
Impul.
Defoc,
Glass
Zoom
Snow
Fog
Brit.
Contr.
Hlastic
Pixel
JPBG
Avg.
No adaptation
89.3
88.3
91.0
67.2
63.5
60.8
59.6
56.1
62.3
67.6
42.7
84.4
50.8
85.5
60.9
68.7
TENT
54.1
51.4
50.7
59.6
BN adapiation
70.5
69.9
68.8
46.6
60.8
48.8
45.9
59.0
56.8
55.1
45.5
51.2
53.5
54.8
62.8
56.7
SHOT
58.6
55.0
41.3
51.9
49.9
41.0
48.6
48.9
46.4
56.0
T3A
89.3
88.4
90.4
64.8
60.9
59.9
57.3
57.2
61.2
65.3
43.2
82.5
50.0
82.9
60.4
67.6
TTT
63.7
63.2
65.1
43.9
57.2
49.9
43.4
54.1
50.8
49.7
38.7
70.2
49.7
45.7
56.1
53.4
NOTE
76.4
74.6
74.5
53.9
57.6
50.7
47.9
52.7
52.3
38.6
67.4
49.0
70.4
57.8
58.7
Conjugate PL
65.6
64.7
65.1
43.9
58.1
47.0
43.6
56.2
54.1
52.3
42.9
49.5
51.4
50.7
59.6
53.6
EATA
56.7
$1,9
55.4
52.1
62.2
SAR
65.8
64.9
65.3
44.2
58.2
47.1
43.8
56.4
54.4
52.5
43.0
49.3
51.4
50.8
59.7
53.8
TEA
64.0
64.2
45.1
59.0
48.4
45.4
56.5
55.1
43.3
53.4
52.7
50.1
60.0
54.2
AEA (ours)
58.2
58.8
59.0
40.9
55.0
43.9
40.5
51.3
49.0
47.4
39.4
44.1
48.4
44.3
54.8
|9.0
4
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
In this section, we demonstrate the effectiveness of our approach through extensive experiments. We follow the online TTA setting (Zhao et al., 2023) for experiments. Given pre-trained model on the source domain, we adapt the model to the incoming batches consisting of samples from the target domain in an online manner. Specifically, the model is persistently adapted to the target domain by accumulating knowledge from past batches, to make predictions for the following batches.
Datasets. For the domain shift datasets, we utilize corrupted image datasets: CIFARIO-C, CIFARI00- C, and Tiny ImageNet--CCC The uncorrupted datasets (i.e., CIFAR 10, CIFAR100, TinyImageNG are used the source domain, while corrupted datasets (i.e., CIFAR1O-C, CIFAR100-C, TinylmageNet- C) with the highest severity (i.e., level 5) serve as the target domain. We also evaluate our method on style shift dataset (i.e., PACS) in Sec. 4.2 and ImageNet ImageNet-C dataset in Sec. B.
To validate the effectiveness of our method, we compare AEA with the following baselines: SHOT (Liang et al. 2020). TENT (Wang al. (0021), I), BN adaptation (Schneider et al., 2020). T3A
Baselines.
[[wasawa & Matsuo 2221)) TTT (Sun et al. 2020). NOTE (Gong et al. 2022), Conjugate PL (Goyal
al. 2022). EATA (Nuu al. 2022). SAR (Niu et al. 2023) and TEA Yuan al. 2024 These are
chosen to cover the diverse TTA methods described in Sec. 2
Implementation details. For a fair comparison, we follow the official setup of the recent TTA benchmark (Zhao et al. 2023) and set the hyperparameters for each bascline consistent with them. For CIFAR10/TON pre-training before the TTA stage, we adopt a self-supervised learning scheme (i.e., the rotation prediction) with an auxiliary head to fairly compare with an auxiliary bask-based methods like TTT (Sun et al. 2020), while TinyImageNet pre-training follows the official Torch Vision setup (manntainers & contributors 2016). For performance metric, we use classification error (%, 1) averaged over all online batches. We utilize ResNet-26 (He et al. 2016) for CIFAR IIFAR1O-C/100-- and 50 for TinylmageNet-C. with additional experiments under different backbones (i.e., Vision transformer (Dosovitskiy et al.) (02T)) ) in Sec. [.2] During TTA, only batch normalization parameters updated. Further implementation details in Sec. of Appendix.
8

---

Published as a conference paper at ICLR 2025
Table 3: Classification emror (%. L) TinylmageNet-C with the highest severity level 5 for ResNet-50.
Noise Shot
Blur
Weather Frost
Digital
Method
Gauss.
Impul.
Defoc.
Class
Motion
Zoom
Snow
Brit.
Contr.
Elastic
Pixel
JPB(
No adaptation
96.6
95.1
97.2
92.5
92.2
77.8
78.5
81.9
78.1
89.5
77.8
98.3
69.5
71.9
55.6
83.5
TENT
66.5
64.3
72.9
63.2
75.1
55.5
54.8
63.6
61.9
67.6
57.6
86.2
56.6
51.5
52.5
63.3
BN adaptation
77.1
57.7
52.2
53.3
SHOT
64.5
62.8
70.6
61.9
73.5
54.5
53.8
62.1
61.1
65.2
56.2
89.6
55.4
50.8
51.8
62.2
T3A
96.6
97.3
92.7
92.1
77.3
78.2
82.3
78.2
77.1
98.5
68.9
70,3
55.8
83.4
NOTE
83.4
80.4
86.4
81.9
85.7
65.4
64.1
69.6
67.4
79.8
64.2
96.0
61.2
56.0
53.1
73.0
Conjugate PL
67.4
65.3
74.0
63.9
76.1
55.8
55.3
64.3
62.7
68.6
58.1
86.2
57.1
51.8
53.0
64.0
EATA
74,2
90.8
56.1
51.1
52.2
62.4
SAR
67.2
65.2
73.6
63.9
75.9
55.8
55.2
64.1
62.6
68.4
58.0
86.0
57.1
51.8
52.9
63.8
TEA
72.3
56.9
52.9
AEA (ours)
63.7
62.1
67.9
60.9
72.8
53.8
$2.9
60.1
58.9
61.9
54.2
90.3
54.5
49,8
51.1
61.0
4.2
EXPERIMENTAL RESULTS
Overall performance. In Tables 1 we report the classification error (%) for the online TTA on CIFARIO-C, CIFARI00-C and TinlmmaeNNet-C dataset, respectively. Experiments are conducted over 3 random trials, and we include the full version with standard deviations in Sec. B.9 of Appendix. They show that our AEA consistently achieves superior performances than other baselines for most of corruption types. This supports claim that reducing the energy gap between the source and target domains contributes to the improvement of TTA performance. In several cases, TTT (Sun et al. 2020), which performs self-supervised learning during the pre-training stage, shows competitive performances. However, this approach has limitations in a practical usage since it necessarily requires a pre-trained auxiliary head during TTA. In contrast, our AEA does not need employ this additional head at the time of TTA. The results demonstrate that our AEA can effectively adapt models via the proposed loss functions, improving TTA performance on various domain shift datasets.
Fast adaptation of AEA. One of the key advantages of our AEA is its ability celerate model adaptation, enabling fast TTA with high performance. This can be done by explicitly reducing the energy gap through our SFEA loss. Simultaneously, our LCS loss further contributes to forming a robust decision boundary the domain- invariant latent space to clearly separate target samples according to their classes. In Fig. 5 we report the temporal TTA per- formance over adaptation batches. The re- sults show that, compared other compet- itive TTA baselines, our AEA consistently achieves superior performance, especially in the early stages of test batches from the target domain. Furthermore, the significant performance gaps between our AEA and TENT (i.e., only EM loss) demonstrate the effectiveness of our proposed objective functions in accelerating TTA.
Figure 5: Fast adaptation of our AEA. We report the model accuracy (%. 1) over adaptation batches on CIFARIO-C and CIFARI00-C datasets (with Gaussian noise). Our AFA con- sistently achieves superior performance, particularly the carly stages of hatches.
Result for style shift dataset. In Table 4 we present the results the style shift dataset, PACS (Li et al., (017)) using ResNet-50. Each column represents the average classification er- ror for the corresponding target domain, with the
Methods
Avg.
No adapt.
24.67
42.74
48.26
42.18
39.46
TENT
16.55
26.23
29.65
34.25
26.67
BN adapt.
16.72
26.68
30.18
36.57
27.54
T3A
16.46
37.98
33.88
36.05
31.09
remaining domains used as independent source domains (i.e, domain generalization set- ting). Our AEA consistently outperforms other baselines, demonstrating its effectiveness across diverse types of domain shifts. This highlights our AEA's versatility in various domain shift scenarios.
SAR
16.63
26.41
29.87
34.98
26.97
AEA (ours)
16.22
24.32
27.22
30.76
24.63
Table 4: Results on style shift dataset (PACS) with different styles: Photo, Art painting, Cartoon and Sketch.

---

Published as a conference paper at ICLR 2025
4.3 FURTHER STUDIES ON AEA
Effect of each component. In Table 5 we conduct an ablation study on each component of our AEA, demonstrating that both proposed loss functions (i.e, LSrEA, fLCS) independently contribute to performance improvement. Compared to using only the EM loss, the SFEA loss significantly enhances TTA performance by accelerating adaptation, while the LCS loss further makes the direction of energy alignment robust, leading to additional gains.
Methods
CIFARIO-C
CIFARIOD
Na aduptation
44.4
68.7
EM
24.0
53.6
SFEA
20.7
51.8
SPEA LCS
49.0
Table Effect of each component in our AEA.
Effect of hyperparameters. In Fig. we per- form an ablation on the hyperparameters intro- duced in AEA, including A1, A2, a, and Co, to evaluate the robustness of our method to varying hyperparameter settings. Each subfigure repre- sents our AEA's performance across specified range of given hyperparameter, while keep- ing the others fixed at their default values. As shown in the results, AEA consistently outper- forms other baselines by notable margin across different hyperparameter settings. This not only confirms the robustness of our method on hyperparameters but also highlights its overall superiority.
AEA (ours)
TENT
Accurccy
80.0
75.9
0.5 1.0 1.5
50
30
50
0.60
0.80
0.99
A1
A2
(%)
Co
Figure 6: Ablation on varying hyperparameters for our AEA. We compare our method with competitive base- lines through the average accuracy on CIFARIO-C.
Why does energy alignment lead to better adaptation? We find that our energy align- ment help mitigate domain dis- parity by promoting feature alignment between the source and target domains, a well- established factor for improv- ing domain adaptation (Yang et al.) 2022; Lu et al.) (022). Specifically, in our implemen- tation, we freeze the classifier of the source pre-trained model while updating the batch normalization (BN) parameters of the feature extractor. As illustrated in Liang et al. (2020). the classifier retains fine-grained knowledge of the source domain, and aligning the target domain's 's representation with the source hypothesis facilitates OOD predictions. Similarly, our energy alignment approach encourages reduction in target energy, which updates the model parameters to ensure that target features align with the classifier's 's weights (i.e., source hypothesis). This, in turn, leads to the feature alignment between the source and target domains, resulting in improved TTA performance with accelerated adaptation. To validate this, in Fig. 7 we provide feature visualizations for source and target samples in the early stage of TTA adaptation. Each color represents different class, while the green and yellow stars indicate class-wise feature means for the target and source domains, respectively. As shown, our AEA (right) effectively aligns source and target features, leading to improved performance, which is not the case in situations without adaptation (left) and with only EM loss (center). This improvement occurs due to the fact that our energy alignment encourages the model to reduce the energies of target samples, matching them source samples, which inherently aligns their feature representations as well. A more in-depth discussion can be found Sec. A.8 of Appendix.
scurce featune moin target more
Compenent:
Campanent-
Camponent--1
Figure 7: TSNE visualization of features for three methods: no adaptation (left), EM loss (center) and AEA (right).
5 CONCLUSION
In this work, we propose an adaptive energy alignment (AEA) scheme to handle OOD samples in TTA setups. Based on our insight that aligning energy levels between the source and target domains facilitates rapid and effective adaptation to the target domains, our AEA strategically reduces the energy gap to enhance TTA performance. By revisiting the EM loss, we discover its limitations in reducing the energy gap. To overcome this, we introduce 1) the SFEA loss, which effectively minimizes the energy gap without access to the source domain, and 2) the LCS loss, which guides the direction of energy alignment by considering class-wise correlations. With its ability to accelerate model adaptation, approach is versatile and leads improved performance in various scenarios.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide our implementation details and setups in Sec. 4 and Sec. C
This includes information on implementation benchmark, hyperparameters, and computing resources. Our source code is publicly available at https//gghhub.oom//vvojennnggg//
ACKNOWLEDGEMENTS
This work supported in part by Samsung Electronics Co., Ltd. and by the NAVER- Co-Lab. Additionally, was funded by the National Research Foundation of Korea (NRF) through grants from the Korea government (MSIT) (No. RS-2024-004080003, RS-2024-00340966 and by the Institute of Information & communications Technology Planning & Evaluation (IITP) under MSIT of Korea (No. RS-2024-004448622 Dong-Jun Han the corresponding author.
REFERENCES
Abien Fred Agarap. Deep learning using rectified linear units (relu), arXiv preprint orXiv:1803.0837. 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d'Alché-Buc, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems, 33:6637-6647, 2020.
Yuntao Du, Haiyang Yang, Mingcai Chen, Hongtao Luo, Juan Jiang, Xin, and Chongjun Wang. Generation, augmentation, and alignment: A pseudo-source domain based method for source-free domain adaptation. Machine Learning, pp. 1-21, 2023.
Zhekai Du, Jingjing Li, Lin Zuo, Lei Zhu, and Ke Lu. Energy-based domain generalization for face anti-spoofing. In Proceedings of the 30th ACM International Conference on Multimedia, pp- 1757,1757, 2022.
Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating second-order functional knowledge for better option pricing. Advances in neural information processing systems, 13, 2000.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323, JMLR Workshop and Conference Proceedings, 2011.
Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Robust continual test-time adaptation against temporal correlation. Advances in Neural Information Processing Systems, 35:27253-27266, 2022.
Taesik Gong, Yewon Kim, Taeckyung Lee, Sorn Chottananurak, and Sung-Ju Lee. SoTTA: Robust test-time adaptation on noisy data streams. In Thirly-seventh Conference Neural Information Processing Systems, 2023.
Sachin Goyal, Mingjie Sun, Aditi Raghunathan, and Zico Kolter. Test time adaptation via conjugate pseudo-labels. Advances in Neural Information Processing Systems, 35:6204-665 6218, 2022.

---

Published as a conference paper at ICLR 2025
Will Grathwohl, Kuan-Chinn Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2020. |
//openrevies eww. net /ate/fforne
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution ex- amples in neural networks. Proceedings of International Conference on Learning Representations, 2017.
Samitha Herath, Basura Fernando, Ehsan Abbasnejad, Munawar Hayat, Shahram Khadivi, Mehrtash Harandi, Hamid Rezatofighi, and Gholamreza Haffari. Energy-based self-training and normal- ization for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11653-11662, October 2023.
Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34:2427-2440, 2021.
Minguk Jang, Sae- Young Chung, and Hye Won Chung. Test-time adaptation via self-training with nearest neighbor information. International Conference on Learning Representations, 2023.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Hugo Larochelle and Yoshua Bengio. Classification using discriminative restricted boltzmann machines. In Proceedings of the 25th international conference on Machine learning, pp. 536-543, 2008.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning, pp. 6028- 6039. PMLR, 2020.
Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey test-time adaptation under distribu- tion shifts. arXiv preprint 2r3iv:2303. 2023.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of distriibbbtto detection. Advances in neural information processing systems, 434214644 2020.
Wang Lu, Jindong Wang, Haoliang Li, Yigiang Chen, and Xing Xie. Nomain-invariane feature exploration for domain generalization. arXiv preprint arXiv:2207.12020, 2022.
Ning Ma. Jiajun Bu, Zhen Zhang, and Sheng Zhou Uncertainty-guider mixup for semi-supervised domain adaptation without source data. arXiv preprint arXiv:2107.06707, 2021.
Vision maintainers and contributors. Torchvision: Pytorch's computer vision library. https
//github. com wwt../o//ttttttoiiiooooeeoo 2016.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. arXiv preprint 1rXiv:1301.388/ 2013.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pp. 16888-16905. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2025
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In Internetional Conference on Learning Representations, 2023.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games, In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 102-118. Springer, 2016.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems, 33: 1539-1111111 1733--11551, 11551, 2020.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pp. 9229-9222999 9248. PMLR, 2020.
Riccardo Volpi, Pau De Jorge, Diane Larlus, and Gabriela Csurka. On the road to online adaptation for semantic image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19184-19195, 2022.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test- time adaptation by entropy minimization. In International Conference on Learning Representations, 2021. URL https: /ttp::/://///eeeeeeeeeeee net forum? -indix1i-iii 3b 3Lkr3e
Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7201-7211, 2022.
Zixin Wang, Yadan Luo, Liang Zheng, Zhuoxiao Chen, Sen Wang, and Zi Huang. In search of lost online test-time adaptation: A survey. arXiv preprint 10.20099 2023.
Zehao Xiao, Xiantong Zhen, Shengeai Liao, and Cees G. M. Snoek. Energy- test sample adaptation for domain generalization. In The Eleventh International Conference on Learning Representations, 2023. httpttss::/ pettttps://opeereeeee
Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, Xinjing Cheng, and Guoren Wang. Active learning for domain adaptation: An energy-basee approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8708-8716 1708-8716,770 2022.
Xiulong Yang and Shihao Ji. Jem++: Improved techniques for training jem. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6494-644000 2021.
Yuzhe Yang, Hao Wang, and Dina Katabi. On multi-domaii long-tailed recognition, imbalanced domain generalization and beyond. In European Conference Computer Vision, pp. 57-75. Springer, 2022.
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15922-15932, 2023.
Yige Yuan, Binghing Xu, Liang Hou, Fei Sun, Huawei Shen, and Xueqi Cheng. Tea: Test-time energy adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo:
Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35:38629-38644 2022.
Hao Zhao, Yuejiang Liu, Alexandre Alahi, and Tao Lin. On pitfalls of test-time adaptation. In International Conference on Machine Learning (ICML), 2023.
Han Zou, Jianfei Yang, and Xiaojian Wu. Unsupervised energy- adversarial domain adaptation for dross-d text classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1208-118,10 2021.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A ADDITIONAL ANALYSIS
A.1 DETAILED DERIVATION OF ENERGY FUNCTION
In this section, we provide detailed derivation of energy function in Sec. 3.1 By defining energy function Ex(x) RD + R, one can design energy-based models (EBMs) (LeCun et al., 2006 with neural networks parameterized by 0. From the defined Es(x) and the Gibbs distribution, the probability density function pa()r) can be derived as
(x) = exp( exp(-Eo(x))
V
(8)
Z(0)
where Z(0) = exp(-Eo(z))dd denotes the partition function. Following the energy based
interpretation of discriminate models (LeCun et al.) 2006 Grathwohl et (2020 Liu et al.) 2202))
consider neural network [o(x)) modeling categorical distribution with softmax function exp(fa(=)(o)) where fo [x/] denotes the network output (i.e., logit) for input æ and po(y)æ>) y "xp[f=
class y². Without changing the model fs. we can represent the joint distribution Pa y) and density
po(x) marginalized over
Po po(x,y) (x)(fo()) 2(0)
po(x) Eg exp(Jo(z)v))
Z(0)
where (0) is the partition function and Ex 8) = - (x) By combining Egns. 8
9 we can define the energy function Ex(x) (also known the free energy) with a log partition function as
Ex(x) = log Σ exp(fe(x)lv))
(10)
A.2 WHY DOES EM LOSS APPEAR IN OUR FINAL LOSS?
In our AEA, we define our final loss as combination of LSFEA, LLCS along with the EM loss LEM (see Eqn. 77). Basically, the foundation of our AEA is the inherent limitations of EM loss in sufficiently reducing the energy gap the early-stage of batch arrivals where the energy gap between source and target domains is pronounced. To address this, our SFEA loss is proposed to adaptively minimize target energy over time (strongly in the initial batches and more weakly later on), thereby reducing the energy gap more quickly, while our LCS loss considers the directional aspect of energy alignment. turn, our method designed to complement the limitations of the EM loss while leveraging its advantages in TTA (rather than replacing it), which is why incorporate EM loss in our final objective.
A.3 ENERGY-LEVED DISCREPANCY BETWEEN THE SOURCE AND TARGET DOMAINS
In Fig. we demonstrate the energy level discrepancy between the source and target domains to further strengthen implications. As shown, the energy levels of the target samples, which is referred to as 'CIFARIO-C" are higher than that the source samples of CIFARIO 10 regardless of the classes. Due to this energy level discrepancy between the source and target datasets, the existing EM loss approaches exhibit limited performance in the early-stage of online batches as the EM loss fails to sufficiently decrease the energy levels of the target samples. In contrast, method strategically reduces this energy gap while precisely guiding the direction of energy alignment by using our proposed losses LSFFA, and LLCS..
A.4 MOTIVATION OF OUR LCS LOSS
Our LCS loss is proposed to address the directional aspect of energy alignment, particularly con- cerning the class-wise adjustment of logit elements. Specifically, our SFEA loss alone minimizes the free energy of the target samples, aligning the energy between the source and target domains in
14

---

Published as a conference paper at ICLR 2025
Energo
CIFAR1O CIFAR1O-C
Basticttttion Frise pée
Zoom Defor.
Figure 8: Energy-level discrepancy CIFARIO (red) and CIFARIO-C (blue) dataset the time.
terms of the magnitude of energy levels. tends to increase the overall magnitude of the logits across all classes rather than considering the relativity of each logit element (i.c., blass-wise correlation). On the other hand, our LCS loss additionally accounts for class-wise correlation by increasing the logit elements of classes with high correlation together. This allows each target sample align with the well-trained source knowledge in class-wise manner, leading to directionally improved energy alignment.
To support our claim, in Fig. 9 we provide sample-wise visualizations (similar to Figs. lb with and without our LCS loss. In Fig. 9b] while our SFEA loss effectively reduces the energy gap, number of target samples are still not clearly separated according their true classes. In contrast, by incorporating our LCS loss in Fig. 9c the target samples align more successfully with the source samples in class-wise manner, guiding the direction of energy alignment in logit space.
source
dlss
SOUTCO
dlds
class
tarpet
dass
tarpet
dlsss
tageet
class
dlass
class
class
-12
-14
-16
20
60
200 y
60
-00
2040 y
22
40
na
2op 204050
y
(a) EM only
(b) EM SFEA
(c) EM + SFEA + LCS
Figure 9: Sample-we wise logit (z, y-axes) and energy (z-axis) distribution in the carly stage of adaptation steps Although our SFEA loss effectively reduces the energy gap, number of target samples are still not clearly separated according to their true classes. In contrast, by incorporating our LCS loss, target samples align more successfully with the source samples in class-wise manner, guiding the direction of energy alignment in a logit space.
(similar to Figs. [b..
A.5 FURTHER ANALYSIS ON THE MODEL OUTPUT
In 6 we present additional results that help understand how our proposed losses affect model outputs. Specifically, CIFAR10-C dataset, consider four methods: No Adaptation, EM, EM + SEFA, and EM + SEFA + LCS, measuring their average energy scores for source/target domains (Esres Exrg), accuracy and entropy of test samples. From the results, we can further recognize the effects of our method as follows: Our proposed methods (i.e., LSEFA, LL.Gs) can effectively reduce the energy gap between the source and target domains, leading improved accuracy. Furthermore, it is observed that our proposed methods produce lower entropy than TENT (EM loss) by guiding target samples to placed in more confident and correct regions.
A.6 ABLATION STUDY ON THE CHOICE OF LOSS FUNCTION
In Table conduct ablation study the choice of loss function for energy alignment, comparing the hinge loss to our SFEA loss. Specifically, we report the results with varying X coefficient of LSFEA, evaluate hyperparameter sensitivity. The results demonstrate that our loss is
15

---

Published as a conference paper at ICLR 2025
No adaptation
EM
EM+SEFA
EM4SFEA+LCC
Esre
10.26
-00.54
-11.26
-10.83
Pirg
-6.85
-7.35
-8.81
-9.28
Bero Eergl
3.41
3.19
2.45
1.55
Accuracy (%)
61.5
64.8
71.0
72.1
Entropy
7.94
7.84
7.67
7.65
Table 6: Analysis the model output.
less sensitive to hyperparameters than hinge loss, achieving better performance with marginal gains. Our intuition that these improvements are attributed to smoothness of the softplus function around the near-zero region, containing more information useful for reducing the energy gap than the hinge loss near that region.
Also, unlike previous energy alignment works that have access to the source domain, the source- frrece-free scenario TTA does not allow for accurate estimation source energy. Under this new setting, have utilized the approximated source energy for energy alignment, wherein smoothed loss design proves to be more feasible and leads to better performance compared to the hinge loss.
A
1.0
1.2
1.4
1.6
1.8
2.0
hinge
80.3
80.2 80.7 (+0.5%)
79.9 80.6 (40.7%)
79.7 80.2 (+0.5%)
79.3
79.0
80.5 (+0.2%)
79.9 (+0.6%)
79.8 (+0.8%)
Table 7: Ablation study the choice of loss function for energy alignment, comparing the hinge loss SFEA loss. We report the average classification accuracy (%, f) on CIFARIOS 10-C dataset. The other experimental details (e.g. pre-trained model, hyperparameters, etc.) are consistent with the main settings in the paper.
A.7 EVIDENCE FOR EM LOSS'S LIMITATION ON ENERGY GAP REDUCTION
Our argument on this topic initiates from an mergy-basee reinterpretation Sec. 3.2 where the EM loss can be decomposed into two contrastive energy-basee based terms. Regarding the 1sr term (i.c., Σø=1 Poly; Eo) ),)). given the absence labels for target samples, this term dispersely reduces the energy given class (i.e., Es (x, Mj)) across each class, whereas the supervised NLL loss concentrates predictive probabilities on ground truth class. Also, the 2nd term (i.e., negative free penalization during adaptation.
energy, (x)) encourages the increase of ensemble of energies
In TTA, due to the typically limited adaptation batches provided for sequential target batches, the EM loss exhibits limitations in sufficiently reducing the energy gap within the constrained iterations, potentially hindering the assertive adaptation TTA. Confirmingly, as shown in Fig. Ic and Fig. 2 the EM loss alone is not sufficient reduce the energy gap between the source and target domains. For a more comprehensive understanding. in Fig. I0 we visualize how these two decomposed terms change during TTA. Here, Esre and Etrg represent the negative free energy (i.e., 2nd term) for the source and target domain, respectively, and the st term corresponds to Σ/= Po(u,) 2x) Eo Vj) as denoted in Eqn.
In the result, among the two terms, the 1st term is dominant, leading to a reduction energy as adaptation progresses. However, the case of EM loss, the energy gap between source and target domains slowly decreases. and it fails to reach the energy level of the source domain. On the other hand, by adding our SFEA loss to EM loss, we can explicitly align the energy, thereby reducing the energy gap more fast. Also, this result is consistent with Fig. 2 in our main paper, our SFEA loss facilitates close approximation to the Oracle case, which assumes knowledge of source domain, achieving superior TTA performance.
A.8 THE LINK BETWEEN ENERGY GAP REDUCTION AND BETTER ADAPTATION
In this paper, we argue that the performance of TTA can improved by aligning the energy levels between the source and target domains. To accomplish this, our AEA adopts the SFEA loss (in Sec.
16

---

Published as a conference paper at ICLR 2025
Loss Decomposition
Loss Magnitude
Esrc (average)
Eng of 'No adaptation'
-5
-Eng of 'EM loss'
-Eig of +SEA loss'
-10
1st term of 'EM loss'
1 term of 'EM+SFEA loss'
20
40
60
80
100
120
140
160
Batch Index
Figure 10: Visualization of how the two decomposed loss terms in Equ.
change during TTA CIFARIO-C.
3.3 and the LCS loss (in Sec. 3.4) which successfully reduce energy gap, thereby accelerating model
adaptation and achieving superior TTA performance on various domain- datasets.
can pose following question: why would reducing this energy gap improve TTA? We figure out that our energy alignment can also facilitate feature aligmment between the source and target domains. Indeed, it has been widely studied that the feature alignment through domain-invariant representation learning effectively addresses domain shift problems (Yang et al.) 2022 Lu et al., 2022). However, they are typically allowed to access both source and target samples, it is straightforward to align features of both domains, while the absence of source domain in our TTA setting makes it quite challenging. Instead, we utilize the energy level that obviously differentiate between the source and target domains as surrogate feature to be aligned. We note that before adaptation, the pre-traime pre-trained model produces low energy (i.e., high logits) for the souree domain and high energy (i.e., low logits) for the target domain (as observed in Fig. Ta As an alternative to directly aligning features, our AEA strategically matches the energy levels between the source and target domains the logit space, encouraging the model to learn aligned feature representations.
As introduced in Sec.
More specifically, the implementation, we freeze the classifier of pre-trained model which is trained on the source domain, while updating model parameters (e.g., BN parameters of feature extractor) prior to the classifier. As discussed in Liang et al. (2020), such a classifier holds fine- grained knowledge the source domain (i.e., source hypothesis), and by aligning the target domain's representation with the source hypothesis, a well-trained classifier can be effectively utilized for performance improvement. In this context, since our energy alignment encourages decreasing the energies (or increasing the overall logits) of target samples, it necessarily makes the model parameters be updated so that the target features are aligned with the classifier's weight itself, which is already sufficiently aligned with the source domain. Consequentially, this inherently results in alignment of feature representations of target domain with those of source domain, leading performance enhancement.
In Fig. 7 we provide the feature visualization of souree/target samples within a few adaptation batches (after 30 steps out of total 157 steps). As can be seen, in the cases without adaptation (left) and with just EM loss (center), the class-wise feature means are not aligned well between source and target; at the same time, these two cases have shown high energy gaps and low performances the target domain (as shown in Fig. Ic and 2). On the other hand, our AEA (right) can significantly align features between the source and target domains, encouraged by our energy alignment approach. From the results, we emphasize that our strategical and effective energy alignment scheme through SFEA loss and LCS loss facilitates feature alignment, which leads to improved TTA performance.
For further analysis, in Table 3 we measured the true distance between source features and target fea- tures during TTA adaptation. Specifically, for each adaptation stage, we (1) estimate the source/target.
17

---

Published as a conference paper at ICLR 2025
10
20
30
40
50
60
70
80
90
100
110
120
130
140
150
No aduptation
5.18
5.18
5.18
5.18
5.18
5.18
5.18
5.18
5.18
5.18
5.18
5.18
5.18
5.18
5.18
5.18
TENT
5.18
1.87
1,83
1.80
1.76
1.73
1.69
1.64
1.61
1,58
1.55
1.52
1.49
1.46
1.44
1.41
AEA (ours)
5.18
1,58
1.28
1.09
0.98
0.94
0.89
0.86
0.82
0.79
0.78
0.77
0.75
0.71
0.69
0.71
Table 8: Feature alignment between the source and target domains. We measure the true distance (i.e, Euclidean distance) between source features and target features during TTA adaptation. Each column represents the number of adaptation batches. Experiments conducted CIFARIO- C-C (gaussian noise with severity level 5).
feature means for each ground-truth class, (2) measure the Euclidean distance between the feature means for each class, and (3) take the average across all classes. These measurements can indicate how well the adapted model performs feature alignment between the source and target domains. The results demonstrate that our method indeed aligns the true source and target features more rapidly. For instance, our AEA achieves better aligmment by reducing the distance between each feature from 5.18 to 0.98 over 40 adaptation batches, whereas TENT does from 5.18 to 1.76. Based on these findings, we argue that our adaptive energy alignment approach facilitates more effective feature alignment from the early online batches, accelerating TTA adaptation.
B ADDITIONAL RESULTS
B.1 EVALUATION ON IMAGENET- C DATASET
In Tables IO we evaluate the online TTA performance on ImageNet-C dataset, comparing our AEA with comparable baselines. For the experiments, we employ ResNet-50 backbone pre-trained model and follow the setup of TENT (Wang et al. (021). We consider two different practical settings using the large-scale ImageNet-C dataset, which contains 50K target samples: First, we assume there are sufficient target samples (50K, full), and second, we examine case where the number of target samples smaller (10K). We report each result in Table and Table TO, respectively. The results show that our AEA still achieves competitive performance both cases. Notably in case of limited target samples (10K), our AEA outperforms the baselines by large margins ( 4.0). This result indeed supports claim that adaptive energy alignment approach can achieve desired performance even a few adaptation batches compared to the baselines, enhancing its applicability in various scenarios.
Table 9: Classification enrors (%, J) on ImageNet-C (50K target samples) with the highest severity level for Re backbone.
Noise Shot
Blur Motion
Weather Frust
Digital Elustic
Method
Gauss,
Impul.
Defoc:
Glass
Zoom
Snow
Fog
Brit.
Contr.
Pixel
JPEG
Avg.
No adaptation
95.5
94.9
95.4
84.9
91.7
86.6
77.1
84.1
79.4
77.3
44.3
95.8
85.2
77.4
66.8
82.4
TENT
77.4
75.2
75.8
78.0
79.9
45.4
88.4
48.9
45.5
53.2
Ba Ba_adapt
88.4
87.8
87.7
88.4
88.0
78.6
64.4
68.1
69.4
55.3
36.8
89.3
59.2
57.3
67.7
72.4
SHOT
77.6
74.9
76.0
80.4
80.9
63.5
53.1
54.0
61.7
45.1
34.0
95.3
47.9
45.0
52.6
62.8
ConjugatePl.
76.5
74.1
75.4
77.4
78.4
64.8
54.1
55.5
61.5
45.2
34.4
85.6
48.6
45.3
52.9
62.0
SAR
75.2
74.4
73.9
77.5
77.7
63.5
54.0
55.6
60.5
45,4
34.5
76.7
48.9
45.5
52.9
61.1
AEA (ours)
73.8
73.2
72.7
75.8
79.2
59.7
51.9
52.7
58.6
44.0
34.3
90.5
46.6
43.3
50.5
60.5
Table 10: Classification errors (%, J) on ImageNet-C (10K target samples) with the highest severity level 5 for ResNet-50 backbone.
Noise Shot
Blur Motion
Weather Frest
Digital Elustic
Method
Gauss.
Impul.
Defoc:
Glass
Zoom
Snow
Fog
Brit.
Contr.
Pixel
JPEG
Avg.
No. adaptation
95.5
94.8
95.4
84.8
91.7
86.2
76.8
84.0
79.3
77.1
44.4
95.6
85.3
77.1
66.5
82.3
TENT
84.1
65.2
85.9
50.3
Ba_adapt
88.4
87.7
87.6
88.4
87.9
78.9
64.5
68.1
69.0
55.0
37.0
89.2
59.3
57.2
67.5
72.4
SHOT
81.1
79.2
80.2
84.2
83.8
70,2
57.3
58.6
63.7
47.8
35.1
92.0
51.9
48.6
56.7
66.0
ConjugatePI.
84.0
82.6
83.3
84.5
84.4
73.3
59.7
62.6
65.4
49.7
35.6
85.6
54.7
50.5
60.1
67.7
SAR
83.7
83.7
82.9
85.3
84.9
72.8
59.9
62.5
65.2
50.0
35.8
83.3
55.0
50.8
60.0
67.7
AEA (ours)
77.2
75.8
76.5
78.8
79.8
63.6
54,2
54.6
59.9
45.6
34.7
83.5
48.5
45.1
$2.6
62.0
18

---

Published as a conference paper at ICLR 2025
B.2 PERFORMANCE UNDER DIFFERENT BACKBONES
In table I1 we report experimental results for different backbone architectures of vision transformer (ViT) family (Dosovitskiy et al. 2021). Our method can be easily applied to various model architec- tures and, as shown in the results, consistently achieves superior TTA performance compared to other comparable baselines.
Mothod
Goass
Shot
Inpul.
Defoc.
Gloss
Motion
Zocas
Snow
Froet
Fag
Contr.
Elistic
Piscl
JPEG
Avg.
No.adapt
33.5
28.3
17.8
22.6
4.9
7.3
10.0
12.7
24.4
15.5
TENT
24.2
18.9
14.0
4.7
16.5
7.8
3.9
5.0
6.3
9.0
2.7
5.8
10.3
8.6
13.4
10.1
SAR
28.1
19.8
14.4
4.8
16.5
8.0
4.0
5.1
6.4
9.5
2.7
6.0
10.4
8.8
13.5
10.5
AEA(ours)
20.3
16.3
11.5
4.7
15.0
6.6
3.7
5.1
5.3
7.0
2.7
3.9
10.1
0.9
13.6
8.8
Method
Ganss.
Shot
Impal.
Defec.
Glass
Mition
Zaom
Snow
Fiost
Contr.
Elesti
Pisel
JPB5
No.adaptation
63.1
66.3
63.6
67.6
77.8
63.4
68.9
13.3
38.8
49.3
68.0
45.5
44.1
61.1
TENT
48.2
47.4
55.5
62.5
33.4
SAR
48.2
47.2
47.1
47.6
53.8
45.1
51.7
80.2
68.6
33.9
27.9
35.4
49.4
33.9
35.7
47.0
AFA(ours)
45.9
44.9
44.4
44.5
48.1
50.7
88.6
74.3
30.8
26.3
35.5
40.9
20.9
33.1
45.3
Table 11: Classification errors (%,, 4) on CIFAR10-CA for ViT small 16 backhone (top) and on ImageNet-C for VIT base 16 backbone (bottom).
B.3 PERFORMANCE ON SEGMENTATION TASK
For further justification, we evaluate our method the segmentation TTA task in Table 12 We report the segmentation performance (i.e., Ml)U(%)) on the Cityscapes dataset (Cordts et al. (2016), following the experimental setup of Volpi et al. (2022). More specifically, given a pre- rre-trained segmentation model on GTA-5 dataset Richter et al. (2016), we perform online test-time adaptation to adapt the pre-trained model to target sequences composed of target samples from each scene (i.e.,
city) in Cityscapes dataset. In Table 12 we report the results for five different scenes and other experimental details (e.g., segmentation model, batch size, etc.) are consistent with Volpi et al. (2022). Also, we use the same hyperparameters in our main results for AEA evaluation. The results show that our AEA still achieves competitive performance in the segmentation TTA task, demonstrating its effectiveness and applicability across various tasks.
Target sequence
Seq.#1
Seq.#2
Seq
Seq.#4
Seq.#5
Avg.
No adaptation
42.8
41.9
47.6
46.3
47.9
45.3
BN adaptation
43.3
42.7
48.1
47.0
48.5
45,9
TENT
43.7
43.6
48.8
47.8
49.1
46.6
AEA (ours)
44.7
44.9
50.3
48.3
49.3
47.5
Table 12: Evaluation under the TTA segmentation task, following the experimental setup of Volpi et al. (2022
We report the segmentation performance (i.e., MloU(%, 1)) on the Cityscapes dataset (oordtt et al. (2016). We adapt a pre-trained segmentation model on GTA-5 dataset to five different target sequences from Cityscapes: #1) aachen, #2) dusseldorf, #3) erfurt, #4) monchengladbach and #5) ulm.
B.4 PERFORMANCE UNDER SOURCE-FREE DOMAIN ADAPTATION SETTING
Our AEA can also be applied to the source-free domain adaptation (SFDA) task and we present the results in Table [3 The main difference between SFDA and TTA is that SFDA allows access to the entire target samples for model adaptation, whereas TTA should conduct adaptation over a continuous stream of target batches. To demonstrate the effectiveness of our AEA, we measured performance by varying the number of target samples for adaptation in the SFDA setting. Specifically, we first adapt a tre-trained model on the given target samples, and subsequently, the performance is evaluated over the entire target samples. The results show that our method still achieves competitive performance in the SFDA setting, with a significant performance improvement when the number of target samples is small. For example, given only 30% of the entire target samples (for adaptation), our AEA outperforms SHOT by 6.9% and even surpasses the performance of SHOT adapted on the full dataset. These findings support the advantages of our method in accelerating adaptation and suggest its applicability in other source-free scenarios.
19

---

Published as a conference paper at ICLR 2025
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
No adaptation
26.9
26.9
26.9
26.9
26.9
26.9
26.9
26.9
26.9
26.9
TENT
61.6
62.9
64.3
64.9
66.6
67.5
68.7
69.1
70.1
70.5
SHOT
63.2
67.8
70.2
71.3
72.6
73.3
74.3
74.5
75.5
75.2
AEA (ours)
67.7
72.9
77.6
77.6
77.8
77.7
76.8
Table Performance under SFDA setting. We report classification accuracy (%, 1) with varying the sample sizes (in column) for model adaptation. Experiments are conducted on CIFARIO (gaussian noise with severity level 5).
B.5 ABLATION STUDY ON UPDATING MODEL PARAMETERS
have further conducted an additional ablation study on the proposed components in our AEA. In Table 14 we show the results (i.e., classification errors) of updating all the parameters in the feature extractor, along with the results of updating only batch normalization parameters, which denoted by 'Batch Normalization', As can be seen, proposed loss functions, i.c., LSFEA A and LLCS, individually contribute to the performance improvement compared to naive EM baselines. This indicates that our approach works effectively regardless of which parameters updated.
Table 14: Ablation study for each component of AEA. The results show that each component (i.e., LSFEA. LLOS) of our method individually contributes to the performance improvement in TTA.
Updated parameters
Methods
CIFARIO-C
No adaptation
44.4
EM
24.0
Batch Normalization
EM + SFEA
20.7
EM + SFEA + I.C.
19.5
EM
22.6
Feature Extractor
EM SFEA
21.7
EM SFEA LCS
20.6
B.6 CHOICE OF IYPER-PARAMETERN λG A2
0,1
20.2
20.0
19.0
19.8
19.7
20.1
0.5
19.6
19.4
19.3
19.4
19.4
19.7
10.8
19.4
19.4
19.5
19.3
19.3
1.0
39.7
19.5
19.4
19.4
19.4
19.6
19.9
19.5
19.5
19.5
19.5
19.9
1.5
20.0
19.9
19.9
19.5
19.9
20.2
A2
Figure 11: Classification (%, CIFARIO-C with varying main hyper- parameters (i.e., X1, A2).
To provide further insight into the choice of parem-paraeeters, we have varied the main hyper- parameters (i.e., Å1, Ag) across a wide range and reported the corresponding results the Fig. Our findings demonstrate that various combinations of X1 X2 still can yield robust and comparable performances, with the best results when X is around 1.0 and X2 is around 25. We note that these results consistent with the hyper-parameter analysis provided in Fig. of the main paper.
20

---

Published as a conference paper at ICLR 2025
B.7 TIME COMPLEXITY AND MEMORY USAGE
In this section, we compare the time complexity and memory usage between our method and baselines. In Fig. 12 measured performance changes according the time required for adaptation on CIFAR IO-C. Also, in Table T5] we report GPU memory usages (MiB) per adaptation batch during TTA. As can be seen from the results, our scheme achieves high performance more rapidly and requires almost the same amount of computations as TENT, which demonstrates that our proposed method requires only negligible additional costs. Since our proposed SFEA and LCS losses are simply computed using only output logits and the weights of the last classifier, proposed method requires only negligible additional costs compared to the conventional EM loss. Therefore, our AEA can be practically utilized in various scenarios and applications compared to the other promising baselines.
Accursty
65
60
SHOT
TENT
TEA
BN, dapa
AEA tourst
SAR
50g
10
30
40
50
70
80
Figure 12: Classification accuracy (%, 1) with respect to the adaptation time stamp on CIFARIO-C.
Method
CIFARIO-C/100-C
TinylmageNet-C
TENT
1,759
9,362
SHOT
1,829
9,973
SAR
1,761
9,367
TEA
1,919
16,449
AEA (ours)
1,761
9,367
Table 15: GPU memory usage (MiB) per batch during adaptation.
B.8 RESULTS UNDER THE CONTINUAL TTA SETTINGS
In Table [6] have compared our AEA with several promising baselines under the continual TTA scenario. The key consideration under the continual TTA is to adapt the model persistently to changing domains while mitigating eтror accumulation catastrophic forgetting (Wang et al.) (022).. We note that our AEA does not primarily target the continual setting and thus has no additional components for those purposes. To compensate for this, we can adopt simple model restoration technique (e.g., stochastic restoration in Wang et al. (2022)) b which stochastically resets the adapted model its initial state. In Table 16 we report our AEA'S results under continual setting with stochastic restoration technique. For fair comparison, we also apply the same technique to baseline that do not include additional components for continual settings, such as TENT (Wang al. 2021). Interestingly, the results confirms that although we simply restore the model to its initial state stochastically. our method can also achieve competitive performance in the continual TTA setting. We attribute this to our AEA's capability of effectively accelerating model adaptation within few batches despite model restorations via adaptive energy alignment scheme. This enables the model to rapidly adapt to new target domains and achieve competent OOD adaptation performance. In this regard, argue that our AEA's acceleration capability could be another beneficial approach to tackle the continual TTA settings.
21

---

Published as a conference paper at ICLR 2025
Table 16: Classification errors (%. 4) CIFARI00-C with the highest severity level 5 for ResNext-22 backbone. The experiment is conducted under the continual TTA setup following Wang et al. (2022), and domain shifts happen from left (Gauss.) to right (JPEG). We report the averaged results over different runs.
Noise Gouss.
Blur Glass Motion Zoom
Weather Frost
Digitl Elastic Pixel
Method
Shot Impul Defoc
[Soo
Fog
Brit, (oonrr
IPEG
Avg.
No adapiation
73.0
68.0
39.4
29.4
54.1
30.8
28.8
39.5
45.8
50.3
29.5
56.1
37.2
74.7
41.2
46.5
BN adapration (Schneider al.
2020
42.3
40.8
43.2
27.7
41.8
29.8
27.9
35.0
34,8
41.7
26.4
30.2
35.6
33.2
41.2
35.4
TENT
(Wang et at
2022
32.2
34,3
36.3
27.3
39.4
30.1
27.1
32.4
33.1
35.8
24,8
28.0
35.2
30.8
39.1
32.7
FATA
Niu et 20721
13.1
36.2
27.8
37.9
29.7
26,9
12.7
31.3
35.1
26,9
28.7
31.4
29.8
37.4
32.3
SAR
NJu cr
0125)
40.5
34,9
37.1
25.9
37.2
28.0
25.6
318
308
35.9
25.2
28.1
32.1
29.2
37.3
32.0
RoTTA
Uun et al 202.3
49.1
45.0
45.5
30.2
42.6
29.5
26.0
32.3
30.6
37.6
24.7
29.2
32.7
30.4
36.9
34.8
COTTA
Waang
40.6
38.0
39.9
27.2
38.2
28.5
26.5
33.3
32.2
40.6
25.1
26.9
32.3
28.3
33,8
32.8
AEA (ours)
36.2
32.8
32.8
25.3
36.3
27,5
24.8
29.7
29.0
31.8
24.0
25.3
32.2
28.1
30.1
B.9 RESULTS WITH STANDARD DEVIATIONS
In Tables 17 19 we present the online TTA results on CIFARIO-C, CIFARI00-C, and TinyImageNet- Cwith standard deviations, respectively. Each method is conducted with three different random seeds, and we report the mean and standard deviation of three runs for each method. Our AEA, which is fundamentally based on the EM loss, shows similar or lower standard deviations compared to TENT. The results still confirm that our method consistently achieves superior TTA performance through our energy alignment approach.
Table 17: Classification errors (%, 1) with standard deviations on CIFAR 10-C with the highest severity level 5 for ResNet-26 backbone.
Noise Shot
Blur
Weather
Digial
Method
Gauss.
Impul.
Defoc..
Glass
Motion
Zoom
Snow
Frost
bog
Beit.
Contr.
Blastic
Pixel
JPEG
Aug.
No aduptation
(+:000)
73.1
(1:000)
66.8
(1:000)
69.4
(=0000)
40.2
(4:0.00
$1.9
(1:0.00
(=00000
(=00000
25.5
(=0000)
50.3
(=0.00)
30.4
(=0.00)
10.1
(+:00)
55.2
(4:00)
26.7
(+000)
61.5
(+:0000)
31.0
(+0000)
44,4
TENT
(+0.52)
32.7
(20034)
30.2
(1:0 10)
39.1
(=00
15.7
16)
(20058)
35.9
(230.3)
18.2
(=:0.28)
16.1
(200
21.8
10)
(20.30
23.1
(20115)
19.4
(=20.29
12.7
(::043)
16.0
(2:0.56
26.1
(2:042)
22.3
(2:0.50)
30.4
(2:033)
24.0
BN aduptazion
39.1
17.2
18.0
25.1
25.5
14.1
17.6
26.6
35.5
27.5
(1:0 11)
(1:0
13)
(1:007)
(220118
18) (2:0.3)
(1:0.13)
(2:0 10)
(200
12) 0.02) (2:0.07
(200099
(1::0 18) (1:0.33
(3:0
(10
(1:0 14)
29.5
26.5
35.4
14.6
33.7
17.0
14.6
19.8
22.2
17.8
11.8
16.3
24.6
2014
26.9
22.1
SHOT
(+00776)
(0))
(::0.52)
(=0004)
(111
(+:0.49)
(=+0334)
(=0.4)
(=00188)
(=:0.49)
(:b00.99
(+0032)
(=10.11)
(+00558
(+:0.50)
65.3
50.6
37.0
34.0
24.8
10.1
50.0
25.1
29.8
40.4
(+:0.2))
(1:0
(5:0015))
(20002)
(10011)
(#:005)
(:0000)
(20))2
(=0115)
(20002)
(5:0.25) (2:00
(+:000
(2:00)
(#:0.10
24.9 23.0
30.2
13.5
34.6
20.4
15.9
19.3
17.9
14.1
9.4
26.4
23.7
160
23.8
20,9
TTT
(2:074)
(2:0.04)
(1:0013) 13)
(:00
(20032
(23.223
(2:0.04)
(:0000)
(20.30
(20.21)
(200.0)
(::042) (1:0.6)
(2:017)
(2:0.17)
(::0.23)
48.8
47.5
25.0
23.7
20.3
23.5
20.2
30.4
(1:0.77)
(9)
(1:004)
(2:0
50)
(5:025)
(1:0.16)
(1:0
(20.116
(223)
(200116
(10.6)
(1:0.29)
(1:0
(2:0.4)
(1:0.19)
(1:0:25)
Conjagate PL
(+:05)) (::0.5)
32.7
30.1
56)
(2:0:12)
39.2
(=0.19)
15.8
(2058))
15.8
(33)13)
1822
(=0025)
16.1
(=00.10)
21.8
(2.32)
23.1
(20.14)
19.4
(=0.29
12.7
(::044) (#0.53)
16.1
26.
(#:04)
22.3
(=:0.48
30.
(+:033)
24.0
38.7
36.2
17.1
17.9
24.9
25.4
14.1
35.3
27.3
EATA
(7:017)
(1:0.4)
40)
(1:0
(9)
(220.1)
(20066
(10.06)
(2:0 12)
(20.22)
(20.9)
(20111)
(20.0))
(1:0.00)
(1:034)
(3)233
(1:0.7)
(2:0.1)
SAR
(1:0.1)
32,9
(1:0.2))
30.6
(1:0.28)
39.5
(2:0.20
15.9
(220.11
26.3
(100005
18.3
(2:0.12
16.3
(20.
22.1
(20000
21.2
(200223
19.3
(220.12
12.7
(100.7) (1002)) (1:0.66
16.5
26.2
22.8
(1:0.2)
30.6
(1:0.20)
24,2
27.7
34.2
15.3
34.8
15.9
200
20.5
12.4
192
26.3
22,0
TEA
(1:0.77) (1:0.40)
(1:0.64)
(:20022)
(1:0.95
(1:0.3)
(1:0.5)
(:10009
(20.31)
(100.1)
(1002)((1000))
(10012)
(110.10)
(110.22)
(1:0.31)
AEA (ours)
(+:0.19)
24.7
(4-0.05)
22.7
(4:0.5)
31.9
0100.10
13.8
0.86 (4-033)
20.8
(4-0.34)
13.8
(-0000)
17.5
(20.17)
17.5
(40.39)
15.4
(#0.20)
1088
(4:.4))
137
(4:22)
233
(400.12
(40.17)
23.8
(+-0.00)
19.5
C IMPLEMENTATION DETAILS
In this section, we provide some detailed implementation settings for experiments. Basically, we follow the basic settings of Zhao et al. (023). including the hyperparameter settings required for each baseline. For all experiments, we use SGD as optimizer and set the learning rate to 0.001, batch size to 64, and the number of adaptation iterations per batch 1, Also. we adapt the model parameters of the batch normalization (BN) layers similar to Wang et al. (202T). Additionally, there are several hyperparameters that need to be tuned for our AEA, i.e., (A1, 0 for the SFEA loss and (A2, Co} for the LCS loss. For all datasets in the experiment section, we apply the SFEA loss with X1 set to 1.0 and a set to 0.5. Also, for the LCS loss, which depends on the confidence scores of the model, we set the threshold Co to 0.99 for CIFARIO-C, 0.66 for CIFAR 100C and Tiny ImageNet-C. Also, A is set to 25 for all experiments. To further provide insights into our hyperparameters, we conduct an ablation study for each hyperparameter in Fig. and Sec. B.6) All the experiments are performed with xNVIDIA GeForce RTX 3090 and the other details of computational resources (c.g. workers,
memory, etc.) are the same with /hhao et al. (2023).
22

---

Published as a conference paper at ICLR 2025
Table 18: Classification errors (%, L) with standard deviations on CIFARI00-C with the highest severity level 5 for ResNet-26 backbone.
Noise Shot
Rlur
Weather
Digital
Method
Goues.
Impul.
Defoc
Glass
Moxion
Zoom
Snuw
Frost
Fog
Drit.
Contr.
Ehasiic
Pixel
JPEG
Avg.
No abpuution
89.3
88.3
91.0
67.2
63.5
60.8
39.6
56.1
62.3
67.6
42.7
84.4
50.
85.5
00.9
68.7
(1:0.00)
(1:000)
(1:0.00)
(2:0,00)
(1:0.00)
(1:0,00)
(1:0.00)
(:40.00)
(:00.00)
(110.00)
(4:0.00)
(1.000)
(4:0.00
(110000)
(:10.00)
(1:0.00)
TENT
(+..18)
65.5
(40.12)
64.7
(4-033)
(400.44)
43,9
(400117)
58.1
(40221)
(4-0
43.6
(+40337)
562
54.1
(+0227)
52.3
(+0043)
42.9
(+0.20)
49.5
(40029)
514
(+00115)
5077
(+00000)
59.6
(+-0266
53.6
BN adaptasion
(+0020)
70.5
(+:0.13)
69.9
(+60333
68.8
(550.15)
46.6
(#00355
60.8
48.8
(#:0.13)
45.9
(+0.333
59.0
(=0099
56.8
(+00335
55.1
(-0.11)
45.5
(-b0300
51.2
(+:031)
53.5
(+:0003)
548
(+00.14)
62.8
(+0021)
56.7
SH1TT
38.6
57.7
58.7
41.4
55.0
44.1
41.3
51.9
49.9
41.0
48.6
46.4
56.0
49,9
(+:064)
(1:0
23)
(::0.60)
(=:0
(10.11
(+0.57)
(#:0
(:0033)
(+0:15)
(20:13)
(6:046)
(::041)
(+0
(#:0. 0.30
(+0.33)
T3A
(+-0.34)
89.3
M8.4
(4500.4)
90.4
(#50.25)
6488
(4-0.19)
60.9
(4:0.25)
59.9
(+-0.17)
57.3
(=+0.05)
57.2
(=00.12)
61.2
(+40.34)
65.3
(=-0.02)
43.2
(--0000)
82.5
(++0.06)
50.0
(++00077
82.9
(=+0.29)
50.4
(+:0.344
67.6
63.7
63.2
43.9
43.4
54.1
50.8
38.7
70.2
49.7
56.1
5344
(+0.14) (4:0009)
(4:0229)
(400.200
(40033)
(+0020)
(+:0.13)
(::0.28)
(:5555)
(+0:10)
(H0.117)
(+:027)
(4:022)
(+0229)
(++0:37)
(+:0.22)
NOTE
(+0029)
76.4
(4:004)
74.6
(:b0.17)
74.5
(=0025)
51.9
(=2222)
57.6
50.7
(=:0.18)
47.9
(=00227
52.7
(=0331)
$2.1
(+0344)
56.7
(=10177
38.6
(+:045)
67.4
(#:025)
40.0
(+00555
70.4
(+00006
$7.8
(+:0.25)
58.7
Conjugate
63.6
64.7
65.1
43,9
58.1
43.6
56.2
42.9
49.5
59.6
53.6
[+0 16)
(1:0 14)
(::0:44
(5)
(17077)
(2:0.2))
(+:0 51)
(
0445
(=025))
(=0.00)
(::0025)
(::0
(:00 (8)
(#:0000)
(#:0.66
EATA
(+22,7)
68.0
(+:2
66.2
(+-1138)
71.5
(+-0.31)
46.0
64.7
(:11111
49.3
(+:1.33)
46.0
(++1.03)
56.7
L:73)
57.2
(+10.13)
53.8
(=10,69)
44.1
(++11.6)
51.9
(+) (:6)
55.4
(#:3323)
52.1
(+00.99)
62.2
(+:144)
56.3
65.8
64.9
65.3
44.2
43.8
564
54.4
43.0
49.3
51.4
50.8
50.7
53.8
(+0000)
(#:0.52)
(::0.26)
(#:0.17)
(4:0.20
(+0000)
(4:446)
(+0333)
(.311
(+0.33)
(+00322)
(::0.37)
(4:0.40
(+0004)
(#:0.28)
(+0027)
TEA
(+043) (4:0000
64.0
63.3
(4:0.57)
64.2
(=00299)
45.1
59.0
(230.23)
48.4
(#:0.3)
454
(0.28)
56.5
(2444)
55.1
(=0445)
52.5
(=20.27
43.3
(+:070)
53.4
(20.22)
52.7
(#00.10
50.1
(7)
60.0
(+0.33)
54.2
58.2
58.8
59.0
40.9
55.0
43.9
40.5
51.3
49.0
47.4
39.4
44.1
48.4
44.3
54.8
49.0
AFA (nurs)
(+0055
(4-0.27) (4-024)
(450.200
(400155
(4-0.44)
(+00.331
(-005))))
(+0.31)
(40.27
(=0.10) (40.5)))) (40.30) (+00448 (=+0.25)
(=00110)
(+-0.30)
Table 19: Classification errors (%, J) with standard deviations on TinyImageNet-C with the highest severity level 5 for ResNet-50 backbone.
Noise Shoc
Blur Motion
Wealher
Digital
Method
Guuss,
Impal.
Defoc:
Glass
Zoom
Snew
Frost
Fog
Brit.
Contr.
Flastic
Pixel
JPBG
Avg.
No aduptation
(:000))
966
(::0.00)
95.1
(20.00)
97.2
(::0000)
92.5
(4:0000
92.2
(::000)
77.8
(4:00))
78.5
(+:0:00)
81.9
(2:0000)
78.1
(=:000)
89.5
(1:0:00)
77.8
(=0000)
98.3
(1:0.00)
09.5
(=00000)
71.9
(=1.00)
55.6
(+0000)
83.5
SHOT
(5) (#00255
64.5
62.8
(220.2)
70.6
(#:0.17)
61.9
(d:0200)
73.5
(2:337)
54.5
(20.48
55.8
(2:0.34
62.1
(1:0.37)
611
(20025)
65.2
36.2
$9.6
55.4
(220.32)
50.8
(=0.20)
51.8
(+00337
62.2
TENT
(tn 12)
66.5
(2:046)
64.3
(2:011)
72.9
(1:00300
63.2
(230113
75.1
(2:0.00
55.5
(20.28)
54.8
(::0.43)
63.6
(2:0116
61.9
(220.28)
62.6
(1:005)
57.6
(200000
86.2
(220.33)
56.6
(22022)
51.5
(=0.15)
52.5
(2:0.4)
63.3
68.8
66.7
75.7
65.0
77.1
564
55.8
64.9
63.4
71.0
59.0
56.55
57.7
52.2
53.3
64.9
(+.33)) (+-0.43)
(::0.27)
(4:0.27)
(40030)
(+0.4)
(4:0.10)
(4:0.17)
(4:0.00)
(=00.12)
(1-0.20)
(.20.26)
(+:0224
13A
(t (::0..9) (2006))
96.6
95.0
97.3
(2:0005)
92.7
(2:011)
92.1
77.3
(2:00)))
78.2
(#:004)
82.3
(2)1133)
78.2
(2008))
80.9
(1:0.18)
77.1
(200.00)
98.
(::0.24)
68.9
(=0.9
70.3
(=1.08)
55.8
(::0.11)
83.4
NOTE
(20 (#:014) (220.66)
83.4 50.4
96.4
(1:0.05
81.9
(1:0
85.7
(2:023
65.4
(1:030)
64.1
69.6
(2:005)
67.4
(2000)
79.8
(100211
64.2
(20.11)
96.0
61.2
56.0
53.1
(::0. 16)
73.0
SAR
(2022) (2:0233 (20017)
672 65.
73.6
(::0028)
63.9
(2.0.8)
75.9
(1:0
55.8
20)
(1:0.22)
(220225)
64.1
(220227
62.6
(22024)
68.4
(10006)
58.0
(220.13)
56.0
(220
57.1
(234.3)
51.8
(=0.25)
52.9
(::0023)
63.8
ConjuganetL
(0.77) (+0222) (20.12)
67.4 65.3 74.0
(#:0.25)
63.9
(2
76.1
(2:022)
55.8
(2:020)
55.3
64.3
62.7
(=00.1)
68.6
(=000))
58.1
(=0.04)
56.2
(200.2))
57.1
51.8
(=0.26
53.0
(+00222)
64.0
65 63.6
69.7
62.3
74.2
55.3
54.1
61.3
61.0
63.9
55.4
90.8
56.1
51.1
52.2
62.4
ENTA
(2277) (::0
(20.28)
(#:0:44)
(2:0.54
(2:000
(2:22))
(=20228)
(::0.14)
(2:44)
(2:0.54
(=20.28
(22044)
(2662)
(=0.10) 10)
(200.3))
TEA
(2000) (1:024) (2:0.1))
66.1 64.6
72.3
(1:0013)
63.2
(2:0
75.1
10)
(1:0
55.8
20)
(1:0
54.9
24)
(220.47
63.1
(1:0.08)
61.2
(20021)
65.9
(10030)
56.9
(:20.14
65.3
(200.1)
56.9
(20.00)
51.8
(=0. 15)
52.9
(20.119
63.1
AEA (ours)
(10.13) (10029) (10.3)
63.7
62.1
67.9
(1:0:20)
60.9
(1:0.32)
72.8
(1006)
53.8
(10.16)
$2.9
(1:0.16)
60.1
(1:0.22)
58,9
(10021)
61,9
(1:011)
54.2
(220.88
90.3
(110.35)
54.5
(:20.25
49.8
020.0
51.1
(10255
61.0
D LIMITATIONS OF OUR WORK
As our AEA is primarily designed for the online TTA setting, it does not include mechanisms to mitigate error accumulation or catastrophic forgetting issues for continual setting. While AEA still achieves competitive performance in continual TTA due to its acceleration capability, promising future direction would be to enhance its applicability to continual setups by integrating model restoration strategies (c.g., stochastic restoration in Wang et al. (2022). We hope that our study. which effectively accelerates model adaptation through adaptive energy alignment, will contribute to making TTA more practical and widely applicable across various real-world scenarios.
23


Reviews:
[{"review_id": "PExQUIhkJa", "reviewer": "Reviewer_aNYF", "summary": "The paper introduces an energy-based approach for addressing the test-time adaptation (TTA) problem. The authors analyze the classical entropy minimization loss widely adopted by existing TTA methods through the lens of energy-based model, pinpointing the potential issue lies within the entropy minimization. To address the issue, two alignment objectives termed source-free energy alignment (SFEA) and logit cosine similarity (LCS) are proposed. SFEA aligns the overall energy magnitudes between source and target domains to reduce the energy gap during adaptation, whereas LCS ensures the class-wise correlation of the target domain maintains consistent during energy alignment. Experiments are conducted on several common TTA benchmarks including CIFAR10-C, CIFAR100-C, TinyImageNet-C and ImageNet-C, as well as style shift dataset PACS. The results of the proposed method generally surpass the baselines.", "questions": "1.\tIn the caption of fig. 3, the ‘relative distance in a logit space’ is confusing. Please elaborate on the meaning of the x-axis in detail.\n2.\tFrom the experiment results, it appears that SHOT achieves better performance than most of the TTA methods. What is the reason?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tAs stated in the paper, Test-time adaptation is an important direction to mitigate the out-of-distribution issue of inference data and worths exploration.\n2.\tThe experiment is extensive on several benchmarks, and the proposed method achieves good results on most tasks within these benchmarks.", "weaknesses": "1.\tIn both the summary and the introduction, the authors argue that “the EM loss can be decomposed into two energy-based terms with conflicting roles” and such confliction “hinders the energy gap reduction when the EM loss is used alone.” However, the authors do not address this issue in the proposed method. Minimizing the target energy does not resolve the conflict. On the contrary, the reviewer thinks that the free-energy maximization term in equation (4) is necessary for avoiding the explosion of the magnitude of logits. The reviewer’s opinion is supported by the fact that the final objective of the proposed method still incorporates EM loss. \n2.\tThe authors highlight the strength of the proposed method that can reduce the energy gap more quickly than EM loss in the early batches. However, it appears from fig. 2 that the proposed method performs similarly to other baselines at the beginning. The result makes the author’s claim questionable.\n3.\tGiven that previous works [1][2] adopt hinge loss for energy alignment, the reviewer suggests that the authors should conduct ablation study on the choice of loss to support their design.\n4.\tFor each target batch, the proposed method computes a new approximated energy of the source domain, which means that the estimated source energy varies over time. This is a seemingly strange design without explanation. In fact, the reviewer suspects that this particular design is the actual reason for performance enhancement and distinguishes it from the previous energy alignment method. In reviewer’s opinion, the target energy is not actually aligned with the source energy, but rather is minimized towards an adaptively changing goal. This allows the energy minimization process to be more flexible and friendly to the model training.\n5.\tThe second line of equation (4) is mistaken. It should include parentheses over the two energy terms. \n\nRef: \n[1] Energy-based out-of-distribution detection.\n[2] Active learning for domain adaptation: An energy-based approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "834Dqg7oIt", "reviewer": "Reviewer_V21s", "summary": "This paper targeted at accelerating test-time adaptation from a new perspective, i.e., reducing the energy gap between the source and target domains. There are two key losses in the proposed method. The one is the source-free energy alignment (SFEA) loss to align the overall energy magnitude and the other is the logit cosine similarity (LCS) loss to guide the class-wise alignment direction. Experimental results on three common benckmarks (CIFAR10-C, CIFAR100-C, and TinyImageNet-C) have demonstrated the effectiveness of the proposed method.", "questions": "- How to define the directional aspects of energy alignment?\n\n- It seems that the source-free energy alignment loss ($\\mathcal{L}_{SFEA}$) is calculated in each target batch. How can this calculation method avoid the noise caused by bath sampling?\n\nMinor comments:\n- Actually, the foundamental of this work is to answer why does energy alignment lead to better adaptation. Thus, in my view, Sec. A.4 should be moved to the main context and the revisit of EMBs can be moved to the appendix.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is easy-to-follow. The topic is essential yet the idea is moderate. Both test-time adaptation and energy-based models are important topics for the community and the paper addresses two formulations simultaneously.", "weaknesses": "- **Limited motivation.** My major concern is about the motivation of this work. First, the necessity of the proposed logit cosine similarity (LCS) loss is not clear. Second, though the authors have tried to describe the correlation between energy gap reduction and better adaptation, it's not clear why TTA needs to do this. Especially, many previous works have proven its effectivebess in addressing distribution shift problems .Third, the proposed method is complicated but the improvements in Tables 1,2,3 is minor. \n\n- **Insufficient justifications.** From main results of Table 1,2,3, it can be observed that the proposed AEA obtains limited improvements in three small-scale image classification datasets (CIFAR10-C, CIFAR100-C, and TinyImageNet-C). Large-scale datasets and other tasks like segmentation or detection might be evaluated to demonstrate the AEA's utlization in various scenarios and applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "epSq76gWPP", "reviewer": "Reviewer_SzkH", "summary": "This paper proposes an Adaptive Energy Alignment (AEA) method that enables fast online test-time adaptation (TTA). The introduced Source-Free Energy Alignment (SFEA) loss strategically aligns the overall energy magnitudes between source and target domains. Additionally, the authors propose a Logit Cosine Similarity (LCS) loss to ensure that class-wise correlations in the target domain align well with those in the source domain during energy alignment. Extensive experiments on three datasets demonstrate the advantages of the proposed AEA method.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*\tThe proposed SFEA loss provide novel insight for TTA, effectively reduce the domain gap.\n*\tThe authors have conducted a thorough ablation study to examine the impact of various components. Additionally, they have made the code and datasets available in the supplementary material.\n*\tThe paper is well-written, well-organized, and easy to read.", "weaknesses": "*\tWhile the authors demonstrate the superiority of the proposed method on TTA, it would be interesting to evaluate its performance in a similar scenario, such as source-free domain adaptation (SFDA), where source data is also inaccessible. Additional evaluations could further illustrate the versatility of the proposed method.\n*\tThe authors did not clearly discuss the distinction between online test-time adaptation (OTTA) and TTA. In line 142, they claim to address the OTTA problem, but in other parts of the paper, the main focus appears to be on TTA.\n*\tA minor issue concerns the punctuation of equations. Eqs. (3), (4), and (9) should include the appropriate punctuation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GuJtC6RZyf", "reviewer": "Reviewer_w8pg", "summary": "This paper proposes Adaptive Energy Alignment (AEA) for fast test-time adaptation (TTA) in out-of-domain scenarios. It reinterprets entropy minimization (EM) loss and shows that its conflicting energy-based components can hinder adaptation. AEA reduces the energy gap between source and target domains by aligning target sample energy levels with familiar source domains and matching class-wise correlations. The approach achieves efficient TTA with minimal computational overhead, demonstrating strong results on datasets like CIFAR10-C, CIFAR100-C, and TinyImageNet-C.", "questions": "Please refer to the weakness part.\n\n---\nI am open to discussing these points with the authors during the response period. If the concerns and questions are adequately addressed, I will consider raising my score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*\tThis paper is well-written, well-organized, and easy to follow.\n*\tThe insight of leveraging the direction of energy alignment with the guidance of the structural relations between different classes is convince, and provide a new solution for accelerating test-time adaptation\n*\tThe proposed SFEA and LCS loss is novel to me, effectively reduce the energy gap and aligns the class-wise correlation across source and target domains.\n*\tOverall, the quality of the paper is commendable. The authors have conducted a thorough comparison experiments to examine the effectiveness of the proposed method. Additionally, they have made the code and datasets available in the supplementary material.", "weaknesses": "*\tWhile the proposed AEA significantly accelerates test-time adaptation (as observed in Figure 1c), the authors provide insufficient discussion on how AEA achieves this acceleration. Further analysis and discussion are needed.\n*\tI noticed that in TEA, the backbone for CIFAR10-C is WRN-28-10, whereas ResNet-26 is used in this work. If the experimental results are reproduced using publicly available code, please provide the relevant details in the table caption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)